Publication:
Vibrotactile captioning of musical effects in audio-visual media as an alternative for deaf and hard of hearing people: An EEG study

Simple item page
dc.affiliation.dptoUC3M. Departamento de Tecnología Electrónicaes
dc.affiliation.dptoUC3M. Departamento de Informáticaes
dc.affiliation.grupoinvUC3M. Grupo de Investigación: Displays y Aplicaciones Fotónicases
dc.affiliation.grupoinvUC3M. Grupo de Investigación: Human Language and Accessibility Technologies (HULAT)es
dc.contributor.authorLucia Mulas, Maria Jose
dc.contributor.authorRevuelta Sanz, Pablo
dc.contributor.authorGarcia Lopez, Alvaro
dc.contributor.authorRuiz Mezcua, María Belén
dc.contributor.authorVergaz Benito, Ricardo
dc.contributor.authorCerdan Martinez, Victor
dc.contributor.authorOrtiz, Tomas
dc.contributor.funderComunidad de Madrides
dc.date.accessioned2022-01-11T11:33:57Z
dc.date.available2022-01-11T11:33:57Z
dc.date.issued2020-10-19
dc.description.abstractStandard captioning for the deaf and hard of hearing people cannot transmit the emotional information that music provides in support of the narrative in audio-visual media. We explore an alternative method using vibrotactile stimulation as a possible channel to transmit the emotional information contained in an audio-visual soundtrack and, thus, elicit a greater emotional reaction in hearing-impaired people. To achieve this objective, we applied two one-minute videos that were based on image sequences that were unassociated with dramatic action, maximizing the effect of the music and vibrotactile stimuli. While viewing the video, using EEG we recorded the brain activity of 9 female participants with normal hearing, and 7 female participants with very severe and profound hearing loss. The results show that the same brain areas are activated in participants with normal hearing watching the video with the soundtrack, and in participants with hearing loss watching the same video with a soft and rhythmic vibrotactile stimulation on the palm and fingertips, although in different hemispheres. These brain areas (auditory cortex, superior temporal cortex, medial frontal cortex, inferior frontal gyrus, superior temporal pole and insula) have been consistently reported as areas involved in the emotional perception of music. We conclude that vibrotactile stimuli can generate cortex activation while watching audio-visual media in a similar way to sound. Thus, a further in-depth study of the possibilities of these stimuli can contribute to an alternative subtitling channel for enriching the audiovisual experience of hearing-impaired people.en
dc.description.sponsorshipThis work was supported in part by the Comunidad de Madrid through the SINFOTON2-CM Research Program under Grant S2018/NMT-4326-SINFOTON2-CM.en
dc.format.extent9
dc.identifier.bibliographicCitationLucia, M. J., Revuelta, P., Garcia, A., Ruiz, B., Vergaz, R., Cerdan, V. & Ortiz, T. (2020). Vibrotactile Captioning of Musical Effects in Audio-Visual Media as an Alternative for Deaf and Hard of Hearing People: An EEG Study. IEEE Access, 8, 190873–190881.en
dc.identifier.doihttps://doi.org/10.1109/ACCESS.2020.3032229
dc.identifier.issn2169-3536
dc.identifier.publicationfirstpage190873
dc.identifier.publicationlastpage190881
dc.identifier.publicationtitleIEEE Accessen
dc.identifier.publicationvolume8
dc.identifier.urihttps://hdl.handle.net/10016/33858
dc.identifier.uxxiAR/0000027665
dc.language.isoengen
dc.publisherIEEEen
dc.relation.projectIDComunidad de Madrid. S2018/NMT-4326es
dc.rights© The authors, 2020. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.en
dc.rightsAtribución-NoComercial-SinDerivadas 3.0 España*
dc.rights.accessRightsopen accessen
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/3.0/es/*
dc.subject.ecienciaElectrónicaes
dc.subject.ecienciaInformáticaes
dc.subject.otherMusic emotion recognitionen
dc.subject.otherHearing impairmenten
dc.subject.otherVibrotactileen
dc.subject.otherAudio-visualen
dc.subject.otherCaptionsen
dc.subject.otherAccessibilityen
dc.subject.otherElectroencephalographyen
dc.titleVibrotactile captioning of musical effects in audio-visual media as an alternative for deaf and hard of hearing people: An EEG studyen
dc.typeresearch article*
dc.type.hasVersionVoR*
dspace.entity.typePublication
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Vibrotactile_IEEEA_2020.pdf
Size:
1.16 MB
Format:
Adobe Portable Document Format
Download
Collections
DI - LABDA - Artículos de Revistas
DTE - GDAF - Artículos de Revistas