Vibrotactile captioning of musical effects in audio-visual media as an alternative for deaf and hard of hearing people: An EEG study

e-Archivo Repository

Show simple item record Lucia Mulas, Maria Jose Revuelta Sanz, Pablo Garcia Lopez, Alvaro Ruiz Mezcua, María Belén Vergaz Benito, Ricardo Cerdan Martinez, Victor Ortiz, Tomas 2022-01-11T11:33:57Z 2022-01-11T11:33:57Z 2020-10-19
dc.identifier.bibliographicCitation Lucia, M. J., Revuelta, P., Garcia, A., Ruiz, B., Vergaz, R., Cerdan, V. & Ortiz, T. (2020). Vibrotactile Captioning of Musical Effects in Audio-Visual Media as an Alternative for Deaf and Hard of Hearing People: An EEG Study. IEEE Access, 8, 190873–190881.
dc.identifier.issn 2169-3536
dc.description.abstract Standard captioning for the deaf and hard of hearing people cannot transmit the emotional information that music provides in support of the narrative in audio-visual media. We explore an alternative method using vibrotactile stimulation as a possible channel to transmit the emotional information contained in an audio-visual soundtrack and, thus, elicit a greater emotional reaction in hearing-impaired people. To achieve this objective, we applied two one-minute videos that were based on image sequences that were unassociated with dramatic action, maximizing the effect of the music and vibrotactile stimuli. While viewing the video, using EEG we recorded the brain activity of 9 female participants with normal hearing, and 7 female participants with very severe and profound hearing loss. The results show that the same brain areas are activated in participants with normal hearing watching the video with the soundtrack, and in participants with hearing loss watching the same video with a soft and rhythmic vibrotactile stimulation on the palm and fingertips, although in different hemispheres. These brain areas (auditory cortex, superior temporal cortex, medial frontal cortex, inferior frontal gyrus, superior temporal pole and insula) have been consistently reported as areas involved in the emotional perception of music. We conclude that vibrotactile stimuli can generate cortex activation while watching audio-visual media in a similar way to sound. Thus, a further in-depth study of the possibilities of these stimuli can contribute to an alternative subtitling channel for enriching the audiovisual experience of hearing-impaired people.
dc.description.sponsorship This work was supported in part by the Comunidad de Madrid through the SINFOTON2-CM Research Program under Grant S2018/NMT-4326-SINFOTON2-CM.
dc.format.extent 9
dc.language.iso eng
dc.publisher IEEE
dc.rights © The authors, 2020. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
dc.rights Atribución-NoComercial-SinDerivadas 3.0 España
dc.subject.other Music emotion recognition
dc.subject.other Hearing impairment
dc.subject.other Vibrotactile
dc.subject.other Audio-visual
dc.subject.other Captions
dc.subject.other Accessibility
dc.subject.other Electroencephalography
dc.title Vibrotactile captioning of musical effects in audio-visual media as an alternative for deaf and hard of hearing people: An EEG study
dc.type article
dc.subject.eciencia Electrónica
dc.subject.eciencia Informática
dc.rights.accessRights openAccess
dc.relation.projectID Comunidad de Madrid. S2018/NMT-4326
dc.type.version publishedVersion
dc.identifier.publicationfirstpage 190873
dc.identifier.publicationlastpage 190881
dc.identifier.publicationtitle IEEE Access
dc.identifier.publicationvolume 8
dc.identifier.uxxi AR/0000027665
dc.contributor.funder Comunidad de Madrid
 Find Full text

Files in this item

*Click on file's image for preview. (Embargoed files's preview is not supported)

The following license files are associated with this item:

This item appears in the following Collection(s)

Show simple item record