Temporal window of integration in auditory-visual speech perception
Section snippets
Participants
Participants (native speakers of American English) were recruited from the University of Maryland undergraduate population and provided informed consent. Two groups of participants took part in this study. The first group included 21 participants (11 females, average 21 years) who were run in the voiced AbVg condition (AbVg: audio /b/ and video /g/). The second group consisted of 22 participants (8 females, average 22.5 years) who were run in the voiceless ApVk condition (ApVk: audio /p/ and
Voiced McGurk pair AbVg
Fig. 1 shows the distribution (in percent) of each of the three possible response categories (/ba/, /ga/, /da/ or /ða/) as a function of SOA (N = 18). Auditory-visual “ga” responses (visually driven responses) were seldom given, whereas /ba/ (auditorily driven responses) and /da/ or /ða/ fusion responses formed the majority of responses. The overall trend shows that as the asynchrony between the AV utterances increases, /ba/ judgments increase, whereas /da/ or /ða/ judgments (fusion responses
Discussion
Two experiments were conducted to examine the effects of audiovisual temporal asynchrony on syllable identification and simultaneity judgment. The major finding was that AV speech inputs are extremely tolerant to bimodal asynchrony, and that bimodal information separated in time by as much as 200 ms is usually perceived as simultaneous. Specifically, both the identification experiment and the subjective simultaneity judgment experiment revealed temporal windows of maximal AV integration of about
Acknowledgments
This work was supported by grants NIH DC 0463801 and NIH DC 05660 to DP. A preliminary report of this work was presented at the 31st Annual Meeting of the Society for Neurosciences, San Diego, November 15, 2001 and the 9th Annual Meeting of the Cognitive Neuroscience Society, San Francisco, April 14th, 2002. The opinions or assertions contained herein are the private views of the authors [KG] and are not to be construed as official or as reflecting the views of the Department of the Army or the
References (59)
- et al.
Timing sight and sound
Vision Research
(2005) - et al.
Evidence from functional magnetic resonance imaging of Crossmodal binding in the human heteromodal cortex
Current Biology
(2000) - et al.
A chronic microelectrode investigation of the tonotopic organization of human auditory cortex
Brain Research
(1996) - et al.
Auditory-visual integration as a function of distance: no compensation for sound-transmission time in human perception
Neuroscience Letters
(2004) - et al.
On the relation of speech to language
Trends in Cognitive Sciences
(2000) - et al.
Temporal integration in auditory sensory memory: neuromagnetic evidence
Electroencephalography and Clinical Neurophysiology
(1996) - et al.
Processing of changes in visual speech in the human auditory cortex
Cognitive Brain Research
(2002) - et al.
Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration
Cognitive Brain Research
(2005) The analysis of speech in different temporal integration windows: cerebral lateralization as ‘asymmetric sampling in time’
Speech Communication
(2003)- et al.
Speech-evoked activity in primary auditory cortex: effects of voice onset time
Electroencephalography and Clinical Neurophysiology
(1994)
Audio-visual speech perception is special
Cognition
Recalibration of temporal order perception by exposure to audio-visual asynchrony
Brain Research. Cognitive Brain Research
Automatic discriminative sensitivity inside temporal window of sensory memory as a function of time
Cognitive Brain Research
Organizing sound sequences in the human brain: the interplay of auditory streaming and temporal integration
Brain Research
Hierarchical and asymmetric temporal sensitivity in human auditory cortices
Nature Neuroscience
Fast visual evoked potential input in to human area V5
Neuroreport
Perception of visible speech: influence of spatial quantization
Perception
Organization of auditory cortical areas in man
Brain
Audiovisual asynchrony detection for speech and nonspeech signals
The detection of auditory visual desynchrony
Perception
Visual-auditory distance constancy
Nature
The parallel visual motion inputs into areas V1 and V5 of human cerebral cortex
Brain
Recalibration of audio-visual simultaneity
Nature Neuroscience
Speech intelligibility derived from asynchronous processing of auditory-visual information
Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration
Journal of the Acoustical Society of America
A multi-tier theoretical framework for understanding spoken language
Auditory and auditory-visual perception of clear and conversational speech
Journal of Speech, Language and Hearing Research
Cited by (465)
Audiovisual speech asynchrony asymmetrically modulates neural binding
2024, NeuropsychologiaThe role of attention in immersion: The two–competitor model
2024, Brain Research BulletinDistinctive features of experiential time: Duration, speed and event density
2024, Consciousness and CognitionSensory profiles of children using cochlear implant and auditory brainstem implant
2023, International Journal of Pediatric Otorhinolaryngology