Temporal window of integration in auditory-visual speech perception

doi:10.1016/j.neuropsychologia.2006.01.001

Neuropsychologia

Volume 45, Issue 3, 2007, Pages 598-607

https://doi.org/10.1016/j.neuropsychologia.2006.01.001 Get rights and content

Abstract

Forty-three normal hearing participants were tested in two experiments, which focused on temporal coincidence in auditory visual (AV) speech perception. In these experiments, audio recordings of/pa/and/ba/were dubbed onto video recordings of /ba/or/ga/, respectively (A_pV_k, A_bV_g), to produce the illusory “fusion” percepts /ta/, or /da/ [McGurk, H., & McDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–747]. In Experiment 1, an identification task using McGurk pairs with asynchronies ranging from −467 ms (auditory lead) to +467 ms was conducted. Fusion responses were prevalent over temporal asynchronies from −30 ms to +170 ms and more robust for audio lags. In Experiment 2, simultaneity judgments for incongruent and congruent audiovisual tokens (A_dV_d, A_tV_t) were collected. McGurk pairs were more readily judged as asynchronous than congruent pairs. Characteristics of the temporal window over which simultaneity and fusion responses were maximal were quite similar, suggesting the existence of a 200 ms duration asymmetric bimodal temporal integration window.

Section snippets

Participants

Participants (native speakers of American English) were recruited from the University of Maryland undergraduate population and provided informed consent. Two groups of participants took part in this study. The first group included 21 participants (11 females, average 21 years) who were run in the voiced A_bV_g condition (A_bV_g: audio /b/ and video /g/). The second group consisted of 22 participants (8 females, average 22.5 years) who were run in the voiceless A_pV_k condition (A_pV_k: audio /p/ and

Voiced McGurk pair A_bV_g

Fig. 1 shows the distribution (in percent) of each of the three possible response categories (/ba/, /ga/, /da/ or /ða/) as a function of SOA (N = 18). Auditory-visual “ga” responses (visually driven responses) were seldom given, whereas /ba/ (auditorily driven responses) and /da/ or /ða/ fusion responses formed the majority of responses. The overall trend shows that as the asynchrony between the AV utterances increases, /ba/ judgments increase, whereas /da/ or /ða/ judgments (fusion responses

Discussion

Two experiments were conducted to examine the effects of audiovisual temporal asynchrony on syllable identification and simultaneity judgment. The major finding was that AV speech inputs are extremely tolerant to bimodal asynchrony, and that bimodal information separated in time by as much as 200 ms is usually perceived as simultaneous. Specifically, both the identification experiment and the subjective simultaneity judgment experiment revealed temporal windows of maximal AV integration of about

Acknowledgments

This work was supported by grants NIH DC 0463801 and NIH DC 05660 to DP. A preliminary report of this work was presented at the 31st Annual Meeting of the Society for Neurosciences, San Diego, November 15, 2001 and the 9th Annual Meeting of the Cognitive Neuroscience Society, San Francisco, April 14th, 2002. The opinions or assertions contained herein are the private views of the authors [KG] and are not to be construed as official or as reflecting the views of the Department of the Army or the

References (59)

D.H. Arnold et al.
Timing sight and sound
Vision Research
(2005)
G.A. Calvert et al.
Evidence from functional magnetic resonance imaging of Crossmodal binding in the human heteromodal cortex
Current Biology
(2000)
M.A. Howard et al.
A chronic microelectrode investigation of the tonotopic organization of human auditory cortex
Brain Research
(1996)
J. Lewald et al.
Auditory-visual integration as a function of distance: no compensation for sound-transmission time in human perception
Neuroscience Letters
(2004)
A.M. Liberman et al.
On the relation of speech to language
Trends in Cognitive Sciences
(2000)
N. Loveless et al.
Temporal integration in auditory sensory memory: neuromagnetic evidence
Electroencephalography and Clinical Neurophysiology
(1996)
R. Möttönen et al.
Processing of changes in visual speech in the human auditory cortex
Cognitive Brain Research
(2002)
J. Navarra et al.
Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration
Cognitive Brain Research
(2005)
D. Poeppel
The analysis of speech in different temporal integration windows: cerebral lateralization as ‘asymmetric sampling in time’
Speech Communication
(2003)
M. Steinschneider et al.
Speech-evoked activity in primary auditory cortex: effects of voice onset time
Electroencephalography and Clinical Neurophysiology
(1994)

J. Tuomainen et al.

Audio-visual speech perception is special

Cognition

(2005)

J. Vroomen et al.

Recalibration of temporal order perception by exposure to audio-visual asynchrony

Brain Research. Cognitive Brain Research

(2004)

H. Yabe et al.

Automatic discriminative sensitivity inside temporal window of sensory memory as a function of time

Cognitive Brain Research

(2001)

H. Yabe et al.

Organizing sound sequences in the human brain: the interplay of auditory streaming and temporal integration

Brain Research

(2001)

Arai, T., & Greenberg, S. (1997). The temporal properties of spoken Japanese are similar to those of English. In...

A. Boemio et al.

Hierarchical and asymmetric temporal sensitivity in human auditory cortices

Nature Neuroscience

(2005)

H. Buchner et al.

Fast visual evoked potential input in to human area V5

Neuroreport

(1997)

C.S. Campbell et al.

Perception of visible speech: influence of spatial quantization

Perception

(1997)

G.G. Celesia

Organization of auditory cortical areas in man

Brain

(1976)

B.L. Conrey et al.

Audiovisual asynchrony detection for speech and nonspeech signals

N.F. Dixon et al.

The detection of auditory visual desynchrony

Perception

(1980)

G.R. Engel et al.

Visual-auditory distance constancy

Nature

(1971)

D.H. Ffytche et al.

The parallel visual motion inputs into areas V1 and V5 of human cerebral cortex

Brain

(1995)

W. Fujisaki et al.

Recalibration of audio-visual simultaneity

Nature Neuroscience

(2004)

K.W. Grant et al.

Speech intelligibility derived from asynchronous processing of auditory-visual information

K. Grant et al.

Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration

Journal of the Acoustical Society of America

(1998)

Greenberg, S. (1996). Understanding speech understanding: towards a unified theory of speech perception. In Proceedings...

S. Greenberg

A multi-tier theoretical framework for understanding spoken language

K.S. Helfer

Auditory and auditory-visual perception of clear and conversational speech

Journal of Speech, Language and Hearing Research

(1997)

Cited by (465)

Audiovisual speech asynchrony asymmetrically modulates neural binding
2024, Neuropsychologia
Previous psychophysical and neurophysiological studies in young healthy adults have provided evidence that audiovisual speech integration occurs with a large degree of temporal tolerance around true simultaneity. To further determine whether audiovisual speech asynchrony modulates auditory cortical processing and neural binding in young healthy adults, N1/P2 auditory evoked responses were compared using an additive model during a syllable categorization task, without or with an audiovisual asynchrony ranging from 240 ms visual lead to 240 ms auditory lead. Consistent with previous psychophysical findings, the observed results converge in favor of an asymmetric temporal integration window. Three main findings were observed: 1) predictive temporal and phonetic cues from pre-phonatory visual movements before the acoustic onset appeared essential for neural binding to occur, 2) audiovisual synchrony, with visual pre-phonatory movements predictive of the onset of the acoustic signal, was a prerequisite for N1 latency facilitation, and 3) P2 amplitude suppression and latency facilitation occurred even when visual pre-phonatory movements were not predictive of the acoustic onset but the syllable to come. Taken together, these findings help further clarify how audiovisual speech integration partly operates through two stages of visually-based temporal and phonetic predictions.
The role of attention in immersion: The two–competitor model
2024, Brain Research Bulletin
Currently, we face an exponentially increasing interest in immersion, especially sensory–driven immersion, mainly due to the rapid development of ideas and business models centered around a digital virtual universe as well as the increasing availability of affordable immersive technologies for education, communication, and entertainment. However, a clear definition of ‘immersion’, in terms of established neurocognitive concepts and measurable properties, remains elusive, slowing research on the human side of immersive interfaces.
To address this problem, we propose a conceptual, taxonomic model of attention in immersion. We argue (a) modeling immersion theoretically as well as studying immersion experimentally requires a detailed characterization of the role of attention in immersion, even though (b) attention, while necessary, cannot be a sufficient condition for defining immersion. Our broader goal is to characterize immersion in terms that will be compatible with established psychophysiolgical measures that could then in principle be used for the assessment and eventually the optimization of an immersive experience. We start from the perspective that immersion requires the projection of attention to an induced reality, and build on accepted taxonomies of different modes of attention for the development of our two–competitor model. The two–competitor model allows for a quantitative implementation and has an easy graphical interpretation. It helps to highlight the important link between different modes of attention and affect in studying immersion.
Distinctive features of experiential time: Duration, speed and event density
2024, Consciousness and Cognition
William James’s use of “time in passing” and “stream of thoughts” may be two sides of the same coin that emerge from the brain segmenting the continuous flow of information into discrete events. Herein, we investigated how the density of events affects two temporal experiences: the felt duration and speed of time. Using a temporal bisection task, participants classified seconds-long videos of naturalistic scenes as short or long (duration), or slow or fast (passage of time). Videos contained a varying number and type of events. We found that a large number of events lengthened subjective duration and accelerated the felt passage of time. Surprisingly, participants were also faster at estimating their felt passage of time compared to duration. The perception of duration scaled with duration and event density, whereas the felt passage of time scaled with the rate of change. Altogether, our results suggest that distinct mechanisms underlie these two experiential times.
Sensory profiles of children using cochlear implant and auditory brainstem implant
2023, International Journal of Pediatric Otorhinolaryngology
Hierarchically nested networks optimize the analysis of audiovisual speech
2023, iScience
In conversational settings, seeing the speaker’s face elicits internal predictions about the upcoming acoustic utterance. Understanding how the listener’s cortical dynamics tune to the temporal statistics of audiovisual (AV) speech is thus essential. Using magnetoencephalography, we explored how large-scale frequency-specific dynamics of human brain activity adapt to AV speech delays. First, we show that the amplitude of phase-locked responses parametrically decreases with natural AV speech synchrony, a pattern that is consistent with predictive coding. Second, we show that the temporal statistics of AV speech affect large-scale oscillatory networks at multiple spatial and temporal resolutions. We demonstrate a spatial nestedness of oscillatory networks during the processing of AV speech: these oscillatory hierarchies are such that high-frequency activity (beta, gamma) is contingent on the phase response of low-frequency (delta, theta) networks. Our findings suggest that the endogenous temporal multiplexing of speech processing confers adaptability within the temporal regimes that are essential for speech comprehension.
The Influence of Multisensory Input On Voice Perception and Production Using Immersive Virtual Reality
2023, Journal of Voice
The purpose was to examine the influence of auditory vs visual vs combined audiovisual input on perception and production of one’s own voice, using immersive virtual reality technology.
Thirty-one vocally healthy men and women were investigated under 18 sensory input conditions, using immersive virtual reality technology. Conditions included two auditory rooms with varying reverberation times, two visual rooms with varying volumes, and the combination of audiovisual conditions. All conditions were repeated with and without background noise. Speech tasks included counting, sustained vowel phonation, an all-voiced sentence from the Consensus Auditory-Perceptual Evaluation of Voice, and the first sentence from the Rainbow Passage, randomly ordered. Perception outcome measures were participants’ self-reported perceptions of their vocal loudness, vocal effort, and vocal comfort in speech. Production outcome measures were sound pressure level (SPL) and spectral moments (spectral mean and standard deviation in Hz, skewness, and kurtosis). Statistical analyses used self-reported vocal effort, vocal loudness, and vocal comfort in percent (0 = “not at all,” 100 = extremely), SPL in dB, and spectral moments in Hz. The reference level was a baseline audiovisual deprivation condition.
Results suggested (i) increased self-perceived vocal loudness and effort, and decreased comfort, with increasing room volume, speaker-to-listener distance, audiovisual input, and background noise, and (ii) increased SPL and fluctuations in spectral moments across conditions.
Not only auditory, but also visual and audiovisual input influenced voice perception and production in ways that have not been previously documented. Findings contribute to the basic science understanding the role of visual, audiovisual and auditory input in voice perception and production, and also to models of voice training and therapy. The findings also set the foundation for the use of virtual reality in voice and speech training, as a potentially power solution to the generalization problem.

View all citing articles on Scopus

View full text

Temporal window of integration in auditory-visual speech perception

Abstract

Section snippets

Participants

Voiced McGurk pair AbVg

Discussion

Acknowledgments

Vision Research

Current Biology

Brain Research

Neuroscience Letters

Trends in Cognitive Sciences

Electroencephalography and Clinical Neurophysiology

Cognitive Brain Research

Cognitive Brain Research

Speech Communication

Electroencephalography and Clinical Neurophysiology

Cognition

Brain Research. Cognitive Brain Research

Cognitive Brain Research

Brain Research

Hierarchical and asymmetric temporal sensitivity in human auditory cortices

Nature Neuroscience

Fast visual evoked potential input in to human area V5

Neuroreport

Perception of visible speech: influence of spatial quantization

Perception

Organization of auditory cortical areas in man

Brain

Audiovisual asynchrony detection for speech and nonspeech signals

The detection of auditory visual desynchrony

Perception

Visual-auditory distance constancy

Nature

The parallel visual motion inputs into areas V1 and V5 of human cerebral cortex

Brain

Recalibration of audio-visual simultaneity

Nature Neuroscience

Speech intelligibility derived from asynchronous processing of auditory-visual information

Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration

Journal of the Acoustical Society of America

A multi-tier theoretical framework for understanding spoken language

Auditory and auditory-visual perception of clear and conversational speech

Journal of Speech, Language and Hearing Research

Voiced McGurk pair A_bV_g