Skip to main content
SpringerLink
Log in

On quantifying multisensory interaction effects in reaction time and detection rate

  • Original Article
  • Published:
Psychological Research Aims and scope Submit manuscript

Abstract

Both mean reaction time (RT) and detection rate (DR) are important measures for assessing the amount of multisensory interaction occurring in crossmodal experiments, but they are often applied separately. Here we demonstrate that measuring multisensory performance using either RT or DR alone misses out on important information. We suggest an integration of RT and DR into a single measure of multisensory performance: the first index (MRE*) is based on an arithmetic combination of RT and DR, the second (MPE) is constructed from parameters derived from fitting a sequential sampling model to RT and DR data simultaneously. Our approach is illustrated by data from two audio–visual experiments. In the first, a redundant targets detection experiment using stimuli of different intensity, both measures yield similar pattern of results supporting the “principle of inverse effectiveness”. The second experiment, introducing stimulus onset asynchrony and differing instructions (focused attention vs. redundant targets task) further supports the usefulness of both indices. Statistical properties of both measures are investigated via bootstrapping procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Note that both a change of response bias and an change of overall performance can provide evidence for multisensory interaction if their occurrence can be attributed to the additional presentation of the auditory stimulus. In this study, however, our interest is directed towards the latter phenomenon.

  2. We are aware of the ongoing debate on whether coactivation effects might be located in the motor component (eg., Diederich & Colonius, 1987; Giray & Ulrich, 1993) or not (eg., Miller, Ulrich & Lamarre, 2001; Mordkoff, Miller, & Roch, 1996). However, the purpose of this paper is not to take a stand in this debate, but rather to provide methods to combine different response measures into one single index of overall performance. Hence, base time as defined here does not include components that are influenced by coactivation effects.

  3. A statistical test for the goodness of fit is given by χ 2(21) = 29.6, p = 0.10 (27 parameters were fitted to 48 data points). Excellent fits were indicated for 5 out of 6 participants (observed χ 2 values of 22.2, 11.9, 21.0, 10.9, and 18.6) and a very poor fit for the sixth participant (observed χ 2 values of 54.6) Note, however, that we did not intend to test the diffusion model with this fit. Instead we utilized the drift rates in a descriptive way to quantify overall performance as indicated by RT and DR.

  4. SOAs where determined individually for each participant in a pilot study. See section “Stimuli” for details.

  5. Note that, for constant θ, β, and T r , all δ i  ∈ Δ are independent from each other, thus can be estimated separately.

References

  • Amlôt, R., Walker, R., Driver, J., & Spence, C. (2003). Multimodal visual–somatosensory integration in saccade generation. Neuropsychologica, 41, 1–15.

    Article  Google Scholar 

  • Arieh, Y., & Marks, L. E. (2008). Cross-modal interaction between vision and hearing: A speed–accuracy analysis. Perception & Psychophysics, 70, 412–421.

    Article  Google Scholar 

  • Arndt, P. A., & Colonius, H. (2003). Two stages in crossmodal saccadic integration: Evidence from a visual–auditory focused attention task. Experimental Brain Research, 150, 417–426.

    Google Scholar 

  • Baird, J. C. (1984). Information theory and information processing. Information Processing and Management, 20, 373–381.

    Article  Google Scholar 

  • Bernstein, I. H., Chu, P. K., Briggs, P., & Schurman, D. L. (1973). Stimulus intensity and foreperiod effects in intersensory facilitation. Journal of Experimental Psychology, 25, 171–181.

    Article  Google Scholar 

  • Bernstein, I. H., Clark, M. E., & Edelstein, B. A. (1969a). Effects of an auditory signal on visual reaction time. Journal of Experimental Psychology, 80(3), 567–569.

    Article  PubMed  Google Scholar 

  • Bernstein, I. H., Clark, M. H., & Edelstein, B. A. (1969b). Intermodal effects in choice reaction time. Journal of Experimental Psychology, 81(2), 405–407.

    Article  Google Scholar 

  • Bernstein, I. H., & Edelstein, B. A. (1971). Effects of some variation in auditory input upon visual choice reaction time. Journal of Experimental Psychology, 87(2), 241–247.

    Article  PubMed  Google Scholar 

  • Bernstein, I. H., Rose, R., & Ashe, V. M. (1970). Energy integration in intersensory facilitation. Journal of Experimental Psychology, 86(2), 196–203.

    Article  PubMed  Google Scholar 

  • Bolognini, N., Frassinetti, F., Serino, A., & Làdavas, E. (2005). “acoustical vision” of below threshold stimuli: interaction among spatially converging audiovisual inputs. Experimental Brain Research, 160, 273–282.

    Article  Google Scholar 

  • Colonius, H., & Diederich, A. (2004). Multisensory interaction in saccadic reaction time: A time-window-of-integration model. Journal of Cognitive Neuroscience, 16(6), 1–10.

    Article  Google Scholar 

  • Corneil, B. D., Van Wanrooij, M., Munoz, D. P., & Van Opstal, A. J. (2002). Auditory–visual interactions subserving goal-directed saccades in a complex scene. Journal of Neurophysiology, 88(1), 438–454.

    PubMed  Google Scholar 

  • Diederich, A. (1992). Intersensory facilitation: Race, superposition, and diffusion models for reaction time to multiple stimuli (Dissertation). Frankfurt: Verlag Peter Lang.

  • Diederich, A. (1995). Intersensory facilitation of reaction time: Evaluation of counter and diffusion coactivation models. Journal of Mathematical Psychology, 39, 197–215.

    Article  Google Scholar 

  • Diederich, A. (1997). Dynamic stochastic models for decision making with time constraints. Journal of Mathematical Psychology, 41, 260–274.

    Article  PubMed  Google Scholar 

  • Diederich, A. (2008). A further test of sequential-sampling models that account for payoff effects on response bias in perceptual decision tasks. Perception & Psychophysics, 70, 229–256.

    Article  Google Scholar 

  • Diederich, A., & Busemeyer, J. R. (2003). Simple matrix methods for analyzing diffusion models of choice probability, choice response time and simple response time. Journal of Mathematical Psychology, 47, 304–322.

    Article  Google Scholar 

  • Diederich, A., & Busemeyer, J. R. (2006). Modeling the effects of payoff on response bias in a perceptual discrimination task: Threshold-bound, drift-rate-change, or two-stage-processing hypothesis. Perception & Psychophysics, 68, 194–207.

    Article  Google Scholar 

  • Diederich, A., & Colonius, H. (1987). Intersensory facilitation in the motor component? A reaction time analysis. Psychological Research, 49, 23–29.

    Article  Google Scholar 

  • Diederich, A., & Colonius, H. (2004). Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time. Perception & Psychophysics, 66(8), 1388–1404.

    Article  Google Scholar 

  • Efron, B., & Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1, 54–75.

    Article  Google Scholar 

  • Eimer, M. (2001). Crossmodal links in spatial attention between vision, audition, and touch: evidence from event-related brain potentials. Neuropsychologia, 39, 1292–1303.

    Article  PubMed  Google Scholar 

  • Fitts, P. M. (1966). Cognitive aspects of information processing. III. Set for speed versus accuracy. Journal of Experimental Psychology, 71, 849–857.

    Article  PubMed  Google Scholar 

  • Frassinetti, F., Bolognini, N., & Làdavas, E. (2002). Enhancement of visual perception by crossmodal visuo-auditory interaction. Experimental Brain Research, 147, 332–343.

    Article  Google Scholar 

  • Frens, M. A., Van Opstal, A. J., & Van der Willigen, R. F. (1995). Spatial and temporal factors determine auditory–visual interaction in human saccadic eye movements. Perception & Psychophysics, 57(6), 802–816.

    Article  Google Scholar 

  • Gielen, S. C. A. M., Schmidt, R. A., & Van den Heuvel, P. J. M. (1983). On the nature of intersensory facilitation of reaction time. Perception & Psychophysics, 34(2), 161–168.

    Article  Google Scholar 

  • Gillmeister, H., & Eimer, M. (2007). Tactile enhancement of auditory detection and perceived loudness. Brain Research, 1160, 58–68.

    Article  PubMed  Google Scholar 

  • Giray, M., & Ulrich, R. (1993). Motor coactivation revealed by response force in divided and focused attention. Journal of Experimental Psychology: Human Perception and Performance, 19(6), 1278–1291.

    Article  PubMed  Google Scholar 

  • Gomez, P., Ratcliff, R., & Perea, M. (2007). A model of the go/no-go task. Journal of Experimental Psychology: General, 136, 389–413.

    Article  Google Scholar 

  • Harrington, L.K., & Peck, C.K. (1998). Spatial disparity affects visual–auditory interactions in human sensorimotor processing. Experimental Brain Research, 122, 247–252.

    Article  Google Scholar 

  • Hershenson, M. (1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63(3), 289–293.

    Article  PubMed  Google Scholar 

  • Hick, W. (1952). On the rate of gain of information. Quarterly Journal of Experimental Psycholology, 4, 11–26.

    Article  Google Scholar 

  • Hilgard, E. R. (1933). Reinforcement and inhibition of eyelid reflexes. Journal of General Psychology, 8, 85–113.

    Article  Google Scholar 

  • Holmes, N. P. (2009). The principle of inverse effectiveness in multisensory integration: Some statistical considerations. Brain Topography, 21, 168–176.

    Article  PubMed  Google Scholar 

  • Jepma, M., Wagenmakers, E.-J., Band, G. P. H., & Nieuwenhuis, S. (2008). The effects of accessory stimuli on information processing: Evidence from electrophysiology and a diffusion model analysis. Journal of Cognitive Neuroscience, 21, 847–864.

    Article  Google Scholar 

  • Kitagawa, N., & Spence, C. (2005). Investigating the effect of a transparent barrier on the crossmodal congruency effect. Experimental Brain Research, 161, 62–71.

    Article  Google Scholar 

  • Lovelace, C. T., Stein, B. E., & Wallace, M. T. (2003). An irrelevant light enhances auditory detection in humans: a psychophysical analysis of multisensory integration in stimulus detection. Cognitive Brain Research, 17, 447–453.

    Article  PubMed  Google Scholar 

  • Luce, R. D. (1986). Response times: Their role in inferring elementary mental organisation. Oxford Psychology Series; no. 8. New York: Oxford University Press.

    Google Scholar 

  • Luce, R. D. (2003). Whatever happened to information theory in psychology? Review of General Psychology, 7, 183–188.

    Article  Google Scholar 

  • Meredith, M. A., Nemitz, J. W., & Stein, B. E. (1987). Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors. Journal of Neuroscience, 7(10), 3215–3229.

    PubMed  Google Scholar 

  • Meredith, M. A. & Stein, B. E. (1986). Visual, auditory and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56(3), 640–662.

    PubMed  Google Scholar 

  • Miller, J. O. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247–279.

    Article  PubMed  Google Scholar 

  • Miller, J. O. (1986). Time course of coactivation in bimodal divided attention. Perception & Psychophysics, 40(5), 331–343.

    Article  Google Scholar 

  • Miller, J. O., Ulrich, R., & Lamarre, Y. (2001). Locus of the redundant-signals effect in bimodal divided attention: a neurophysiological analysis. Perception & Psychophysics, 63(3), 555–562.

    Article  Google Scholar 

  • Mordkoff, J. T., Miller, J., & Roch, A.-C. (1996). Absence of coactivation in the motor component: evidence from psychophysiological measures of target detection. JJournal of Experimental Psychology: Human Perception and Performance, 22(1), 25–41.

    Article  PubMed  Google Scholar 

  • Morrell, E. K. (1968). Temporal characteristics of sensory interaction in choice reaction times. Journal of Experimental Psychology, 77(1), 14–18.

    Article  PubMed  Google Scholar 

  • Nickerson, R. S. (1973). Intersensory facilitation of reaction time: Energy summation or preparation enhancement. Psychological Review, 80, 489–509.

    Article  PubMed  Google Scholar 

  • Perrault, T. J., Vaughan, J. W., Stein, B. E., & Wallace, M. T. (2005). Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J Neurophysiology, 93, 2575–2586.

    Article  Google Scholar 

  • Proctor, R. W., & Vu, K.-P.L. (2006). The cognitive revolution at age 50: Has the promise of the human information-processing approach been fulfilled? International Journal of Human–Computer Interaction, 21, 253–284.

    Article  Google Scholar 

  • Rach, S., & Diederich, A. (2006). Visual–tactile integration: does stimulus duration influence the relative amount of response enhancement? Experimental Brain Research, 173, 514–520.

    Article  Google Scholar 

  • Ratcliff, R., & Smith, P. L. (2004). A comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111(2), 333–367.

    Article  PubMed  Google Scholar 

  • Röder, B., Kusmierek, A., Spence, C., & Schicke, T. (2007). Developmental vision determines the reference frame for the multisensory control of action. PNAS, 104, 4753–4758.

    Article  PubMed  Google Scholar 

  • Schwarz, W. (1994). Diffusion, superposition, and the redundant-targets effect. Journal of Mathematical Psychology, 38, 504–520.

    Article  Google Scholar 

  • Shore, D. I., Barnes, M. E., & Spence, C. (2006). Temporal aspects of the visuotactile congruency effect. Neuroscience Letters, 392, 96–100.

    Article  PubMed  Google Scholar 

  • Spence, C., McGlone, F. P., Kettenmann, B., & Kobal, G. (2001). Attention to olfaction: A psychophysical investigation. Experimental Brain Research, 138, 432–437.

    Article  Google Scholar 

  • Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. London, Cambridge: The MIT Press.

    Google Scholar 

  • Townsend, J. T., & Ashby, F. G. (1983). Stochastic modelling of elementary psychological processes. New York: Cambridge Univ Press.

    Google Scholar 

  • Townsend, J. T., & Honey, C. J. (2007). Consequences of base time for redundant signals experiments. Journal of Mathematical Psychology, 51, 242–265.

    Article  PubMed  Google Scholar 

  • Walker, R., Deubel, H., Schneider, W. X., & Findlay, J. M. (1997). Effect of remote distractors on saccade programming: Evidence for an extended fixation zone. Journal of Neurophysiology, 78, 1108–1119.

    PubMed  Google Scholar 

  • Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: II. Bootstrap-based confidence intervals and sampling. Perception & Psychophysics, 63, 1314–1329.

    Article  Google Scholar 

  • Woodworth, R. S., & Schlosberg, H. (1956). Experimental psychology (rev. ed.). New York: Holt.

Download references

Acknowledgments

This research was supported by Deutsche Forschungsgemeinschaft (DFG) Grant No. Di 506/8-1 to A.D. and by SFB/TR31 “Active Hearing”, Teilprojekt B4 to H.C.

Author information

Authors and Affiliations

  1. Department of Psychology, University of Oldenburg, 26111, Oldenburg, Germany

    Stefan Rach & Hans Colonius

  2. Jacobs University Bremen, Bremen, Germany

    Stefan Rach & Adele Diederich

Authors
  1. Stefan Rach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adele Diederich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hans Colonius
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Rach.

Additional information

This paper is part of a dissertation submitted by S.R. in partial fulfillment of the requirements for the degree of Doctor of Philosophy at Jacobs University Bremen, Germany.

Appendix

Appendix

Bootstrap procedure

For each of 6 participants, the original data set consisted of responses to 32 trials for each of 24 conditions. RTs for trials without a valid response were set to 0. For the non-parametric bootstrap, 1,000 random samples were generated as follows.

Repeat 1,000 times:

  1. 1.

    Draw a set of six participants with replacement from the original set of participants.

  2. 2.

    For each of these participants:

    1. (a)

      for each of 24 conditions, draw a random sample with replacement from the 32 responses and calculate mean RT (mean of non-zero elements) and DR (number of non-zero elements divided by 32), resulting in a simulated data set (24 pairs of mean RT and DR).

    2. (b)

      calculate RT* and MRE* from the simulated data set.

    3. (c)

      fit a sequential sampling model to the simulated data set to obtain drift rates and calculate RDC.

  3. 3.

    Average the obtained measures across the set of six participants yields RT*, MRE*, the drift rates, and RDC for one random sample.

Parameter estimation

For the original data set and each of 1,000 resampled non-parametric bootstrap data sets, 24 drift rates δ i  ∈ Δ, one boundary separation θ, one β, and one residual time T r were estimated from 24 RT and 24 DRs (8 intensity levels × 3 modalities). The estimation routine utilized the following constraints on parameters: each δ i  ∈ Δ was bound to 0.001 < δ i  < 0.999, β was bound to − 1 <  δ i  < 1, and the residual time was bound to T r ≥0. Parameter estimation was attempted in a three-step procedure:

  • Step 1: Estimate parameters θ, β, T r , and Δ by minimizing χ 2. Store χ 2.

  • Step 2: Hold constant θ, β, and T r , while estimating each δ i  ∈ Δ individually by minimizing χ 2. Footnote 5

  • Step 3: Estimate parameters θ, β, T r , and Δ by minimizing χ 2. If the new χ 2 is smaller than the stored one, jump to Step 2, or break, otherwise.

MatLab’s (The Mathworks, Natick, MA) routine FMINSEARCH was used for parameter estimation.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rach, S., Diederich, A. & Colonius, H. On quantifying multisensory interaction effects in reaction time and detection rate. Psychological Research 75, 77–94 (2011). https://doi.org/10.1007/s00426-010-0289-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00426-010-0289-0

Keywords

Search

Navigation