Skip to main content
SpringerLink
Log in

Audiovisual temporal order judgments

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

In two experiments, we examined the extent to which audiovisual temporal order judgments (TOJs) were affected by spatial factors and by the dimension along which TOJs were made. Pairs of auditory and visual stimuli were presented from either the left and/or right of fixation at varying stimulus onset asynchronies (SOAs), and participants made unspeeded TOJs regarding either "Which modality was presented first?" (experiment 1), or "Which side was presented first?" (experiment 2). Modality TOJs were more accurate (i.e. just-noticeable differences, JNDs, were smaller) when the auditory and visual stimuli were presented from different spatial positions rather than from the same position, highlighting an important potential confound inherent in previous research. By contrast, spatial TOJs were unaffected by whether or not the two stimuli were presented in different modalities. A between-experiments comparison revealed more accurate performance (i.e. smaller JNDs) when people reported which modality came first than when they reported which side came first for identical bimodal stimulus pairs. These results demonstrate that multisensory TOJs are critically dependent on both the relative spatial position from which stimuli are presented and on the particular dimension being judged.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Similar content being viewed by others

Notes

  1. In the Introduction to their paper, Hirsh and Sherrick (1961) point out that performance on simultaneous/successive judgment tasks will be affected by the dimension along which stimuli vary, despite the fact that they do not consider this point for the TOJ studies they report. However, this apparent inconsistency may simply reflect another of Hirsh and Sherrick's underlying assumptions, namely, that TOJs and simultaneous/successive judgment tasks rely on very different neural mechanisms.

  2. The +200 ms points were excluded from this computation, because most participants performed nearly perfectly at this interval, so no additional variance was accounted for by these points. The inclusion of these points would actually have resulted in an artefactual reduction in slope.

  3. This hemispheric redundancy account can be considered as a specific example of a more general "spatial channel" account (see Regan 1982 for a review). That is, stimuli presented from the same spatial location may activate a common limited capacity spatial channel, whereas stimuli presented to different spatial positions may activate different spatial channels. However, we prefer the hemispheric redundancy account over the spatial channel account given the problems that are associated with the appropriate definition of a channel (see Regan 1982; Spence et al. 2001b).

  4. Because light travels through air far more rapidly than sound, these differences in transduction latencies can be offset, or even reversed, for pairs of audiovisual stimuli that occur very far from us (see Pöppel et al. 1990). For example, we often see distant lightening before hearing the associated thunder (Spence and Squire 2003). However, such differences are unlikely to have had any significant effect in the present studies since the auditory and visual stimuli were presented relatively close to the participants (distance=62 cm; see also Sugita and Suzuki 2003).

References

  • Allan LG (1975) The relationship between judgments of successiveness and judgments of order. Percept Psychophys 18:29–36

    Google Scholar 

  • Bald L, Berrien FK, Price JB, Sprague RO (1942) Errors in perceiving the temporal order of auditory and visual stimuli. J Appl Psychol 26:382–388

    Google Scholar 

  • Banich MT (1998) The missing link: the role of interhemispheric interaction in attentional processing. Brain Cogn 36:128–157

    Article  CAS  PubMed  Google Scholar 

  • Bergenheim M, Johansson H, Granlund B, Pedersen J (1996) Experimental evidence for a sensory synchronization of sensory information to conscious experience. In: Hameroff SR, Kaszniak AW, Scott AC (eds) Toward a science of consciousness: the first Tucson discussions and debates. MIT Press, London, UK, pp 303–310

  • Bertelson P (1998) Starting from the ventriloquist: the perception of multimodal events. In: Sabourin M, Craik FIM, Robert M (eds) Biological and cognitive aspects. (Advances in psychological science, Vol 2) Psychological, Hove, UK, pp 419–439

  • Bertelson P, Gelder B de (2003) The psychology of multimodal perception. In: Spence C, Driver J (eds) Crossmodal space and crossmodal attention. Oxford University Press, Oxford, UK

  • Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onset asynchrony detection. J Neurosci 21:300–304

    CAS  PubMed  Google Scholar 

  • Calvert GA, Brammer MJ, Iversen SD (1998) Crossmodal identification. Trends Cognit Sci 2:247–253

    Article  Google Scholar 

  • Dinnerstein AJ, Zlotogura P (1968) Intermodal perception of temporal order and motor skills: effects of age. Percept Mot Skill 26:987–1000

    CAS  Google Scholar 

  • Drew F (1896) Attention: experimental and critical. Am J Psychol 7:533–573

    Google Scholar 

  • Driver J, Spence C (1998) Crossmodal links in spatial attention. Philos Trans R Soc B 353:1319–1331

    CAS  Google Scholar 

  • Driver J, Spence C (2000) Multisensory perception: beyond modularity and convergence. Curr Biol 10:731–735

    PubMed  Google Scholar 

  • Exner S (1875) Experimentelle Untersuchung der einfachsten psychischen Processe (Experimental study of the most simple psychological processes). Pflugers Arch 11:403–432

    Google Scholar 

  • Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophys 63:719–725

    CAS  PubMed  Google Scholar 

  • Finger R, Davis AW (2001) Measuring video quality in videoconferencing systems. Technical Report SN187-D. Pixel Instrument Corporation, Los Gatos, CA

  • Finney DJ (1964) Probit analysis: statistical treatment of the sigmoid response curve. Cambridge University Press, London, UK

    Google Scholar 

  • Friedman A, Polson MC (1981) Hemispheres as independent resource systems: limited-capacity processing and cerebral specialization. J Exp Psychol Hum Percept Perform 7:1031–1058

    Article  CAS  PubMed  Google Scholar 

  • Gebhardt JW, Mowbray GH (1959) On discriminating the rate of visual flicker and auditory flutter. Am J Psychol 72:521–528

    CAS  Google Scholar 

  • Gengel RW, Hirsh IJ (1970) Temporal order: the effect of single versus repeated presentations, practice, and verbal feedback. Percept Psychophys 7:209–211

    Google Scholar 

  • Grossberg S, Grunewald A (1997) Cortical synchronization and perceptual framing. J Cognit Neurosci 9:117–132

    Google Scholar 

  • Hamlin AJ (1895) On the least observable interval between stimuli addressed to disparate senses and to different organs of the same sense. Am J Psychol 6:564–575

    Google Scholar 

  • Herdman CM, Friedman A (1985) Multiple resources in divided attention: a cross modal test of independence of hemispheric resources. J Exp Psychol Hum Percept Perform 11:40–49

    Article  CAS  PubMed  Google Scholar 

  • Hirsh IJ (1959) Auditory perception of temporal order. J Acoust Soc Am 31:759–767

    Google Scholar 

  • Hirsh IJ, Sherrick CE Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432

    CAS  Google Scholar 

  • ITU-T (1990) Television and sound transmission: tolerances for transmission time differences between the vision and sound components of a television signal. Telecommunication standardization sector of ITU, Recommendation J100, CMTT 717 in CCIR Recommendations 12. International Telecommunication Union, Düsseldorf, Germany

    Google Scholar 

  • Jaskowski P, Jaroszyk F, Hojan-Jerierska D (1990) Temporal-order judgments and reaction time for stimuli of different modalities. Psychol Res 52:35–38

    CAS  PubMed  Google Scholar 

  • Kahneman D, Treisman A, Gibbs BJ (1992) The reviewing of object files: object-specific integration of information. Cognit Psychol 24:175–219

    CAS  PubMed  Google Scholar 

  • King AJ, Palmer AR (1985) Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus. Exp Brain Res 60:492–500

    CAS  PubMed  Google Scholar 

  • Kinsbourne M, Cook J (1971) Generalized and lateralized effects of concurrent verbalization on a unimanual skill. Q J Exp Psychol 23:341–345

    Google Scholar 

  • Kinsbourne M, Hicks RE (1978) Functional cerebral space: a model for overflow, transfer and interference effects in human performance: a tutorial review. In: Requin J (ed) Attention and performance, VII. Erlbaum, Hillsdale, NJ, pp 345–362

  • Kristofferson AB (1967) Attention and psychophysical time. Acta Psychol 27:93–100

    Article  CAS  Google Scholar 

  • Lewald J, Ehrenstein WH, Guski R (2001) Spatio-temporal constraints for auditory-visual integration. Behav Brain Res 121:69–79

    Article  CAS  PubMed  Google Scholar 

  • McFarland DJ, Cacace AT, Setzen G (1998) Temporal-order discrimination for selected auditory and visual stimulus dimensions. J Speech, Lang Hearing Res 41:300–314

  • McGrath M, Summerfield Q (1985) Intermodal timing relations and audio-visual speech recognition by normal-hearing adults. J Acoust Soc Am 77:678–685

    CAS  PubMed  Google Scholar 

  • Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391

    CAS  PubMed  Google Scholar 

  • Mitrani L, Shekerdjiski S, Yakimoff N (1986) Mechanisms and asymmetries in visual perception of simultaneity and temporal order. Biol Cybern 54:159–165

    CAS  PubMed  Google Scholar 

  • Morein-Zamir S, Soto-Faraco S, Kingstone AF (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17:154–163

  • Mortlock AN, Machin D, McConnell S, Sheppard P (1997) Virtual conferencing. BT Technol J 15:120–129

    Article  Google Scholar 

  • Moutoussis K, Zeki S (1997) A direct demonstration of perceptual asynchrony in vision. Proc R Soc Lond B Biol 264:393–399

    Article  CAS  Google Scholar 

  • Neumann O, Niepel M (2003) Timing of "perception" and perception of "time". In: Kaernbach C, Schröger E, Müller H (eds) Psychophysics beyond sensation: laws and invariants of human cognition. (Scientific Psychology Series) Erlbaum, Hove, UK

  • Pandev PC, Kunov H, Abel SM (1986) Disruptive effects of auditory signal delay on speech perception with lip-reading. J Aud Res 26:27–41

    PubMed  Google Scholar 

  • Pöppel E, Schill K, Steinbüchel N von (1990) Sensory integration within temporally neutral system states: a hypothesis. Naturwissenschaften 77:89–91

    PubMed  Google Scholar 

  • Posner MI, Nissen MJ, Klein RM (1976) Visual dominance: an information-processing account of its origins and significance. Psychol Rev 83:157–171

    CAS  PubMed  Google Scholar 

  • Rains JD (1963) Signal luminance and position effects in human reaction time. Vision Res 3:239–251

    Article  Google Scholar 

  • Raizada RDS, Poldrack RA (2001) Event-related fMRI of audio-visual simultaneity perception. Soc Neurosci Abstr 27:511.14

    Google Scholar 

  • Rao SM, Mayer AR Harrington DL (2001) The evolution of brain activation during temporal processing. Nat Neurosci 4:317–323

    Article  CAS  PubMed  Google Scholar 

  • Reeves B, Voelker D (1993) Effects of audio-video asynchrony on viewer's memory, evaluation of content and detection ability. Research report. Pixel Instruments, Los Gatos, CA

  • Regan D (1982) Visual information channeling in normal and disordered vision. Psychol Rev 89:407–444

    Article  CAS  PubMed  Google Scholar 

  • Regan D, Spekreijse H (1977) Auditory-visual interactions and the correspondence between perceived auditory space and perceived visual space. Perception 6:133–138

    CAS  PubMed  Google Scholar 

  • Rihs S (1995) The influence of audio on perceived picture quality and subjective audio-visual delay tolerance. In: Hamberg R, Ridder H de (eds) Report 1071. Proc MOSAIC workshop, Eindhoven, 18–19 Sept: Advanced methods for the evaluation of television picture quality. Inst Percept Res, pp 133–137

  • Roberson GE, Hairston WD, Wallace MT, Stein BE, Laurienti PJ, Schirillo JA (2001) Unifying multisensory signals across time and space. Soc Neurosci Abstr 27:511.19

    Google Scholar 

  • Rutschmann J, Link R (1964) Perception of temporal order of stimuli differing in sense mode and simple reaction time. Percept Mot Skills 18:345–352

    CAS  Google Scholar 

  • Sade A, Spitzer H (1998) The effects of attentional spread and attentional effort on orientation discrimination. Spatial Vision 11:367–383

    CAS  PubMed  Google Scholar 

  • Scheier CR, Nijhawan R, Shimojo S (1999) Sound alters visual temporal resolution. Invest Ophthalmol Vis Sci 40:S792

    Google Scholar 

  • Shore DI, Spence C, Klein RM (2001) Visual prior entry. Psychol Sci 12:205–212

    Article  CAS  PubMed  Google Scholar 

  • Shore DI, Spry E, Spence C (2002) Confusing the mind by crossing the hands. Cogn Brain Res 14:153–163

    Article  Google Scholar 

  • Slutsky DA, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. Neuroreport 12:7–10

    CAS  PubMed  Google Scholar 

  • Smith WF (1933) The relative quickness of visual and auditory perception. J Exp Psychol 16:239–257

    Google Scholar 

  • Spence C (2001) Crossmodal attentional capture: a controversy resolved? In: Folk C, Gibson B (eds) Attention, distraction and action: multiple perspectives on attentional capture. Elsevier Science, Amsterdam, pp 231–262

  • Spence C, Driver J (1999) A new approach to the design of multimodal warning signals. In: Harris D (ed) Job design, product design and human-computer interaction. (Engineering psychology and cognitive ergonomics, Vol 4) Ashgate: Hampshire, pp 455-461

  • Spence C, Squire SB (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13:R519–R521

  • Spence C, Nicholls MER, Driver J (2001a) The cost of expecting events in the wrong sensory modality. Percept Psychophys 63:330–336

    CAS  PubMed  Google Scholar 

  • Spence C, Shore DI, Klein RM (2001b) Multisensory prior entry. J Exp Psychol Gen 130:799–832

    CAS  PubMed  Google Scholar 

  • Spence C, Baddeley R, Zampini M, James R, Shore DI (2003) Crossmodal temporal order judgments: when two locations are better than one. Percept Psychophys

    Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA

  • Sternberg S, Knoll RL, Gates BA (1971) Prior entry reexamined: effect of attentional bias on order perception. Presented, meeting Psychon Soc, November, St. Louis, MO

    Google Scholar 

  • Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M, Port M, Porter NR (2001) When is now? Perception of simultaneity. Proc R Soc B 268:31–38

    Article  CAS  Google Scholar 

  • Stone SA (1926) Prior entry in the auditory-tactual complication. Am J Psychol 37:284–287

    Google Scholar 

  • Sugita Y, Suzuki Y (2003) Implicit estimation of sound-arrival time. Nature 421:911

    Article  CAS  Google Scholar 

  • Teatini G, Farnè M, Verzella F, Berruecos P Jr (1976) Perception of temporal order: visual and auditory stimuli. G Ital Psicol 3:157–164

    Google Scholar 

  • Ulrich R (1987) Threshold models of temporal-order judgments evaluated by a ternary response task. Percept Psychophys 42:224–239

    CAS  PubMed  Google Scholar 

  • Urbach D, Spitzer H (1995) Attentional effort modulated by task difficulty. Vision Res 35:2169–2177

    Article  CAS  PubMed  Google Scholar 

  • Van de Par S, Juola JF, Kohlrausch A (1999) Judged synchrony/asynchrony for light-tone pairs. Presented 40th Ann meeting Psychon Soc, Los Angeles, CA

  • Welch RB, DuttonHurt LD, Warren DH (1986) Contributions of audition and vision to temporal rate perception. Percept Psychophys 39:294–300

    Google Scholar 

  • Whipple GM, Sanford EC, Colgrove FW (1899) Minor studies from the psychological laboratory of Clark University: on nearly simultaneous clicks and flashes: the time required for recognition: notes on mental standards of length. Am J Psychol 10:280–295

    Google Scholar 

  • Zampini M, Shore DI, Spence C (2003) Multisensory temporal order judgments: the role of hemispheric redundancy. Int J Psychophysiol (in press)

Download references

Acknowledgements

Charles Spence and David I. Shore were funded by a Network Grant from the McDonnell-Pew Centre for Cognitive Neuroscience, University of Oxford. David I. Shore was also funded by an operating grant from the Natural Science and Engineering Research Council of Canada.

Author information

Authors and Affiliations

  1. Department of Experimental Psychology, University of Oxford, South Parks Road, Oxford, OX1 3UD, UK

    Massimiliano Zampini & Charles Spence

  2. University of Verona, Italy

    Massimiliano Zampini

  3. Department of Psychology, McMaster University, Hamilton, Canada

    David I. Shore

Authors
  1. Massimiliano Zampini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David I. Shore
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charles Spence
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Massimiliano Zampini.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zampini, M., Shore, D.I. & Spence, C. Audiovisual temporal order judgments. Exp Brain Res 152, 198–210 (2003). https://doi.org/10.1007/s00221-003-1536-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-003-1536-z

Keywords

Search

Navigation