Abstract
Research on the crossmodal correspondences has revealed that seemingly unrelated perceptual information can be matched across the senses in a manner that is consistent across individuals. An interesting extension of this line of research is to study how sensory information biases action. In the present study, we investigated whether different sounds (i.e. tones and piano chords) would bias participants’ hand movements in a free movement task. Right-handed participants were instructed to move a computer mouse in order to represent three tones and two chords. They also had to rate each sound in terms of three visual analogue scales (slow–fast, unpleasant–pleasant, and weak–strong). The results demonstrate that tones and chords influence hand movements, with higher-(lower-)pitched sounds giving rise to a significant bias towards upper (lower) locations in space. These results are discussed in terms of the literature on forward models, embodied cognition, crossmodal correspondences, and mental imagery. Potential applications sports and rehabilitation are discussed briefly.
Similar content being viewed by others
Notes
Note here that a chord is composed of three notes: a base or root note, a third, and a perfect fifth. What determines whether a chord is major or minor depends on whether the third is four (major chord) or three (minor chord) semitones higher than the root note (see Bakker and Martin 2014; Parker 2009).
These sensations can be related to the mechanical response of tissues to vibration. The range of natural frequencies for different body parts (e.g., hands, eyes, abdominal mass, and chest) is from two to 200 Hz. Above 250 Hz, Pacini receptors are unresponsive and no longer represent sinusoidal frequencies as vibrations (Chaffin et al. 2006; Makous et al. 1995).
References
Aglioti SM, Pazzaglia M (2010) Representing actions through their sound. Exp Brain Res 206:141–151
Arnott SR, Alain C (2011) The auditory dorsal pathway: orienting vision. Neurosci Biobehav Rev 35:2162–2173
Bakker DR, Martin FH (2014) Musical chords and emotion: major and minor triads are processed for emotion. Cognit Affect Behav Neurosci 15:15–31
Barca L, Pezzulo G (2015) Tracking second thoughts: continuous and discrete revision processes during visual lexical decision. PLoS ONE 10:e0116193
Bergeron V, Lopes D (2009) Hearing and seeing musical expression. Philos Phenomenol Res 78:1–16
Bértolo H (2014) Visual imagery without visual perception: lessons from blind subjects. In: Martins Costa MFPC, Nogueira RN (eds) Second international conference on applications of optics and photonics. International Society for Optics and Photonics, p 92862K
Bolognini N, Rasi F, Coccia M, Làdavas E (2005) Visual search improvement in hemianopic patients after audio–visual stimulation. Brain 128:2830–2842
Bood RJ, Nijssen M, van der Kamp J, Roerdink M (2013) The power of auditory-motor synchronization in sports: enhancing running performance by coupling cadence with the right beats. PLoS ONE 8:e70758
Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13:400–404
Cattaneo Z, Vecchi T, Cornoldi C, Mammarella I, Bonino D, Ricciardi E, Pietrini P (2008) Imagery and spatial processes in blindness and visual impairment. Neurosci Biobehav Rev 32:1346–1360
Chaffin DB, Andersson G, Martin BJ (2006) Occupational biomechanics. Wiley, Hoboken, NJ
D’Ausilio A, Brunetti R, Delogu F, Santonico C, Belardinelli MO (2010) How and when auditory action effects impair motor performance. Exp Brain Res 201:323–330
De Volder AG, Toyama H, Kimura Y, Kiyosawa M, Nakano H, Vanlierde A, Wanet-Defalque MC, Mishina M, Oda K, Ishiwata K, Senda M (2001) Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage 14:129–139
DeChaine DR (2002) Affect and embodied understanding in musical experience. Text Perform Q 22:79–98
Deroy O, Fernandez-Prieto I, Navarra J, Spence C (in press) Unravelling the paradox of spatial pitch. In Hubbard TL (ed) Spatial biases in perception and cognition. Cambridge University Press, Cambridge (To appear)
Dieter KC, Hu B, Knill DC, Blake R, Tadin D (2013) Kinesthesis can make an invisible hand visible. Psychol Sci 25:66–75
Dimitriou M, Wolpert DM, Franklin DW (2013) The temporal evolution of feedback gains rapidly update to task demands. J Neurosci 33:10898–10909
Eitan Z, Timmers R (2010) Beethoven’s last piano sonata and those who follow crocodiles: cross-domain mappings of auditory pitch in a musical context. Cognition 114:405–422
Evans KK, Treisman A (2010) Natural cross-modal mappings between visual and auditory features. J Vis 10:1–12
Farah MJ, Hammond KM, Levine DN, Calvanio R (1988) Visual and spatial mental imagery: dissociable systems of representation. Cogn Psychol 20:439–462
Ferri F, Tajadura-Jiménez A, Väljamäe A, Vastano R, Costantini M (2015) Emotion-inducing approaching sounds shape the boundaries of multisensory peripersonal space. Neuropsychologia 70:468–475
Franklin DW, Wolpert DM (2011) Computational mechanisms of sensorimotor control. Neuron 72:425–442
Frassinetti F, Pavani F, Làdavas E (2002) Acoustical vision of neglected stimuli: interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci 14:62–69
Freeman JB, Ambady N (2010) MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav Res Methods 42:226–241
Freeman JB, Dale R, Farmer TA (2011) Hand in motion reveals mind in motion. Frontiers Psychol 2:59
Gazzola V, Keysers C (2009) The observation and execution of actions share motor and somatosensory voxels in all tested subjects: single-subject analyses of unsmoothed fMRI data. Cereb Cortex 19:1239–1255
Harfield P, Halkon B, Mitchell S, Phillips I, May A (2014) A novel, real-time biomechanical feedback system for use in rowing. Proc Eng 72:126–131
Harris LR, Carnevale MJ (2016) Which direction is up for a high pitch? Multisens Res 29:113–132
Heffner RS, Heffner HE (1992a) Evolution of sound localization in mammals. In: Webster DB, Fay RR, Popper AN (eds) The evolutionary biology of hearing. Springer, New York, NY, pp 691–715
Heffner RS, Heffner HE (1992b) Visual factors in sound localization in mammals. J Comp Neurol 317:219–232
Henriques J (2003) Sonic dominance and the reggae sound system session. In: Bull M, Black L (eds) The auditory culture reader. Berg, Oxford, pp 451–480
Holstege G, Huynh HK (2013) Emotions studied by imaging of the human brain: the somatic and emotional motor systems. In: Pfaff DW (ed) Neuroscience in the 21st century: from basic to clinical. Springer, New York, NY, pp 2045–2068
Karwoski TF, Odbert HS, Osgood CE (1942) Studies in synesthetic thinking: II. The role of form in visual responses to music. J Gen Psychol 26:199–222
Keysers C, Kohler E, Umilta MA, Nanetti L, Fogassi L, Gallese V (2003) Audiovisual mirror neurons and action recognition. Exp Brain Res 153:628–636
King AJ, Schnupp JWH, Doubell TP (2001) The shape of ears to come: dynamic coding of auditory space. Trends Cogn Sci 5:261–270
Kohler E, Keysers C, Umiltà MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848
Lambert S, Sampaio E, Mauss Y, Scheiber C (2004) Blindness and brain plasticity: Contribution of mental imagery? An fMRI study. Cogn Brain Res 20:1–11
Maeda F, Kanai R, Shimojo S (2004) Changing pitch induced visual motion illusion. Curr Biol 14:R990–R991
Maga JA (1974) Influence of color on taste thresholds. Chem Senses Flavor 1:115–119
Makous JC, Friedman RM, Vierck CJ (1995) A critical band filter in touch. J Neurosci 15:2808–2818
Marmolejo-Ramos F, Tian U (2010) A simple method for the detection of outliers. Electron J Appl Stat Anal 2:67–76
Meier BP, Robinson MD (2004) Why the sunny side is up: associations between affect and vertical position. Psychol Sci 15:243–247
Meier BP, Robinson MD, Caven AJ (2008) Why a Big Mac is good Mac: associations between affect and size. Basic Appl Soc Psychol 30:46–55
Melara RD, O’Brien TP (1987) Interaction between synesthetically corresponding dimensions. J Exp Psychol Gen 116:323–336
Miall RC, Wolpert DM (1996) Forward models for physiological motor control. Neural Netw 9:1265–1279
Montoro P, Contreras MJ, Elosúa MJ, Marmolejo-Ramos F (2015) Cross-modal metaphorical mapping of spoken emotion words onto vertical space. Frontiers Psychol 6:1205
Mudd SA (1963) Spatial stereotypes of four dimensions of pure tone. J Exp Psychol 66:347–352
Niedenthal PM, Augustinova M, Rychlowska M (2010) Body and mind: zajonc's (re)introduction of the motor system to emotion and cognition. Emot Rev 2:340–347
Palmer SE, Schloss KB, Xu Z, Prado-León LR (2013) Music-color associations are mediated by emotion. Proc Natl Acad Sci USA 110:8836–8841
Parise CV (2016) Crossmodal correspondences: standing issues and experimental guidelines. Multisens Res 29:7–28
Parise CV, Knorre K, Ernst MO (2014) Natural auditory scene statistics shapes human spatial hearing. Proc Natl Acad Sci USA 111:6104–6108
Parise C, Spence C, Deroy O (2015) Understanding the correspondences: introduction to the special issue on crossmodal correspondences. Multisens Res 29:1–6
Parker B (2009) Good vibrations: the physics of music. John Hopkins University Press, Baltimore, MD
Penfield W, Rasmussen T (1950) The cerebral cortex of man. MacMillan, New York, NY
Phillips-Silver J, Trainor LJ (2005) Feeling the beat: movement influences infants’ rhythm perception. Science 308:1430
Phillips-Silver J, Trainor LJ (2007) Hearing what the body feels: auditory encoding of rhythmic movement. Cognition 105:533–546
Pratt CC (1930) The spatial character of high and low tones. J Exp Psychol 13:278–285
Press C, Gherri E, Heyes C, Eimer M (2009) Action preparation helps and hinders perception of action. J Cogn Neurosci 22:2198–2211
Ricciardi E, Bonino D, Sani L, Vecchi T, Guazzelli M, Haxby JV, Fadiga L, Pietrini P (2009) Do we really need vision? how blind people “see” the actions of others. J Neurosci 29:9719–9724
Rigas D, Alty J (2005) The rising pitch metaphor: an empirical study. Int J Hum Comput Stud 62:1–20
Rusconi E, Kwan B, Giordano BL, Umiltà C, Butterworth B (2006) Spatial representation of pitch height: the SMARC effect. Cognition 99:113–129
Salgado-Montejo A, Alvarado JA, Velasco C, Salgado CJ, Hasse K, Spence C (2015) The sweetest thing: the influence of angularity, symmetry, and the number of elements on shape-taste matches. Frontiers Psychol 6:1382
Sasaki K, Yamada Y, Miura K (2016) Emotion biases voluntary vertical action only with visible cues. Acta Psychol 163:97–106
Schaffert N, Mattes K (2015a) Effects of acoustic feedback training in elite-standard Para-Rowing. J Sports Sci 33:411–418
Schaffert N, Mattes K (2015b) Interactive sonification in rowing: acoustic feedback for on-water training. IEEE Multimed 22:58–67
Schaffert N, Mattes K, Effenberg AO (2011) An investigation of online acoustic information for elite rowers in on-water training conditions. J Hum Sport Exerc 6:392–405
Scherer KR, Oshinsky JS (1977) Cue utilization in emotion attribution from auditory stimuli. Motiv Emot 1:331–346
Sherman A, Grabowecky M, Suzuki S (2013) Auditory rhythms are systemically associated with spatial-frequency and density information in visual scenes. Psychon Bull Rev 20:740–746
Sigrist R, Rauter G, Marchal-Crespo L, Riener R, Wolf P (2014) Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233:909–925
Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995
Spence C, Deroy O (2013a) Crossmodal mental imagery. In: Lacey S, Lawson R (eds) Multisens Imag. Springer, New York, NY, pp 157–183
Spence C, Deroy O (2013b) Taste and shapes: a review of four hypotheses. Histor Sci 25:42–49
Stahl B, Thoshkahna B (2016) Design and evaluation of the effectiveness of a sonification technique for real time heart-rate data. J Multimodal User Interfaces 10:207–219
Stumpf K (1883) Tonpsychologie [Psychology of tones]. S. Hirzel, Leipzig
Tajadura-Jiménez A, Larsson P, Väljamäe A, Västfjäll D, Kleiner M (2010) When room size matters: acoustic influences on emotional responses to sounds. Emotion 10:416–442
Tajadura-Jiménez A, Pantelidou G, Rebacz P, Västfjäll D, Tsakiris M (2011) I-space: the effects of emotional valence and source of music on interpersonal distance. PLoS ONE 6:e26083
Thinus-Blanc C, Gaunet F (1997) Representation of space in blind persons: Vision as a spatial sense? Psychol Bull 121:20–42
Tinga AM, Visser-Meily JMA, van der Smagt MJ, van der Stigchel S, van Ee R, Nijboer TCW (2015) Multisensory stimulation to improve low- and higher-level sensory deficits after stroke: a systematic review. Neuropsychol Rev 26:73–91
Treitler L (1982) The early history of music writing in the West. J Am Musicol Soc 35:237–279
Velasco C, Woods AT, Deroy O, Spence C (2015) Hedonic mediation of the crossmodal correspondence between taste and shape. Food Qual Prefer 41:151–158
Vuokowski J, Spence C, Thompson M, Clarke EF (2016) Interaction of sight and sound in the perception and experience of musical performance. Music Psychol 33:457–471
Vuoskoski JK, Thompson MR, Clarke EF, Spence C (2014) Crossmodal interactions in the perception of expressivity in musical performance. Atten Percept Psychophys 76:591–604
Watson L (1971) The omnivorous ape. Coward, McCann, & Geohhegan, New York, NY
Wilcox RR (2005) Introduction to robust estimation and hypothesis testing. Elsevier, San Diego, CA
Wilke C, Synofzik M, Lindner A (2012) The valence of action outcomes modulates the perception of one’s actions. Conscious Cogn 21:18–29
Wolpert DM, Flanagan JR (2001) Motor prediction. Curr Biol 11:R729–R732
Yohai V (1987) High breakdown-point and high efficiency estimates for regression. Ann Stat 15:642–656
Zajonc RB, Markus H (1984) Affect and cognition: The hard interface. In: Izard C, Kagan J, Zajonc RB (eds) Emotion, cognition, and behavior. Cambridge University Press, Cambridge, pp 73–102
Zatorre RJ, Chen JL, Penhune VB (2007) When the brain plays music: auditory-motor interactions in music perception and production. Nat Rev Neurosci 8:547–558
Acknowledgments
Alejandro Salgado-Montejo would like to thank COLCIENCIAS for funding his DPhil. Charles Spence would like to acknowledge the AHRC Rethinking the Senses grant (AH/L007053/1).
Author information
Authors and Affiliations
Corresponding author
Additional information
Alejandro Salgado-Montejo and Fernando Marmolejo-Ramos have contributed equally to this work.
Rights and permissions
About this article
Cite this article
Salgado-Montejo, A., Marmolejo-Ramos, F., Alvarado, J.A. et al. Drawing sounds: representing tones and chords spatially. Exp Brain Res 234, 3509–3522 (2016). https://doi.org/10.1007/s00221-016-4747-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-016-4747-9