Skip to main content
Log in

Drawing sounds: representing tones and chords spatially

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Research on the crossmodal correspondences has revealed that seemingly unrelated perceptual information can be matched across the senses in a manner that is consistent across individuals. An interesting extension of this line of research is to study how sensory information biases action. In the present study, we investigated whether different sounds (i.e. tones and piano chords) would bias participants’ hand movements in a free movement task. Right-handed participants were instructed to move a computer mouse in order to represent three tones and two chords. They also had to rate each sound in terms of three visual analogue scales (slow–fast, unpleasant–pleasant, and weak–strong). The results demonstrate that tones and chords influence hand movements, with higher-(lower-)pitched sounds giving rise to a significant bias towards upper (lower) locations in space. These results are discussed in terms of the literature on forward models, embodied cognition, crossmodal correspondences, and mental imagery. Potential applications sports and rehabilitation are discussed briefly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Note here that a chord is composed of three notes: a base or root note, a third, and a perfect fifth. What determines whether a chord is major or minor depends on whether the third is four (major chord) or three (minor chord) semitones higher than the root note (see Bakker and Martin 2014; Parker 2009).

  2. These sensations can be related to the mechanical response of tissues to vibration. The range of natural frequencies for different body parts (e.g., hands, eyes, abdominal mass, and chest) is from two to 200 Hz. Above 250 Hz, Pacini receptors are unresponsive and no longer represent sinusoidal frequencies as vibrations (Chaffin et al. 2006; Makous et al. 1995).

References

  • Aglioti SM, Pazzaglia M (2010) Representing actions through their sound. Exp Brain Res 206:141–151

    Article  PubMed  Google Scholar 

  • Arnott SR, Alain C (2011) The auditory dorsal pathway: orienting vision. Neurosci Biobehav Rev 35:2162–2173

    Article  PubMed  Google Scholar 

  • Bakker DR, Martin FH (2014) Musical chords and emotion: major and minor triads are processed for emotion. Cognit Affect Behav Neurosci 15:15–31

    Article  Google Scholar 

  • Barca L, Pezzulo G (2015) Tracking second thoughts: continuous and discrete revision processes during visual lexical decision. PLoS ONE 10:e0116193

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Bergeron V, Lopes D (2009) Hearing and seeing musical expression. Philos Phenomenol Res 78:1–16

    Article  Google Scholar 

  • Bértolo H (2014) Visual imagery without visual perception: lessons from blind subjects. In: Martins Costa MFPC, Nogueira RN (eds) Second international conference on applications of optics and photonics. International Society for Optics and Photonics, p 92862K

  • Bolognini N, Rasi F, Coccia M, Làdavas E (2005) Visual search improvement in hemianopic patients after audio–visual stimulation. Brain 128:2830–2842

    Article  PubMed  Google Scholar 

  • Bood RJ, Nijssen M, van der Kamp J, Roerdink M (2013) The power of auditory-motor synchronization in sports: enhancing running performance by coupling cadence with the right beats. PLoS ONE 8:e70758

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Buccino G, Binkofski F, Fink GR, Fadiga L, Fogassi L, Gallese V, Seitz RJ, Zilles K, Rizzolatti G, Freund HJ (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. Eur J Neurosci 13:400–404

    CAS  PubMed  Google Scholar 

  • Cattaneo Z, Vecchi T, Cornoldi C, Mammarella I, Bonino D, Ricciardi E, Pietrini P (2008) Imagery and spatial processes in blindness and visual impairment. Neurosci Biobehav Rev 32:1346–1360

    Article  PubMed  Google Scholar 

  • Chaffin DB, Andersson G, Martin BJ (2006) Occupational biomechanics. Wiley, Hoboken, NJ

    Google Scholar 

  • D’Ausilio A, Brunetti R, Delogu F, Santonico C, Belardinelli MO (2010) How and when auditory action effects impair motor performance. Exp Brain Res 201:323–330

    Article  PubMed  Google Scholar 

  • De Volder AG, Toyama H, Kimura Y, Kiyosawa M, Nakano H, Vanlierde A, Wanet-Defalque MC, Mishina M, Oda K, Ishiwata K, Senda M (2001) Auditory triggered mental imagery of shape involves visual association areas in early blind humans. Neuroimage 14:129–139

    Article  PubMed  Google Scholar 

  • DeChaine DR (2002) Affect and embodied understanding in musical experience. Text Perform Q 22:79–98

    Article  Google Scholar 

  • Deroy O, Fernandez-Prieto I, Navarra J, Spence C (in press) Unravelling the paradox of spatial pitch. In Hubbard TL (ed) Spatial biases in perception and cognition. Cambridge University Press, Cambridge (To appear)

  • Dieter KC, Hu B, Knill DC, Blake R, Tadin D (2013) Kinesthesis can make an invisible hand visible. Psychol Sci 25:66–75

    Article  PubMed  PubMed Central  Google Scholar 

  • Dimitriou M, Wolpert DM, Franklin DW (2013) The temporal evolution of feedback gains rapidly update to task demands. J Neurosci 33:10898–10909

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Eitan Z, Timmers R (2010) Beethoven’s last piano sonata and those who follow crocodiles: cross-domain mappings of auditory pitch in a musical context. Cognition 114:405–422

    Article  PubMed  Google Scholar 

  • Evans KK, Treisman A (2010) Natural cross-modal mappings between visual and auditory features. J Vis 10:1–12

    Article  PubMed  Google Scholar 

  • Farah MJ, Hammond KM, Levine DN, Calvanio R (1988) Visual and spatial mental imagery: dissociable systems of representation. Cogn Psychol 20:439–462

    Article  CAS  PubMed  Google Scholar 

  • Ferri F, Tajadura-Jiménez A, Väljamäe A, Vastano R, Costantini M (2015) Emotion-inducing approaching sounds shape the boundaries of multisensory peripersonal space. Neuropsychologia 70:468–475

    Article  PubMed  Google Scholar 

  • Franklin DW, Wolpert DM (2011) Computational mechanisms of sensorimotor control. Neuron 72:425–442

    Article  CAS  PubMed  Google Scholar 

  • Frassinetti F, Pavani F, Làdavas E (2002) Acoustical vision of neglected stimuli: interaction among spatially converging audiovisual inputs in neglect patients. J Cogn Neurosci 14:62–69

    Article  PubMed  Google Scholar 

  • Freeman JB, Ambady N (2010) MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav Res Methods 42:226–241

    Article  PubMed  Google Scholar 

  • Freeman JB, Dale R, Farmer TA (2011) Hand in motion reveals mind in motion. Frontiers Psychol 2:59

    Article  Google Scholar 

  • Gazzola V, Keysers C (2009) The observation and execution of actions share motor and somatosensory voxels in all tested subjects: single-subject analyses of unsmoothed fMRI data. Cereb Cortex 19:1239–1255

    Article  PubMed  Google Scholar 

  • Harfield P, Halkon B, Mitchell S, Phillips I, May A (2014) A novel, real-time biomechanical feedback system for use in rowing. Proc Eng 72:126–131

    Article  Google Scholar 

  • Harris LR, Carnevale MJ (2016) Which direction is up for a high pitch? Multisens Res 29:113–132

    Article  PubMed  Google Scholar 

  • Heffner RS, Heffner HE (1992a) Evolution of sound localization in mammals. In: Webster DB, Fay RR, Popper AN (eds) The evolutionary biology of hearing. Springer, New York, NY, pp 691–715

    Chapter  Google Scholar 

  • Heffner RS, Heffner HE (1992b) Visual factors in sound localization in mammals. J Comp Neurol 317:219–232

    Article  CAS  PubMed  Google Scholar 

  • Henriques J (2003) Sonic dominance and the reggae sound system session. In: Bull M, Black L (eds) The auditory culture reader. Berg, Oxford, pp 451–480

    Google Scholar 

  • Holstege G, Huynh HK (2013) Emotions studied by imaging of the human brain: the somatic and emotional motor systems. In: Pfaff DW (ed) Neuroscience in the 21st century: from basic to clinical. Springer, New York, NY, pp 2045–2068

    Chapter  Google Scholar 

  • Karwoski TF, Odbert HS, Osgood CE (1942) Studies in synesthetic thinking: II. The role of form in visual responses to music. J Gen Psychol 26:199–222

    Article  Google Scholar 

  • Keysers C, Kohler E, Umilta MA, Nanetti L, Fogassi L, Gallese V (2003) Audiovisual mirror neurons and action recognition. Exp Brain Res 153:628–636

    Article  CAS  PubMed  Google Scholar 

  • King AJ, Schnupp JWH, Doubell TP (2001) The shape of ears to come: dynamic coding of auditory space. Trends Cogn Sci 5:261–270

    Article  PubMed  Google Scholar 

  • Kohler E, Keysers C, Umiltà MA, Fogassi L, Gallese V, Rizzolatti G (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297:846–848

    Article  CAS  PubMed  Google Scholar 

  • Lambert S, Sampaio E, Mauss Y, Scheiber C (2004) Blindness and brain plasticity: Contribution of mental imagery? An fMRI study. Cogn Brain Res 20:1–11

    Article  CAS  Google Scholar 

  • Maeda F, Kanai R, Shimojo S (2004) Changing pitch induced visual motion illusion. Curr Biol 14:R990–R991

    Article  CAS  PubMed  Google Scholar 

  • Maga JA (1974) Influence of color on taste thresholds. Chem Senses Flavor 1:115–119

    Article  Google Scholar 

  • Makous JC, Friedman RM, Vierck CJ (1995) A critical band filter in touch. J Neurosci 15:2808–2818

    CAS  PubMed  Google Scholar 

  • Marmolejo-Ramos F, Tian U (2010) A simple method for the detection of outliers. Electron J Appl Stat Anal 2:67–76

    Google Scholar 

  • Meier BP, Robinson MD (2004) Why the sunny side is up: associations between affect and vertical position. Psychol Sci 15:243–247

    Article  PubMed  Google Scholar 

  • Meier BP, Robinson MD, Caven AJ (2008) Why a Big Mac is good Mac: associations between affect and size. Basic Appl Soc Psychol 30:46–55

    Article  Google Scholar 

  • Melara RD, O’Brien TP (1987) Interaction between synesthetically corresponding dimensions. J Exp Psychol Gen 116:323–336

    Article  Google Scholar 

  • Miall RC, Wolpert DM (1996) Forward models for physiological motor control. Neural Netw 9:1265–1279

    Article  PubMed  Google Scholar 

  • Montoro P, Contreras MJ, Elosúa MJ, Marmolejo-Ramos F (2015) Cross-modal metaphorical mapping of spoken emotion words onto vertical space. Frontiers Psychol 6:1205

    Article  Google Scholar 

  • Mudd SA (1963) Spatial stereotypes of four dimensions of pure tone. J Exp Psychol 66:347–352

    Article  CAS  PubMed  Google Scholar 

  • Niedenthal PM, Augustinova M, Rychlowska M (2010) Body and mind: zajonc's (re)introduction of the motor system to emotion and cognition. Emot Rev 2:340–347

    Article  Google Scholar 

  • Palmer SE, Schloss KB, Xu Z, Prado-León LR (2013) Music-color associations are mediated by emotion. Proc Natl Acad Sci USA 110:8836–8841

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Parise CV (2016) Crossmodal correspondences: standing issues and experimental guidelines. Multisens Res 29:7–28

    Article  PubMed  Google Scholar 

  • Parise CV, Knorre K, Ernst MO (2014) Natural auditory scene statistics shapes human spatial hearing. Proc Natl Acad Sci USA 111:6104–6108

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Parise C, Spence C, Deroy O (2015) Understanding the correspondences: introduction to the special issue on crossmodal correspondences. Multisens Res 29:1–6

    Article  Google Scholar 

  • Parker B (2009) Good vibrations: the physics of music. John Hopkins University Press, Baltimore, MD

    Google Scholar 

  • Penfield W, Rasmussen T (1950) The cerebral cortex of man. MacMillan, New York, NY

    Google Scholar 

  • Phillips-Silver J, Trainor LJ (2005) Feeling the beat: movement influences infants’ rhythm perception. Science 308:1430

    Article  CAS  PubMed  Google Scholar 

  • Phillips-Silver J, Trainor LJ (2007) Hearing what the body feels: auditory encoding of rhythmic movement. Cognition 105:533–546

    Article  PubMed  Google Scholar 

  • Pratt CC (1930) The spatial character of high and low tones. J Exp Psychol 13:278–285

    Article  Google Scholar 

  • Press C, Gherri E, Heyes C, Eimer M (2009) Action preparation helps and hinders perception of action. J Cogn Neurosci 22:2198–2211

    Article  Google Scholar 

  • Ricciardi E, Bonino D, Sani L, Vecchi T, Guazzelli M, Haxby JV, Fadiga L, Pietrini P (2009) Do we really need vision? how blind people “see” the actions of others. J Neurosci 29:9719–9724

    Article  CAS  PubMed  Google Scholar 

  • Rigas D, Alty J (2005) The rising pitch metaphor: an empirical study. Int J Hum Comput Stud 62:1–20

    Article  Google Scholar 

  • Rusconi E, Kwan B, Giordano BL, Umiltà C, Butterworth B (2006) Spatial representation of pitch height: the SMARC effect. Cognition 99:113–129

    Article  PubMed  Google Scholar 

  • Salgado-Montejo A, Alvarado JA, Velasco C, Salgado CJ, Hasse K, Spence C (2015) The sweetest thing: the influence of angularity, symmetry, and the number of elements on shape-taste matches. Frontiers Psychol 6:1382

    Article  Google Scholar 

  • Sasaki K, Yamada Y, Miura K (2016) Emotion biases voluntary vertical action only with visible cues. Acta Psychol 163:97–106

    Article  Google Scholar 

  • Schaffert N, Mattes K (2015a) Effects of acoustic feedback training in elite-standard Para-Rowing. J Sports Sci 33:411–418

    Article  PubMed  Google Scholar 

  • Schaffert N, Mattes K (2015b) Interactive sonification in rowing: acoustic feedback for on-water training. IEEE Multimed 22:58–67

    Article  Google Scholar 

  • Schaffert N, Mattes K, Effenberg AO (2011) An investigation of online acoustic information for elite rowers in on-water training conditions. J Hum Sport Exerc 6:392–405

    Article  Google Scholar 

  • Scherer KR, Oshinsky JS (1977) Cue utilization in emotion attribution from auditory stimuli. Motiv Emot 1:331–346

    Article  Google Scholar 

  • Sherman A, Grabowecky M, Suzuki S (2013) Auditory rhythms are systemically associated with spatial-frequency and density information in visual scenes. Psychon Bull Rev 20:740–746

    Article  PubMed  PubMed Central  Google Scholar 

  • Sigrist R, Rauter G, Marchal-Crespo L, Riener R, Wolf P (2014) Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233:909–925

    Article  PubMed  Google Scholar 

  • Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995

    Article  PubMed  Google Scholar 

  • Spence C, Deroy O (2013a) Crossmodal mental imagery. In: Lacey S, Lawson R (eds) Multisens Imag. Springer, New York, NY, pp 157–183

    Chapter  Google Scholar 

  • Spence C, Deroy O (2013b) Taste and shapes: a review of four hypotheses. Histor Sci 25:42–49

    Google Scholar 

  • Stahl B, Thoshkahna B (2016) Design and evaluation of the effectiveness of a sonification technique for real time heart-rate data. J Multimodal User Interfaces 10:207–219

    Article  Google Scholar 

  • Stumpf K (1883) Tonpsychologie [Psychology of tones]. S. Hirzel, Leipzig

    Google Scholar 

  • Tajadura-Jiménez A, Larsson P, Väljamäe A, Västfjäll D, Kleiner M (2010) When room size matters: acoustic influences on emotional responses to sounds. Emotion 10:416–442

    Article  PubMed  Google Scholar 

  • Tajadura-Jiménez A, Pantelidou G, Rebacz P, Västfjäll D, Tsakiris M (2011) I-space: the effects of emotional valence and source of music on interpersonal distance. PLoS ONE 6:e26083

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Thinus-Blanc C, Gaunet F (1997) Representation of space in blind persons: Vision as a spatial sense? Psychol Bull 121:20–42

    Article  CAS  PubMed  Google Scholar 

  • Tinga AM, Visser-Meily JMA, van der Smagt MJ, van der Stigchel S, van Ee R, Nijboer TCW (2015) Multisensory stimulation to improve low- and higher-level sensory deficits after stroke: a systematic review. Neuropsychol Rev 26:73–91

    Article  PubMed  PubMed Central  Google Scholar 

  • Treitler L (1982) The early history of music writing in the West. J Am Musicol Soc 35:237–279

    Article  Google Scholar 

  • Velasco C, Woods AT, Deroy O, Spence C (2015) Hedonic mediation of the crossmodal correspondence between taste and shape. Food Qual Prefer 41:151–158

    Article  Google Scholar 

  • Vuokowski J, Spence C, Thompson M, Clarke EF (2016) Interaction of sight and sound in the perception and experience of musical performance. Music Psychol 33:457–471

    Google Scholar 

  • Vuoskoski JK, Thompson MR, Clarke EF, Spence C (2014) Crossmodal interactions in the perception of expressivity in musical performance. Atten Percept Psychophys 76:591–604

    Article  PubMed  Google Scholar 

  • Watson L (1971) The omnivorous ape. Coward, McCann, & Geohhegan, New York, NY

    Google Scholar 

  • Wilcox RR (2005) Introduction to robust estimation and hypothesis testing. Elsevier, San Diego, CA

    Google Scholar 

  • Wilke C, Synofzik M, Lindner A (2012) The valence of action outcomes modulates the perception of one’s actions. Conscious Cogn 21:18–29

    Article  PubMed  Google Scholar 

  • Wolpert DM, Flanagan JR (2001) Motor prediction. Curr Biol 11:R729–R732

    Article  CAS  PubMed  Google Scholar 

  • Yohai V (1987) High breakdown-point and high efficiency estimates for regression. Ann Stat 15:642–656

    Article  Google Scholar 

  • Zajonc RB, Markus H (1984) Affect and cognition: The hard interface. In: Izard C, Kagan J, Zajonc RB (eds) Emotion, cognition, and behavior. Cambridge University Press, Cambridge, pp 73–102

    Google Scholar 

  • Zatorre RJ, Chen JL, Penhune VB (2007) When the brain plays music: auditory-motor interactions in music perception and production. Nat Rev Neurosci 8:547–558

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

Alejandro Salgado-Montejo would like to thank COLCIENCIAS for funding his DPhil. Charles Spence would like to acknowledge the AHRC Rethinking the Senses grant (AH/L007053/1).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alejandro Salgado-Montejo.

Additional information

Alejandro Salgado-Montejo and Fernando Marmolejo-Ramos have contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Salgado-Montejo, A., Marmolejo-Ramos, F., Alvarado, J.A. et al. Drawing sounds: representing tones and chords spatially. Exp Brain Res 234, 3509–3522 (2016). https://doi.org/10.1007/s00221-016-4747-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-016-4747-9

Keywords

Navigation