Elsevier

Cortex

Volume 92, July 2017, Pages 304-319
Cortex

Research report
Sensorimotor experience and verb-category mapping in human sensory, motor and parietal neurons

https://doi.org/10.1016/j.cortex.2017.04.021Get rights and content

Abstract

Semantic grounding is the process of relating meaning to symbols (e.g., words). It is the foundation for creating a representational symbolic system such as language. Semantic grounding for verb meaning is hypothesized to be achieved through two mechanisms: sensorimotor mapping, i.e., directly encoding the sensorimotor experiences the verb describes, and verb-category mapping, i.e., encoding the abstract category a verb belongs to. These two mechanisms were investigated by examining neuronal-level spike (i.e. neuronal action potential) activities from the motor, somatosensory and parietal areas in two human participants. Motor and a portion of somatosensory neurons were found to be involved in primarily sensorimotor mapping, while parietal and some somatosensory neurons were found to be involved in both sensorimotor and verb-category mapping. The time course of the spike activities and the selective tuning pattern of these neurons indicate that they belong to a large neural network used for semantic processing. This study is the first step towards understanding how words are processed by neurons.

Introduction

Upon hearing the word grasp, we quickly understand the action described. In everyday language, words are commonly used to point to real-world experiences (Meteyard, Cuadrado, Bahrami, & Vigliocco, 2012). The process of establishing “meaning-to-symbol” pointers in the brain is called semantic grounding. The semantic grounding mechanisms for concrete action verbs were investigated in this study at the neuronal level. Specifically, we asked how human neurons map action concepts experienced in the real world to language, and vice versa.

We propose that two complementary grounding mechanisms are involved in the mapping between verbs and their meanings. The first of these is Sensorimotor Mapping: under this mechanism, verb concepts are grounded through direct reference to concrete sensorimotor experience (Hauk, Johnsrude, & Pulvermüller, 2004). This mechanism is what is behind embodied approaches to verb meaning. The second mechanism is Verb-Category Mapping: in this mechanism, verb meanings are grounded in their relationship to other verb meanings, based on the overlapping set of sentence contexts that related verbs may occur in (Mahon and Caramazza, 2008, Pulvermüller, 2013a). This mechanism is connected to the taxonomy of verb classes that has been the basis of much linguistic and psycholinguistic work on verb meaning (e.g., Levin, 1993, Pinker, 1994).

The distinction between these two mechanisms can be illustrated by considering their relationship to the meaning of a single verb, kick. In relation to sensorimotor mapping, the concept for the verb kick is derived (in part) from the Body Part or Force associated with performing the action of kicking. Body Part or Force, therefore, are defined as Features of sensorimotor mapping, as each of these features describes one important sensorimotor aspect of the verb's meaning. These features relevant to sensorimotor mappings are referred as sensorimotor features for the remainder of this paper. Verb-category mapping, on the other hand, requires a taxonomy of verbal categories, based on the sentence contexts in which a set of verbs may appear. These verb categories may be used to facilitate learning of new verbs and concepts (Kemmerer, 2006, Mahon and Caramazza, 2008, Meteyard et al., 2012, Pulvermüller, 2013a). Take the same verb kick. It belongs to the same category as slap and knock in a verb taxonomy, because it occurs in similar sentence contexts as kick (e.g., Levin, 1993). This grouping of similar verbs (kick with slap and knock) defines a category, one which is different from verb groupings based on sensorimotor features: kick, slap, and knock have different Body Part sensorimotor features but belong to the same abstract taxonomic verb category. We will refer to the taxonomic verb categories that verbs belong to as verb-category features for the remainder of this paper. They form an abstract, grammatically-active semantic representation that is no longer strictly tied to the specific sensorimotor experiences associated with verbs.

These two kinds of semantic mappings have been argued to tap into two different levels of verb meanings (Kemmerer & Gonzalez-Castillo, 2010), and to recruit different neural networks. The sensorimotor mapping is proposed to involve motor and somatosensory areas. For example, previous functional Magnetic Resonance Imaging (fMRI) studies found that a sensorimotor feature of verbs, Body Part, modulated the activations in the motor cortex by somatotopic organizations (Buccino et al., 2001, Hauk et al., 2004, Kemmerer et al., 2008, van Ackeren et al., 2014). For example, the hand area in the motor cortex activated selectively in response to hand verbs (e.g. pick, grasp) and the mouth area activated selectively in response to mouth verbs (Hauk et al., 2004). Magnetoencephalography (MEG) studies further show that these responses occurred earlier than the typical semantic processing window for words (Mollo, Pulvermüller, & Hauk, 2016). This supports the hypothesis that sensorimotor mapping is not merely from post-comprehension processes (see also Vukovic, Feurra, Shpektor, Myachykov, & Shtyrov, 2017, for evidence that repetitive Transcranial Magnetic Stimulation (rTMS) to motor cortex impairs retrieval of actions verbs). Rather, these sensorimotor mappings may in fact be a fundamental neural mechanism of verb-semantic processing. Some aspects of sensorimotor mapping have also been found to be connected to associative cortices such as the inferior frontal (Barrós-Loscertales et al., 2012, Kemmerer and Gonzalez-Castillo, 2010, Moody and Gennari, 2010), temporal (Papeo et al., 2014, Romagno et al., 2012) and parietal areas (Buccino et al., 2001, Cattaneo et al., 2015, Fogassi et al., 2005).

Verb-category mapping, on the other hand, is hypothesized to primarily recruit associative cortices such as inferior frontal cortex (Bak & Chandran, 2012), temporal lobe (Damasio et al., 1996, Kiefer and Pulvermüller, 2012), and inferior parietal cortex (Kemmerer, 2006, Noordzij et al., 2008). Damage to these regions in post-stroke aphasia is commonly associated with deficits in access to these abstract verb-category features (e.g., Kemmerer, 2006, Kim and Thompson, 2000, Kim and Thompson, 2004).

These previous studies provide evidence of the existence of the two kinds of mapping at macroscopic level, in terms of broad cortical regions. However, due in part to the difficulty in examining activities of individual neurons in human participants, there is still debate regarding whether these two grounding mechanisms are distinct, and whether they are jointly active during online language processing (Arévalo, Baldo, & Dronkers, 2012). This difficulty is magnified by the fact that much data to date regarding sensorimotor and verb-category mapping comes from either fMRI evidence or lesion-deficit studies, neither of which directly addresses how these mechanisms affect real-time processing or are connected to neuronal-level activity. Similarly, it remains unclear whether the two grounding mechanisms overlap in their cortical distribution and temporal properties (Mahon and Caramazza, 2008, Meteyard et al., 2012). Some previous findings have suggested that they may overlap cortically, for example in temporal or parietal associative cortex. Understanding how these hypothesized grounding mechanisms are represented at the neuronal level, and whether they overlap in their temporal or cortical distributions, is a critical next step in understanding the neural bases of verb-semantic representation.

To elucidate these issues, four questions need to be answered. First, do the sensorimotor experiences described in verbs directly modulate spike activities (i.e. neuronal action potentials) of individual neurons in the primary motor and somatosensory areas, as actual body movement or sensation does (Georgopoulos et al., 1986, Wang et al., 2010)? Second, does verb-category information of the verbs modulate spike activities of individual neurons, specifically in associative areas like parietal cortex (Kemmerer, 2006, Noordzij et al., 2008)? Third, what is the time window of these neuronal responses? Does this time window precede or overlap with semantic processing time windows (Kutas & Hillyard, 1980b), and is it similar for the two different grounding mechanisms (sensorimotor and verb-category)? Fourth, are those neurons involved in semantic grounding (either sensorimotor or verb-category) specialists or generalists? That is, does a neuron encode a variety of semantic features, or specialize in just one or two? This question is related to a long-standing issue in neuroscience: are individual neurons broadly or narrowly tuned (Anderson et al., 2015, Quiroga et al., 2013).

To answer these questions, we recorded spike activities from implanted microelectrode arrays in two human participants, in response to visually-presented verbs. Spike activities were recorded from neurons in the primary motor cortex, the primary somatosensory cortex, and the superior and inferior parietal lobules near the intraparietal sulcus. The recorded spike activities from each channel of the implanted arrays were sorted into units based on spike waveform morphology similarity. A unit is presumed to be a collection of spikes from one or a small number of neighbor neurons. Four specific hypotheses were tested. Hypothesis I: the spike activities of somatosensory and motor units will be modulated by sensorimotor features (e.g. Body Part, Duration) of presented verbs (Kemmerer & Gonzalez-Castillo, 2010). This would be evidence of embodied verb meaning at the neuronal level. We anticipate that some of these modulated units will also be modulated by attempted motor movements, exhibiting “bimodal” properties. Hypothesis II: the spike activities of parietal units will be modulated by both sensorimotor features and verb-category features of presented verbs (Aziz-Zadeh and Damasio, 2008, Cattaneo et al., 2015, Jirak et al., 2010). This would be evidence that two mapping mechanisms (sensorimotor and verb-category) overlap in their cortical distribution, recruiting different neuronal populations within the same macroscopic region. Hypothesis III: these spike modulations will occur prior to the semantic processing window, consistent with previous MEG findings (Mollo et al., 2016). This would be evidence that sensorimotor activations are not simply epiphenomenal but are critical to verb-semantic processing (see also Vukovic et al., 2017). We do not have strong hypotheses regarding whether spike modulations in response to verb-category features will also occur in early time windows, or will overlap in time with sensorimotor spike modulations; these questions have not been examined to date. Hypothesis IV: units are more likely to be Specialists, modulated only by a very selective set of sensorimotor features or verb-category features, similar to the neuronal tuning properties to other kinds of complex information, such as persons or objects (Anderson et al., 2015, Quiroga et al., 2013) by highly specialized neurons (Quiroga et al., 2013).

Section snippets

Human participants and intracortical microelectrode array placements

The implanted device used in this study was under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration and registered on clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/NCT01364480 and http://clinicaltrials.gov/ct2/show/NCT01894802). The study was approved by the Institutional Review Board of the University of Pittsburgh (Pittsburgh, PA, USA), the Space and Naval Warfare Systems Center Pacific (San Diego, CA, USA) and the Department of Defense.

Results

We present three sets of results: 1) neuronal activity changes (i.e. firing rate modulation) with respect to both different semantic features (sensorimotor and verb-category) and attempted movements in the verb-reading and video following tasks; 2) the spatiotemporal pattern of the neuronal tuning: when and where the firing rate variances were explained significantly by the coded sensorimotor features or verb-category features; and 3) tuning pattern analyses: which and how many semantic

Spatial distribution of the two grounding mechanisms

We found that neurons in the motor, somatosensory, superior and inferior parietal lobules selectively responded to sensorimotor features or verb-category features when human participants silently read action verbs. These responses are the neuronal reflex of verb-related embodied cognition (Hauk, et al., 2004), and of higher-order verb taxonomies (Levin, 1993). Furthermore, these units' tuning properties followed a trend: gradually decreasing representation of sensorimotor mapping and increasing

Conclusion

Neuronal spike activities in the motor, somatosensory, superior and inferior parietal lobules respond to either sensorimotor or verb-category features of concrete action verbs. The pattern of spike-rate modulation revealed gradually decreasing sensitivity to sensorimotor features and increased tuning to verb-category features moving from motor to somatosensory to superior/inferior parietal lobules. Furthermore, the majority of neurons were found to be specialists: they were tuned to only one or

Acknowledgments

This work was supported by the University of Pittsburgh Medical Center (UPMC), UPMC Rehabilitation Institute. This work was developed with the funding from the National Institutes of Health (NIH) (Grants 3R01NS050256-05S1 and 8KL2TR000146), and the Defense Advanced Research Projects Agency (DARPA) Revolutionizing Prosthetics Program (SPAWAR Contract N66001-10-C-4056). This material is also supported in part by the Office of Research and Development, Rehabilitation Research & Development Service

References (61)

  • D. Kemmerer et al.

    The Two-Level Theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system

    Brain and Language

    (2010)
  • M. Kiefer et al.

    Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions

    Cortex

    (2012)
  • M. Kim et al.

    Patterns of comprehension and production of nouns and verbs in agrammatism: Implications for lexical organization

    Brain and Language

    (2000)
  • M. Kim et al.

    Verb deficits in Alzheimer's disease and agrammatism: Implications for lexical organization

    Brain and Language

    (2004)
  • M. Kutas et al.

    Event-related brain potentials to semantically inappropriate and surprisingly large words

    Biological Psychology

    (1980)
  • B.Z. Mahon et al.

    A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content

    Journal of Physiology Paris

    (2008)
  • L. Meteyard et al.

    Coming of age: A review of embodiment and the neuroscience of semantics

    Cortex

    (2012)
  • G. Mollo et al.

    Movement priming of EEG/MEG brain responses for action-words characterizes the link between language and action

    Cortex

    (2016)
  • C.L. Moody et al.

    Effects of implied physical effort in sensory-motor and pre-frontal cortex during language comprehension

    NeuroImage

    (2010)
  • M.L. Noordzij et al.

    Neural correlates of locative prepositions

    Neuropsychologia

    (2008)
  • S. Pinker

    How could a child use verb syntax to learn verb semantics?

    Lingua

    (1994)
  • N. Postle et al.

    Action word meaning representations in cytoarchitectonically defined primary and premotor cortices

    NeuroImage

    (2008)
  • F. Pulvermüller

    How neurons make meaning: Brain mechanisms for embodied and abstract-symbolic semantics

    Trends in Cognitive Sciences

    (2013)
  • F. Pulvermüller

    Semantic embodiment, disembodiment or misembodiment? In search of meaning in modules and neuron circuits

    Brain and Language

    (2013)
  • D. Romagno et al.

    Where the brain appreciates the final state of an event: The neural correlates of telicity

    Brain and Language

    (2012)
  • N. Vukovic et al.

    Primary motor cortex functionally contributes to language comprehension: An online rTMS study

    Neuropsychologia

    (2017)
  • M.J. van Ackeren et al.

    Oscillatory neuronal activity reflects lexical-semantic feature integration within and across sensory modalities in distributed cortical networks

    Journal of Neuroscience

    (2014)
  • R.A. Andersen

    Multimodal integration for the representation of space in the posterior parietal cortex

    Philosophical Transactions of the Royal Society of London B: Biological Sciences

    (1997)
  • A.J. Anderson et al.

    Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text

    NeuroImage

    (2015)
  • R.H. Baayen et al.

    The CELEX lexical database (release 1)

    (1993)
  • Cited by (0)

    1

    The two corresponding authors contributed equally to this work.

    View full text