Research reportSensorimotor experience and verb-category mapping in human sensory, motor and parietal neurons
Introduction
Upon hearing the word grasp, we quickly understand the action described. In everyday language, words are commonly used to point to real-world experiences (Meteyard, Cuadrado, Bahrami, & Vigliocco, 2012). The process of establishing “meaning-to-symbol” pointers in the brain is called semantic grounding. The semantic grounding mechanisms for concrete action verbs were investigated in this study at the neuronal level. Specifically, we asked how human neurons map action concepts experienced in the real world to language, and vice versa.
We propose that two complementary grounding mechanisms are involved in the mapping between verbs and their meanings. The first of these is Sensorimotor Mapping: under this mechanism, verb concepts are grounded through direct reference to concrete sensorimotor experience (Hauk, Johnsrude, & Pulvermüller, 2004). This mechanism is what is behind embodied approaches to verb meaning. The second mechanism is Verb-Category Mapping: in this mechanism, verb meanings are grounded in their relationship to other verb meanings, based on the overlapping set of sentence contexts that related verbs may occur in (Mahon and Caramazza, 2008, Pulvermüller, 2013a). This mechanism is connected to the taxonomy of verb classes that has been the basis of much linguistic and psycholinguistic work on verb meaning (e.g., Levin, 1993, Pinker, 1994).
The distinction between these two mechanisms can be illustrated by considering their relationship to the meaning of a single verb, kick. In relation to sensorimotor mapping, the concept for the verb kick is derived (in part) from the Body Part or Force associated with performing the action of kicking. Body Part or Force, therefore, are defined as Features of sensorimotor mapping, as each of these features describes one important sensorimotor aspect of the verb's meaning. These features relevant to sensorimotor mappings are referred as sensorimotor features for the remainder of this paper. Verb-category mapping, on the other hand, requires a taxonomy of verbal categories, based on the sentence contexts in which a set of verbs may appear. These verb categories may be used to facilitate learning of new verbs and concepts (Kemmerer, 2006, Mahon and Caramazza, 2008, Meteyard et al., 2012, Pulvermüller, 2013a). Take the same verb kick. It belongs to the same category as slap and knock in a verb taxonomy, because it occurs in similar sentence contexts as kick (e.g., Levin, 1993). This grouping of similar verbs (kick with slap and knock) defines a category, one which is different from verb groupings based on sensorimotor features: kick, slap, and knock have different Body Part sensorimotor features but belong to the same abstract taxonomic verb category. We will refer to the taxonomic verb categories that verbs belong to as verb-category features for the remainder of this paper. They form an abstract, grammatically-active semantic representation that is no longer strictly tied to the specific sensorimotor experiences associated with verbs.
These two kinds of semantic mappings have been argued to tap into two different levels of verb meanings (Kemmerer & Gonzalez-Castillo, 2010), and to recruit different neural networks. The sensorimotor mapping is proposed to involve motor and somatosensory areas. For example, previous functional Magnetic Resonance Imaging (fMRI) studies found that a sensorimotor feature of verbs, Body Part, modulated the activations in the motor cortex by somatotopic organizations (Buccino et al., 2001, Hauk et al., 2004, Kemmerer et al., 2008, van Ackeren et al., 2014). For example, the hand area in the motor cortex activated selectively in response to hand verbs (e.g. pick, grasp) and the mouth area activated selectively in response to mouth verbs (Hauk et al., 2004). Magnetoencephalography (MEG) studies further show that these responses occurred earlier than the typical semantic processing window for words (Mollo, Pulvermüller, & Hauk, 2016). This supports the hypothesis that sensorimotor mapping is not merely from post-comprehension processes (see also Vukovic, Feurra, Shpektor, Myachykov, & Shtyrov, 2017, for evidence that repetitive Transcranial Magnetic Stimulation (rTMS) to motor cortex impairs retrieval of actions verbs). Rather, these sensorimotor mappings may in fact be a fundamental neural mechanism of verb-semantic processing. Some aspects of sensorimotor mapping have also been found to be connected to associative cortices such as the inferior frontal (Barrós-Loscertales et al., 2012, Kemmerer and Gonzalez-Castillo, 2010, Moody and Gennari, 2010), temporal (Papeo et al., 2014, Romagno et al., 2012) and parietal areas (Buccino et al., 2001, Cattaneo et al., 2015, Fogassi et al., 2005).
Verb-category mapping, on the other hand, is hypothesized to primarily recruit associative cortices such as inferior frontal cortex (Bak & Chandran, 2012), temporal lobe (Damasio et al., 1996, Kiefer and Pulvermüller, 2012), and inferior parietal cortex (Kemmerer, 2006, Noordzij et al., 2008). Damage to these regions in post-stroke aphasia is commonly associated with deficits in access to these abstract verb-category features (e.g., Kemmerer, 2006, Kim and Thompson, 2000, Kim and Thompson, 2004).
These previous studies provide evidence of the existence of the two kinds of mapping at macroscopic level, in terms of broad cortical regions. However, due in part to the difficulty in examining activities of individual neurons in human participants, there is still debate regarding whether these two grounding mechanisms are distinct, and whether they are jointly active during online language processing (Arévalo, Baldo, & Dronkers, 2012). This difficulty is magnified by the fact that much data to date regarding sensorimotor and verb-category mapping comes from either fMRI evidence or lesion-deficit studies, neither of which directly addresses how these mechanisms affect real-time processing or are connected to neuronal-level activity. Similarly, it remains unclear whether the two grounding mechanisms overlap in their cortical distribution and temporal properties (Mahon and Caramazza, 2008, Meteyard et al., 2012). Some previous findings have suggested that they may overlap cortically, for example in temporal or parietal associative cortex. Understanding how these hypothesized grounding mechanisms are represented at the neuronal level, and whether they overlap in their temporal or cortical distributions, is a critical next step in understanding the neural bases of verb-semantic representation.
To elucidate these issues, four questions need to be answered. First, do the sensorimotor experiences described in verbs directly modulate spike activities (i.e. neuronal action potentials) of individual neurons in the primary motor and somatosensory areas, as actual body movement or sensation does (Georgopoulos et al., 1986, Wang et al., 2010)? Second, does verb-category information of the verbs modulate spike activities of individual neurons, specifically in associative areas like parietal cortex (Kemmerer, 2006, Noordzij et al., 2008)? Third, what is the time window of these neuronal responses? Does this time window precede or overlap with semantic processing time windows (Kutas & Hillyard, 1980b), and is it similar for the two different grounding mechanisms (sensorimotor and verb-category)? Fourth, are those neurons involved in semantic grounding (either sensorimotor or verb-category) specialists or generalists? That is, does a neuron encode a variety of semantic features, or specialize in just one or two? This question is related to a long-standing issue in neuroscience: are individual neurons broadly or narrowly tuned (Anderson et al., 2015, Quiroga et al., 2013).
To answer these questions, we recorded spike activities from implanted microelectrode arrays in two human participants, in response to visually-presented verbs. Spike activities were recorded from neurons in the primary motor cortex, the primary somatosensory cortex, and the superior and inferior parietal lobules near the intraparietal sulcus. The recorded spike activities from each channel of the implanted arrays were sorted into units based on spike waveform morphology similarity. A unit is presumed to be a collection of spikes from one or a small number of neighbor neurons. Four specific hypotheses were tested. Hypothesis I: the spike activities of somatosensory and motor units will be modulated by sensorimotor features (e.g. Body Part, Duration) of presented verbs (Kemmerer & Gonzalez-Castillo, 2010). This would be evidence of embodied verb meaning at the neuronal level. We anticipate that some of these modulated units will also be modulated by attempted motor movements, exhibiting “bimodal” properties. Hypothesis II: the spike activities of parietal units will be modulated by both sensorimotor features and verb-category features of presented verbs (Aziz-Zadeh and Damasio, 2008, Cattaneo et al., 2015, Jirak et al., 2010). This would be evidence that two mapping mechanisms (sensorimotor and verb-category) overlap in their cortical distribution, recruiting different neuronal populations within the same macroscopic region. Hypothesis III: these spike modulations will occur prior to the semantic processing window, consistent with previous MEG findings (Mollo et al., 2016). This would be evidence that sensorimotor activations are not simply epiphenomenal but are critical to verb-semantic processing (see also Vukovic et al., 2017). We do not have strong hypotheses regarding whether spike modulations in response to verb-category features will also occur in early time windows, or will overlap in time with sensorimotor spike modulations; these questions have not been examined to date. Hypothesis IV: units are more likely to be Specialists, modulated only by a very selective set of sensorimotor features or verb-category features, similar to the neuronal tuning properties to other kinds of complex information, such as persons or objects (Anderson et al., 2015, Quiroga et al., 2013) by highly specialized neurons (Quiroga et al., 2013).
Section snippets
Human participants and intracortical microelectrode array placements
The implanted device used in this study was under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration and registered on clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/NCT01364480 and http://clinicaltrials.gov/ct2/show/NCT01894802). The study was approved by the Institutional Review Board of the University of Pittsburgh (Pittsburgh, PA, USA), the Space and Naval Warfare Systems Center Pacific (San Diego, CA, USA) and the Department of Defense.
Results
We present three sets of results: 1) neuronal activity changes (i.e. firing rate modulation) with respect to both different semantic features (sensorimotor and verb-category) and attempted movements in the verb-reading and video following tasks; 2) the spatiotemporal pattern of the neuronal tuning: when and where the firing rate variances were explained significantly by the coded sensorimotor features or verb-category features; and 3) tuning pattern analyses: which and how many semantic
Spatial distribution of the two grounding mechanisms
We found that neurons in the motor, somatosensory, superior and inferior parietal lobules selectively responded to sensorimotor features or verb-category features when human participants silently read action verbs. These responses are the neuronal reflex of verb-related embodied cognition (Hauk, et al., 2004), and of higher-order verb taxonomies (Levin, 1993). Furthermore, these units' tuning properties followed a trend: gradually decreasing representation of sensorimotor mapping and increasing
Conclusion
Neuronal spike activities in the motor, somatosensory, superior and inferior parietal lobules respond to either sensorimotor or verb-category features of concrete action verbs. The pattern of spike-rate modulation revealed gradually decreasing sensitivity to sensorimotor features and increased tuning to verb-category features moving from motor to somatosensory to superior/inferior parietal lobules. Furthermore, the majority of neurons were found to be specialists: they were tuned to only one or
Acknowledgments
This work was supported by the University of Pittsburgh Medical Center (UPMC), UPMC Rehabilitation Institute. This work was developed with the funding from the National Institutes of Health (NIH) (Grants 3R01NS050256-05S1 and 8KL2TR000146), and the Defense Advanced Research Projects Agency (DARPA) Revolutionizing Prosthetics Program (SPAWAR Contract N66001-10-C-4056). This material is also supported in part by the Office of Research and Development, Rehabilitation Research & Development Service
References (61)
- et al.
What do brain lesions tell us about theories of embodied semantics and the human mirror neuron system?
Cortex
(2012) - et al.
Embodied semantics for actions: Findings from functional brain imaging
Journal of Physiology Paris
(2008) - et al.
What wires together dies together: Verbs, actions and neurodegeneration in motor neuron disease
Cortex
(2012) - et al.
High-performance neuroprosthetic control by an individual with tetraplegia
Lancet
(2013) - et al.
Neuroimaging of cognitive functions in human parietal cortex
Current Opinion in Neurobiology
(2001) - et al.
Somatotopic representations of action words in human motor and premotor cortex
Neuron
(2004) - et al.
A continuous semantic space describes the representation of thousands of object and action categories across the human brain
Neuron
(2012) - et al.
Grasping language - a short story on embodiment
Consciousness and Cognition
(2010) The semantics of space: Integrating linguistic typology and cognitive neuroscience
Neuropsychologia
(2006)- et al.
Neuroanatomical distribution of five semantic components of verbs: Evidence from fMRI
Brain and Language
(2008)
The Two-Level Theory of verb meaning: An approach to integrating the semantics of action with the mirror neuron system
Brain and Language
Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions
Cortex
Patterns of comprehension and production of nouns and verbs in agrammatism: Implications for lexical organization
Brain and Language
Verb deficits in Alzheimer's disease and agrammatism: Implications for lexical organization
Brain and Language
Event-related brain potentials to semantically inappropriate and surprisingly large words
Biological Psychology
A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content
Journal of Physiology Paris
Coming of age: A review of embodiment and the neuroscience of semantics
Cortex
Movement priming of EEG/MEG brain responses for action-words characterizes the link between language and action
Cortex
Effects of implied physical effort in sensory-motor and pre-frontal cortex during language comprehension
NeuroImage
Neural correlates of locative prepositions
Neuropsychologia
How could a child use verb syntax to learn verb semantics?
Lingua
Action word meaning representations in cytoarchitectonically defined primary and premotor cortices
NeuroImage
How neurons make meaning: Brain mechanisms for embodied and abstract-symbolic semantics
Trends in Cognitive Sciences
Semantic embodiment, disembodiment or misembodiment? In search of meaning in modules and neuron circuits
Brain and Language
Where the brain appreciates the final state of an event: The neural correlates of telicity
Brain and Language
Primary motor cortex functionally contributes to language comprehension: An online rTMS study
Neuropsychologia
Oscillatory neuronal activity reflects lexical-semantic feature integration within and across sensory modalities in distributed cortical networks
Journal of Neuroscience
Multimodal integration for the representation of space in the posterior parietal cortex
Philosophical Transactions of the Royal Society of London B: Biological Sciences
Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text
NeuroImage
The CELEX lexical database (release 1)
Cited by (0)
- 1
The two corresponding authors contributed equally to this work.