Abstract
Some open theoretical questions are addressed on how the mind and brain represent and process concepts, particularly as they are instantiated in particular human languages. Recordings of neuroimaging data should provide a suitable empirical basis for investigating this topic, but the complexity and variety of language demands appropriate data-driven approaches. In this review we argue for a particular suite of methodologies, based on multivariate classification techniques which have proven to be powerful tools for distinguishing neural and cognitive states in fMRI. A combination of larger scale neuroimaging studies are introduced with different monolingual and bilingual populations, and hybrid computational analyses that use encoded implementations of existing theories of conceptual organisation to probe those data. We develop a suite of methodologies that holds the promise of being able to holistically elicit, record and model neural processing during language comprehension and production.
Similar content being viewed by others
Notes
For instance, WordNet, a large scale computational lexicon of contemporary English, contains over 200 thousand word-sense entries. See http://wordnet.princeton.edu/wordnet/man/wnstats.7WN.html.
It should be remembered that conventional contrastive analyses generalise at the cost of fidelity to the original data. They usually use a variety of spatial smoothing (using Gaussian filters), temporal smoothing (through use of a single assumed hemodynamic model) and averaging, which blurs real and partly systematic patterns in the data.
The concepts used were the following: Mammals Anteater, armadillo, beaver, camel, deer, elephant, fox, giraffe, gorilla, hare, hedgehog, hippopotamus, kangaroo, koala, mole, monkey, panda, rhinoceros skunk, zebra.
Tools Allen key, axe, chainsaw, craft knife, file, hammer, nail, paint roller, plaster trowel, pliers, plunger, power drill, rake, saw, scraper, scissors, screw sickle, spanner, tape measure.
The corresponding Chinese and Korean terms are detailed in Akama et al. (2014).
Python package developed to apply machine-learning to human neurological recordings http://www.pymvpa.org/.
References
Akama H, Murphy B, Li N, Shimizu Y, Poesio M (2012) Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study. Frontiers Neuroinformatics 6:24
Akama H, Murphy B, Lei MM, Poesio M (2014) Cross-participant modelling based on joint or disjoint feature selection: an fMRI conceptual decoding study. In: Applied Informatics, vol 1, no 1. Springer, New York, p 1
Anderson AJ, Binder J, Fernandino L, Humphries C, Conant L, Aguilar M, Wang X, Doko D, Raizada RDS (2016) Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation. Cereb Cortex. doi:10.1093/cercor/bhw240 (Advance Online Publication)
Brennan J, Nir Y, Hasson U, Malach R, Heeger DJ, Pylkkänen L (2012) Syntactic structure building in the anterior temporal lobe during natural story listening. Brain Lang 120(2):163–173
Buchweitz A, Keller TA, Meyler A, Just MA (2012) Brain activation for language dual‐tasking: listening to two people speak at the same time and a change in network timing. Hum Brain Mapp 33:1868–1882
Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka ER, Mitchell TM (2010) Toward an architecture for never-ending language learning. Artif Intell 2(4):3
Charest I, Kievit RA, Schmitz TW, Deca D, Kriegeskorte N (2014) Unique semantic space in the brain of each beholder predicts perceived similarity. Proc Natl Acad Sci USA 111(40):14565–14570
Chomsky N (1981) Lectures on government and binding: the Pisa lectures. Walter de Gruyter, USA
Dmochowski JP, Sajda P, Dias J, Parra LC (2012) Correlated components of ongoing EEG point to emotionally laden attention—a possible marker of engagement? Front Human Neurosci 6:112
Fairhall SL, Caramazza A (2013) Brain regions that represent amodal conceptual knowledge. J Neurosci 33(25):10552–10558
Fernandino L, Humphries CJ, Seidenberg MS, Gross WL, Conant LL, Binder JR (2015) Predicting brain activation patterns associated with individual lexical concepts based on five sensory-motor attributes. Neuropsychologia 76:17–26
Fernandino L, Binder JR, Desai RH, Pendl SL, Humphries CJ, Gross WL, Conant LL, Seidenberg MS (2016) Concept representation reflects multimodal abstraction: a framework for embodied semantics. Cereb Cortex 26(5):2018–2034
Frank MC, Everett DL, Fedorenko E, Gibson E (2008) Number as a cognitive technology: evidence from Pirahã language and cognition. Cognition 108(3):819–824
Friston KJ, Holmes AP, Worsley KJ, Poline JP, Frith CD, Frackowiak RS (1994) Statistical parametric maps in functional imaging: a general linear approach. Hum Brain Mapp 2(4):189–210
Fyshe A, Wehbe L, Talukdar P, Murphy B, Mitchell T (2015) A Compositional and Interpretable Semantic Space. In: Proceedings of NAACL, pp 32–41
Griffiths TL, Steyvers M, Tenenbaum JB (2007) Topics in semantic representation. Psychol Rev 114(2):211–244
Hanke M, Halchenko YO, Sederberg PB, Hanson SJ, Haxby JV, Pollmann S (2009) PyMVPA: a python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics 7(1):37–53
Hasson U, Landesman O, Knappmeyer B, Vallines I, Rubin N, Heeger DJ (2008) Neurocinematics: the neuroscience of film. Projections 2(1):1–26
Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539):2425–2430
Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, Ramadge PJ (2011) A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 72(2):404–416
Kriegeskorte N, Goebel R, Bandettini P (2006) Information-based functional brain mapping. Proc Natl Acad Sci USA 103(10):3863–3868
Kriegeskorte N, Mur M, Bandettini P (2008a) Representational similarity analysis–connecting the branches of systems neuroscience. Front Syst Neurosci 2:4
Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J, Esteky H, Tanaka K, Bandettini PA (2008b) Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60(6):1126–1141
Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI (2009) Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 12(5):535–540
Landauer T, Dumais S (1997) A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol Rev 104(2):211–240
Lei M, Akama H, Murphy B (2014) Neural basis of language switching in the brain: fMRI evidence from Korean–Chinese early bilinguals. Brain Lang 138:12–18
Liuzzi AG, Bruffaerts R, Dupont P, Adamczuk K, Peeters R, De Deyne S, Storms G, Vandenberghe R (2015) Left perirhinal cortex codes for similarity in meaning between written words: comparison with auditory word input. Neuropsychologia 76:4–16
Lund K, Burgess C (1996) Producing high-dimensional semantic spaces from lexical co-occurrence. Behav Res Methods Instrum Comput 28:203–208
Meteyard L, Cuadrado SR, Bahrami B, Vigliocco G (2012) Coming of age: a review of embodiment and the neuroscience of semantics. Cortex 48(7):788–804
Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA (2008) Predicting human brain activity associated with the meanings of nouns. Science 320(5880):1191–1195
Murphy G (2004) The big book of concepts. MIT Press, USA, p 555
Murphy B, Poesio M, Bovolo F, Bruzzone L, Dalponte M, Lakany H (2011) EEG decoding of semantic category reveals distributed representations for single concepts. Brain Lang 117(1):12–22
Murphy B, Talukdar P, Mitchell T (2012a) Selecting corpus-semantic models for neurolinguistic decoding. In: Proceedings of First Joint Conference on Lexical and Computational Semantics (*SEM). Association for Computational Linguistics, pp 114–123
Murphy B, Talukdar PP, Mitchell TM (2012b) Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding. In: COLING. pp 1933–1950
Nevins A, Pesetsky D, Rodrigues C (2009) Pirahã exceptionality: a reassessment. Language 85(2):355–404
Newell A (1973) You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium
Pliatsikas C, Luk G (2016) Executive control in bilinguals: a concise review on fMRI studies. Language and Cognition, Bilingualism, pp 1–7
Pulvermüller F (2005) Brain mechanisms linking language and action. Nat Rev Neurosci 6:576–582
Raizada RD, Connolly AC (2012) What makes different people’s representations alike: neural similarity space solves the problem of across-subject fMRI decoding. J Cogn Neurosci 24(4):868–877
Regier T, Kay P, Khetarpal N (2007) Color naming reflects optimal partitions of color space. Proc Natl Acad Sci USA 104:1436–1441
Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N et al (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1):273–289
Wehbe L, Murphy B, Talukdar P, Fyshe A, Ramdas A, Mitchell T (2014a) Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS One 9(11):e112575
Wehbe L, Vaswani A, Knight K, Mitchell T (2014b) Aligning context-based statistical models of language with brain activity during reading. Proceedings of EMNLP
Wilkins J (1668) An essay towards a real character, and a philosophical language. English linguistics, a collection of facsimile reprints, no. 119. Scholar Press, Germany, pp 1500–1800
Zinszer BD, Anderson AJ, Kang O, Wheatley T, Raizada RD (2016) Semantic structural alignment of neural representational spaces enables translation between English and Chinese words. J Cogn Neurosci. doi:10.1162/jocn_a_01000 (Advance Online Publication)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
None.
Additional information
Communicated by Andrzej Cichocki.
Appendix
Appendix
1.1 Methods
These experiments partially replicate the paradigm developed in Mitchell et al. (2008), using fast whole-brain fMRI imaging, and a slow event-related property generation task. Details of the variations in the language and modality specific to each experiment are described below. In Experiment I, native Japanese speakers performed a monolingual task. In Experiments IIa and IIb, Korean–Chinese and Chinese–Japanese bilinguals performed a language switching task.
1.2 Task and stimuli
The same slow event-related design was used in all experiments. Each session had 6 repeated runs for a total of 240 trials. In each trial, a concept was presented for 3 s followed by a fixation cross for 7 s. There were six additional presentations of a fixation cross of 40 s each, distributed just after each run, to establish a signal baseline for subsequent analysis. During the stimuli presentation, participants were asked to do a silent property generation task by thinking of appropriate features of the corresponding concept. During the fixation cross, participants were asked to fixate their eyes on the cross silently and no response was required.
We employed a stimulus set representing concepts in the two categories of land-mammals and work tools based on an earlier EEG experiment (Murphy et al. 2011), with 20 concepts in each of the two categories.Footnote 3 For trials that used image stimuli, 40 contrast-normalized grey-scale photographs were used.
In Experiment I (Akama et al. 2012) five native Japanese speakers (ages 39–53 years) participated, each performing two separate scanning sessions on two different days separated by at least 1 week. The sessions alternated the language stimulus modality: first viewing pictures while listening to the spoken word describing the represented object (the auditory condition), and next viewing pictures with an accompanying caption (the orthographic condition). They were asked to silently enumerate properties that are characteristic of the presented concept.
In Experiment IIa and IIb (Akama et al. 2014; see also Lei et al. 2014) bilinguals performed the same property generation task, with an added language switch condition, which alternated between each of two sessions. All 14 participants were native speakers of Mandarin Chinese (aged 22–28). In experiment IIa, seven early Korean-Chinese bilinguals performed both Korean–Chinese and Chinese–Korean switch conditions (see Fig. 9), and in experiment IIb seven late Chinese–Japanese bilinguals performed the corresponding switch conditions involving those two languages. The experimental stimuli used in both variants are summarized in Fig. 10.
1.3 Data acquisition
Functional MRI scans were performed with a 3.0-T General Electric Signa scanner at Tokyo Institute of Technology, Japan, with an 8-channel high-resolution head coil. Functional scanning was performed using an echo planar imaging sequence with a 1000-ms repetition time (TR), 30 ms echo time (TE), and 60° flip angle (FA). Each volume consisted of 15* 6-mm-thick slices with an interslice gap of 1 mm; FOV:, 20 × *20 cm; size of acquisition matrix, 64 × *64; NEX 1.00. The parameter values of the anatomical scans were TR = 7.284 ms, TE = 2.892 ms, FA = 11°, band width = 31.25 kHz, and voxel size = 1 mm isotropic. Following the settings used by Mitchell et al. (personal communication), we set oblique slices in the sagittal view with a tilt of −20 to −30° such that the most inferior slice was passed above the eyes and through the posterior cerebellum.
1.4 Preprocessing and contrastive group analysis with GLM
fMRI initial data processing was performed with Statistical Parametric Mapping software (SPM8, Wellcome Department of Cognitive Neurology, London, UK). The data were motion corrected, co-registered to the anatomical images, segmented to identify grey matter, and normalized into standard Montreal Neurological Institute (MNI) space at a resliced voxel size of 3 × 3 × 6 mm, coarse rendering for temporal resolution acuity. A General Linear Model (Friston et al. 1994) was used for conventional contrastive analyses (e.g. task > rest; mammal > tool—see Fig. 1 for an example). Both single-session analyses, and random-effects group analyses were performed. Some figures were produced using the XjView toolboxFootnote 4 in addition to SPM8.
1.5 Machine learning analyses
The PyMVPA 2.0 package (Hanke et al. 2009)Footnote 5 was used for machine learning (i.e. multivariate pattern) analyses. The realigned, co-registered, segmented, and grey-matter-masked (but not smoothed) images of each participant in each session were used. The time-course of each voxel was z normalized and detrended, and unless otherwise noted, trialwise estimates were calculated with a boxcar average. In all analyses datasets were split into a training set and an evaluation set using a leave-one-out six-fold cross-validation. The classifier used was a penalised logistic regression (PLR) using L2-norm regularization. The regularization term deals with both high dimensionality and redundancy in data, and its logistic function is optimized to fit discrete data categories. In decoding analyses a univariate feature-selection (ANOVA, which is monotonic with the t statistic) preceded training, and performance was evaluated as the mean percentage correct classification of the semantic category (mammal or tool) of left-out data trials. In sensitivity analyses either the raw ANOVA feature selection ranking was used, or mammal/tool classification accuracy in a search-light sphere (Kriegeskorte et al. 2006).
About this article
Cite this article
Akama, H., Murphy, B. Emerging methods for conceptual modelling in neuroimaging. Behaviormetrika 44, 117–133 (2017). https://doi.org/10.1007/s41237-016-0009-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41237-016-0009-1