Skip to main content

Advertisement

Log in

Emerging methods for conceptual modelling in neuroimaging

Behaviormetrika Aims and scope Submit manuscript

Abstract

Some open theoretical questions are addressed on how the mind and brain represent and process concepts, particularly as they are instantiated in particular human languages. Recordings of neuroimaging data should provide a suitable empirical basis for investigating this topic, but the complexity and variety of language demands appropriate data-driven approaches. In this review we argue for a particular suite of methodologies, based on multivariate classification techniques which have proven to be powerful tools for distinguishing neural and cognitive states in fMRI. A combination of larger scale neuroimaging studies are introduced with different monolingual and bilingual populations, and hybrid computational analyses that use encoded implementations of existing theories of conceptual organisation to probe those data. We develop a suite of methodologies that holds the promise of being able to holistically elicit, record and model neural processing during language comprehension and production.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. For instance, WordNet, a large scale computational lexicon of contemporary English, contains over 200 thousand word-sense entries. See http://wordnet.princeton.edu/wordnet/man/wnstats.7WN.html.

  2. It should be remembered that conventional contrastive analyses generalise at the cost of fidelity to the original data. They usually use a variety of spatial smoothing (using Gaussian filters), temporal smoothing (through use of a single assumed hemodynamic model) and averaging, which blurs real and partly systematic patterns in the data.

  3. The concepts used were the following: Mammals Anteater, armadillo, beaver, camel, deer, elephant, fox, giraffe, gorilla, hare, hedgehog, hippopotamus, kangaroo, koala, mole, monkey, panda, rhinoceros skunk, zebra.

    Tools Allen key, axe, chainsaw, craft knife, file, hammer, nail, paint roller, plaster trowel, pliers, plunger, power drill, rake, saw, scraper, scissors, screw sickle, spanner, tape measure.

    The corresponding Chinese and Korean terms are detailed in Akama et al. (2014).

  4. http://www.alivelearn.net/xjview8/.

  5. Python package developed to apply machine-learning to human neurological recordings http://www.pymvpa.org/.

References

  • Akama H, Murphy B, Li N, Shimizu Y, Poesio M (2012) Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA study. Frontiers Neuroinformatics 6:24

    Article  Google Scholar 

  • Akama H, Murphy B, Lei MM, Poesio M (2014) Cross-participant modelling based on joint or disjoint feature selection: an fMRI conceptual decoding study. In: Applied Informatics, vol 1, no 1. Springer, New York, p 1

  • Anderson AJ, Binder J, Fernandino L, Humphries C, Conant L, Aguilar M, Wang X, Doko D, Raizada RDS (2016) Predicting neural activity patterns associated with sentences using a neurobiologically motivated model of semantic representation. Cereb Cortex. doi:10.1093/cercor/bhw240 (Advance Online Publication)

    Google Scholar 

  • Brennan J, Nir Y, Hasson U, Malach R, Heeger DJ, Pylkkänen L (2012) Syntactic structure building in the anterior temporal lobe during natural story listening. Brain Lang 120(2):163–173

    Article  Google Scholar 

  • Buchweitz A, Keller TA, Meyler A, Just MA (2012) Brain activation for language dual‐tasking: listening to two people speak at the same time and a change in network timing. Hum Brain Mapp 33:1868–1882

    Article  Google Scholar 

  • Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka ER, Mitchell TM (2010) Toward an architecture for never-ending language learning. Artif Intell 2(4):3

    Google Scholar 

  • Charest I, Kievit RA, Schmitz TW, Deca D, Kriegeskorte N (2014) Unique semantic space in the brain of each beholder predicts perceived similarity. Proc Natl Acad Sci USA 111(40):14565–14570

    Article  Google Scholar 

  • Chomsky N (1981) Lectures on government and binding: the Pisa lectures. Walter de Gruyter, USA

    Google Scholar 

  • Dmochowski JP, Sajda P, Dias J, Parra LC (2012) Correlated components of ongoing EEG point to emotionally laden attention—a possible marker of engagement? Front Human Neurosci 6:112

    Article  Google Scholar 

  • Fairhall SL, Caramazza A (2013) Brain regions that represent amodal conceptual knowledge. J Neurosci 33(25):10552–10558

    Article  Google Scholar 

  • Fernandino L, Humphries CJ, Seidenberg MS, Gross WL, Conant LL, Binder JR (2015) Predicting brain activation patterns associated with individual lexical concepts based on five sensory-motor attributes. Neuropsychologia 76:17–26

    Article  Google Scholar 

  • Fernandino L, Binder JR, Desai RH, Pendl SL, Humphries CJ, Gross WL, Conant LL, Seidenberg MS (2016) Concept representation reflects multimodal abstraction: a framework for embodied semantics. Cereb Cortex 26(5):2018–2034

    Article  Google Scholar 

  • Frank MC, Everett DL, Fedorenko E, Gibson E (2008) Number as a cognitive technology: evidence from Pirahã language and cognition. Cognition 108(3):819–824

    Article  Google Scholar 

  • Friston KJ, Holmes AP, Worsley KJ, Poline JP, Frith CD, Frackowiak RS (1994) Statistical parametric maps in functional imaging: a general linear approach. Hum Brain Mapp 2(4):189–210

    Article  Google Scholar 

  • Fyshe A, Wehbe L, Talukdar P, Murphy B, Mitchell T (2015) A Compositional and Interpretable Semantic Space. In: Proceedings of NAACL, pp 32–41

  • Griffiths TL, Steyvers M, Tenenbaum JB (2007) Topics in semantic representation. Psychol Rev 114(2):211–244

    Article  Google Scholar 

  • Hanke M, Halchenko YO, Sederberg PB, Hanson SJ, Haxby JV, Pollmann S (2009) PyMVPA: a python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics 7(1):37–53

    Article  Google Scholar 

  • Hasson U, Landesman O, Knappmeyer B, Vallines I, Rubin N, Heeger DJ (2008) Neurocinematics: the neuroscience of film. Projections 2(1):1–26

    Article  Google Scholar 

  • Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539):2425–2430

    Article  Google Scholar 

  • Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, Ramadge PJ (2011) A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 72(2):404–416

    Article  Google Scholar 

  • Kriegeskorte N, Goebel R, Bandettini P (2006) Information-based functional brain mapping. Proc Natl Acad Sci USA 103(10):3863–3868

    Article  Google Scholar 

  • Kriegeskorte N, Mur M, Bandettini P (2008a) Representational similarity analysis–connecting the branches of systems neuroscience. Front Syst Neurosci 2:4

    Article  Google Scholar 

  • Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J, Esteky H, Tanaka K, Bandettini PA (2008b) Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60(6):1126–1141

    Article  Google Scholar 

  • Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI (2009) Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 12(5):535–540

    Article  Google Scholar 

  • Landauer T, Dumais S (1997) A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol Rev 104(2):211–240

    Article  Google Scholar 

  • Lei M, Akama H, Murphy B (2014) Neural basis of language switching in the brain: fMRI evidence from Korean–Chinese early bilinguals. Brain Lang 138:12–18

    Article  Google Scholar 

  • Liuzzi AG, Bruffaerts R, Dupont P, Adamczuk K, Peeters R, De Deyne S, Storms G, Vandenberghe R (2015) Left perirhinal cortex codes for similarity in meaning between written words: comparison with auditory word input. Neuropsychologia 76:4–16

    Article  Google Scholar 

  • Lund K, Burgess C (1996) Producing high-dimensional semantic spaces from lexical co-occurrence. Behav Res Methods Instrum Comput 28:203–208

    Article  Google Scholar 

  • Meteyard L, Cuadrado SR, Bahrami B, Vigliocco G (2012) Coming of age: a review of embodiment and the neuroscience of semantics. Cortex 48(7):788–804

    Article  Google Scholar 

  • Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA (2008) Predicting human brain activity associated with the meanings of nouns. Science 320(5880):1191–1195

    Article  Google Scholar 

  • Murphy G (2004) The big book of concepts. MIT Press, USA, p 555

    Google Scholar 

  • Murphy B, Poesio M, Bovolo F, Bruzzone L, Dalponte M, Lakany H (2011) EEG decoding of semantic category reveals distributed representations for single concepts. Brain Lang 117(1):12–22

    Article  Google Scholar 

  • Murphy B, Talukdar P, Mitchell T (2012a) Selecting corpus-semantic models for neurolinguistic decoding. In: Proceedings of First Joint Conference on Lexical and Computational Semantics (*SEM). Association for Computational Linguistics, pp 114–123

  • Murphy B, Talukdar PP, Mitchell TM (2012b) Learning Effective and Interpretable Semantic Models using Non-Negative Sparse Embedding. In: COLING. pp 1933–1950

  • Nevins A, Pesetsky D, Rodrigues C (2009) Pirahã exceptionality: a reassessment. Language 85(2):355–404

    Article  Google Scholar 

  • Newell A (1973) You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium

  • Pliatsikas C, Luk G (2016) Executive control in bilinguals: a concise review on fMRI studies. Language and Cognition, Bilingualism, pp 1–7

    Google Scholar 

  • Pulvermüller F (2005) Brain mechanisms linking language and action. Nat Rev Neurosci 6:576–582

    Article  Google Scholar 

  • Raizada RD, Connolly AC (2012) What makes different people’s representations alike: neural similarity space solves the problem of across-subject fMRI decoding. J Cogn Neurosci 24(4):868–877

    Article  Google Scholar 

  • Regier T, Kay P, Khetarpal N (2007) Color naming reflects optimal partitions of color space. Proc Natl Acad Sci USA 104:1436–1441

    Article  Google Scholar 

  • Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N et al (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15(1):273–289

    Article  Google Scholar 

  • Wehbe L, Murphy B, Talukdar P, Fyshe A, Ramdas A, Mitchell T (2014a) Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS One 9(11):e112575

    Article  Google Scholar 

  • Wehbe L, Vaswani A, Knight K, Mitchell T (2014b) Aligning context-based statistical models of language with brain activity during reading. Proceedings of EMNLP

  • Wilkins J (1668) An essay towards a real character, and a philosophical language. English linguistics, a collection of facsimile reprints, no. 119. Scholar Press, Germany, pp 1500–1800

  • Zinszer BD, Anderson AJ, Kang O, Wheatley T, Raizada RD (2016) Semantic structural alignment of neural representational spaces enables translation between English and Chinese words. J Cogn Neurosci. doi:10.1162/jocn_a_01000 (Advance Online Publication)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hiroyuki Akama.

Ethics declarations

Conflicts of interest

None.

Additional information

Communicated by Andrzej Cichocki.

Appendix

Appendix

1.1 Methods

These experiments partially replicate the paradigm developed in Mitchell et al. (2008), using fast whole-brain fMRI imaging, and a slow event-related property generation task. Details of the variations in the language and modality specific to each experiment are described below. In Experiment I, native Japanese speakers performed a monolingual task. In Experiments IIa and IIb, Korean–Chinese and Chinese–Japanese bilinguals performed a language switching task.

1.2 Task and stimuli

The same slow event-related design was used in all experiments. Each session had 6 repeated runs for a total of 240 trials. In each trial, a concept was presented for 3 s followed by a fixation cross for 7 s. There were six additional presentations of a fixation cross of 40 s each, distributed just after each run, to establish a signal baseline for subsequent analysis. During the stimuli presentation, participants were asked to do a silent property generation task by thinking of appropriate features of the corresponding concept. During the fixation cross, participants were asked to fixate their eyes on the cross silently and no response was required.

We employed a stimulus set representing concepts in the two categories of land-mammals and work tools based on an earlier EEG experiment (Murphy et al. 2011), with 20 concepts in each of the two categories.Footnote 3 For trials that used image stimuli, 40 contrast-normalized grey-scale photographs were used.

In Experiment I (Akama et al. 2012) five native Japanese speakers (ages 39–53 years) participated, each performing two separate scanning sessions on two different days separated by at least 1 week. The sessions alternated the language stimulus modality: first viewing pictures while listening to the spoken word describing the represented object (the auditory condition), and next viewing pictures with an accompanying caption (the orthographic condition). They were asked to silently enumerate properties that are characteristic of the presented concept.

In Experiment IIa and IIb (Akama et al. 2014; see also Lei et al. 2014) bilinguals performed the same property generation task, with an added language switch condition, which alternated between each of two sessions. All 14 participants were native speakers of Mandarin Chinese (aged 22–28). In experiment IIa, seven early Korean-Chinese bilinguals performed both Korean–Chinese and Chinese–Korean switch conditions (see Fig. 9), and in experiment IIb seven late Chinese–Japanese bilinguals performed the corresponding switch conditions involving those two languages. The experimental stimuli used in both variants are summarized in Fig. 10.

Fig. 9
figure 9

‘Language A → Language B’ indicates that the participant was presented with stimuli captions in language A and was asked to perform covert property generation in language B

Fig. 10
figure 10

Trial structure and stimulus format for experiments IIa and IIb

1.3 Data acquisition

Functional MRI scans were performed with a 3.0-T General Electric Signa scanner at Tokyo Institute of Technology, Japan, with an 8-channel high-resolution head coil. Functional scanning was performed using an echo planar imaging sequence with a 1000-ms repetition time (TR), 30 ms echo time (TE), and 60° flip angle (FA). Each volume consisted of 15* 6-mm-thick slices with an interslice gap of 1 mm; FOV:, 20 × *20 cm; size of acquisition matrix, 64 × *64; NEX 1.00. The parameter values of the anatomical scans were TR = 7.284 ms, TE = 2.892 ms, FA = 11°, band width = 31.25 kHz, and voxel size = 1 mm isotropic. Following the settings used by Mitchell et al. (personal communication), we set oblique slices in the sagittal view with a tilt of −20 to −30° such that the most inferior slice was passed above the eyes and through the posterior cerebellum.

1.4 Preprocessing and contrastive group analysis with GLM

fMRI initial data processing was performed with Statistical Parametric Mapping software (SPM8, Wellcome Department of Cognitive Neurology, London, UK). The data were motion corrected, co-registered to the anatomical images, segmented to identify grey matter, and normalized into standard Montreal Neurological Institute (MNI) space at a resliced voxel size of 3 × 3 × 6 mm, coarse rendering for temporal resolution acuity. A General Linear Model (Friston et al. 1994) was used for conventional contrastive analyses (e.g. task > rest; mammal > tool—see Fig. 1 for an example). Both single-session analyses, and random-effects group analyses were performed. Some figures were produced using the XjView toolboxFootnote 4 in addition to SPM8.

1.5 Machine learning analyses

The PyMVPA 2.0 package (Hanke et al. 2009)Footnote 5 was used for machine learning (i.e. multivariate pattern) analyses. The realigned, co-registered, segmented, and grey-matter-masked (but not smoothed) images of each participant in each session were used. The time-course of each voxel was z normalized and detrended, and unless otherwise noted, trialwise estimates were calculated with a boxcar average. In all analyses datasets were split into a training set and an evaluation set using a leave-one-out six-fold cross-validation. The classifier used was a penalised logistic regression (PLR) using L2-norm regularization. The regularization term deals with both high dimensionality and redundancy in data, and its logistic function is optimized to fit discrete data categories. In decoding analyses a univariate feature-selection (ANOVA, which is monotonic with the t statistic) preceded training, and performance was evaluated as the mean percentage correct classification of the semantic category (mammal or tool) of left-out data trials. In sensitivity analyses either the raw ANOVA feature selection ranking was used, or mammal/tool classification accuracy in a search-light sphere (Kriegeskorte et al. 2006).

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Akama, H., Murphy, B. Emerging methods for conceptual modelling in neuroimaging. Behaviormetrika 44, 117–133 (2017). https://doi.org/10.1007/s41237-016-0009-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41237-016-0009-1

Keywords

Navigation