Elsevier

NeuroImage

Volume 56, Issue 2, 15 May 2011, Pages 699-708
NeuroImage

Within- and cross-participant classifiers reveal different neural coding of information

https://doi.org/10.1016/j.neuroimage.2010.03.057Get rights and content

Abstract

Analyzing distributed patterns of brain activation using multivariate pattern analysis (MVPA) has become a popular approach for using functional magnetic resonance imaging (fMRI) data to predict mental states. While the majority of studies currently build separate classifiers for each participant in the sample, in principle a single classifier can be derived from and tested on data from all participants. These two approaches, within- and cross-participant classification, rely on potentially different sources of variability and thus may provide distinct information about brain function. Here, we used both approaches to identify brain regions that contain information about passively received monetary rewards (i.e., images of currency that influenced participant payment) and social rewards (i.e., images of human faces). Our within-participant analyses implicated regions in the ventral visual processing stream—including fusiform gyrus and primary visual cortex—and ventromedial prefrontal cortex (VMPFC). Two key results indicate these regions may contain statistically discriminable patterns that contain different informational representations. First, cross-participant analyses implicated additional brain regions, including striatum and anterior insula. The cross-participant analyses also revealed systematic changes in predictive power across brain regions, with the pattern of change consistent with the functional properties of regions. Second, individual differences in classifier performance in VMPFC were related to individual differences in preferences between our two reward modalities. We interpret these results as reflecting a distinction between patterns showing participant-specific functional organization and those indicating aspects of brain organization that generalize across individuals.

Introduction

Humans can rapidly identify, categorize, and evaluate environmental stimuli. Identifying the neural mechanisms that underlie stimulus evaluation is a fundamental goal of cognitive neuroscience. Part of that research agenda includes the identification of functional changes in the brain that predict the characteristics of perceived stimuli. An important recent approach involves analyzing functional magnetic resonance imaging (fMRI) data for task-related patterns of information (Kriegeskorte and Bandettini, 2007), often through the application of techniques from machine learning, called multivariate pattern analysis (MVPA). Although still less popular than standard univariate techniques, MVPA continues to grow in scope, as evidenced by recent overviews (Haynes and Rees, 2006, Mitchell et al., 2004, Norman et al., 2006, O'Toole et al., 2007), tutorials (Etzel et al., 2009, Mur et al., 2009, Pereira et al., 2009), and consideration of potential applications (deCharms, 2007, Friston, 2009, Haynes, 2009, Spiers and Maguire, 2007).

Studies employing MVPA now cover a diverse set of topics. The earliest and most-common targets were feature representations and topographies in the visual cortex (Carlson et al., 2003, Cox and Savoy, 2003, Haynes and Rees, 2005, Kamitani and Tong, 2005). More recent studies have broadened the application of MVPA to many other types of information: hidden intentions (Haynes et al., 2007), free will (Soon et al., 2008), odor processing (Howard et al., 2009), scene categorization (Peelen et al., 2009), components of working memory (Harrison and Tong, 2009), individual differences in perception (Raizada et al., 2010), basic choices (Hampton and O'Doherty, 2007), purchasing decisions (Grosenick et al., 2008), and economic value (Clithero et al., 2009, Krajbich et al., 2009). In striking examples, feature spaces determined using MVPA have been extended to decode the content of complex brain states, such as identifying specific pictures (Kay et al., 2008) and reconstructing the contents of visual experience (Miyawaki et al., 2008, Naselaris et al., 2009).

Nearly all MVPA studies that employ classifiers build an independent classification model for each participant, based on the trial-to-trial variability in the fMRI signal. This approach is well-suited to identify brain regions that play a consistent functional role within-participants, but it cannot make claims about common cross-participant representation. While relatively few studies have adopted the latter approach, some early applications have targeted deception (Davatzikos et al., 2005), different object categories (Shinkareva et al., 2008), mental states that are consistent across a wide variety of tasks (Poldrack et al., 2009), attention (Mourao-Miranda et al., 2005), biomarkers for psychosis (Sun et al., 2009), and Alzheimer's disease (Vemuri et al., 2008). To date, however, no study has systematically evaluated whether within- and cross-participant analyses provide distinct information about brain function.

There may be important functional differences between the results of within- and cross-participant MVPA. The popularity and promise of MVPA stems from the notion that its analyses go beyond demonstrating the involvement of a region in a particular task; they provide important information about the representational content of brain regions (Mur et al., 2009). Accordingly, joint examination of within- and cross-participants patterns may clarify how information is represented within a region. Regions that contribute to the same task may do so for different reasons. One may be consistently recruited but represent participant-specific information, while another's functional organization may reflect both common recruitment and common information across individuals. The objective of the current study was to provide such comparisons in brain regions whose functional contributions to a task might reflect general or idiosyncratic effects, across individuals.

Here, we employed the “searchlight” method (Kriegeskorte et al., 2006) to extract local spatial information from small spheres of brain voxels while measuring fMRI activation in participants who passively received monetary and social rewards (Hayden et al., 2007, Smith et al., 2010). We then employed a popular machine-learning implementation, support vector machines (SVM), to generate and evaluate classifiers for searchlights throughout the brain. Our goals were to identify the brain regions that contain information that can distinguish the reward modality of each trial, and then to identify potential functional organization within those regions based on the relative classification power and information content of within- and cross-participant analyses.

Section snippets

Participants

Twenty healthy participants (mean age: 23 years, range: 18–30 years) completed a session involving both behavioral and fMRI data collection. All participants were male and indicated a heterosexual orientation, via self-report. Four of these participants were dropped from the sample prior to data analyses: three for excessive head motion and one because of equipment failure, leaving a final sample of 16 participants. Prescreening excluded individuals with prior psychiatric or neurological illness.

Behavior

Participants performed well on the background target-detection task (average hit rate of 84.2%). The post-scanning ratings of face attractiveness were highly correlated with those from an independent sample (mean r = 0.71, range 0.50 to 0.85; see Supplementary Materials), supporting our a priori division of stimuli into attractiveness categories. Moreover, on a large fraction of trials during the exchange task, participants were willing to sacrifice money to see a face with a higher

Discussion

Our results support a novel conclusion: within-participant and cross-participant MVPA classification implicate distinct sets of brain regions. When classifying social and non-social rewards, both models identified key regions for the perception and valuation of social information. Regions showing maximal classification performance (e.g., VC, FFA, and VMPFC) have been implicated in previous studies of face and object recognition (Grill-Spector and Malach, 2004, Tsao and Livingstone, 2008), as

Conclusion

Using machine-learning techniques and multivariate pattern analysis of fMRI data, we demonstrated that classifier performance differs between within-participant and cross-participant training. We emphasize that we are not concerned with the level changes in classifier performance; there are obvious additional sources of variability for cross-participant classification. Instead, our results indicate that relative classifier sensitivity may reflect the contributions of different brain regions to

Acknowledgments

We thank Justin Meyer for help with data collection and analysis. This research was supported by an Incubator Award from the Duke Institute for Brain Sciences (SAH), by NIMH grant P01-41328 (SAH), by NINDS training grant T32-51156 (RMC), and by NIMH National Research Service Awards F31-086255 (JAC) and F31-086248 (DVS). The authors have no competing financial interests.

References (85)

  • K. Izuma et al.

    Processing of social and monetary rewards in the human striatum

    Neuron

    (2008)
  • M. Jenkinson et al.

    Improved optimization for the robust and accurate linear registration and motion correction of brain images

    Neuroimage

    (2002)
  • Y. Kamitani et al.

    Spatial smoothing hurts localization but not information: pitfalls for brain mappers

    Neuroimage

    (2010)
  • N. Kriegeskorte et al.

    Analyzing for information, not activation, to exploit high-resolution fMRI

    Neuroimage

    (2007)
  • M.L. Kringelbach et al.

    The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology

    Prog. Neurobiol.

    (2004)
  • S. LaConte et al.

    Support vector machines for temporal classification of block design fMRI data

    Neuroimage

    (2005)
  • M. Lebreton et al.

    An automatic valuation system in the human brain: evidence from functional neuroimaging

    Neuron

    (2009)
  • Y. Miyawaki et al.

    Visual image reconstruction from human brain activity using a combination of multiscale local image decoders

    Neuron

    (2008)
  • J. Mourao-Miranda et al.

    Classifying brain states and determining the discriminating activation patterns: support vector machine on functional MRI data

    Neuroimage

    (2005)
  • T. Naselaris et al.

    Bayesian reconstruction of natural images from human brain activity

    Neuron

    (2009)
  • K.A. Norman et al.

    Beyond mind-reading: multi-voxel pattern analysis of fMRI data

    Trends Cogn. Sci.

    (2006)
  • J. O'Doherty et al.

    Beauty in a smile: the role of medial orbitofrontal cortex in facial attractiveness

    Neuropsychologia

    (2003)
  • J.P. O'Doherty et al.

    Temporal difference models and reward-related learning in the human brain

    Neuron

    (2003)
  • F. Pereira et al.

    Machine learning classifiers and fMRI: a tutorial overview

    Neuroimage

    (2009)
  • J.T. Serences

    Value-based modulations in human visual cortex

    Neuron

    (2008)
  • B. Seymour et al.

    Anchors, scales and the relative coding of value in the brain

    Curr. Opin. Neurobiol.

    (2008)
  • S.M. Smith et al.

    Advances in functional and structural MR image analysis and implementation as FSL

    Neuroimage

    (2004)
  • H.J. Spiers et al.

    Decoding human brain activity during real-world experiences

    Trends Cogn. Sci.

    (2007)
  • D. Sun et al.

    Elucidating a magnetic resonance imaging-based neuroanatomic biomarker for psychosis: classification analysis using probabilistic brain atlas and machine learning algorithms

    Biol. Psychiatry

    (2009)
  • P. Vemuri et al.

    Alzheimer's disease diagnosis in individual subjects using structural MR images: validation studies

    Neuroimage

    (2008)
  • T. Allison et al.

    Electrophysiological studies of human face perception: I. Potentials generated in occipitotemporal cortex by face and non-face stimuli

    Cereb. Cortex

    (1999)
  • T.E.J. Behrens et al.

    Associative learning of social value

    Nature

    (2008)
  • D.H. Brainard

    The psychophysics toolbox

    Spat. Vis.

    (1997)
  • M. Brett et al.

    The problem of functional localization in the human brain

    Nat. Rev. Neurosci.

    (2002)
  • T.A. Carlson et al.

    Patterns of activity in the categorical representations of objects

    J. Cogn. Neurosci.

    (2003)
  • J. Cloutier et al.

    Are attractive people rewarding? Sex differences in the neural substrates of facial attractiveness

    J. Cogn. Neurosci.

    (2008)
  • D.M. Drucker et al.

    Different spatial scales of shape similarity representation in lateral and ventral LOC

    Cereb. Cortex

    (2009)
  • E. Eger et al.

    fMRI activity patterns in human LOC carry information about object exemplars within category

    J. Cogn. Neurosci.

    (2008)
  • R. Epstein et al.

    A cortical representation of the local visual environment

    Nature

    (1998)
  • K.J. Friston

    Modalities, modes, and models in functional neuroimaging

    Science

    (2009)
  • K. Grill-Spector et al.

    The human visual cortex

    Annu. Rev. Neurosci.

    (2004)
  • L. Grosenick et al.

    Interpretable classifiers for fMRI improve prediction of purchases

    IEEE Trans. Neural Syst. Rehabil. Eng.

    (2008)
  • Cited by (46)

    • Inter-subject pattern analysis for multivariate group analysis of functional neuroimaging. A unifying formalization

      2020, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      In fact, it is already common practice for ISPA studies. Indeed, it has been reported in several articles (see e.g. [8,11,24,29,36,46]), but we believe it is also vastly used without being reported because researchers might consider it to be trivial practice. Our formalization highlights that it is in fact not trivial, and we therefore recommend researchers to explicitly report in their publications whether they perform subject-by-subject feature standardization using this multi-source transductive setting.

    • What do across-subject analyses really tell us about neural coding?

      2020, Neuropsychologia
      Citation Excerpt :

      Making inferences about neural coding implicating fine-scale patterns of information by testing for generalization of brain patterns across individuals might seem impossible, given the well documented anatomical variability within healthy populations (e.g., Frost and Goebel, 2013; Zhen et al., 2017, 2015). In fact, several studies have provided empirical evidence that spatially-structured information generalizes poorly across subjects if spatial normalization is used to bring activation patterns from different subjects into correspondence (Clithero et al., 2011; Cox and Savoy, 2003; Haxby et al., 2011). Haxby et al. however, showed that a novel approach to “hyperalign” brain activation patterns across subjects, could, in principle, enable researchers to harness fine-scale structure while testing for generalization across subjects.

    View all citing articles on Scopus
    View full text