What's on your mind?

From stage magicians to the Vulcan mind meld, mind reading has been the stuff of magic and science fiction. Recent developments in neuroimaging might be bringing it one step closer to reality, however, as increasingly sophisticated analysis techniques move toward the decoding of mental states from functional imaging data in humans. Two companies even offer fMRI-based lie detectors¹. Although legal applications are still premature, these pattern classification techniques represent a new way of looking at neuroimaging data² and may extend the power of functional imaging substantially.

Conventional neuroimaging analysis correlates external regressors such as task condition with activity in specific brain areas. Pattern classification inverts this methodology and instead predicts the external stimulus based on neuroimaging data. Unlike conventional analyses, these pattern-based analyses take into account the full spatial pattern of brain activity rather than concentrating on specific regions. Thus, even if activity at a particular voxel does not distinguish different cognitive states, the pattern of activity distributed over many regions can do so, increasing sensitivity. This multivariate approach generates pattern vectors corresponding to specific cognitive states, and a classifier is trained to discriminate between these states. This classifier can then be used to predict the cognitive state on the basis of brain activity alone. Such approaches have been used to predict what percept is dominant in a binocular rivalry protocol³ or what orientation subjects are viewing⁴, even when they are not consciously aware of the stimulus.

These techniques make it easier to evaluate responses to naturalistic stimuli, as pattern classification algorithms are designed to analyze activity over the whole brain without attempting to localize function. In a recent competition at the University of Pittsburgh (http://www.ebc.pitt.edu/competition.html), participants were given fMRI data and subjective ratings from observers as they viewed two short film clips. Competitors then had to produce an algorithm that could predict what the subjects were seeing based on a third fMRI data set. The winning entries achieved correlations as high as 0.86 for basic features such as the presence of music. Uri Hasson from New York University, one of the researchers who judged the competition, says, “I am much more optimistic as to the power of fMRI to read and predict human experience. Many of the participants managed to predict the twelve features chosen for the competition (such as language, music, emotion), as well as the specific observers who coded the movie. Moreover, a few groups managed to predict the identity of the actor being seen or the location the subject is watching.”

The power of this approach extends beyond just predicting cognitive states from brain activity. Pattern classification techniques can provide clues about how this information is processed as well. For example, this technique shows that object categories with shared image-based attributes have shared neural representation, even when multiple views of objects are included or when line drawings without much detail are used⁵. This kind of information would be difficult to uncover using a conventional analysis technique. James Haxby of Princeton University says, “I find myself working with a whole new community of people: computer scientists, electrical engineers, scientists who come from an applied physics and mathematics background. The fact that we are finally looking for patterns of activity across the whole population rather than areas with different function means that we are taking a more information-based approach. This is a fundamental shift.”

Pattern classification can also overcome another criticism of functional imaging: that it lacks the temporal resolution of other whole-brain imaging techniques such as EEG. Pattern classification has been used to identify distributed patterns of activity associated with different categories of objects⁶. During later free recall, the patterns of activity associated with the specific object category reappear several seconds before the verbal recall of the object. The greater sensitivity of pattern classification techniques compared to conventional imaging analysis results in a temporal resolution approaching that of EEG.

As yet, however, these techniques cannot generate the sort of brain maps that we all know from conventional neuroimaging studies, showing spots of localized activity associated with particular functions. Efforts are underway to develop such maps based on pattern classification techniques, but conventional image analysis is likely to remain the preferred method for generating activity maps to understand where in the brain a process is occurring. Localization via conventional brain imaging complements the information about how a process is occurring that can be provided by pattern classification techniques, so both approaches will continue to be useful.

Researchers are attempting to use pattern classification techniques to predict brain states in real-world applications, such as lie detection, but this endeavor seems much less promising. One problem is that activity is more likely to be variable but, more importantly, an fMRI lie detector would rely crucially on the compliance of its subjects. To train a classifier to categorize lying and truth telling, a suspect would essentially be asked to calibrate the instrument for his own conviction, potentially violating the fifth amendment to the United States Constitution, which protects people from being forced to incriminate themselves. Even though the accuracy of the technology is likely to improve, it is unclear if such an fMRI lie detector can surpass conventional polygraph and EEG lie detectors, with which it is likely to share drawbacks such as noise introduced by arousal or emotional responses.

Although the applicability of pattern classification techniques to lie detection is uncertain, their influence on basic research is likely to be important. Neuroimaging's obsession with localization has often led to accusations that it is little more than phrenology. By using population responses across the whole brain to ask how rather than where information is processed, neuroimaging may be starting to come of age.