Independent features form integrated objects: Using a novel shape-color “conjunction task” to reconstruct memory resolution for multiple object features simultaneously☆
Introduction
An ability fundamental to human experience is visual object recognition. For example, buying ingredients for a dinner party requires recognizing and purchasing multiple objects from a grocery list: apples rather than oranges, potatoes rather than onions, and eggs rather than garlic. Although these objects can have many overlapping features (e.g., apples and oranges can be spherical, apples and onions can be red), we effortlessly and correctly integrate featural information (“red”, “circle”) to form coherent object representations (“apple”). How the mind combines disparate multimodal features to form integrated objects is known as the binding problem (Riesenhuber & Poggio, 1999; Roskies, 1999; von der Malsburg, 1981), a theoretical and computational problem with a rich history in the cognitive sciences (Feldman, 2013; Garnelo & Shanahan, 2019; Treisman, 1998). Understanding how the mind solves the binding problem may be important for many real-world scenarios, such as in the design of increasingly sophisticated autonomous systems that identify complex objects from simple features (LeCun, Bengio, & Hinton, 2015). In the psychological sciences, feature integration has been a central topic of study in a variety of domains: attention (Kristjánsson & Egeth, 2020), visual search (Wolfe & Horowitz, 2017), immediate and delayed memory (Erez, Cusack, Kendall, & Barense, 2016; Horner, Bisby, Bush, Lin, & Burgess, 2015; Liang, Erez, Zhang, Cusack, & Barense, 2020; Ma, Husain, & Bays, 2014), as well as in disorders associated with binding impairments, such as Alzheimer's disease (Cecchini et al., 2017; Parra et al., 2009) and autism spectrum disorder (Simon & Wallace, 2016; Stevenson et al., 2014; Stevenson et al., 2019). Though there has been much progress made in understanding feature integration, there remains no strong consensus among researchers regarding how objects are represented in the mind based on their constituent features. Below, we roughly group the models of object representation into three categories based on their representational format: object-based, feature-based, and object+features.
Purely object-based models (Fig. 1a) hold that features such as shape and color are integrated to form representations of object identity (e.g., “grandmother-like cells”, Gross, 2002; Mahon, 2015; Rey et al., 2020; Quiroga, Kreiman, Koch, & Fried, 2008; “slot models”, Fukuda, Awh, & Vogel, 2010; Luck & Vogel, 1997; Zhang & Luck, 2008; “object files”, Balaban, Drew, & Luria, 2019; Balaban, Assaf, Arad Meir, & Luria, 2019). Previous behavioral evidence suggests that memories are constrained by the number of objects held in mind rather than by the number of features. In classic change-detection tasks, participants can detect changes of sixteen features distributed across four objects just as well as four features distributed across four objects (Luck & Vogel, 1997), suggesting that objects may be the “building blocks” of memory. In the domain of attention, cognitive resources seem to be constrained by the number of objects rather than by the number of features (e.g., “object-based attention”, Gao et al., 2017; Shen, Huang, & Gao, 2015). Not only are markers of object-based representations found in behavioral experiments, but reliable object-based markers have been observed in neuroimaging. For example, contralateral delay activity present in electroencephalography indexes the number of objects held in mind rather than the number of features (Fukuda et al., 2010; Luria, Balaban, Awh, & Vogel, 2016). Further supporting these results, both rodent neurophysiology and single-cell recording in epilepsy patients have uncovered cells in the medial temporal lobe that respond specifically to objects invariant of viewpoint (Chang & Huerta, 2012; Cohen et al., 2013; Jacklin, Cloke, Potvin, Garrett, & Winters, 2016; Quiroga, 2012; Valdez et al., 2015; Vannucci et al., 2003; Winters, Bartko, Saksida, & Bussey, 2010). This diversity of literature across many domains demonstrates robust evidence in favor of integrated object-based representations.
In contrast, purely feature-based models (Fig. 1b) predict that objects are formed through direct reactivations of sensory features (e.g., “embodied cognition”, Barsalou, 2016; Martin, 2016; Pezzulo et al., 2013; “sensory recruitment model”, Katus, Grubert, & Eimer, 2015, Sreenivasan, Curtis, & D'Esposito, 2014; Ester, Anderson, Serences, and Awh, 2014; “binding-by-synchrony”; Palanca & DeAngelis, 2005; Pina, Bodner, & Ermentrout, 2018; Rabagliati, Doumas, & Bemis, 2017; Singer & Gray, 1995). For example, objects have been proposed to be grounded in perceptual and action systems, such that objects are linear combinations of feature-based representations (Barsalou, 2016). Behavioral evidence in favor of feature-based models include evidence of independent forgetting, whereby a single feature of an object can be successfully retrieved without access of other features (Brady, Konkle, Alvarez, & Oliva, 2013; Fougnie & Alvarez, 2011; Utochkin & Brady, 2020). Moreover, features associated within the same object can be held at different resolutions, suggesting that there is not necessarily a single all-or-nothing object representation (Bays, 2015; Fougnie & Alvarez, 2011; Ma et al., 2014; van Berg, Shin, Chou, George, & Ma, 2012). This body of work suggests that features are represented independently because separate features of the same object can be accessed at varying resolutions (e.g., one feature of an object can be remembered in perfect detail, whereas other features from the same object can be seemingly inaccessible).
Importantly, we emphasize a third class of object + features models (Fig. 1c) which posit that multiple representations of an object can be activated simultaneously in a hierarchical and distributed manner. Independent features are represented in early stages of the object processing pathway, whereas the integrated representation of the object as a cohesive whole is found at later stages of the pathway (e.g., “representational-hierarchical view”; Barense et al., 2012; Binder, 2016; Cowell, Barense, & Sadil, 2019; “binding-in-context”; Cooper and Ritchey, 2019, Cooper and Ritchey, 2020; Diana, Yonelinas, & Ranganath, 2007; Ranganath, 2010; Yonelinas, Ranganath, Ekstrom, & Wiltgen, 2019; “cognitive map”; Behrens et al., 2018; Hawkins, Lewis, Klukas, Purdy and Ahmad, 2019; Mok & Love, 2019). This viewpoint may account for both the predictions of a purely object-based (Fig. 1a) or purely feature-based (Fig. 1b) model in a single theoretical framework. In an object+features model, sensory regions represent independent features such as shape (Kourtzi & Kanwisher, 2000), sound (Feng & Wang, 2017), and color (Brouwer & Heeger, 2009), whereas anterior regions of neocortex represent feature-based information in a transformed and integrated format (Coutanche & Thompson-Schill, 2015; Erez et al., 2016; Jung, Larsen, & Walther, 2018; Liang et al., 2020; Martin, Douglas, Newsome, Man, & Barense, 2018; Rademaker, Chunharas, & Serences, 2019; Schapiro, Turk-Browne, Botvinick, & Norman, 2017). In particular, the perirhinal cortex within the medial temporal lobe is thought to contain an explicit conjunctive code for the integrated features composing an object, such that the whole is different from the sum of the parts (Erez et al., 2016; Liang et al., 2020; Martin et al., 2018). Importantly, an object+features model also predicts feature independence, because each feature is represented in sensory cortex and can be susceptible to interference from competing feature-based information. For this reason, it is entirely possible to observe independent forgetting and varying resolution between features of the same object (Bays, 2015; Brady et al., 2013; Fougnie & Alvarez, 2011; Ma et al., 2014; van Berg et al., 2012), despite observed integration of those features in other task contexts. Indeed, recent evidence has increasingly favored object+features accounts which can reconcile seemingly contradictory findings of independence and integration in memory (Fig. 1c; Cowan, Saults, & Blume, 2014; Hardman & Cowan, 2015; Markov, Tiurina, & Utochkin, 2019; Markov, Utochkin, & Brady, 2021; Oberauer & Eichenberger, 2013; Olson & Jiang, 2002; Sone, Kang, Li, Tsubomi, & Fukuda, 2021; Wheeler & Treisman, 2002).
In the present study, we adjudicate between these three models of object representation (Fig. 1) using a novel shape-color “conjunction task” to characterize the resolution of memory for the shape and color features of an object (Fig. 2; for a previous version of this task using simpler orientation and color, see Sone et al., 2021). Critically, our shape-color conjunction task can reconstruct multiple object features simultaneously from the same behavioral response, as opposed to tasks which reconstruct each feature sequentially (Ma et al., 2014). The simultaneous property of our shape-color conjunction task is important because tasks which reconstruct features sequentially (e.g., color first and then orientation second, and vice versa) can induce independence between object features even though this information may be held in an integrated manner (Sone et al., 2021). In Experiment 1, we first replicate and then extend a classic paradigm originally tested using a change detection task (Luck & Vogel, 1997) with our shape-color conjunction task. As traditional change detection tasks are discrete (i.e., memory is often quantified through a binary “yes/no” response), the continuous nature of our shape-color conjunction task allows us to examine memory resolution in a more sensitive manner. Accounting for both higher- and lower-resolution memories, we show that visual objects can contain both an integrated code as well as independent feature representations. Applying a data-driven statistical model of feature integration based on stochastic dependence, we then provide evidence that memory for features can be higher in resolution than memory for the integrated code, suggesting that individual and integrated features from the same object can differ in memory resolution.
Although Experiment 1 provides evidence in favor of both an integrated code and independent features underlying the object representation, we did not control for the influence of spatial location (Fig. 3). Features in the environment do not only co-occur (e.g., an apple is both spherical and red), but visual features also occupy the same spatial location (e.g., “red” and “spherical” features can be located in the same position in the environment). Indeed, previous work has highlighted the critical influence of spatial location on object identity (Cai, Sheldon, Yu, & Postle, 2019; Cai et al., 2020; Golomb, Kupitz, & Thiemann, 2014; Pertzov & Husain, 2014; Schneegans & Bays, 2017), raising the possibility that visual features may be bound to a shared spatial location rather than to each other in memory. In Experiment 2 we varied the nature of interference, examining the influence of task-irrelevant visual feature and spatial location information on the integrated object code. In this second experiment, we found that task-irrelevant feature information but not spatial location disrupted shape-color integration. Taken together, our work reveals that visual features can be bound directly to each other in a manner that cannot be entirely accounted by spatial location in memory, results which constrain models of object representation.
Section snippets
Participants
Thirty participants were recruited from the undergraduate student pool at the University of Toronto and from the community. One participant was excluded without further data analysis as they did not complete all trials of the experiment. The final sample contained 29 individuals (Mage = 20.93 years, SDage = 1.22 years, Females = 24). Participants recruited from the undergraduate student pool received course credit whereas individuals recruited from the community received $20 CAD as
Results
Anonymized data and commented code are available on the Open Science Framework: https://osf.io/976ta/.
Interim discussion
We developed a novel shape-color conjunction task to capture memory resolution for objects containing multiple component features (shape and color). We found evidence for both integrated object-level representations as well as independent feature-level representations. Consistent with an object-based account, increasing the number of objects (set size 1 to set size 2) reduced both fine-grained responses that were near-perfect in resolution to the target, as well as “yes” responses that
Participants
Thirty participants were recruited from the undergraduate participant pool at the University of Toronto and from the community. Participants from the undergraduate pool received course credit whereas participants from the community received $20 CAD. One participant was excluded prior to data analysis as they did not complete all trials of the experiment. Two participants performed at chance and were excluded. The final sample size contained 27 individuals (Mean age = 19.85 years, SD
Results
Anonymized data are available on the Open Science Framework: https://osf.io/976ta/.
General discussion
In two experiments, we sought to understand how constituent visual features are integrated to form objects in memory. We developed a novel shape-color “conjunction task” to reconstruct the memory resolution of object features simultaneously in a single response (Fig. 2). Relative to traditional discrete tasks (e.g., change detection), our task allows for more precise quantification of both high- and low-resolution memories. Furthermore, our task assessed the resolution of multiple object
CRediT authorship contribution statement
Aedan Y. Li: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization. Keisuke Fukuda: Conceptualization, Methodology, Writing – review & editing, Supervision. Morgan D. Barense: Conceptualization, Methodology, Writing – review & editing, Supervision, Funding acquisition.
Declaration of Competing Interest
None.
Acknowledgements
We thank Julia Duder and Lu Wang for their assistance with data collection. We thank three anonymous reviewers for their thorough and constructive feedback. MATLAB code for the shape-color conjunction task, pre-generated shape-color object stimuli, and anonymized data are available on the Open Science Framework: https://osf.io/976ta/.
AYL is supported by an Alexander Graham Bell Canada Graduate Scholarship-Doctoral from the Natural Sciences and Engineering Research Council of Canada (NSERC CGS-D
References (134)
- et al.
Forgetting across a hierarchy of episodic representations
Neurobiology
(2021) - et al.
Neural evidence for an object-based pointer system underlying working memory
Cortex
(2019) - et al.
Intact memory for irrelevant information impairs perception in amnesia
Neuron
(2012) Spikes not slots: Noise in neural populations limits working memory
Trends in Cognitive Sciences
(2015)- et al.
What is a cognitive map? Organizing Knowledge for Flexible Behavior
Neuron
(2018) - et al.
Boundaries shape cognitive representations of spaces and events
Trends in Cognitive Sciences
(2018) - et al.
The rodent hippocampus is essential for nonspatial object memory
Current Biology
(2013) - et al.
Imaging recollection and familiarity in the medial temporal lobe: A three-component model
Trends in Cognitive Sciences
(2007) - et al.
Precision, binding, and the hippocampus: Precisely what are we talking about?
Neuropsychologia
(2020) - et al.
Discrete capacity limits in visual working memory
Current Opinion in Neurobiology
(2010)
Reconciling deep learning with symbolic artificial intelligence: Representing objects and relations
Current Opinion in Behavioral Sciences
Chapter four—Using multidimensional encoding and retrieval contexts to enhance our understanding of stochastic dependence in source memory
What to believe: Bayesian methods for data analysis
Trends in Cognitive Sciences
The contralateral delay activity as a neural measure of visual working memory
Neuroscience & Biobehavioral Reviews
Different features are stored independently in visual working memory but mediated by object-based representations
Acta Psychologica
Does neuronal synchrony underlie visual feature grouping?
Neuron
Sparse but not ‘grandmother-cell’ coding in the medial temporal lobe
Trends in Cognitive Sciences
Representing composed meanings through temporal binding
Cognition
Visual working memory represents a fixed number of items regardless of complexity
Psychological Science
Why some colors appear more memorable than others: A model combining categories and particulars in color working memory
Journal of Experimental Psychology. General
Different features of real-world objects are represented in a dependent manner in long-term memory
Journal of Experimental Psychology
On staying grounded and avoiding quixotic dead ends
Psychonomic Bulletin & Review
The effects of refreshing and elaboration on working memory performance, and their contributions to long-term memory formation
Memory and Cognition
Fitting Linear Mixed-Effects Models Using lme4
Journal of Statistical Software
The precision of visual working memory is set by allocation of a shared resource
J Vis
Dissociating memory accessibility and precision in forgetting
Nat Hum Behav
Variability in encoding precision accounts for visual short-term memory limitations
Proceedings of the National Academy of Sciences
In defense of abstract conceptual representations
Psychonomic Bulletin & Review
Real-world objects are not represented as bound units: Independent forgetting of different object details from visual memory
Journal of Experimental Psychology: General
Decoding and reconstructing color from responses in human visual cortex
The Journal of Neuroscience
The role of location-context binding in nonspatial visual working memory
ENeuro
Overlapping and distinct contributions of stimulus location and of spatial context to nonspatial visual short-term memory
Journal of Neurophysiology
Recalling feature bindings differentiates Alzheimer's disease from frontotemporal dementia
Journal of Neurology
Neurophysiological correlates of object recognition in the dorsal subiculum
Frontiers in Behavioral Neuroscience
Cortico-hippocampal network connections support the multidimensional quality of episodic memory
ELife
Progression from feature-specific brain activity to hippocampal binding during episodic encoding
Journal of Neuroscience
Creating concepts from converging features in human cortex
Cerebral Cortex
Central and peripheral components of working memory storage
Journal of Experimental Psychology. General
A roadmap for understanding memory: Decomposing cognitive processes into operations and representations
ENeuro
Conjunctive coding of complex object features
Cerebral Cortex
A neural measure of precision in visual working memory
Journal of Cognitive Neuroscience
The neural binding problem(s)
Cognitive Neurodynamics
Harmonic template neurons in primate auditory cortex underlying complex sound processing
Proceedings of the National Academy of Sciences
Object features fail independently in visual working memory: Evidence for a probabilistic feature-store model
Journal of Vision
Bindings in working memory: The role of object-based attention
Attention, Perception, & Psychophysics
A weakly informative default prior distribution for logistic and other regression models
The Annals of Applied Statistics
The influence of object location on identity: A “spatial congruency bias”
Journal of Experimental Psychology: General
A specificity principle of memory: Evidence from aging and associative memory
Psychological Science
Cited by (6)
Similarity-based clustering of multifeature objects in visual working memory
2023, Attention, Perception, and PsychophysicsAwareness of the relative quality of spatial working memory representations
2023, Attention, Perception, and PsychophysicsThe effect of memory load on object reconstruction: Insights from an online mouse-tracking task
2023, Attention, Perception, and PsychophysicsObject-Based Encoding Constrains Storage in Visual Working Memory
2023, Journal of Experimental Psychology: GeneralMapping visual working memory models to a theoretical framework
2023, Psychonomic Bulletin and ReviewFeature Identity Determines Representation Structure in Working Memory
2023, Journal of Experimental Psychology: General
- ☆
An earlier version of this work was presented at the Lake Ontario Visionary Establishment and the Vision Sciences Society in 2019. A preprint is available at https://psyarxiv.com/szjr3/. MATLAB code for the task, pre-generated object stimuli, and anonymized data are available at https://osf.io/976ta/.