Elsevier

Cognition

Volume 223, June 2022, 105024
Cognition

Independent features form integrated objects: Using a novel shape-color “conjunction task” to reconstruct memory resolution for multiple object features simultaneously

https://doi.org/10.1016/j.cognition.2022.105024Get rights and content

Abstract

Though much progress has been made to understand feature integration, debate remains regarding how objects are represented in mind based on their constituent features. Here, we advance this debate by introducing a novel shape-color “conjunction task” to reconstruct memory resolution for multiple object features simultaneously. In a first experiment, we replicate and extend a classic paradigm originally tested using a change detection task. Replicating previous work, memory resolution for individual features was reduced when the number of objects increased, regardless of the number of to-be-remembered features. Extending previous work, we found that high resolution memory near perfect in resemblance to the target was selectively impacted by the number of to-be-remembered features. Applying a data-driven statistical model of stochastic dependence, we found robust evidence of integration for lower-resolution feature memories, but less evidence for integration of high-resolution feature memories. These results suggest that memory resolution for individual features can be higher than memory resolution for their integration. In a second experiment which manipulated the nature of distracting information, we examined whether object features were directly bound to each other or by virtue of shared spatial location. Feature integration was disrupted by distractors sharing visual features of target objects but not when distractors shared spatial location – suggesting that feature integration can be driven by direct binding between shape and color features in memory. Our results constrain theoretical models of object representation, providing empirical support for hierarchical representations of both integrated and independent features.

Introduction

An ability fundamental to human experience is visual object recognition. For example, buying ingredients for a dinner party requires recognizing and purchasing multiple objects from a grocery list: apples rather than oranges, potatoes rather than onions, and eggs rather than garlic. Although these objects can have many overlapping features (e.g., apples and oranges can be spherical, apples and onions can be red), we effortlessly and correctly integrate featural information (“red”, “circle”) to form coherent object representations (“apple”). How the mind combines disparate multimodal features to form integrated objects is known as the binding problem (Riesenhuber & Poggio, 1999; Roskies, 1999; von der Malsburg, 1981), a theoretical and computational problem with a rich history in the cognitive sciences (Feldman, 2013; Garnelo & Shanahan, 2019; Treisman, 1998). Understanding how the mind solves the binding problem may be important for many real-world scenarios, such as in the design of increasingly sophisticated autonomous systems that identify complex objects from simple features (LeCun, Bengio, & Hinton, 2015). In the psychological sciences, feature integration has been a central topic of study in a variety of domains: attention (Kristjánsson & Egeth, 2020), visual search (Wolfe & Horowitz, 2017), immediate and delayed memory (Erez, Cusack, Kendall, & Barense, 2016; Horner, Bisby, Bush, Lin, & Burgess, 2015; Liang, Erez, Zhang, Cusack, & Barense, 2020; Ma, Husain, & Bays, 2014), as well as in disorders associated with binding impairments, such as Alzheimer's disease (Cecchini et al., 2017; Parra et al., 2009) and autism spectrum disorder (Simon & Wallace, 2016; Stevenson et al., 2014; Stevenson et al., 2019). Though there has been much progress made in understanding feature integration, there remains no strong consensus among researchers regarding how objects are represented in the mind based on their constituent features. Below, we roughly group the models of object representation into three categories based on their representational format: object-based, feature-based, and object+features.

Purely object-based models (Fig. 1a) hold that features such as shape and color are integrated to form representations of object identity (e.g., “grandmother-like cells”, Gross, 2002; Mahon, 2015; Rey et al., 2020; Quiroga, Kreiman, Koch, & Fried, 2008; “slot models”, Fukuda, Awh, & Vogel, 2010; Luck & Vogel, 1997; Zhang & Luck, 2008; “object files”, Balaban, Drew, & Luria, 2019; Balaban, Assaf, Arad Meir, & Luria, 2019). Previous behavioral evidence suggests that memories are constrained by the number of objects held in mind rather than by the number of features. In classic change-detection tasks, participants can detect changes of sixteen features distributed across four objects just as well as four features distributed across four objects (Luck & Vogel, 1997), suggesting that objects may be the “building blocks” of memory. In the domain of attention, cognitive resources seem to be constrained by the number of objects rather than by the number of features (e.g., “object-based attention”, Gao et al., 2017; Shen, Huang, & Gao, 2015). Not only are markers of object-based representations found in behavioral experiments, but reliable object-based markers have been observed in neuroimaging. For example, contralateral delay activity present in electroencephalography indexes the number of objects held in mind rather than the number of features (Fukuda et al., 2010; Luria, Balaban, Awh, & Vogel, 2016). Further supporting these results, both rodent neurophysiology and single-cell recording in epilepsy patients have uncovered cells in the medial temporal lobe that respond specifically to objects invariant of viewpoint (Chang & Huerta, 2012; Cohen et al., 2013; Jacklin, Cloke, Potvin, Garrett, & Winters, 2016; Quiroga, 2012; Valdez et al., 2015; Vannucci et al., 2003; Winters, Bartko, Saksida, & Bussey, 2010). This diversity of literature across many domains demonstrates robust evidence in favor of integrated object-based representations.

In contrast, purely feature-based models (Fig. 1b) predict that objects are formed through direct reactivations of sensory features (e.g., “embodied cognition”, Barsalou, 2016; Martin, 2016; Pezzulo et al., 2013; “sensory recruitment model”, Katus, Grubert, & Eimer, 2015, Sreenivasan, Curtis, & D'Esposito, 2014; Ester, Anderson, Serences, and Awh, 2014; “binding-by-synchrony”; Palanca & DeAngelis, 2005; Pina, Bodner, & Ermentrout, 2018; Rabagliati, Doumas, & Bemis, 2017; Singer & Gray, 1995). For example, objects have been proposed to be grounded in perceptual and action systems, such that objects are linear combinations of feature-based representations (Barsalou, 2016). Behavioral evidence in favor of feature-based models include evidence of independent forgetting, whereby a single feature of an object can be successfully retrieved without access of other features (Brady, Konkle, Alvarez, & Oliva, 2013; Fougnie & Alvarez, 2011; Utochkin & Brady, 2020). Moreover, features associated within the same object can be held at different resolutions, suggesting that there is not necessarily a single all-or-nothing object representation (Bays, 2015; Fougnie & Alvarez, 2011; Ma et al., 2014; van Berg, Shin, Chou, George, & Ma, 2012). This body of work suggests that features are represented independently because separate features of the same object can be accessed at varying resolutions (e.g., one feature of an object can be remembered in perfect detail, whereas other features from the same object can be seemingly inaccessible).

Importantly, we emphasize a third class of object+features models (Fig. 1c) which posit that multiple representations of an object can be activated simultaneously in a hierarchical and distributed manner. Independent features are represented in early stages of the object processing pathway, whereas the integrated representation of the object as a cohesive whole is found at later stages of the pathway (e.g., “representational-hierarchical view”; Barense et al., 2012; Binder, 2016; Cowell, Barense, & Sadil, 2019; “binding-in-context”; Cooper and Ritchey, 2019, Cooper and Ritchey, 2020; Diana, Yonelinas, & Ranganath, 2007; Ranganath, 2010; Yonelinas, Ranganath, Ekstrom, & Wiltgen, 2019; “cognitive map”; Behrens et al., 2018; Hawkins, Lewis, Klukas, Purdy and Ahmad, 2019; Mok & Love, 2019). This viewpoint may account for both the predictions of a purely object-based (Fig. 1a) or purely feature-based (Fig. 1b) model in a single theoretical framework. In an object+features model, sensory regions represent independent features such as shape (Kourtzi & Kanwisher, 2000), sound (Feng & Wang, 2017), and color (Brouwer & Heeger, 2009), whereas anterior regions of neocortex represent feature-based information in a transformed and integrated format (Coutanche & Thompson-Schill, 2015; Erez et al., 2016; Jung, Larsen, & Walther, 2018; Liang et al., 2020; Martin, Douglas, Newsome, Man, & Barense, 2018; Rademaker, Chunharas, & Serences, 2019; Schapiro, Turk-Browne, Botvinick, & Norman, 2017). In particular, the perirhinal cortex within the medial temporal lobe is thought to contain an explicit conjunctive code for the integrated features composing an object, such that the whole is different from the sum of the parts (Erez et al., 2016; Liang et al., 2020; Martin et al., 2018). Importantly, an object+features model also predicts feature independence, because each feature is represented in sensory cortex and can be susceptible to interference from competing feature-based information. For this reason, it is entirely possible to observe independent forgetting and varying resolution between features of the same object (Bays, 2015; Brady et al., 2013; Fougnie & Alvarez, 2011; Ma et al., 2014; van Berg et al., 2012), despite observed integration of those features in other task contexts. Indeed, recent evidence has increasingly favored object+features accounts which can reconcile seemingly contradictory findings of independence and integration in memory (Fig. 1c; Cowan, Saults, & Blume, 2014; Hardman & Cowan, 2015; Markov, Tiurina, & Utochkin, 2019; Markov, Utochkin, & Brady, 2021; Oberauer & Eichenberger, 2013; Olson & Jiang, 2002; Sone, Kang, Li, Tsubomi, & Fukuda, 2021; Wheeler & Treisman, 2002).

In the present study, we adjudicate between these three models of object representation (Fig. 1) using a novel shape-color “conjunction task” to characterize the resolution of memory for the shape and color features of an object (Fig. 2; for a previous version of this task using simpler orientation and color, see Sone et al., 2021). Critically, our shape-color conjunction task can reconstruct multiple object features simultaneously from the same behavioral response, as opposed to tasks which reconstruct each feature sequentially (Ma et al., 2014). The simultaneous property of our shape-color conjunction task is important because tasks which reconstruct features sequentially (e.g., color first and then orientation second, and vice versa) can induce independence between object features even though this information may be held in an integrated manner (Sone et al., 2021). In Experiment 1, we first replicate and then extend a classic paradigm originally tested using a change detection task (Luck & Vogel, 1997) with our shape-color conjunction task. As traditional change detection tasks are discrete (i.e., memory is often quantified through a binary “yes/no” response), the continuous nature of our shape-color conjunction task allows us to examine memory resolution in a more sensitive manner. Accounting for both higher- and lower-resolution memories, we show that visual objects can contain both an integrated code as well as independent feature representations. Applying a data-driven statistical model of feature integration based on stochastic dependence, we then provide evidence that memory for features can be higher in resolution than memory for the integrated code, suggesting that individual and integrated features from the same object can differ in memory resolution.

Although Experiment 1 provides evidence in favor of both an integrated code and independent features underlying the object representation, we did not control for the influence of spatial location (Fig. 3). Features in the environment do not only co-occur (e.g., an apple is both spherical and red), but visual features also occupy the same spatial location (e.g., “red” and “spherical” features can be located in the same position in the environment). Indeed, previous work has highlighted the critical influence of spatial location on object identity (Cai, Sheldon, Yu, & Postle, 2019; Cai et al., 2020; Golomb, Kupitz, & Thiemann, 2014; Pertzov & Husain, 2014; Schneegans & Bays, 2017), raising the possibility that visual features may be bound to a shared spatial location rather than to each other in memory. In Experiment 2 we varied the nature of interference, examining the influence of task-irrelevant visual feature and spatial location information on the integrated object code. In this second experiment, we found that task-irrelevant feature information but not spatial location disrupted shape-color integration. Taken together, our work reveals that visual features can be bound directly to each other in a manner that cannot be entirely accounted by spatial location in memory, results which constrain models of object representation.

Section snippets

Participants

Thirty participants were recruited from the undergraduate student pool at the University of Toronto and from the community. One participant was excluded without further data analysis as they did not complete all trials of the experiment. The final sample contained 29 individuals (Mage = 20.93 years, SDage = 1.22 years, Females = 24). Participants recruited from the undergraduate student pool received course credit whereas individuals recruited from the community received $20 CAD as

Results

Anonymized data and commented code are available on the Open Science Framework: https://osf.io/976ta/.

Interim discussion

We developed a novel shape-color conjunction task to capture memory resolution for objects containing multiple component features (shape and color). We found evidence for both integrated object-level representations as well as independent feature-level representations. Consistent with an object-based account, increasing the number of objects (set size 1 to set size 2) reduced both fine-grained responses that were near-perfect in resolution to the target, as well as “yes” responses that

Participants

Thirty participants were recruited from the undergraduate participant pool at the University of Toronto and from the community. Participants from the undergraduate pool received course credit whereas participants from the community received $20 CAD. One participant was excluded prior to data analysis as they did not complete all trials of the experiment. Two participants performed at chance and were excluded. The final sample size contained 27 individuals (Mean age = 19.85 years, SD

Results

Anonymized data are available on the Open Science Framework: https://osf.io/976ta/.

General discussion

In two experiments, we sought to understand how constituent visual features are integrated to form objects in memory. We developed a novel shape-color “conjunction task” to reconstruct the memory resolution of object features simultaneously in a single response (Fig. 2). Relative to traditional discrete tasks (e.g., change detection), our task allows for more precise quantification of both high- and low-resolution memories. Furthermore, our task assessed the resolution of multiple object

CRediT authorship contribution statement

Aedan Y. Li: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization. Keisuke Fukuda: Conceptualization, Methodology, Writing – review & editing, Supervision. Morgan D. Barense: Conceptualization, Methodology, Writing – review & editing, Supervision, Funding acquisition.

Declaration of Competing Interest

None.

Acknowledgements

We thank Julia Duder and Lu Wang for their assistance with data collection. We thank three anonymous reviewers for their thorough and constructive feedback. MATLAB code for the shape-color conjunction task, pre-generated shape-color object stimuli, and anonymized data are available on the Open Science Framework: https://osf.io/976ta/.

AYL is supported by an Alexander Graham Bell Canada Graduate Scholarship-Doctoral from the Natural Sciences and Engineering Research Council of Canada (NSERC CGS-D

References (134)

  • M. Garnelo et al.

    Reconciling deep learning with symbolic artificial intelligence: Representing objects and relations

    Current Opinion in Behavioral Sciences

    (2019)
  • J.L. Hicks et al.

    Chapter four—Using multidimensional encoding and retrieval contexts to enhance our understanding of stochastic dependence in source memory

  • J.K. Kruschke

    What to believe: Bayesian methods for data analysis

    Trends in Cognitive Sciences

    (2010)
  • R. Luria et al.

    The contralateral delay activity as a neural measure of visual working memory

    Neuroscience & Biobehavioral Reviews

    (2016)
  • Y.A. Markov et al.

    Different features are stored independently in visual working memory but mediated by object-based representations

    Acta Psychologica

    (2019)
  • B.J.A. Palanca et al.

    Does neuronal synchrony underlie visual feature grouping?

    Neuron

    (2005)
  • R.Q. Quiroga et al.

    Sparse but not ‘grandmother-cell’ coding in the medial temporal lobe

    Trends in Cognitive Sciences

    (2008)
  • H. Rabagliati et al.

    Representing composed meanings through temporal binding

    Cognition

    (2017)
  • E. Awh et al.

    Visual working memory represents a fixed number of items regardless of complexity

    Psychological Science

    (2007)
  • G.-Y. Bae et al.

    Why some colors appear more memorable than others: A model combining categories and particulars in color working memory

    Journal of Experimental Psychology. General

    (2015)
  • H. Balaban et al.

    Different features of real-world objects are represented in a dependent manner in long-term memory

    Journal of Experimental Psychology

    (2019)
  • L.W. Barsalou

    On staying grounded and avoiding quixotic dead ends

    Psychonomic Bulletin & Review

    (2016)
  • L.M. Bartsch et al.

    The effects of refreshing and elaboration on working memory performance, and their contributions to long-term memory formation

    Memory and Cognition

    (2018)
  • D. Bates et al.

    Fitting Linear Mixed-Effects Models Using lme4

    Journal of Statistical Software

    (2015)
  • P.M. Bays et al.

    The precision of visual working memory is set by allocation of a shared resource

    J Vis

    (2009)
  • S.C. Berens et al.

    Dissociating memory accessibility and precision in forgetting

    Nat Hum Behav

    (2020)
  • R. van den Berg et al.

    Variability in encoding precision accounts for visual short-term memory limitations

    Proceedings of the National Academy of Sciences

    (2012)
  • J.R. Binder

    In defense of abstract conceptual representations

    Psychonomic Bulletin & Review

    (2016)
  • T.F. Brady et al.

    Real-world objects are not represented as bound units: Independent forgetting of different object details from visual memory

    Journal of Experimental Psychology: General

    (2013)
  • G.J. Brouwer et al.

    Decoding and reconstructing color from responses in human visual cortex

    The Journal of Neuroscience

    (2009)
  • Y. Cai et al.

    The role of location-context binding in nonspatial visual working memory

    ENeuro

    (2020)
  • Y. Cai et al.

    Overlapping and distinct contributions of stimulus location and of spatial context to nonspatial visual short-term memory

    Journal of Neurophysiology

    (2019)
  • M.A. Cecchini et al.

    Recalling feature bindings differentiates Alzheimer's disease from frontotemporal dementia

    Journal of Neurology

    (2017)
  • E.H. Chang et al.

    Neurophysiological correlates of object recognition in the dorsal subiculum

    Frontiers in Behavioral Neuroscience

    (2012)
  • R.A. Cooper et al.

    Cortico-hippocampal network connections support the multidimensional quality of episodic memory

    ELife

    (2019)
  • R.A. Cooper et al.

    Progression from feature-specific brain activity to hippocampal binding during episodic encoding

    Journal of Neuroscience

    (2020)
  • M.N. Coutanche et al.

    Creating concepts from converging features in human cortex

    Cerebral Cortex

    (2015)
  • Cowan, N., Saults, J.S., & Blume, C.L. (2014). Central and peripheral components of working memory storage. Journal of...
  • N. Cowan et al.

    Central and peripheral components of working memory storage

    Journal of Experimental Psychology. General

    (2014)
  • R.A. Cowell et al.

    A roadmap for understanding memory: Decomposing cognitive processes into operations and representations

    ENeuro

    (2019)
  • J. Erez et al.

    Conjunctive coding of complex object features

    Cerebral Cortex

    (2016)
  • E.F. Ester et al.

    A neural measure of precision in visual working memory

    Journal of Cognitive Neuroscience

    (2013)
  • Ester, E.F., Ho, T.C., Brown, S.D., & Serences, J.T. (2014). Variability in visual working memory ability limits the...
  • J. Feldman

    The neural binding problem(s)

    Cognitive Neurodynamics

    (2013)
  • L. Feng et al.

    Harmonic template neurons in primate auditory cortex underlying complex sound processing

    Proceedings of the National Academy of Sciences

    (2017)
  • D. Fougnie et al.

    Object features fail independently in visual working memory: Evidence for a probabilistic feature-store model

    Journal of Vision

    (2011)
  • Z. Gao et al.

    Bindings in working memory: The role of object-based attention

    Attention, Perception, & Psychophysics

    (2017)
  • A. Gelman et al.

    A weakly informative default prior distribution for logistic and other regression models

    The Annals of Applied Statistics

    (2008)
  • J.D. Golomb et al.

    The influence of object location on identity: A “spatial congruency bias”

    Journal of Experimental Psychology: General

    (2014)
  • N.R. Greene et al.

    A specificity principle of memory: Evidence from aging and associative memory

    Psychological Science

    (2020)
  • An earlier version of this work was presented at the Lake Ontario Visionary Establishment and the Vision Sciences Society in 2019. A preprint is available at https://psyarxiv.com/szjr3/. MATLAB code for the task, pre-generated object stimuli, and anonymized data are available at https://osf.io/976ta/.

    View full text