Abstract
Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant’s responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Rangel, A., Camerer, C., & Montague., P. R. (2008, July). A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience, 9(7), 545–556. ISSN 1471-0048. https://doi.org/10.1038/nrn2357. http://www.ncbi.nlm.nih.gov/pubmed/18545266
Gold, J. I., & Shadlen, M. N. (2007, January). The neural basis of decision making. Annual Review of Neuroscience, 30, 535–74. ISSN 0147-006X. https://doi.org/10.1146/annurev.neuro.29.051605.113038. http://www.ncbi.nlm.nih.gov/pubmed/17600525
Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85(2), 59–108.
Gordon, A. M., Rissman, J., Kiani, R., & Wagner, A. D. (2013). Cortical reinstatement mediates the relationship between content-specific encoding activity and subsequent recollection decisions. Cerebral Cortex, 24(12), 3350–3364. ISSN 1460-2199. https://doi.org/10.1093/cercor/bht194. http://www.ncbi.nlm.nih.gov/pubmed/23921785
van Ravenzwaaij, D., Mulder, M. J., Tuerlinckx, F., & Wagenmakers, E-J. (2012, January). Do the dynamics of prior information depend on task context? An analysis of optimal performance and an empirical test. Frontiers in Psychology, 3(May), 132. ISSN 1664-1078. https://doi.org/10.3389/fpsyg.2012.00132
Drugowitsch, J., Moreno-Bote, R., Churchland, A. K., Shadlen, M. N., & Pouget, A. (2012, March). The cost of accumulating evidence in perceptual decision making. Journal of Neuroscience, 32(11), 3612–28. ISSN 1529-2401. https://doi.org/10.1523/JNEUROSCI.4010-11.2012. http://www.ncbi.nlm.nih.gov/pubmed/22423085
Moran, R. (2015). Optimal decision making in heterogeneous and biased environments. Psychonomic Bulletin & Review, 38–53. https://doi.org/10.3758/s13423-014-0669-3
Van Ravenzwaaij, D., Mulder, M. J., Tuerlinckx, F., & Wagenmakers, E. J. (2015). Paradoxes of optimal decision making: A response to Moran (2015). Psychonomic Bulletin & Review, 22, 307–308.
Perugini, A., Ditterich, J., & Basso, M. A. (2016). Patients with Parkinson’s disease show impaired use of priors in conditions of sensory uncertainty. Current Biology, 26(14), 1902–1910. ISSN 09609822. https://doi.org/10.1016/j.cub.2016.05.039
Perugini, A., & Basso, M. A. (2017). Perceptual Decisions Based on Previously Learned Information are Independent of Dopaminergic Tone. Journal of Neurophysiology, jn.00761.2017. ISSN 0022-3077. https://doi.org/10.1152/jn.00761.2017. http://www.physiology.org/doi/10.1152/jn.00761.2017
Hanks, T. D., Mazurek, M. E., Kiani, R., Hopp, E., Michael, N., Shadlen, M. N. (2011, April). Elapsed decision time affects the weighting of prior probability in a perceptual decision task. Journal of Neuroscience, 31 (17): 6339–6352. ISSN 0270-6474. https://doi.org/10.1523/JNEUROSCI.5613-10.2011. http://www.jneurosci.org/cgi/doi/10.1523/JNEUROSCI.5613-10.2011
Biderman, N., Bakkour, A., & Shohamy, D. (2020). What are memories for? the hippocampus bridges past experience with future decisions. Trends in Cognitive Sciences, 24(7), 542–556.
Bornstein, A. M., & Daw, N. D. (2013, December). Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans. PLoS Computational Biology, 9(12), e1003387. ISSN 1553-7358. https://doi.org/10.1371/journal.pcbi.1003387. http://dx.plos.org/10.1371/journal.pcbi.1003387
Shadlen, M. N., & Shohamy, D. (2016). Decision making and sequential sampling from memory. Neuron, 90 (5): 927–939. ISSN 0896-6273. https://doi.org/10.1016/j.neuron.2016.04.036
Bornstein, A. M., Khaw, M. W., Shohamy, D., & Daw, N. D. (2017). Reminders of past choices bias decisions for reward in humans. Nature Communications, 8, 15958. https://doi.org/10.1038/ncomms15958
Bornstein, A. M., & Norman, K. A. (2017). Reinstated episodic context guides sampling-based decisions for reward. Nature Neuroscience. https://doi.org/10.1038/nn.4573
Bakkour, A., Palombo, D. J., Zylberberg, A., Kang, Y. H. R., Reid, A., Verfaellie, M., ... Shohamy, D. (2019). The hippocampus supports deliberation during value-based decisions. elife, 8, e46080.
Bornstein, A. M., & Daw, N. D. (2012, April). Dissociating hippocampal and striatal contributions to sequential prediction learning. European Journal of Neuroscience, 35(7), 1011–1023. ISSN 0953816X. https://doi.org/10.1111/j.1460-9568.2011.07920.x. http://doi.wiley.com/10.1111/j.1460-9568.2011.07920.x
Graham, K. S., Barense, M. D., & Lee, A. C. H. (2010). Going beyond LTM in the MTL: A synthesis of neuropsychological and neuroimaging findings on the role of the medial temporal lobe in memory and perception. Neuropsychologia, 48(4), 831-853.
Sullivan, N., Hutcherson, C., Harris, A., & Rangel, A. (2015). Dietary self-control is related to the speed with which attributes of healthfulness and tastiness are processed. Psychological Science, 26(2), 122–134.
Srivastava, V., Feng, S. F., Cohen, J. D., Leonard, N. E., & Shenhav, A. (2016). A martingale analysis of first passage times of time-dependent Wiener diffusion models. Journal of Mathematical Psychology. ISSN 00222496. https://doi.org/10.1016/j.jmp.2016.10.001
Lombardi, G., & Hare, T. (2021). Piecewise constant averaging methods allow for fast and accurate hierarchical bayesian estimation of drift diffusion models with time-varying evidence accumulation rates.
Hartigan, J. A., & Hartigan, P. M. (1985). The dip test of unimodality. The Annals of Statistics, 13(1), 70–84. ISSN 0090-5364. https://doi.org/10.1214/aos/1176346577
Bogacz, R., Brown, E., Moehlis, J., Holmes, P., & Cohen, J. D. (2006, October). The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychological Review, 113(4), 700–65. ISSN 0033-295X. https://doi.org/10.1037/0033-295X.113.4.700. http://www.ncbi.nlm.nih.gov/pubmed/17014301
Kanwisher, N., Mcdermott, J., & Chun, M. M. (1997). The Fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11), 4302–4311.
Epstein, R., & Kanwisher, N. (1998, April). A cortical representation of the local visual environment. Nature, 392(6676), 598–601. ISSN 0028-0836. https://doi.org/10.1038/33402. http://www.ncbi.nlm.nih.gov/pubmed/9560155
Bogacz, R., Hu, P. T., Holmes, P. J., & Cohen, J. D. (2010). Do humans produce the speed-accuracy trade-off that maximizes reward rate? Quarterly Journal of Experimental Psychology, 63(5), 863–91. ISSN 1747–0226. https://doi.org/10.1080/17470210903091643
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. London: Springer.
Dunovan, K. E., Tremel, J. J., & Wheeler, M. E. (2014, June). Prior probability and feature predictability interactively bias perceptual decisions. Neuropsychologia, 61(1), 210–221. ISSN 18733514. https://doi.org/10.1016/j.neuropsychologia.2014.06.024. http://linkinghub.elsevier.com/retrieve/pii/S002839321400205X. http://www.ncbi.nlm.nih.gov/pubmed/24978303
Palestro, J. J., Weichart, E., Sederberg, B., & Turner, B. M. (2018). Some task demands induce collapsing bounds: Evidence from a behavioral analysis. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-018-1479-9
Leite, F. P., & Ratcliff, R. (2011). What cognitive processes drive response biases? A diffusion model analysis. Judgement and Decision Making, 6(7), 651–687.
Ratcliff, R., & Tuerlinckx, F. (2002, September). Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability. Psychonomic Bulletin & Review, 9(3), 438–81. ISSN 1069-9384. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2474747 &tool=pmcentrez &rendertype=abstract
Ratcliff, R., & McKoon, G. (2008, April). The diffusion decision model: theory and data for two-choice decision tasks. Neural computation, 20(4), 873–922. ISSN 0899-7667. https://doi.org/10.1162/neco.2008.12-06-420. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2474742 &tool=pmcentrez &rendertype=abstract
Hauser, C. K., Zhu, D., Stanford, T. R., & Salinas, E. (2018). Motor selection dynamics in FEF explain the reaction time variance of saccades to single targets. eLife, 1–32.
Deneve, S. (2012). Making decisions with unknown sensory reliability. Frontiers in neuroscience, 6, 75.
Rahnev, D., Lau, H., & de Lange, F. P. (2011, July). Prior Expectation Modulates the Interaction between Sensory and Prefrontal Regions in the Human Brain. Journal of Neuroscience, 31 (29): 10741–10748. ISSN 0270-6474. https://doi.org/10.1523/JNEUROSCI.1478-11.2011. http://www.jneurosci.org/cgi/doi/10.1523/JNEUROSCI.1478-11.2011
Wyart, V., de Gardelle, V., Scholl, J., Summerfield, C., De Gardelle, V., Scholl, J., & Summerfield, C. (2012, November). Rhythmic Fluctuations in Evidence Accumulation during Decision Making in the Human Brain. Neuron, 76(4), 847–858. ISSN 08966273. https://doi.org/10.1016/j.neuron.2012.09.015. URL http://linkinghub.elsevier.com/retrieve/pii/S0896627312008483
Yu, A. J., & Cohen, J. D. (2009). Sequential effects: Superstition or rational behavior? Advances in Neural Information Processing Systems, 21, 1873–80. ISSN 1049-5258 (Print). URL http://www.pni.princeton.edu/ncc/publications/2009/YuNIPS09.pdf
Silver, D., & Veness, J. (2010). Monte-Carlo planning in large POMDPs. Neural Information Processing Systems, 1–9.
Barto, A. C., & Sutton, R. S. (1998). April). Reinforcement learning: An introduction. MIT Press, Cambridge, MA. https://doi.org/10.1007/s00134-010-1760-5
Lengyel, Mate, & Dayan, Peter. (2008). Hippocampal Contributions to Control: The Third Way. Advances in Neural Information Processing Systems, 20, 889–896.
Ritter, S. (2018). Jane X Wang, Zeb Kurth-nelson Siddhant, Charles Blundell, Razvan Pascanu, and Matthew Botvinick. Been There, Done That: Meta-Learning with Episodic Recall. In ICML.
Vikbladh, O., Shohamy, D., & Daw, N. D. (2017). Episodic contributions to model - based reinforcement learning. Cognitive Computational Neuroscience Conference, 2016–2017.
Kiani, R., Churchland, A. K., & Shadlen, M. N. (2013). Integration of direction cues is invariant to the temporal gap between them. Journal of Neuroscience, 33(42), 16483–16489. https://doi.org/10.1523/JNEUROSCI.2094-13.2013
Kiani, R., Hanks, T. D., & Shadlen, M. N. (2008). Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment. Journal of Neuroscience, 28(12), 3017–3029. https://doi.org/10.1523/JNEUROSCI.4761-07.2008
Hoskin, A. N., Bornstein, A. M., Norman, K. A., & Cohen, J. D. (2019). Refresh my memory: Episodic memory reinstatements intrude on working memory maintenance. Cognitive, Affective, & Behavioral Neuroscience, 19(2), 338–354.
Gao, J., Tortell, R., & McClelland, J. L. (2011, January). Dynamic integration of reward and stimulus information in perceptual decision-making. PloS one, 6(3), e16749. ISSN 1932-6203. https://doi.org/10.1371/journal.pone.0016749. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3048391 &tool=pmcentrez &rendertype=abstract
Braun, A., Urai, A. E., & Donner, T. H. (2018). Adaptive history biases result from confidence-weighted accumulation of past choices. Journal of Neuroscience, 38(10), 2418–2429. https://doi.org/10.1523/JNEUROSCI.2189-17.2017
Odegaard, B., Grimaldi, P., Hah, S., Peters, M. A. K., & Lau, H. (2017). Superior colliculus neuronal ensemble activity signals optimal rather than subjective confidence. Proceedings of the National Academy of Sciences, 1–10 https://doi.org/10.1073/pnas.1711628115
Kiani, R., & Shadlen, M. N. (2009, May). Representation of confidence associated with a decision by neurons in the parietal cortex. Science, 324(5928): 759–64. ISSN 1095-9203. https://doi.org/10.1126/science.1169405. http://www.ncbi.nlm.nih.gov/pubmed/19423820
Resulaj, A., Kiani, R., Wolpert, D. M., & Shadlen, M. N. (2009). Changes of mind in decision-making. Nature, 461(7261), 263–266. ISSN 1476-4687. https://doi.org/10.1038/nature08275. http://www.ncbi.nlm.nih.gov/pubmed/19693010
van den Berg, R., Zylberberg, A., Kiani, R., Shadlen, M. N., Wolpert, D. M., Link, S. W., ... Ratcliff, R. (2016). Confidence is the bridge between multi-stage decisions. Current Biology, 0(0), 114–135. ISSN 09609822. https://doi.org/10.1016/j.cub.2016.10.021. http://linkinghub.elsevier.com/retrieve/pii/S0960982216312064
Mccurdy, L. Y., Maniscalco, B., Metcalfe, J., Liu, K. Y., De Lange, F. P., & Lau, H. (2013). Anatomical coupling between distinct metacognitive systems for memory and visual perception. Journal of Neuroscience, 33(5), 1897–1906. https://doi.org/10.1523/JNEUROSCI.1890-12.2013
Ye, Q., Zou, F., Lau, H., Hu, Y., & Kwok, S. C. (2018). Causal evidence for mnemonic metacognition in human precuneus. Journal of Neuroscience, 38(28), 6379–6387. https://doi.org/10.1523/JNEUROSCI.0660-18.2018
Khoudary, A., Peters, M. A. K., & Bornstein, A. M. (2022). Precision-weighted evidence integration predicts time-varying influence of memory on perceptual decisions. Cognitive Computational Neuroscience.
Wang, S., Feng, S. F., & Bornstein, A. M. (2022). Mixing memory and desire: How memory reactivation supports deliberative decision-making. Wiley Interdisciplinary Reviews: Cognitive Science, 13(2), e1581
Zheng, X. Y., Hebart, M. N., Dolan, R. J., Doeller, C. F., Cools, R., & Garvert, M. M. (2022). Parallel cognitive maps for short-term statistical and long-term semantic relationships in the hippocampal formation. bioRxiv, 2022–08.
Stine, G. M., Zylberberg, A., Ditterich, J., & Shadlen, M. N. (2020). Differentiating between integration and non-integration strategies in perceptual decision making. Elife, 9, e55365
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016). Characterizing a psychiatric symptom dimension related to deficits in goaldirected control. eLife, 5(MARCH2016): 1–24. ISSN 2050084X. https://doi.org/10.7554/eLife.11305
Davies, D. J., Teufel, C., & Fletcher, P. C. (2017). Anomalous perceptions and beliefs are associated with shifts toward different types of prior knowledge in perceptual inference. Schizophrenia Bulletin, (January). ISSN 0586-7614. https://doi.org/10.1093/schbul/sbx177. http://academic.oup.com/schizophreniabulletin/advance-article/doi/10.1093/schbul/sbx177/4779898
Brainard, D. H. (1997, January). The psychophysics toolbox. Spatial Vision, 10(4), 433–6. ISSN 0169-1015. http://www.ncbi.nlm.nih.gov/pubmed/9176952
Watson, A. B., & Pelli, D. G. (1983). QUEST: A Bayesian adaptive psychometric method. Perception & Psychophysics, 33(2), 113–120.
Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E. J., Johansen-Berg, H., Bannister, P. R., De Luca, M., Drobnjak, I., Flitney, D. E., Niazy, R. K., Saunders, J., Vickers, J., Zhang, Y., De Stefano, N., Brady, J. M., & Matthews, P. M. (2004). Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage, 23(SUPPL. 1), 208–219. ISSN 10538119. https://doi.org/10.1016/j.neuroimage.2004.07.051
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012). FSL. NeuroImage, 62(2), 782–790. ISSN 10538119. https://doi.org/10.1016/j.neuroimage.2011.09.015
Price, N., & Mechler, F. (2002). HartigansDipTest. http://www.nicprice.net/diptest/
Kim, G., Lewis-Peacock, J. A., Norman, K. A., & Turk-Browne, N. B. (2014). Pruning of memories by context-based prediction error. Proceedings of the National Academy of Sciences, 111(24). https://doi.org/10.1073/pnas.1319438111
Cohen, J. (1988). Statistical power analysis for the behavioural sciences. NJ: Lawrence Earlbaum Associates, Hillside, NJ.
Acknowledgements
The authors wish to thank Abigail Hoskin, Amitai Shenhav, Judith Fan, Phillip Holmes, Michael Waskom and Roozbeh Kiani for helpful conversations, Ghootae Kim for providing ranked face and scene stimuli, Nicholas Hindy for providing fractal stimuli, and Charlotte Townsend for extensive assistance with data collection. This publication was made possible through the support of funding from the Intel corporation, and a grant from the John Templeton Foundation (Grant ID #57,876; K.A.N. and J.D.C.). The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the John Templeton Foundation.
Author information
Authors and Affiliations
Contributions
A.M.B. and M.A. conceived experiment; A.M.B., M.A., and S.F.F. designed experiment and analyses, with input from N.B.T., K.A.N. and J.D.C. A.M.B. and M.A. wrote the experiment code; A.M.B. and M.A. ran the experiment; A.M.B. and S.F.F. contributed analytic tools; A.M.B. and S.F.F. performed analyses; A.M.B. wrote the paper, with input from M.A., N.B.T., K.A.N., and J.D.C.. All authors approved the final manuscript.
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Bornstein, A.M., Aly, M., Feng, S.F. et al. Associative memory retrieval modulates upcoming perceptual decisions. Cogn Affect Behav Neurosci 23, 645–665 (2023). https://doi.org/10.3758/s13415-023-01092-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3758/s13415-023-01092-6