Abstract
Exploring the spatial patterns and temporal dynamics of human brain activity has been of great interest, in the quest to better understand connectome-scale brain networks. Though modeling spatial and temporal patterns of functional brain networks have been researched for a long time, the development of a unified and simultaneous spatial-temporal model has yet to be realized. For instance, although some deep learning methods have been proposed recently in order to model functional brain networks, most of them can only represent either spatial or temporal perspective of functional Magnetic Resonance Imaging (fMRI) data and rarely model both domains simultaneously. Due to the recent success in applying sequential auto-encoders for brain decoding, in this paper, we propose a deep sparse recurrent auto-encoder (DSRAE) to be applied unsupervised to learn spatial patterns and temporal fluctuations of brain networks at the same time. The proposed DSRAE was evaluated and validated based on three tasks of the publicly available Human Connectome Project (HCP) fMRI dataset, resulting with promising evidence. To the best of our knowledge, the proposed DSRAE is among the early efforts in developing unified models that can extract connectome-scale spatial-temporal networks from 4D fMRI data simultaneously.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Archbold, K. H., Ph, D., Borghesani, P. R., Ph, D., Mahurin, R. K., Ph, D., & Kapur, V. K. (2009). Neural activation patterns during working memory tasks and OSA disease severity: Preliminary findings. Journal of Clinical Sleep Medicine, 5(1), 21–27 https://www.ncbi.nlm.nih.gov/pubmed/19317377.
Barch, D. M., Burgess, G. C., Harms, M. P., Petersen, S. E., Schlaggar, B. L., Corbetta, M., et al. (2013). Function in the human connectome: Task-fMRI and individual differences in behavior. NeuroImage, 80, 169–189. https://doi.org/10.1016/j.neuroimage.2013.05.033.
Binder, J. R., Gross, W. L., Allendorfer, J. B., Chapin, J., Edwards, J. C., Thomas, J., et al. (2011). Mapping anterior temporal lobe language areas with FMRI: A multi-center normative StudyMulti-center normative study. NeuroImage, 54(2), 1465–1475. https://doi.org/10.1016/j.neuroimage.2010.09.048.
van den Bos, R., Koot, S., & de Visser, L. (2014). A rodent version of the Iowa gambling task: 7 years of progress. Frontiers in Psychology, 5, 1–6. https://doi.org/10.3389/fpsyg.2014.00203.
Caceres, A., Hall, D. L., Zelaya, F. O., Williams, S. C. R., & Mehta, M. A. (2009). Measuring fMRI reliability with the intra-class correlation coefficient Alejandro. NeuroImage, 45(3), 758–768. https://doi.org/10.1016/j.neuroimage.2008.12.035.
Cui, Y., Zhao, S., Chen, Y., Han, J., & Guo, L. (2019). Modeling brain diverse and complex hemodynamic response patterns via deep recurrent autoencoder. IEEE Transactions on Cognitive and Developmental Systems, 12, 1–11. https://doi.org/10.1109/TCDS.2019.2949195.
Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology, 84(6), 3072–3077. https://doi.org/10.1152/jn.2000.84.6.3072.
Derado, G., Bowman, F. D., & Kilts, C. D. (2010). Modeling the spatial and temporal dependence in fMRI data. Biometrics, 66(3), 949–957. https://doi.org/10.1016/j.dcn.2011.01.002.
Dimsdale-Zucker, H. R., & Ranganath, C. (2019). Representational similarity analyses: A practical guide for functional MRI applications. Handbook of Behavioral Neuroscience, 28, 509–525. https://doi.org/10.1016/B978-0-12-812028-6.00027-6.
Drobyshevsky, A., Baumann, S. B., & Schneider, W. (2006). A rapid fMRI task battery for mapping of visual , motor , cognitive , and emotional function. NeuroImage, 31, 732–744. https://doi.org/10.1016/j.neuroimage.2005.12.016.
Elsken, T., Sep, M. L., & Hutter, F. (2018). Neural Architecture Search : A Survey. Retrieved from https://arxiv.org/pdf/1808.05377
Forbes, E. E., Hariri, A. R., Martin, S. L., Silk, J. S., Moyles, D. L., Fisher, P. M., et al. (2009). Altered striatal activation predicting real-world positive affect in adolescent major depressive disorder. American Journal of Psychiatry, 166(1), 64–73. https://doi.org/10.1176/appi.ajp.2008.07081336.
Fox, M. D., Corbetta, M., Snyder, A. Z., Vincent, J. L., & Raichle, M. E. (2006). Spontaneous neuronal activity distinguishes human, Proceedings of the National Academy of Sciences., 103(25), 9381–9386. https://doi.org/10.1073/pnas.0604187103.
Fox, M. D., Snyder, A. Z., Vincent, J. L., Corbetta, M., Van Essen, D. C., & Raichle, M. E. (2005). The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proceedings of the National Academy of Science, 102(27), 9673–9678. https://doi.org/10.1002/poc.610010207.
Friston, K. J. (1997). Transients, metastability, and neuronal dynamics. NeuroImage, 5, 164–171. https://doi.org/10.1006/nimg.1997.0259.
Giesbrecht, B., Woldorff, M. G., Song, A. W., & Mangun, G. R. (2003). Neural mechanisms of top-down control during spatial and feature attention. NeuroImage, 19(3), 496–512. https://doi.org/10.1016/S1053-8119(03)00162-9.
Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Andersson, J. L., et al. (2013). The minimal preprocessing pipelines for the human Connectome project. NeuroImage, 80, 105–124. https://doi.org/10.1016/j.neuroimage.2013.04.127.
Graves, A. (2014). Generating sequences with recurrent neural networks. https://doi.org/10.1145/2661829.2661935.
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). DRAW: A recurrent neural network for image generation. In International Conference on Machine Learning – ICML 2015 (Vol. 2, pp. 1462–1471). https://arxiv.org/abs/1502.04623
Hochreiter, S., & Schmidhuber, J.U. (1997). Long short-term memory. Neural Computation. Retrieved from http://www7.informatik.tu-muenchen.de/~hochreit, http://www.idsia.ch/~juergen
Hu, X., Huang, H., Peng, B., Han, J., Liu, N., Lv, J., Guo, L., Guo, C., & Liu, T. (2018). Latent source mining in FMRI via restricted Boltzmann machine. Human Brain Mapping, 39(6), 2368–2380. https://doi.org/10.1002/hbm.24005.
Huang, H., Hu, X., Zhao, Y., Makkie, M., Dong, Q., Zhao, S., Guo, L., & Liu, T. (2018). Modeling task fMRI data via deep convolutional autoencoder. IEEE Transactions on Medical Imaging, 37(7), 1551–1561. https://doi.org/10.1109/TMI.2017.2715285.
Jiang, X., Li, X., Lv, J., Zhang, T., Zhang, S., Guo, L., & Liu, T. (2015). Sparse representation of HCP grayordinate data reveals novel functional architecture of cerebral cortex. Human Brain Mapping, 36(12), 5301–5319. https://doi.org/10.1002/hbm.23013.
Kennedy, J., Eberhart, R., & Gov, B. (1995). Particle swarm optimization. In International Conference on Neural Networks – ICNN 1995 (pp. 1942–1948). http://ci.nii.ac.jp/naid/10015518367
Kim, Y. K., Park, E., Lee, A., Im, C. H., & Kim, Y. H. (2018). Changes in network connectivity during motor imagery and execution. PLoS One, 13(1), 1–18. https://doi.org/10.1371/journal.pone.0190715.
Kingma, D. P., & Ba, J. L. (2015). ADAM: A method for stochastic optimization. In International Conference on Learning Representations – ICLR 2015 (pp. 1–15). https://doi.org/10.1063/1.4902458.
Koshino, H., Minamoto, T., Yaoi, K., Osaka, M., & Osaka, N. (2014). Coactivation of the default mode network regions and working memory network regions during task preparation. Scientific Reports, 4, 34–39. https://doi.org/10.1038/srep05954.
Kriegeskorte, N. (2015). Deep neural networks: A new framework for modeling biological vision and brain information processing. Annual Review of Vision Science, 1(1), 417–446. https://doi.org/10.1146/annurev-vision-082114-035447.
Kriegeskorte, N., Mur, M., & Bandettini, P. (2008). Representational similarity analysis – Connecting the branches of systems neuroscience. Frontiers in System Neuroscience, 2, 1–26. https://doi.org/10.3389/neuro.06.004.2008.
Li, Q., Dong, Q., Ge, F., Qiang, N., Zhao, Y., Wang, H., … Liu, T. (2019). Simultaneous spatial-temporal decomposition of Connectome-scale brain networks by deep sparse recurrent auto-encoders. In International Conference on Information Processing in Medical Imaging – IPMI 2019 (pp. 1–12). https://link.springer.com/chapter/10.1007/978-3-030-20351-1_45
Lin, C. H., Chiu, Y. C., Cheng, C. M., & Hsieh, J. C. (2008). Brain maps of Iowa gambling task. BMC Neuroscience, 9, 1–15. https://doi.org/10.1186/1471-2202-9-72.
Lipton, Z. C., Berkowitz, J., & Elkan, C. (2015). A Critical Review of Recurrent Neural Networks for Sequence Learning. Retrieved from https://arxiv.org/abs/1506.00019
Logothetis, N. K. (2008). What we can do and what we cannot do with fMRI. Nature, 453, 869–878. https://doi.org/10.1038/nature06976.
Luiz, P. (2014). Understanding brain networks and brain organization. Physics of Life Reviews, 11(3), 400–435. https://doi.org/10.1016/j.plrev.2014.03.005.
Lv, J., Jiang, X., Li, X., Zhu, D., Zhang, S., Zhao, S., Chen, H., Zhang, T., Hu, X., Han, J., Ye, J., Guo, L., & Liu, T. (2015). Holistic atlases of functional networks and interactions reveal reciprocal organizational architecture of cortical function. IEEE Transactions on Biomedical Engineering, 62(4), 1120–1131. https://doi.org/10.1109/TBME.2014.2369495.
Pandarinath, C., O’Shea, D. J., Collins, J., Jozefowicz, R., Stavisky, S. D., Kao, J. C., Trautmann, E. M., Kaufman, M. T., Ryu, S. I., Hochberg, L. R., Henderson, J. M., Shenoy, K. V., Abbott, L. F., & Sussillo, D. (2018). Inferring single-trial neural population dynamics using sequential auto-encoders. Nature, 15, 805–815. https://doi.org/10.1038/s41592-018-0109-9.
Pei, W., & Tax, D. M. J. (2018). Unsupervised Learning of Sequence Representations by Autoencoders. Retrieved from https://arxiv.org/abs/1804.00946
Plis, S. M., Hjelm, D. R., Slakhutdinov, R., Allen, E. A., Bockholt, H. J., Long, J. D., et al. (2014). Deep learning for neuroimaging: A validation study. Frontiers in Neuroscience, 8, 1–11. https://doi.org/10.3389/fnins.2014.00229.
Prilipko, O., Huynh, N., Schwartz, S., Tantrakul, V., Kim, J. H., Peralta, A. R., Kushida, C., Paiva, T., & Guilleminault, C. (2011). Task positive and default mode networks during a parametric working memory task in obstructive sleep apnea patients and healthy controls. Sleep, 34(3), 293–301. https://doi.org/10.1093/sleep/34.3.293.
Qian, P., Qiu, X., & Huang, X. (2016). Bridging LSTM architecture and the neural dynamics during reading. In International Joint Conference on Artificial Intelligence – IJCAI 2016 (pp. 1953–1959).
Raichle, M. E., MacLeod, A. M., Snyder, A. Z., Powers, W. J., Gusnard, D. A., & Shulman, G. L. (2001). A default mode of brain function. Proceedings of the National Academy of Sciences, 98(2), 676–682. https://doi.org/10.1002/jso.20675.
Sak, H., Senior, A., & Beaufays, F. (2014). Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition. Retrieved from arXiv:1402.1128.
Salmon, E., Linden, M. Van der, Collette, F., Delfiore, G., Maquet, P., Degueldre, C., … Franck, G. (1996). Regional Brain Activity During Working Memory Tasks. Brain, 119(5), 1617–1625. Retrieved from https://doi.org/10.1093/brain/119.5.1617
Shen, Y., Mayhew, S. D., Kourtzi, Z., & Tiňo, P. (2014). Spatial-temporal modelling of fMRI data through spatially regularized mixture of hidden process models. NeuroImage, 84, 657–671. https://doi.org/10.1016/j.neuroimage.2013.09.003.
Shimony, J. S., Zhang, D., James, M. J., Fox, M. D., Roy, A., Eric, C., & L. (2009). Resting state spontaneous fluctuations in brain activity: A new paradigm for Presurgical planning using fMRI. Academic Radiology, 16(5), 578–583. https://doi.org/10.1016/j.acra.2009.02.001.
Smith, S. M., Miller, K. L., Moeller, S., Xu, J., Auerbach, E. J., Woolrich, M. W., Beckmann, C. F., Jenkinson, M., Andersson, J., Glasser, M. F., van Essen, D. C., Feinberg, D. A., Yacoub, E. S., & Ugurbil, K. (2012). Temporally-independent functional modes of spontaneous brain activity. Proceedings of the National Academy of Sciences, 109(8), 3131–3136. https://doi.org/10.1073/pnas.1121329109.
Srivastava, N. (2014). Unsupervised Learning of Video Representations using LSTMs. Retrieved from https://arxiv.org/abs/1502.04681
Tricomi, E. M., Delgado, M. R., & Fiez, J. A. (2004). Modulation of caudate activity by action contingency. Neuron, 41, 281–292. https://doi.org/10.3873/j.issn.1000-1328.2009.05.018.
Wang, H., Zhao, S., Dong, Q., Cui, Y., Chen, Y., Han, J., Xie, L., & Liu, T. (2018). Recognizing brain states using deep sparse recurrent neural network. IEEE Transactions on Medical Imaging, 38(4), 1058–1068. https://doi.org/10.1109/TMI.2018.2877576.
Woodman, G. F., Luck, S. J., & Schall, J. D. (2007). The role of working memory representations in the control of attention. Cerebral Cortex, 17, 118–124. https://doi.org/10.1093/cercor/bhm065.
Woolrich, M. W., Jenkinson, M., Brady, J. M., & Smith, S. M. (2004). Fully Bayesian Spatio-temporal modeling of FMRI data. IEEE Transactions on Medical Imaging, 23(2), 213–231. https://doi.org/10.1109/TMI.2003.823065.
Wu, X., Li, Q., Yu, X., Chen, K., Fleisher, A. S., Guo, X., Zhang, J., Reiman, E. M., Yao, L., & Li, R. (2016). A triple network connectivity study of large-scale brain Systems in Cognitively Normal APOE4 carriers. Frontiers in Aging Neuroscience, 8, 1–9. https://doi.org/10.3389/fnagi.2016.00231.
Xu, K., Courville, A., Zemel, R. S., & Bengio, Y. (2015). Show , Attend and Tell : Neural Image Caption Generation with Visual Attention. Retrieved from https://arxiv.org/abs/1502.03044
Xue, W., Brahm, G., Pandey, S., Leung, S., & Li, S. (2018). Full left ventricle quantification via deep multitask relationships learning. Medical Image Analysis, 43, 54–65. https://doi.org/10.1016/j.media.2017.09.005.
Zhang, W., Lv, J., Li, X., Zhu, D., Jiang, X., Zhang, S., Zhao, Y., Guo, L., Ye, J., Hu, D., & Liu, T. (2018). Experimental comparisons of sparse dictionary learning and independent component analysis for brain network inference from fMRI data. IEEE Transactions on Biomedical Engineering, 66(1), 289–299. https://doi.org/10.1109/TBME.2018.2831186.
Zhao, Y., Dong, Q., Zhang, S., Zhang, W., Chen, H., Jiang, X., Guo, L., Hu, X., Han, J., & Liu, T. (2018). Automatic recognition of fMRI-derived functional networks using 3-D convolutional neural networks. IEEE Transactions on Biomedical Engineering, 65(9), 1975–1984. https://doi.org/10.1109/TBME.2017.2715281.
Zhao, Y., Li, X., Huang, H., Zhang, W., Zhao, S., Makkie, M., Zhang, M., Li, Q., & Liu, T. (2019). 4D modeling of fMRI data via Spatio-temporal convolutional neural networks (ST-CNN). IEEE Transactions on Cognitive and Developmental Systems, 12, 1–11. https://doi.org/10.1109/TCDS.2019.2916916.
Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B, 67(part 2), 301–320. https://doi.org/10.1037/h0100860.
Acknowledgments
We would like to thank the Human Connectome Project (HCP) for sharing the datasets used in this work.
Funding
This work was supported by General Program of National Natural Science Foundation of China (Grant No. 61876021), Beijing Municipal Natural Science Foundation (Grant No. 4212037), the program of China Scholarships Council (No. 201806040083), National Institutes of Health (DA033393, AG042599) and National Science Foundation (IIS-1149260, CBET-1302089, BCS-1439051 and DBI-1564736).
Author information
Authors and Affiliations
Contributions
Author contributions included conception and study design (XW and TL), data acquisition (QD), statistical analysis (QL and QD), interpretation of results (QL and TL), drafting the manuscript work or revising it critically for important intellectual content (QL, QD, FG, NQ and TL) and approval of final version to be published and agreement to be accountable for the integrity and accuracy of all aspects of the work (All authors).
Corresponding author
Ethics declarations
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Conflict of interest
The authors declare that they have no conflict of interest. Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Qing Li and Qinglin Dong are Co-first authors; Xia Wu and Tianming Liu are Co-corresponding authors
Supplementary Information
ESM 1
(DOCX 17997 kb)
Rights and permissions
About this article
Cite this article
Li, Q., Dong, Q., Ge, F. et al. Simultaneous spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder. Brain Imaging and Behavior 15, 2646–2660 (2021). https://doi.org/10.1007/s11682-021-00469-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11682-021-00469-w