Abstract
Deep networks have been shown to achieve impressive accuracy for some medical image analysis tasks where large datasets and annotations are available. However, tasks involving learning over new sets of classes arriving over extended time is a different and difficult challenge due to the tendency of reduction in performance over old classes while adapting to new ones. Controlling such a ‘forgetting’ is vital for deployed algorithms to evolve with new arrivals of data incrementally. Usually, incremental learning approaches rely on expert knowledge in the form of manual annotations or active feedback. In this paper, we explore the role that other forms of expert knowledge might play in making deep networks in medical image analysis immune to forgetting over extended time. We introduce a novel framework for mitigation of this forgetting effect in deep networks considering the case of combining ultrasound video with point-of-gaze tracked for expert sonographers during model training. This is used along with a novel weighted distillation strategy to reduce the propagation of effects due to class imbalance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cai, Y., Sharma, H., Chatelain, P., Noble, J.A.: Sonoeyenet: Standardized fetal ultrasound plane detection informed by eye tracking. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1475–1478 (2018)
Chen, H., Ni, D., Yang, X., Li, S., Heng, P.A.: Fetal abdominal standard plane localization through representation learning with knowledge transfer. In: Wu, G., Zhang, D., Zhou, L. (eds.) MLMI 2014. LNCS, vol. 8679, pp. 125–132. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10581-9_16
Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: Proceedings of the 23rd International Conference on Machine Learning. p. 233–240. ICML 2006, Association for Computing Machinery, New York, NY, USA (2006). https://doi.org/10.1145/1143844.1143874
Dhar, P., Singh, R.V., Peng, K.C., Wu, Z., Chellappa, R.: Learning without memorizing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5138–5146 (2019)
Díaz-Rodríguez, N., Lomonaco, V., Filliat, D., Maltoni, D.: Don’t forget, there is more than forgetting: new metrics for continual learning. arXiv:1810.13166 (2018)
Droste, R., et al.: Ultrasound image representation learning by modeling sonographer visual attention. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 592–604. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20351-1_46
Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv:1312.6211 (2013)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS 2014 Deep Learning Workshop (2014)
Hou, S., Pan, X., Change Loy, C., Wang, Z., Lin, D.: Lifelong learning via progressive distillation and retrospection. In: ECCV (2018)
Kemker, R., Kanan, C.: Fearnet: Brain-inspired model for incremental learning. arXiv:1711.10563 (2017)
Kim, H.-E., Kim, S., Lee, J.: Keep and learn: continual learning by constraining the latent space for knowledge preservation in neural networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 520–528. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_59
Kingma, D.P., Adam, J.B.: A method for stochastic optimization. arXiv:1412.6980
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. In: Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521–3526 (2017)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)
Li, Z., Zhong, C., Wang, R., Zheng, W.-S.: Continual learning of new diseases with dual distillation and ensemble strategy. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 169–178. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_17
Omar, H., Patra, A., Domingos, J., Upton, R., Leeson, P., Noble, J.: Myocardial wall motion assessment in stress echocardiography by quantification of principal strain bulls eye maps: P299. Eur. Heart J. Cardiovascular Imaging 18 (2017)
Omar, H.A., Domingos, J.S., Patra, A., Leeson, P., Noble, J.A.: Improving visual detection of wall motion abnormality with echocardiographic image enhancing methods. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1128–1131. IEEE (2018)
Omar, H.A., Domingos, J.S., Patra, A., Upton, R., Leeson, P., Noble, J.A.: Quantification of cardiac bull’s-eye map based on principal strain analysis for myocardial wall motion assessment in stress echocardiography. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1195–1198. IEEE (2018)
Omar, H.A., Patra, A., Domingos, J.S., Leeson, P., Noblel, A.J.: Automated myocardial wall motion classification using handcrafted features vs a deep CNN-based mapping. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3140–3143. IEEE (2018)
Ozdemir, F., Fuernstahl, P., Goksel, O.: Learn the new, keep the old: extending pretrained models with new anatomy and images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 361–369. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_42
Patra, A., Huang, W., Noble, J.A.: Learning spatio-temporal aggregation for fetal heart analysis in ultrasound video. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS-2017. LNCS, vol. 10553, pp. 276–284. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_32
Patra, A., et al.: Sequential anatomy localization in fetal echocardiography videos. arXiv preprint arXiv:1810.11868 (2018)
Patra, A., et al.: Efficient ultrasound image analysis models with sonographer gaze assisted distillation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 394–402. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_43
Patra, A., Chakraborti, T.: Learn more, forget less: Cues from human brain. In: Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020
Patra, A., Noble, J.A.: Incremental learning of fetal heart anatomies using interpretable saliency maps. In: Zheng, Y., Williams, B.M., Chen, K. (eds.) MIUA 2019. CCIS, vol. 1065, pp. 129–141. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-39343-4_11
Patra, A., Noble, J.A.: Multi-anatomy localization in fetal echocardiography videos. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1761–1764. IEEE (2019)
Patra, A., Noble, J.A.: Hierarchical class incremental learning of anatomical structures in fetal echocardiography videos. IEEE J. Biomed. Health Informatics (2020)
PULSE: Perception ultrasound by learning sonographic experience (2018). www.eng.ox.ac.uk/pulse
Ravishankar, H., et al.: Understanding the mechanisms of deep transfer learning for medical images. In: Deep Learning and Data Labeling for Medical Applications, pp. 188–196. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46976-8_20
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: ICARL: Incremental classifier and representation learning. In: Proceedings of the IEEE CVPR, pp. 2001–2010 (2017)
Rusu, A.A., et al.: Progressive neural networks. arXiv:1606.04671 (2016)
Zhang, J., Wang, Y.: Continually modeling Alzheimer’s disease progression via deep multi-order preserving weight consolidation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 850–859. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_94
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Patra, A. et al. (2021). Multimodal Continual Learning with Sonographer Eye-Tracking in Fetal Ultrasound. In: Noble, J.A., Aylward, S., Grimwood, A., Min, Z., Lee, SL., Hu, Y. (eds) Simplifying Medical Ultrasound. ASMUS 2021. Lecture Notes in Computer Science(), vol 12967. Springer, Cham. https://doi.org/10.1007/978-3-030-87583-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-87583-1_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87582-4
Online ISBN: 978-3-030-87583-1
eBook Packages: Computer ScienceComputer Science (R0)
