Abstract
Cardiovascular disease (CVD) is the number one killer in the USA, yet it is largely preventable (World Health Organization 2011). To prevent CVD, carotid intima-media thickness (CIMT) imaging, a noninvasive ultrasonography method, has proven to be clinically valuable in identifying at-risk persons before adverse events. Researchers are developing systems to automate CIMT video interpretation based on deep learning, but such efforts are impeded by the lack of large annotated CIMT video datasets. CIMT video annotation is not only tedious, laborious, and time consuming, but also demanding of costly, specialty-oriented knowledge and skills, which are not easily accessible. To dramatically reduce the cost of CIMT video annotation, this paper makes three main contributions. Our first contribution is a new concept, called Annotation Unit (AU), which simplifies the entire CIMT video annotation process down to six simple mouse clicks. Our second contribution is a new algorithm, called AFT (active fine-tuning), which naturally integrates active learning and transfer learning (fine-tuning) into a single framework. AFT starts directly with a pre-trained convolutional neural network (CNN), focuses on selecting the most informative and representative AU s from the unannotated pool for annotation, and then fine-tunes the CNN by incorporating newly annotated AU s in each iteration to enhance the CNN’s performance gradually. Our third contribution is a systematic evaluation, which shows that, in comparison with the state-of-the-art method (Tajbakhsh et al., IEEE Trans Med Imaging 35(5):1299–1312, 2016), our method can cut the annotation cost by >81% relative to their training from scratch and >50% relative to their random selection. This performance is attributed to the several advantages derived from the advanced active, continuous learning capability of our AFT method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
World Health Organization. Global atlas on cardiovascular disease prevention and control. Available at http://www.who.int/cardiovascular_diseases/publications (September 19, 2011)
Al Rahhal Ms, Bazi Y, AlHichri H, Alajlan N, Melgani F, Yager R: Deep learning approach for active classification of electrocardiogram signals. Inf Sci 345: 340–354, 2016
Carneiro G, Nascimento J, Bradley A: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: (Navab N, Hornegger J, Wells WM, Frangi AF, Eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science, vol. 9351, pp. 652–660. Springer International Publishing, 2015. https://doi.org/10.1007/978-3-319-24574-4_78
Chen H, Ni D, Qin J, Li S, Yang X, Wang T, Heng PA: Standard plane localization in fetal ultrasound via domain transferred deep neural networks. IEEE J Biomed Health Inform 19 (5): 1627–1636, 2015
Delsanto S, Molinari F, Giustetto P, Liboni W, Badalamenti S, Suri JS: Characterization of a completely user-independent algorithm for carotid artery segmentation in 2-d ultrasound images. IEEE Trans Instrum Meas 56 (4): 1265–1274, 2007
Deng J, Dong W, Socher R, Li LJ, Li K: Fei-fei, L.: ImageNet: A Large-Scale Hierarchical Image Database CVPR09, 2009
Gao M, Bagci U, Lu L, Wu A, Buty M, Shin HC, Roth H, Papadakis GZ, Depeursinge A, Summers RM, et al: Holistic classification of ct attenuation patterns for interstitial lung diseases via deep convolutional neural networks.. In: The 1st workshop on deep learning in medical image analysis, international conference on medical image computing and computer assisted intervention, at MICCAI-DLMIA’15, 2015
Gepner AD, Young R, Delaney JA, Tattersall MC, Blaha MJ, Post WS, Gottesman RF, Kronmal R, Budoff MJ, Burke GL, et al: A comparison of coronary artery calcium presence, carotid plaque presence, and carotid intima-media thickness for cardiovascular disease prediction in the multi-ethnic study of atherosclerosis (mesa). Circulation Cardiovascular Imaging 8 (1): e002262, 2015
Greenspan H, van Ginneken B, Summers RM: Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35 (5): 1153–1159, 2016
Guyon I, Cawley G, Dror G, Lemaire V, Statnikov A (2011) JMLR Workshop and conference proceedings (volume 16): Active learning challenge microtome publishing
He K, Zhang X, Ren S, Sun J: Deep residual learning for image recognition.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016
Hoo-Chang S, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM: Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging 35 (5): 1285, 2016
Hurst RT, Burke RF, Wissner E, Roberts A, Kendall CB, Lester SJ, Somers V, Goldman ME, Wu Q, Khandheria B: Incidence of subclinical atherosclerosis as a marker of cardiovascular risk in retired professional football players. Am J Cardiol 105 (8): 1107–1111, 2010
Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. arXiv:http://arXiv.org/abs/1408.5093
Kass M, Witkin A, Terzopoulos D: Snakes: Active contour models. Int J Comput Vis 1 (4): 321–331, 1988
Krizhevsky A, Sutskever I, Hinton GE: Imagenet classification with deep convolutional neural networks.. In: Advances in neural information processing systems, pp 1097–1105, 2012
Li J: Active learning for hyperspectral image classification with a stacked autoencoders based neural network.. In: 2016 IEEE International conference on image processing (ICIP), pp 1062–1065, 2016. https://doi.org/10.1109/ICIP.2016.7532520
Liang J, McInerney T, Terzopoulos D: United snakes. Med Image Anal 10 (2): 215–233, 2006
Liang Q, Wendelhag I, Wikstrand J, Gustavsson T: A multiscale dynamic programming procedure for boundary detection in ultrasonic artery images. IEEE Trans Med Imaging 19 (2): 127–142, 2000
Loizou CP, Pattichis CS, Pantziaris M, Nicolaides A: An integrated system for the segmentation of atherosclerotic carotid plaque. IEEE Trans Inf Technol Biomed 11 (6): 661–667, 2007
Long J, Shelhamer E, Darrell T: Fully convolutional networks for semantic segmentation.. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440, 2015
Lorenz MW, Markus HS, Bots ML, Rosvall M, Sitzer M: Prediction of clinical cardiovascular events with carotid intima-media thickness. Circulation 115 (4): 459–467, 2007
Margeta J, Criminisi A, Cabrera Lozoya R, Lee DC, Ayache N: Fine-tuned convolutional neural nets for cardiac mri acquisition plane recognition. Comput Methods Biomech Biomed Eng: Imaging Visual 5 (5): 339–349, 2017
Menchón-Lara RM, Bastida-Jumilla MC, González-López A, Sancho-Gómez JL: Automatic evaluation of carotid intima-media thickness in ultrasounds using machine learning.. In: Natural and artificial computation in engineering and medical applications, pp 241–249. Springer, 2013
Menchón-Lara RM, Sancho-Gómez JL: Fully automatic segmentation of ultrasound common carotid artery images based on machine learning. Neurocomputing 151: 161–167, 2015
Schlegl T, Ofner J, Langs G: Unsupervised pre-training across image domains improves lung tissue classification.. In: Medical computer vision: Algorithms for big data, pp 82–93. Springer, 2014
Shin HC, Lu L, Kim L, Seff A, Yao J, Summers RM: Interleaved text/image deep mining on a very large-scale radiology database.. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1090–1099, 2015
Shin J, Tajbakhsh N, Todd Hurst R, Kendall CB, Liang J: Automating carotid intima-media thickness video interpretation with convolutional neural networks.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2016
Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition.. In: ICLR, 2015
Stark F, Hazırbas C, Triebel R., Cremers D.: Captcha recognition with active deep learning.. In: Workshop new challenges in neural computation 2015, p 94. Citeseer, 2015
Stein JH, Korcarz CE, Hurst RT, Lonn E, Kendall CB, Mohler ER, Najjar SS, Rembold CM, Post WS: Use of carotid ultrasound to identify subclinical vascular disease and evaluate cardiovascular disease risk: a consensus statement from the american society of echocardiography carotid intima-media thickness task force endorsed by the society for vascular medicine. J Am Soc Echocardiogr 21 (2): 93–111, 2008
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A: Going deeper with convolutions.. In: The IEEE conference on computer vision and pattern recognition (CVPR), 2015
Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J: Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging 35 (5): 1299–1312, 2016
Tajbakhsh N, Shin JY, Hurst RT, Kendall CB, Liang J: Automatic interpretation of carotid intima–media thickness videos using convolutional neural networks.. In: Deep learning for medical image analysis, pp 105–131. Elsevier, 2017
Wang D, Shang Y: A new active labeling method for deep learning.. In: 2014 International joint conference on neural networks (IJCNN), pp 112–119, 2014. https://doi.org/10.1109/IJCNN.2014.6889457
Yang L, Zhang Y, Chen J, Zhang S, Chen DZ: Suggestive annotation: a deep active learning framework for biomedical image segmentation.. In: International conference on medical image computing and computer-assisted intervention, pp 399–407. Springer, 2017
Zhou Z, Shin J, Zhang L, Gurudu S, Gotway M, Liang J: Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally.. In: IEEE Conference on computer vision and pattern recognition, hawaii, pp 7340–7349, 2017
Acknowledgements
This research has been supported partially by NIH under Award Number R01HL128785 and partially by ASU and Mayo Clinic through the Discovery Translation Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhou, Z., Shin, J., Feng, R. et al. Integrating Active Learning and Transfer Learning for Carotid Intima-Media Thickness Video Interpretation. J Digit Imaging 32, 290–299 (2019). https://doi.org/10.1007/s10278-018-0143-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10278-018-0143-2