An ultrasound standard plane detection model of fetal head based on multi-task learning and hybrid knowledge graph

Highlights

• A multi-task learning model that jointly optimizes key structure recognition and plane classification tasks is proposed for fetal head ultrasound standard plane detection.

• Prior clinical knowledge is integrated into our method to reduce the missed and error detection rates.

• Unlike most “end-to-end” automatic detection modes, our method provides explanations for its decisions.

• Compared with several competitive benchmarks, our method displays a promising performance.

Abstract

Prenatal ultrasound examination is a powerful tool to prevent birth defects and assess fetal health. Obtaining ultrasound standard planes is a prerequisite for prenatal ultrasound diagnosis. However, ultrasound standard plane detection depends heavily on the sonographer’s sufficient clinical experience and solid knowledge of fetal anatomy. In this study, to lighten the workload of the sonographer and promote the accuracy, efficiency, and interpretability of ultrasound standard plane detection, we propose an ultrasound standard plane detection (USPD) model based on multi-task learning and a hybrid knowledge graph. We first design a multi-task learning strategy to learn the shared features of fetal ultrasound images through convolutional blocks. Then, we optimize the generalization performance by extending the shared features into the task-specific output streams. In addition, USPD integrates clinical prior knowledge graphs to reduce the error rate and missed detection rate. The USPD model can recognize the key anatomical structures of fetal heads and analyze the types of ultrasound planes. Furthermore, unlike most “end-to-end” automatic detection models, the USPD model not only outputs the prediction results but also provides consistent interpretation for professional sonographers, thereby increasing the interpretability of the model without the sonographer’s intervention. We conduct extensive experiments on a fetal head ultrasound image dataset to assess the proposed USPD model via comparison with competitive methods. Experimental results illustrate that the proposed USPD model outperforms the competitive methods with regard to accuracy and performance, and it can meet the clinical requirements in practical application.

Introduction

Benefiting from low cost, safety, and convenience, ultrasound is widely used in routine prenatal examinations. Prenatal ultrasound examination is a powerful tool to prevent birth defects and assess the health of the fetus [1]. In the prenatal ultrasound examination, the fetus is scanned by an ultrasound instrument to obtain ultrasound images and video streams. Sonographers manually select the standard planes from the ultrasound images or video stream [2] because the ultrasound standard planes contain key anatomical structures for biometric measurement or disease diagnosis. Thus, ultrasound standard plane detection is an important step for subsequent diagnosis. At present, this task mainly depends on the experience of the sonographer, and for inexperienced sonographers, it is a daunting task. In addition, because of the diversity of ultrasound standard planes and the presence of many highly similar anatomical structures, ultrasound standard plane detection also takes a great deal of time in prenatal examinations. Therefore, it is critical to provide an automatic and accurate detection method for the fetal head ultrasound standard planes.

Morphological changes of the key anatomical structures in the fetal head ultrasound standard planes are of great clinical significance for assessing fetal central nervous system development. Fetal central nervous system malformations are some of the most common congenital malformations. According to a long-term surveillance study in Europe and the United States, the incidence of neurological malformations is as high as 3%–4%, and even higher in spontaneous abortion cases. Certain fetal head abnormalities found by ultrasound in the second trimester may be signs of spina bifida and Arnold-Chiari malformation. For example, the abnormal structure of the cerebellum (called the banana sign) is related to the disappearance of key anatomical structures (e.g., posterior fossa, PCF), and is one of the serious fetal congenital malformations. With the advent of high-resolution ultrasound equipment, we can observe the development of the fetal nervous system at an early stage. Four types of fetal head ultrasound planes are systematically studied in the prenatal examination, namely the ultrasound planes of the thalamus, lateral ventricle, cerebellum, and parietal lobe, so these are the key points of our research. Fig. 1 shows the corresponding key anatomical structures of these planes (i.e., brain falx (CF), sepum septi pellucidi (CSP), thalamus (T), lateral sulcus (LS), choroid plexus (CP), posterior horn of lateral ventricle (PH), cerebellar hemisphere (CH), PCF, midline of brain (BM), and parieto-occipital sulcus (PS) ). In our work, we aim to alleviate the daily burdens of sonographers and assist with prenatal examinations in healthcare facilities with less experienced sonographers by improving the efficiency and accuracy of ultrasound interpretation.

At present, the challenges of ultrasound standard plane detection can be summarized in three main points [3], [4]. (1) Due to the different scanning technology of the sonographer and position of the fetus, the quality of the ultrasound image is easily influenced by rotation scale and shadow noise, as shown in Fig. 2. This challenge results from the limitations of ultrasonic imaging technology, which is not the concern of our work. (2) Due to the high intra-class and low inter-class variations of ultrasound images, sometimes ultrasound non-standard images are very similar to ultrasound standard images, so they are difficult to distinguish, as shown in Fig. 6. (3) Due to many factors such as data confidentiality, sample collection, and annotation workload, it is very hard, or sometimes impossible, to collect large-scale datasets with annotation information in the field of medical image analysis. Because a pure data-driven model may not meet the constraints of the definition of clinical prior knowledge, when dealing with under-trained ultrasound data, the use of pure data-driven automatic detection methods still has its limitations. (4) Most “end-to-end” automatic detection modes only output the predicted results without any explanations, so the interpretability is weak. Doctors generally find it hard to trust decisions made by an uninterpretable black-box model. There has been a call for explainable artificial intelligence approaches to better understand the black box, especially in high-stakes decision-making areas such as medical image analysis.

In recent studies, well-designed neural networks [5], [6] have been designed to effectively extract features for classification and recognition in response to the above challenges. A classification model [7] that achieves quite impressive performance has been developed, but the interpretability of the model is weak. The working process of a key structure detection model [8] is consistent with the cognition of professional sonographers, but there remain some errors due to the misdetection of structures (such as BM and CSP) with high scores. Automatic detection tasks for the fetal head ultrasound plane are still quite complicated.

In order to improve the reliability and interpretability of end-to-end automatic detection, and avoid misdetection of structures that affect the standard plane detection, we propose an end-to-end ultrasound standard plane detection (USPD) model of the fetal head based on multi-task learning and hybrid knowledge graphs. The USPD model divides the complicated process into two related tasks: key anatomical structure recognition and plane classification, which are executed by different modules. The proposed USPD model includes two related modules: the key anatomical structure recognition (KASR) module and the ultrasound plane classification (UPC) module. Our model can effectively learn the shared features through the convolution layers and then optimize the two related modules synchronously. The working process of the USPD model is consistent with the cognition of professional sonographers (the basic principle professional sonographers use to detect ultrasound standard planes is that all necessary anatomical structures should be complete, obvious, and well-defined). Doctors can easily understand the reasons for the difference when the detected results are inconsistent with the model, which can greatly improve the reliability and interpretability of automatic detection. In addition, we further integrate prior clinical knowledge to reduce the missed detection and error detection rates, and also improve the detection accuracy of relatively smaller and morphologically variable anatomical structures. To sum up, the innovative points of this paper are as follows:

• We first propose a multi-task learning model called the USPD model for fetal head examination, which integrates plane classification and key structure detection. The model has strong robustness to interference from diverse images and similar structures, and the detection speed meets the clinical requirements. Unlike most “end-to-end” automatic detection modes, the USPD model provides explanations for its decisions to better inform the decision-making of the professional sonographer.

• We directly integrate hybrid knowledge in an adaptive manner to improve the accuracy of key anatomical structure recognition by enhancing the representation of intermediate features. Our framework performs better semantic reasoning on key anatomical structure recognition and achieves semantic consistency with ultrasound plane classification, which is quite innovative compared with other advanced methods.

• We conduct experiments to evaluate the application of the USPD model to fetal head ultrasound images. The detection and classification tasks of the USPD model outperform existing methods in terms of performance, scalability, and interpretability.

The rest of this paper is organized as follows. Section 2 summarizes related work. Section 3 describes the proposed model in detail. Section 4 introduces the experimental results and analysis. Section 5 discusses the limitations and advantages of the proposed model. Section 6 summarizes the article.