1.

Introduction

It is estimated that there are 161,360 new cases of prostate cancer and 26,730 deaths from prostate cancer in the USA in 2017.¹ Magnetic resonance image (MRI) becomes a routine modality for prostate examination.^2–5 Accurate segmentation of the prostate and lesions from MRI has many applications in prostate cancer diagnosis and treatment. However, manual segmentation can be time consuming and subject to inter- and intrareader variations. In this study, a deep learning method is proposed to automatically segment the prostate on T2-weighted (T2W) MRI.

Recently, deep learning has dramatically changed the landscape of computer vision. The initial work was proposed for image classification⁶ using a big data set of natural images called ImageNet.⁷ Currently, most deep learning models are for image-level classification.^6,8 To obtain a pixel-wise segmentation, some researchers^9–15 proposed a patch-wise segmentation method, which extracts small patches (e.g., $32\times 32$) from images and then trains a patch-wise convolutional neural network (CNN) model. In the training stage, each patch extracted from training image is assigned a label, which can be directly fed into the image-level classification framework to learn a CNN model. If the center pixel of the patch belongs to the foreground, the label of this patch is 1. In the testing stage, the patches are extracted from the testing image first. The learned CNN model can be used to infer the label of testing patches. The label is assigned to the center pixel of the testing patches. Tajbakhsh et al.⁹ proposed a deep learning method for intima-media boundary segmentation from ultrasound image. They formulated the boundary segmentation task as a pixel-level classification problem. To train the CNNs, 200,000 training small patches were extracted from the images and AlexNet was used in their study. Zhang et al.¹⁰ proposed to use deep CNNs for segmenting isointense stage brain tissues using multimodality MRI. For each subject, they generated $>\mathrm{10,000}$ patches centered at each pixel from T1, T2, and fractional anisotropy images. The patches were considered as training and testing samples in their study. Ciresan et al.¹² presented a deep neural network method to segment neuronal membranes in electron microscopy images. They used a special type of deep artificial neural network as a pixel classifier. The label of each pixel was predicted from raw pixel values in a square patch centered on it. Pereira et al.¹³ proposed a CNN-based method for segmentation of brain tumors in MRI. To train the CNN for low grade gliomas (LGG) and high grade gliomas (HGG), they extracted around 450,000 and 335,000 small patches, respectively. Kooi et al.¹⁴ proposed a CNN method for breast lesion detection in mammography. 39,872 patches were extracted for training the CNNs, which the size of patch is $250\times 250$. Milletari et al.¹⁵ proposed a patch-wise deep learning method for segmentation of deep brain regions in MRI and ultrasound. They collected patches from both foreground and background and train a CNN.

The performance of patch-wise CNN models can be affected by the patch size. A large patch size reduces the localization accuracy and a small patch size only can see a small context. In addition, when the number of patches is large (each pixel/voxel assigned a patch), there is a high redundant computation that needs to be performed for neighboring patches. To solve these problems, Long et al.¹⁶ proposed an end-to-end pixel-wise, natural image segmentation method based on Caffe,¹⁷ a deep learning software. They modified an existing classification CNN to a fully convolutional network (FCN) for object segmentation. A coarse label map can be obtained from the network by classifying every local region, and then performing a simple deconvolution based on bilinear interpolation for pixel-wise segmentation. This method does not make use of postprocessing or posthoc refinement by random fields.^18,19

Because the FCN algorithm achieves a good performance, researchers proposed various FCN-based methods for medical image segmentation.^20–23 Ronneberger et al.²⁰ took the idea of the FCN one step further and presented an framework called U-Net, which is a regular CNN followed by an up-sampling operation, where up-convolutions are used to increase the size of feature maps. Çiçek et al.²⁴ extended the U-Net to obtain three-dimensional (3-D) segmentation. Yu et al.²² proposed a volumetric ConvNet to segment prostate from MRI. They extended a two-dimensional (2-D) FCN into a volumetric fully ConvNet (3D-FCN) to enable volume-to-volume segmentation prediction. Milletari et al.²³ proposed a 3-D variant of U-Net architecture called V-Net for prostate segmentation. In the deep neural networks with several convolutional layers, it is important to provide a good initialization, which can improve segmentation performance. In addition, the efficiency of the deep learning architecture is nontrivial. To solve these problems, a pretrained model trained on a large number of natural images is used to initialize the parameters of the proposed network, which can also accelerate the model converges to minimum.

In this paper, we propose the use of FCN for the segmentation of the prostate on MRI. The contribution includes the modification of the FCN and its validation on prostate MRI. The preliminary study of the work was presented at the 2017 SPIE Medical Imaging Conference²⁵ and the authors were requested to submit a full article to the Journal of Medical Imaging (JMI). As compared to the SPIE conference paper, this JMI article made major improvements: (1) we extended the method from 20 patients to 140 patients, (2) we added more literature review in Sec. 1, (3) we performed more segmentation experiments with three databases, and (4) we added significantly more results in this article. The rest of this paper is organized as follows: in Sec. 2, we present the details of the proposed algorithm. Section 3 evaluates the performance of the proposed algorithm and discusses the results of our experiments. We conclude the paper in Sec. 4.

2.

Method

2.1.

Convolutional Neural Network

In practice, few people train an entire CNN from scratch, since it is difficult to collect a dataset of sufficient size, especially for medical images. In contrast, to learn from scratch, it is common to use a pretrained CNN on a large data set as an initialization and then retrain an own classifier on top of the CNN for the data set in hand, named as fine-tuning. Tajbakhsh et al.⁹ showed that knowledge transfer from natural images to medical image is possible based on CNN. Therefore, we fine-tune Long’s FCN model¹⁶ trained on PASCAL VOC data set²⁶ and retrain it based on our medical image for the prostate segmentation, named as PSNet. Figure 1 shows the proposed deep learning method.

Fig. 1

The framework of the proposed deep CNN. There are seven hidden layers in the CNN. The number shows the feature or channel dimension of each hidden layer.

The early layers of PSNet learn low-level generic features that are applicable to most tasks. The late layers learn high-level specific features that are applicable to the application at hand.^27,28 Therefore, we only fine-tune the last three layers of the FCN in this work. Figure 2 shows the filters and the outputs of the first hidden layer. The first layer learns simple features, such as edge, junction, and corner. Figure 3 shows the outputs of late hidden layers. The figure shows that the late layers learn high-level features, which yield highly abstracted output images.

Fig. 2

Filters and outputs of the first hidden layer of the PSNet. (a) Filters of the first hidden layer ($3\times 3$ filters) and (b) outputs of the first hidden layer (first 36 only).

Fig. 3

Output of (a) the fourth layer and (b) the fifth layer of the PSNet.

The proposed PSNet predicts the probability of each voxel belonging to the prostate or background. For each prostate MRI, the prostate only has a small region compared with the background, which means that the number of the foreground voxels is less than that of the background voxels. This unbalance between the prostate and background regions will cause the learning algorithm to get trapped in local minima with the use of the softmax loss function in Caffe.¹⁷ Therefore, the prostate is often missed and the prediction tends to classify prostate voxels as the background. In this work, we use a weighted cross entropy loss function.²⁹ The loss function is formulated as follows:

$$L=\frac{-1}{n}\sum _{i=1}^N{W}_i^C[P_i\log {\widehat{P}}_i+(1-P_i)\log (1-{\widehat{P}}_i)],$$

where $P_i$ represents the ground truth or golden standard, ${\widehat{P}}_i$ denotes the probability of the voxel $i$ belonging to the prostate, and $W_i^C$ is the weight, which is set as $1/|\text{pixels of class}{x}_i=C|$. Based on the weighted cross entropy loss function, the class unbalancing problem can be alleviated.

Data augmentation has been proven to improve the performance of deep learning.^6,23,30 To obtain robustness and increased precision on the test data set, we augment the original training data set using image translations and horizontal reflections.

3.

Experimental Results

3.3.

Qualitative Evaluation Results

The performance of the proposed deep learning method was evaluated qualitatively by visual comparison with the manually segmented contours. Figure 4 shows the qualitative results.

Fig. 4

The qualitative results of the proposed method. The red curves represent the prostate contours obtained by the proposed method, while the blue curves represent the contours obtained from manual segmentation by an experienced radiologist.

4.

Conclusions

We proposed an automatic deep learning method to segment the prostate on MRI. An end-to-end deep CNN model was trained on three prostate MR data sets and achieved good performance for prostate MRI segmentation. To the best of our knowledge, this is the first study to fine-tune an FCN that has been pretrained using a large set of labeled natural images for segmenting the prostate on MRI.

Based on the experimental results, we found that the use of pretrained FCN with fine-tuning could yield satisfactory segmentation results. The performances are expected to be further improved by adding more training data sets. The proposed algorithm is efficient and does not require any handcrafted features. Currently, the deep learning algorithm learns the features automatically. Fortunately, other works^37–39 have developed new CNN models for segmentation, which can be used to improve our segmentation algorithm in the future work. Deep learning-based segmentation methods rely not only on the selection of neural network architecture but also on the selection of loss function. In future, we will investigate the behavior of custom loss functions and their performances for segmentation task. The deep learning method can be generalized to various organs and lesions segmentation problems beyond prostate segmentation. It can be applied not only to MRI but also other imaging modalities such as CT and ultrasound images.