Inter-subject Registration-based Segmentation of Thoracic-Abdominal Organs in 4 Dimensional Magnetic Resonance Imaging

4 Dimensional Magnetic Resonance Imaging ( 4D MRI ) is currently gaining attention as an imaging modality which is able to capture inter-cycle variability of respiratory motion. Such information is beneficial for example in radiotherapy planning and delivery. In the latter case, there may be a need for organ segmentation, however 4D MRI are of low contrast, which complicates automated organ segmentation. This paper proposes a multi-subject thoracic-abdominal organ segmentation propagation scheme for 4D MRI . The proposed scheme is registration based, hence different combinations of deformation and similarity measures are used. For deformation we used either just an affine transformation or additionally free form deformation on top of an affine transform. For similarity measure, either the sum of squared intensity differences or normalised mutual information is used. Segmentations from multiple subjects are registered to a target MRI and the average segmentation is found. The result of the method is compared with the ground truth which is generated from a semi-automated segmentation method. The results are quantified using the Jaccard index and Hausdorff distance. The results show that using free form deformation with a sum of squared intensity differences similarity measure produces an acceptable segmentation of the organs with an overall Jaccard index of over 0.5. Hence, the proposed scheme can be used as a basis for automated organ segmentation in 4D MRI.


INTRODUCTION
Respiratory motion effects are inevitable in thoracicabdominal imaging and image guided interventions.For example in radiotherapy planning, radiation dose calculations will be affected by tumour motion due respiratory motion.Because of respiratory motion, 4 dimensional (4D) imaging such as 4D computed tomography (4D CT) are increasingly used to account for respiratory motion in such procedures (Czerska et al. 2021).However, the drawback and limitation of 4D-CT imaging is that it is not time-resolved i.e. it can only capture the average motion during many respiratory cycles and hence information about its inter cycle variability is not available.Additionally 4D CT imaging leads to higher dose exposure.
In light of the limitations of conventional 4D CT, there have been efforts at utilising time-resolved 4D imaging such as 4D Magnetic Resonance Imaging (4D MRI) (Li et al. 2019;Stemkens et al. 2018;Wang & Yin, 2019;Liney & van der Heide 2019).Such time-resolved 4D imaging are able to capture the inter cycle variability of respiratory motion.With such imaging modalities, there may be a need for organ segmentation, for instance in radiotherapy dose calculation, similar to when CT imaging is used.However, in the case of 4D MRI, while there is the added advantage of time-resolved volumetric imaging, image quality is typically lower than compared to for example 4D CT and this complicates accurate organ segmentation.Manual segmentation of organs is tedious and timeconsuming.Various efforts have been performed to automate segmentation (Li et al. 2020 ;Kayur et al. 2021).Amongst those are ones based on prior information, such as those based on generic deformable models in ultrasound imaging (Zhan & Shen, 2006) and more specifically statistical shape models for multiple imaging modalities (Wu et al. 2019;Cerrolaza et al. 2016).However, in the case of shape models based on prior knowledge, the limitation is that new cases may somewhat vary considerably from the training cases.An alternative which is not constrained by training data is utilising image registration for segmentation (Kéchichian et al. 2018;Oliveira et al. 2018).Both Kéchichian (2018) and Oliveira (2018) applied their method on CT and MRI.However, the MR images used are the normal 3D images with high resolution.In our work, we wish to perform segmentation on a frame from a 4D MRI sequence which is of lower resolution.
In this paper, we are proposing a multi-organ segmentation framework for 4D-MR data.The organs that are segmented are the lungs, liver, heart, spleen, stomach and kidneys.Semi-automated segmentation of these organs in training images constitute prior information instead of an explicit atlas.The training images are then registered to the test image using affine and free form deformation (FFD)-based registration.These steps are describe in further detail in the methodology below.

METHODOLOGY
In our proposed method, there are two categories of images, sources images which are pre-segmented and the target image to which we wish to propagate those segmented organs.The source MR images are registered to the target image using either affine or free form deformation.The deformation found is then utilized to propagate the segmented organs to the target image.The mean of the propagated organs then forms the output which are the target segmented organs.The overall process for a particular organ is as shown in Figure 1.
In our proposed method, the source images are presegmented using a semi-automated organ segmentation method which is described below, after which the registration methods used itself is described in more detail.
We then describe the evaluation data and evaluation scheme and metric respectively.

SEMI-AUTOMATED ORGAN SEGMENTATION
A semi-automated segmentation method is based on active contours (Chan & Vese, 2001) and it is selected for the pre-segmentation of the source images.It is based on an energy minimization inside and outside the curve representation.The formulation used is the Chan-Vese energy or uniform modelling as shown in Equation ( 1): where δ(x,y) refers to the image, and I_1,I_2 are average intensities inside and outside of the curve C respectively.An example of the semiautomated segmentation process on a liver for one image slice is shown in Figure 2.Here the process starts from manual rough delineation of the liver (Figure 2(a)), which is then refined by the active contour method (Figure 2(b)).This manual initialization is performed by an expert.This semi-automated segmentation is performed on all slices of an image volume.A final render of all organs using this technique is shown in Figure 3(c).The organs which are included are the lungs, heart, liver, spleen, stomach and kidneys.Note that the respiratory system itself (lungs and airways) was segmented using a more automated procedure as described in (Abd. Rahni et al. 2013).FIGURE 2. An example of the semi-automated organ segmentation process for a liver on a specific slice (white outline).(a) The liver is roughly delineated with manually selected points, (b) segmentation refinement using the active contour method and, (c) renders of the segmented organs, namely the lungs (blue), liver (magenta), stomach (peach), heart (red), spleen (cyan), kidneys (dark red)

REGISTRATION
Registration of the whole body is performed to propagate the pre-segmented organs in the source images to the target images.This is performed via an affine transformation which preserves the straightness of lines.Additionally, to overcome residual misalignment from affine registration, registration based on free form deformation (FFD) using B-Splines (Modat et al. 2010) is performed as an additional option on top of affine registration.Registration is performed directly on the 4D MR images without prior preprocessing as it is deemed adequate.
For both affine and free form deformation methods, we either use the sum of squared intensity differences (SSD) or normalized mutual information (NMI) as the similarity measure.It is shown that SSD is the optimal measure when the voxel differences between an image α and registered image β' has a Gaussian distribution (Sonka & Fitzpatrick, 2000).The SSD measure is defined as in Equation ( 2): (2 where N is the number of voxels in α∩β' (Sonka & Fitzpatrick 2000).
The other similarity measure used, NMI, is based on the joint histogram of image α and registered image β', which should be maximized to find the desired transformation.NMI is defined as in Equation ( 3): (3) where H is entropy.

EVALUATION DATA
To demonstrate and evaluate the proposed scheme, dynamic MR images are used.These were acquired during normal free breathing.The images of five volunteers were used.The volunteers consist of three males and two females with no abnormality (Tsoumpas et al. 2011).The MR images were acquired using a 1.5 T Philips AchievaTM scanner with a 32-channel coil.As a result, thirty five volumes were captured in a fast dynamic MR acquisition over multiple frames during free breathing with a 0.7 s frame rate, an image dimension of 336×45×336 voxels and a 1.48214 ×5.5 ×1.48214 mm3 voxel size.For the purpose of this paper, the first volume of each volunteer were used for evaluating our method.

EVALUATION SCHEME AND METRICS
For evaluation a leave one out cross validation scheme is used.For each targeted volunteer, source MRI images from the other volunteers are used and registered to the target image.For each organ, the average of the propagated organs from each source MRI is used as the final propagated segmentation.This is then compared to the pre-segmented organs of the targeted volunteer which is considered as the ground truth.To evaluate the agreement of the segmented organs with the ground truth, we use two approaches.The first approach is to measure overlap using the Jaccard index (Tan et al. 2006).For single organs, this is defined as in Equation ( 4): (4) where the ratio is the total number of voxels, denoted by the modulus operator |X|, of the organ-wise intersection, ∩, over the organ-wise union, .β' denotes the segmented organ whereas α denotes the adopted ground truth.
Additionally, we also measure the aggregate overlap over all organs, using a multi-label Jaccard index (MLJI), defined similarly as in (Crum et al. 2006), according to Equation (5): (5) which is defined over all the organs considered (i.e.labels l).
Alternatively, we also measure segmentation accuracy using the Hausdorff distance for each organ.This is defined as the maximum distance between corresponding surface ∩ points of the segmented organ Sβ' and the ground truth surface points Sα.The corresponding point to a ground truth surface point pα є Sα is defined as the nearest surface point in the segmentation pβ' є Sβ' to it.Formally, using Euclidean distances to find point correspondences, the Hausdorff distance d can thus be defined as: (6) However, Equation ( 6) is not symmetrical as .Therefore the symmetrical Hausdorff distance (Cignoni et al. 1998) which we use in this paper is defined by finding the maximum, H, of both the forward and backward Hausdorff distances:

RESULTS
Using the methodology described above, we compare the result of the segmentation scheme using different combinations of deformation and similarity measures, namely: affine with NMI, affine with SSD, FFD with NMI and FFD with SSD. Figure 3 shows the result of this segmentation scheme on the liver in a particular coronal slice of one volunteer, using the above mentioned combinations.As mentioned in Section 2, the organs which were evaluated using the proposed scheme were the lungs, liver, heart, spleen, stomach and kidneys.
Figure 4 on the other hand shows the translucent 3D renders of the result of the proposed scheme (blue), overlaid over the adopted ground truth (red), for one volunteer, using the different combinations of deformation and similarity measure.Visually, it can be seen that for all organs, FFD performs better than just using affine deformation.Additionally, using SSD as a similarity measure results in a better overlap than using NMI.
Figure 5 shows the mean Jaccard indices of the overlap of the segmented organs with the adopted ground truth over the five volunteers.The error bars show the standard deviation over the volunteers.Quantitatively, the best overlap is achieved using FFD with the SSD measure for all organs.The numerical values are as shown in Table 1.A higher Jaccard index signify a better overlap.As in Figure 5, Figure 6 show the multi-label Jaccard indices over all volunteers for each combination of deformation and similarity measure.
To investigate the segmentation results in more details, we look at the Hausdorff distance error (Equation 7), H, for each organ segmented using FFD with the SSD measure.The numerical values are shown in Table 2.
Using this metric, on average the lungs show the largest error compared to the kidneys which has the smallest error.The left lung has an average error value of H=19.44 mm whereas the left kidney has an average error value of H=14.20 mm.This can be attributed to the large size of the lungs and relatively small size of the kidneys.

DISCUSSION
The scheme described in the methodology is proposed for segmentation propagation of thoracic-abdominal organs in low contrast 4D-MR images.Due to organ variability, it does not use explicit shape models.As the proposed scheme is registration based, different combinations of deformation and similarity measure are used.The result shows that the scheme can segment multiple organs of various sizes, shapes and intensities in low contrast MR data.
Bigger organs such as the lungs, liver and heart have better overlap indices with the adopted ground truth compared to the small organs such as kidneys, however this may be due to the nature of overlap indices which is based on percentage of overlap rather than actual volume e.g.mm3.This is corroborated with the Hausdorff distance error shown in Table 2. Nevertheless, it was found that using FFD with the SSD measure produces the best perorgan as well as overall segmentation.The advantage of using FFD over just affine deformation is obvious, given inter-subject variability.On the other hand, the advantage of SSD over NMI in this application can be explained by the fact that the differences in the registered source images and the target images can be represented by a Gaussian distribution [12].The results thus suggest the proposed scheme can be used as a basis for automated organ segmentation in 4D MRI, such as when one needs to perform radiotherapy treatment planning.

CONCLUSION
The presented segmentation propagation scheme based on intensity based image registration is proposed as an organ segmentation method for low resolution 4D-MR data.The proposed scheme is fully automated for the target image and it is tested for multiple thoracic-abdominal organs namely the lungs, liver, heart, spleen, stomach and kidneys.
The results were evaluated with different combinations of deformation (affine or FFD) and similarity measure (SSD or NMI).It was found that FFD with the SSD measure produces the best segmentation per-organ as well as overall organs.It is proposed that the scheme can be used where there is a need for automated organ segmentation in 4D MRI such as for radiotherapy treatment planning.

FIGURE 1 .
FIGURE 1.The proposed automated organ segmentation scheme

FIGURE 3 .
FIGURE 3. The result of the proposed segmentation scheme on the liver in a particular coronal slice of one volunteer (white outline).The combination of deformation and similarity measure used are: (a) affine and NMI, (b) affine and SSD, (c) FFD and NMI and (d) FFD and SSD

TABLE 1 .
Mean Jaccard indices of the segmented organs

TABLE 2 .
Hausdorff distance in mm for each organ segmented using FFD-SSD