Registration-base morphing of active contours for segmentation of CT scans

We present a new algorithm for segmenting organs in CT scans for radiotherapy treatment planning. Given a contour of an organ that is segmented in one image, our algorithm proceeds to segment contours that identify the same organ in the consecutive images. Our technique combines partial diferential equations-based morphing active contours with algorithms for joint segmentation and registration. The coupling between these diferent techniques is done in order to deal with the complexity of segmenting "real" images, where boundaries are not always well defined, and the initial contour is not an isophote of the image.

1. Introduction.In this paper, we are concerned with the segmentation of computed tomography (CT) scans for radiotherapy treatment planning.The CT scans are a sequence of two-dimensional images that correspond to parallel cross-sections of part of the patient's body.They are used both for diagnosis and for planning the radiotherapy treatment.
The treatment planning is a two-step procedure.First, a radiation oncologist has to identify in every image two types of objects: tumors that are to be destroyed (with a sufficient dose of radiation) and healthy tissues that are to be protected (by trying to minimize the amount of radiation they receive).In the second step, the segmented information is used as an input to various optimization algorithms, to determine an optimal treatment.Typically, at this stage one would like to calibrate the number of beams, their directions, their intensities, modulations, shapes, and so forth.The entire process is time consuming.It takes about four to eight hours to prepare a radiotherapy plan for each patient.About half of this time is spent on the segmentation phase.
The accuracy of the treatment depends critically on the identification and on the segmentation of the organs and the tumor in the CT scans.Surprisingly, despite the recent advances in imaging processing technologies, the clinical segmentation process is still performed using manual tools.Our goal in this paper is to present 80 Y.-N.YOUNG AND D. LEVY a method that will partially automate the segmentation process, and by that we hope to reduce substantially the time that this phase requires.
We would like to note that this segmentation problem is expected only to worsen with the arrival of the new generation of CT scanners.The resolution of the new machines in the transverse direction is about three times better than that of the present CT scanners (1 mm compared with 3 mm).Although one expects dense sets of data to provide more information for processing the images, the lack of successful automated algorithms implies that at least in the near future, a radiation oncologist would expect to spend significantly more time in segmenting the images.
The basic assumptions that guide us when approaching the design of an algorithm for solving this problem are the following: First, the number of images is relatively small (on the order of 40 to 80) and all of them are given up front.Second, we would like to have a fast algorithm that can still be used off-line (and not necessarily in real time).Finally, we assume that a radiation oncologist can provide certain information that will be outlined later.
Given these assumptions, a framework that seems to be particularly suitable for solving this problem is the machinery of "morphing active contours" introduced in [1,2,3].This algorithm assumes as data two images and a segmented contour that identifies the object of interest in the first image.The goal is then to identify in the second image a contour that corresponds to the same object that was segmented in the first image.This is done with a system of two partial differential equations.One equation is an advection equation with a velocity proportional to the difference between the two images.This equation is in charge of morphing the two images to each other.The segmented contour in the first image is represented implicitly as a zero contour of a level set function.The second equation evolves this level set function with the same velocity given by the morphing equation.The hope is that at the end of this process the resulting zero level set will represent the boundary of the target object in the second image.
Unfortunately, as shall be demonstrated later, a straightforward application of morphing active contours algorithms fails to produce acceptable results in cases where the initial curve is not a level set of the first image.As expected, most of the objects we are interested in segmenting in our problem do not satisfy this requirement: in some cases, part of their boundary is not well defined; in other cases, they are connected with other organs and it seems to be impossible to separate them with no prior knowledge.
We would like to mention segmentation-based registration algorithms.These "feature-based" methods provide a reference shape for the segmentation based on prior knowledge.Given some preliminary input, either from low-level segmentation (such as edge detection) or customized segmentation for specific anatomical structures (such as physician's segmentation), the common features of the object to be identified are registered to a common reference frame.The object can then be segmented according to the registered features.Recently an explicit combination of registration with segmentation has been implemented in a variational framework through active contours by Yezzi et al. [16].Moelich and Chan [9] extend the algorithm of Yezzi et al. [16] to joint segmentation of an object in two images, and the prior knowledge is implicitly incorporated through the simultaneous evolution of the registration maps and the active contour in the variation framework.
What we propose in this work is to combine the morphing active contours algorithm with the joint segmentation and registration algorithm.This will result in a two-step segmentation algorithm.In the first step, we expect to segment the object based on the registered features as given by a prior segmentation.This phase of joint segmentation leaves us with an object boundary whose shape is very similar to the one in the prior segmentation.We then refine the segmentation through morphing of active contours to incorporate new features of the object in the new frame.By repeating this process for several consecutive frames, this algorithm in effect refines the registration to incorporate new features introduced by morphing, which makes it suitable for segmenting a sequence of medical images.
The structure of this paper is as follows: in section 2 we briefly describe conventional morphing active contours algorithm.We then review the joint segmentation and registration algorithm in section 3. Our way of combining these two algorithms is then presented in section 4. The results of our algorithm as implemented in several test cases are presented in section 5. We show that in some cases our algorithm produces results equivalent to those obtained with the conventional morphing active contours algorithm, but in other cases, the segmentation performed with our algorithm is significantly better than the segmentation performed with the conventional morphing algorithm.We end with several concluding remarks in section 6.

2.
Morphing active contours.We start with some basic notations.We assume that we have a sequence of d CT scans, which we denote as {I} j , j = 1, . . ., d.All images are assumed to have an identical size of N × N voxels (with a typical value of N = 512).Every image is a two-dimensional map that assigns a non-negative scalar (gray scale value) to every pixel, that is, where Z N = {1, . . ., N }.Since we intend to approach the problem with continuum methods, we will assume that every image is a sample (with a prescribed resolution) of an underlying function that is defined almost everywhere.Hence, we can assume a squared domain, which will be arbitrarily normalized to the unit square [0, 1] × [0, 1], on which a function I j (x, y) is defined.(of course we do not know the underlying function, though we can always reconstruct such a function from the image, for example, as a piecewise-constant function).
The second type of objects we will deal with are contours that represent objects of interest in the different images.To simplify the notations, we assume that in every image we are interested in only one object.This simplification does not modify the nature of the problem, as the algorithm we present can be used to capture all the objects of interest in any given image simultaneously.We therefore assume that in every image I j there is one object of interest (either a tumor or a critical organ) for which the boundary is denoted by C I j .The boundary of the object is assumed to be a closed (though not necessarily connected or simple) contour.
With these notations in hand, we assume that two images, namely I 1 and I 2 , are given.We also assume that an object of interest was segmented in the first image I 1 , and is given in terms of the contour C I 1 .The goal is to locate the same object in the second image, I 2 , obtaining the contour C I 2 (consult Fig. 1).
The method of morphing active contours as a way to approach this problem was proposed by Bertalmio, Sapiro, and Randall [2], (see also [1]).In this algorithm, the idea is to morph the first image I 1 to the second image I 2 .As I 1 is morphed into I 2 , the object boundary C I 1 is simultaneously evolved using the same velocity, and ideally settles to C I 2 , which is the object boundary in I 2 .The whole process is carried out with a system of partial differential equations.This requires an additional time variable, which is being introduced to the problem by setting the initial image I 1 to be the information that is known at "time" t = 0.The morphing of one image to the other is then treated as an evolution in time.The second image I 2 will be obtained in this process as t → ∞, though from a practical point of view, the execution of any such algorithm stops once the difference between the morphed image and the target image is smaller than a certain threshold.
With this in mind, we can denote the first image as I 1 (x, y, 0).The morphing process either directly morphs the image I 1 to I 2 , or it can be applied to features of these images.Hence, in general, we can define a "features map" F j = F(I j ) and evolve F 1 according to with a time-dependent "morphing velocity" β(x, y, t) yet to be determined.To simultaneously evolve the contour C I 1 , we consider a function u(x, y, t) for which C I 1 is a level-set (isophote) at time t = 0. We then evolve u(x, y, t) according to ∂u(x, y, t) ∂t = β(x, y, t)||∇u(x, y, t)||. ( The velocity β(x, y, t) is obtained by projecting the morphing velocity β onto the normal direction of the level set C I (i.e., the zero level set of u at time t).Hence β ≡ β N F 1 • N u , and N F 1 and N u are the normals to the level sets of F 1 and u, respectively.
It is natural to assume that the "morphing velocity" β(x, y, t) is to proportional to the difference between F 1 and F 2 (features of I 2 ).The speed function that was proposed in [2] is where F i are the features of image i that are defined as Here, L(•) can be an edge map of a band around the level set C I (see [2], [13] and [10,14,15] for more details).This segmentation scheme by morphing active contours was shown to work well for cases where the images are almost bi-level, and have simple features around the object boundary [2].For more complicated images, where the image is far from bilevel, Bertalmio et al. [2] suggested that more intermediate frames in the sequence will enhance the performance and result in better segmentation.Although this idea might work in certain cases, there are common instances where the object boundary may not correspond to an isophote in the image I.In these cases, most samples of the data will not solve the problem.
To further elucidate this point, we implement the above algorithm and apply it to real CT scans.Figure 2a shows a CT scan of abdominal organs and Figure 2b is a manual segmentation by a radiation oncologist of the liver based on his medical knowledge and experience.The normalized, de-noised image intensity along the liver boundary as a function of the curve length is shown in Figure 2c.It is clear that the liver boundary that was segmented by a radiation oncologist does not correspond to an isophote.Furthermore, we notice that the denoised image is not bilevel.The physician's manual segmentation of the next frame is shown in Figure 3.If we utilize the morphing of contours by morphing the scans of Figure 2a to Figure 3a, and simultaneously tracking the contour in Figure 2b using the velocity in Equation ( 2), we find that the final segmentation contour from morphing and tracking is far from the physician's segmentation.The physician's segmentation is shown as the solid line in Figure 4 while the result of the morphing active contours algorithm is shown as the dashed line in Figure 4.
This typical example makes it clear that for segmenting internal organs in a sequence of scans, the simple morphing and tracking algorithm requires certain improvements and refinements.Once again, we would like to stress that increasing the number of images is not a valid option in this case, and even if more frames were available, the contour that represents the boundary of the liver would still be far from being an isophote, and the results of this algorithm would probably be unsatisfactory.3. Joint segmentation and registration.As illustrated in section 2, the cause for the failure in segmenting internal organs by morphing active contours is the weak correlation between the organ boundary and the isophote of image intensity.Consequently, the morphing velocity based on the difference in image intensities evolves the contour toward an isophote that does not correspond to the organ boundary, and eventually leads to great discrepancy between the morphed active contour and the physician's segmentation as illustrated in Figure 4. To segment images where the object boundary does not correspond to an isophote, a significant modification to the conventional variational approach of the level set method is necessary.Furthermore, the radiation oncologist relies on prior knowledge of the expected shape of the organ, as well as the relative location of the scan in the anatomy.Thus, we are particularly interested in segmentation of a sequence of images based on prior knowledge and object features in the variational framework.
Many segmentation algorithms use statistics of an ensemble of images by registering various features in the images prior to segmenting (see [5] and [6,7] and references therein).In this context, registration amounts to finding a pointwise transformation between the source image (I 1 ) and the target image (I 2 ) that minimizes a certain dissimilarity measure.Feature-based registration requires an input of prior shapes and accurate feature detectors.On the other hand, direct registration involves estimating the transformation between the source and targeted images without feature extraction.In direct (nonrigid) registration, as demonstrated by Paragios et al., the (dis)similarity measure can be taken as the sum of squared differences over pixel intensities.Such a similarity measure works well for non-rigid registration even when the registered source shapes differ slightly from the target objects, or when the target image is damaged by occlusions and local deformation due to noise [11].However, it may be impractical to apply this algorithm to segment an organ in a sequence of images, because the shape of the boundary depends on the location and orientation of the image; therefore many different prior shapes are required for the registration process.
Since our purpose is to segment an organ in a sequence of CT scans where the organ boundary changes smoothly, it helps to evolve both the segmentation and the registration simultaneouslly so that the exact anatomical position of the organ plays little role in the registration process.This can be done by evolving the prior shape.As shown by Yezzi et al. [16,17], one can conduct simultaneous registration and segmentation through a similarity measure, of which the (local) minimum is located by evolving the active contour in the variational framework.This interdependence between registration and segmentation has been discussed and explored by Yezzi et al., who combined registration with active contours and applied their algorithm to knowledge-based segmentation of medical images [16,17].
Based on the work of Yezzi et al., Moelich and Chan [9] proposed a refined algorithm to jointly segment an object in two images in the framework of the variational level set method with registration of an active contour.While Yezzi et al. segment one image, Moelich and Chan simultaneously segment two images.Instead of one registration map as in [16,17], two affine mappings for registration are used for better performance in terms of locating the minimum of the variational energy.The energy to be minimized is constructed with logic models that are taken from Sandberg and Chan [12].This is somewhat similar to the multichannel algorithm by Vese and Chan [15].Overall they have demonstrated how a blurred object of a similar shape and features to an object in another image can be jointly segmented based on the minimizer for the variational energy.The algorithm can be summarized as follows.
Following our previous notations, I 1 : Z N × Z N → R + and I 2 : Z N × Z N → R + denote two unregistered images, and each image contains a common object of interest to be segmented.Let C I1 and C I2 denote the contours that segment the object in I 1 and I 2 , respectively.The key idea is to relate these two contours through a parametrized mapping g to a contour C. We will consider a specific mapping at the end of the section.In the approach of Yezzi et al. [16,17], C I 2 = g(C I1 ; p), where p are the parameters for g.This approach is adapted in Moelich and Chan's algorithm; they consider C I1 and C I2 to be a Euclidean transformation of a contour C under g for two different sets of registration parameters p 1 and p 2 .That is, The use of two mappings allows for more flexibility in avoiding local minima in the search of registration parameters.The goal then is to find a contour C and parameters p 1 and p 2 that minimize the energy where E int is the internal energy and E ext is the exterior energy, defined as The energy densities F int (x; p 1 , p 2 ) and F ext (x; p 1 , p 2 ) depend on which logic model is being used [9].The logic models, developed by Sandberg and Chan [12], are designed to segment multichannel images according to logical combination of the two images.Each logical relation and interpretation of the two images may correspond to a different minimum in the multi-channel algorithm.A great advantage in choosing the logical models over the multi-channel algorithm is to minimize the dependence of the final segmentation on the initial conditions.For more details we refer interested readers to [9] and references therein.From Sandberg and Chan [12], we obtain the logical OR model where The logical AND model is φ is the level set function (of which the zero contour corresponds to the active contour C), and the Heaviside function H(φ) is calculated using the construction scheme in [4].k j i are normalizing scale factors, chosen so that F j i ≤ 1.The coefficients λ j i , λ int and λ ext are design parameters.In our implementation, the values for these parameters are fixed for each test case presented in section 5.
For the registration, we use the Euclidean transformation as in [9], where The parameters of g are p i = (∆x i , ∆y i , ∆θ i , m i ), where ∆x i and ∆y i are the translations, ∆θ i is the rotation about the center of C I i , and m i is the magnification.
As in [16], anisotropic magnification is possible, but here we adopted isotropic magnification for both maps.When p = (0, 0, 0, 1), the transformation g(x; p) is the identity map.The energy from Equation ( 6), which depends on the segmentation contour C and the registration parameters p 1 and p 2 , is minimized by "interleaving" segmentation and registration, as proposed in [9].During each iteration of the algorithm, the estimates of C, p 1 and p 2 gradually improve.After updating the parameters p 1 and p 2 , an updated gradient flow can be calculated and used to evolve the contour C. We refer interested readers to [9] for a more detailed illustration of how well this algorithm works in various examples.
4. An algorithm for segmenting CT scans.Results from the joint segmentation and registration [9] show that even a blurred object in the target image can be accurately segmented without much prior knowledge of the object's shape and features.Of course an a priori condition for the algorithm to work properly is that the object of interest be of similar shape in both images.In essence, the joint segmentation and registration algorithm seeks the "optimized" common features in both images that minimize the intensity-based energy.From this point of view, the joint segmentation algorithm is more flexible than the non-rigid registration algorithm proposed in [11].For example, it is possible to jointly segment images without exact prior knowledge as the algorithm seeks the common features and the registration simultaneously by minimizing the energy.It also allows the possibility of segmenting more than one object in the target image.Furthermore, specifying the logical relations between the two images allows the segmented object to have a different shape than the source object.
The algorithm of [9] seems ideal for our purpose, because we can segment an organ not in just one target image, but also in a few subsequent frames as its shape and geometry change smoothly.We propose to iterate the joint segmentation and registration process as follows: First, we construct a bilevel image I 1 from a given object boundary.Knowing that the target object (with possibly slightly different boundary) is located near the source object in the second image I 2 , we jointly segment both I 1 and I 2 using the logical AND model in Moelich and Chan's algorithm.We choose the AND model to emphasize the similar features between the two objects.In this fashion, we can determine the boundary of the same organ in I 2 despite the difficulties discussed previously in section2.Once the segmentation is complete for the first target image, we use the resulting segmentation as the source object for segmenting the next target image by repeating the above process.
The main drawback to this procedure is that the shape of the segmented object must be similar to the source object because of the AND logic relation.We typically expect the features to be similar between the different frames, but there still are some differences that we are interested in capturing.The alternative OR model is in some sense too flexible, for it allows many irrelevant features to accumulate.To track the changes in the object's shape and features, we propose to combine the joint segmentation and registration with morphing of active contours as follows.We start with two consecutive images, I 1 and I 2 , and a segmented contour C I 1 in I 1 , which is either given by manual segmentation or is taken from a previous iteration of our algorithm.First, we generate a bi-level image from C I 1 .We then jointly Figure 4 illustrates the proposed algorithm.After we complete the process for the first target image, we use the segmentation in the first target image as an initial condition to segment the next image.As already seen in section 2, the organ boundary C 2 does not correspond to an isophote in I 2 .We also note that different features can be found around the organ to be segmented (liver, in this case) in both images, and some of these features will be included using the conventional level set method for segmentation.
Following [9], we start the iteration with registration.This seems to reduce the sensitivity of the algorithm to local minima.During each iteration, we first hold C fixed and improve the estimates of the registration parameters.As in [9], we use a line search for the joint segmentation algorithm.The direction of the search is found by taking numerical derivatives of the energy E int (C; p 1 , p 2 ) with respect to the components of p 1 and p 2 .For example, the components of p 1 are updated by numerically integrating where q stands for the quantities ∆x 1 , ∆y 1 , ∆θ 1 , and m 1 .Next, we hold p 1 and p 2 fixed and improve the estimate of C by evolving it along the gradient flow, which in this context is given by where n is the exterior unit normal vector of the contour C.This process is continued until a minimum is found.We would like to emphasize that morphing active contours does refine the segmentation results when compared with the results of joint segmentation and registration.This will be illustrated in section 5. Remarks 1.In [9], the internal energy is used for evolving the mapping parameters in (14).We replace the internal energy by the total energy, which seems to perform better in this application.2. We can restrict our search for local minimum to a reasonable range of values for both sets of mapping parameters.For example, since we know that the distance between the images is less than 3 mm, we can search for a minimum of the energy within a range for the scaling factor.3. We would like to stress the differences between our algorithm and the algorithm of Yezzi et al. [17].First, they use only one registration map that evolves simultaneously with the level set.Moelich and Chan [9] already noted that one registration map is not always enough for segmenting blurred images and mentioned that their logic model for the energy works better than the energy that is used in [16].In our work, we use a joint segmentation and registration with the addition of morphing for fine tuning of the segmentation.We also start with a bilevel image that is generated from the known contour in the first image.This is unlike any of the previous works.4. For our application, the initial registration phase in [9] is not as essential because the objects in the two images are close to each other.For results presented in the following session, the initial conditions for the contour C are the given segmentation from oncologists, or from the previous joint segmentation.For the registration maps, it seems to work well, because we use unit maps for initial conditions.

5.
Results.In this section, we present results obtained from applying our algorithm to several test cases, all of which involve segmentation of an organ in a series of CT scans.In all test cases, we compare our results to the results of the conventional morphing active contours algorithm, and to a manual segmentation (whenever available) that was done by a radiation oncologist from the radiation oncology department at Stanford's medical school.This manual segmentation incorporates the anatomical knowledge about the particular organ and is considered as our gold standard.It is important to note that there is a difference between the way different physicians segment the same organ in the same image.
Our first example is of a segmentation of a liver.In Figure 6, we present the consecutive four slices that follow the initial slice for which a manual segmentation of the liver is given.In all figures, the dashed lines represent the manual segmentation by the radiation oncologist.The solid lines in the top images in Figure 6 represent the segmentation by the morphing active contours algorithm, while the solid lines in the bottom images represent the segmentation obtained by our algorithm.In all four images, our algorithm generally produces significantly better results than those from the morphing of active contours.In particular, we do not observe the oscillations on the boundary that are present in the results obtained by the morphing active contours algorithm.We further note that the results from our algorithm, as shown in the bottom part of Figure 6, agree closely with the oncologist's segmentation.In some cases (e.g. the third figure from the left), our automatic segmentation seems better than the manual segmentation.
We would like to emphasize that for images such as the CT scans in Figure 6, there is no hope that conventional level set algorithms could segment the liver: the boundary of the liver is not always well defined, and there are certain regions where, with no prior knowledge, one would be tempted to connect the liver with neighboring structures.In this case, if a physician trusts the results and decides to proceed with a radiotherapy treatment planning based on these results, we save 80% of the manual segmentation time.
A more significant deviation between our segmentation and the physician's segmentation is found after the fourth scan in the series.This may be attributed to an accumulated error in our algorithm or to a significant deformation of the boundaries of the liver between these images.
Many methods have been proposed to quantify performance of the segmentation algorithms in standard image processing, where target objects have well-defined boundaries and the correct object contour can be readily determined without any subjective human input.However, in our applications, a good segmentation is the contour that matches well with organ contours from physicians, among whom the definition of a "uniquely correct" contour of an organ is unavailable and very often subject to their experience, training, and different circumstances.A comprehensive comparison between the algorithm and different manual segmentations will be essential before any clinical tests and is left for the future.In the meantime, we propose to quantify the segmentation from different algorithms as follows.First, we define three markers, of which the point values are assigned according to their  6 relative location to the numerical segmentation contour C n (either from morphing active contours or from our algorithm) and the manual segmentation C p from a physician The integrals of m 1 , m 2 , and m 3 over the whole domain can be used as indicators of the segmentation performance when they are scaled to M 0 , the total area enclosed by the contour from the manual segmentation: The closer M 1 is to unity, the better the match between the numerical segmentation C n and the manual contour C p .
In Table 1, we tabulate the values of the three integrals from both our algorithm (left side) and the traditional morphing of active contours (right side).From these values, we can conclude that segmentation from our algorithm is quantitatively better than the traditional morphing scheme, because M 1 remains close to 80% for all four slices using our algorithm.Also M 2 , the portion of the manual contour that does not overlap with segmentation from the numerical algorithms, is significantly smaller for our algorithm.We further note that our algorithm gives a larger M 3 , which may be appropriate for application to radiation treatment given the safety factors that are involved when protecting essential organs from overexposure to radiation.
We now demonstrate the effect of morphing in our algorithm.As a typical example, we pick the second frame in Figure 6 and display the difference in the segmentation due to morphing and tracking in Figure 7.The left panel shows the active contour (solid line) from the joint segmentation before the morphing.In the right panel, we show the active contour after morphing and tracking, and the dashed lines are the oncologist's manual segmentation in both panels.The morphing and tracking seems to moderately improve the joint segmentation.
Our next example, in which we segment a bladder, is shown in Figure 8. Similar to the previous example, we show four consecutive segmentations that follow the first image with a given manual segmentation of the bladder.The solid line in the  top images in Figure 8 shows the results obtained with the morphing active contours algorithm, while the solid line in the bottom images shows the segmentation results obtained with our algorithm.This is a relatively difficult segmentation problem, because there is no clear boundary between the bladder and the prostate.
As in the previous case, we tabulate the integrals M 1 , M 2 , and M 3 in Table 2. Comparing the first image in the top and bottom rows, the morphing algorithm seems to work better for the first two slices.Clearly there is a difference between the first two slices and the last two slices in Figure 8.For the first two slices, our algorithm oversegments significantly more than the morphing algorithm, although they both cover the whole manual contours.However, for the next two slices,  8 the morphing algorithm gives much less overlap with the manual contours (M 1 ), whereas our algorithm still gives 90% overlap with slightly larger over-segmented areas.Furthermore, our algorithm seems to produce better results in the subsequent images both in terms of capturing the evolution of the organ boundary and in the general shape, compared with oncologists' segmentations.In particular, in the last frame of the sequence, the morphing result bears no resemblance to the manually segmented bladder boundary, while the segmentation from our algorithm appears to be closer to the manual segmentation.We further note that the segmentation contours from our algorithm capture the concave curve near the bottom tip of the bladder where it is adjacent to the prostate (see second and third images in bottom row of Figure 8).The corresponding contours from the same images in the top row remain the same and fail to capture the evolution of the boundary as suggested by the manual segmentation.
It is unclear why a sideways deviation appears from the manual segmentation from the third to the fourth frame in both algorithms.This phenomenon may be attributed to the strong gradient flow in the lateral direction during the morphing process.
In this test case, the effect of morphing is shown in Figure 9.In both panels, solid lines are the final segmentation after morphing of the contours from the joint segmentation, dashed lines are the segmentation before the morphing, and dashdotted lines are the oncologist's manual segmentation.The last example is of a segmentation of a left kidney; the results are shown in Figure 10.Here we zoom on the part of the images that contains the kidney.Again, we show the four consecutive segmentations that follow the first image with a given manual segmentation.The top images show the results obtained by the morphing active contour algorithm.The bottom images show the results obtained with our algorithm.In this case, we have no reference manual segmentation by a physician that we can compare with; we generated the initial manual segmentation ourselves.As with the liver, the background is noisy and structures around the kidney are likely to be included by conventional level set algorithms.This is probably why the observed contours from the morphing algorithm in the top images of Figure 10 are wiggly.In contrast, the results from our algorithm are quite smooth and do not suffer from the noisy surroundings as much.We also note that the results from morphing active contours deviate from the obvious kidney boundary near the tip of the organ in the last two images.This is not the case with our algorithm.

Conclusion.
In this work, we present a new algorithm for the segmentation of target objects in CT scans.This algorithm combines the conventional morphing active contour algorithms and the more recent joint registration and segmentation algorithm.Given a sequence of images and a segmented object in the first image, we use our algorithm to segment the same object in the consecutive images.We find that our algorithm accurately segments at least three consecutive slices in a sequence, and the quality of the segmentation deteriorates slightly for the fourth slice in both examples in section 5. Thus, we can reiterate this segmentation process for the whole sequence, provided we have physicians' manual contours every several slices in the sequence and our algorithm can automatically fill the gap.For this paper, we focus more on the new algorithm than on optimizing its performance.We note that segmenting a slice usually requires a few minutes, and the variation in run time is mostly due to the convergence of the joint segmentation component.
We demonstrated the results obtained this way in several test cases, and showed that in most test cases the approach performs better than morphing active contours algorithms.It is noticeable that in all test cases our algorithm produced a segmentation consistent with the manual segmentation for four consecutive images.This is not the case with the traditional algorithms we checked against the same data.From our quantitative comparison between the traditional algorithm and our algorithm, it is evident that segmentation from morphing alone scarcely matches the manual contour in most slices, whereas our algorithm generates segmentation that still resembles the manual contours in terms of shape and of the overlapped area.We would like to emphasize that even though we demonstrated the segmentation of one organ at a time, nothing in the algorithm limits one from simultaneously segmenting several objects.
Currently, we are focusing on estimating the error propagation in our algorithm when applied to various different cases.Some de-noising preprocessing may be necessary to help control the error propagation.We also explore the possibility of combining forward and backward image flows to refine the segmentation.
In most of our test cases, whenever a slight deviation between the automatic and the manual segmentations appeared, the automatic segmentation tended to generate a contour that enclosed the manual contour.We do not know the source of this seemingly consistent behavior.However, we would like to note that for the purpose of planning radiotherapy treatment, over-segmenting an organ is much better than under-segmenting that organ (whether that organ should be radiated or protected).
The ideas presented here are not limited to CT scans, and we expect them to work equally well with other types of medical images, such as MRI images.We conclude by noting that in this work we have not addressed efficiency issues, and we consider them a fundamental topic for future research.

Figure 2 .
Figure 2. (a) Two adjacent frames from a sequence of CT scans of a patient; (b) segmentation by physicians; (c) image intensity along contour in (b).

Figure 3 .
Figure 3. (a) Two adjacent frames from a sequence of CT scans; (b) segmentation by physicians; (c) image intensity along contour in (b).

Figure 4 .
Figure 4. Physician's segmentation (solid line) versus results from morphing and tracking (dashed line) of the contour in Figure 2.

1 2Figure 5 .
Figure 5.The proposed procedure combining joint segmentation and morphing of active contours to segment internal organs based on oncologists' input.

Figure 6 .
Figure 6.Liver.Top: Segmentation with morphing active contours.Bottom: Segmentation with our algorithm.The dashed lines indicate the manual segmentation.

Figure 7 .
Figure 7. Morphing of active contour for fine-tuning of the joint segmentation.The dashed lines indicate the manual segmentaiton.

Figure 8 .
Figure 8. Bladder.Top: Segmentation with morphing active contours.Bottom: Segmentations with our algorithm.The dashed lines are the manual segmentation.

Figure 9 .
Figure 9. Morphing of active contour for fine-tuning of the joint segmentation.

Table 1 .
Table of the integrals M 1 , M 2 , and M 3 for segmentations in Figure

Table 2 .
Comparison of the integrals M 1 , M 2 , and M 3 for segmentations in Figure