Abstract
We propose a convex image segmentation model in a variational level set formulation. Both the local information and the global information are taken into consideration to get better segmentation results. We first propose a globally convex energy functional to combine the local and global intensity fitting terms. The proposed energy functional is then modified by adding an edge detector to force the active contour to the boundary more easily. We then apply the split Bregman method to minimize the proposed energy functional efficiently. By using a weight function that varies with location of the image, the proposed model can balance the weights between the local and global fitting terms dynamically. We have applied the proposed model to synthetic and real images with desirable results. Comparison with other models also demonstrates the accuracy and superiority of the proposed model.
1. Introduction
Image segmentation is a fundamental task in image processing and computer vision. Active contour models have become one of the most successful methods for image segmentation [1–5]. Some of active contour models [1, 4–8] are based on the edge information. These models use the image gradient information to stop the evolving contours on the object boundaries. We call them edge-based models. Typical edge-based active contour models [4, 5] have an edge-based stopping term and a balloon force term to control the motion of the contour. There are also some active contour models [2, 9–13] which are based on the region information of the image instead of the edge information. We call them region-based models. Models of this kind use certain region descriptors to segment different regions or identify interested regions of an image.
Region-based active contour models have the following advantages over edge-based models. Firstly, they do not utilize the image gradient and therefore have better performance for the image with weak object boundaries. Secondly, they are significantly less sensitive to the location of initial contours. One of the most popular region-based active contour models is the Chan-Vese (CV) model [2]. This model has been successful for images with two regions, each having a distinct mean of pixel intensity. In [13], Vese and Chan extended their original model in [2] by using a multiphase level set formulation, in which multiple regions can be represented by multiple level set functions. These models are called piecewise constant (PC) models. However, the PC models and other popular region-based active contour models [9–11] tend to rely on intensity homogeneity in each of the regions to be segmented. Thus they cannot cope with images with intensity inhomogeneity.
To overcome this problem, Vese and Chan [13] and Tsai et al. [12] independently proposed two similar region-based models for more general images. These models, widely known as piecewise smooth (PS) models, have exhibited certain capability of handling intensity inhomogeneity. However, the PS models are computationally expensive and suffer from other difficulties.
Recently, Li et al. proposed a local binary fitting (LBF) model [14, 15] to overcome the difficulty in segmentation caused by intensity inhomogeneity. By using a kernel function in the data fitting term, intensity information in local regions is extracted to guide the motion of the contour, which thereby enables their model to deal with intensity inhomogeneity. However, these methods [14, 15] are to some extent sensitive to the initialization, which limits their practical applications. Then Wang et al. [16] combined the advantages of the CV model and the LBF model to propose an active contour model based on the local and global intensity fitting (LGIF) energy for image segmentation. The LGIF model can segment images more accurately. However, the authors need to balance the weights between the two models appropriately according to different images. This will be a little boring in practice to choose an appropriate value for the weight.
The split Bregman method [17–19] has also been applied to image segmentation problems recently in [20–22]. In [21], the authors proposed a convex and fast segmentation method by applying the split Bregman concept to the CV model. Their method is mainly for homogenous images. Thus, Yang et al. [22] applied the split Bregman method to the region-scalable fitting (RSF) energy model to deal with images with inhomogeneity efficiently.
In this paper, we proposed a new convex region-based image segmentation method based on the LGIF model to consider the local and global information together. We first use the globally convex segmentation idea from Chan et al. [23] to propose a convex energy functional for image segmentation. To minimize the proposed energy functional more efficiently, we use the split Bregman method just as our previous work in [22]. Instead of using a constant weight for the global fitting term, a weight function that varies dynamically with location of the image is applied in this paper. In this way, the proposed model can balance the weights between the local fitting term and the global fitting term by itself. Therefore, the proposed model can segment more general images accurately and efficiently.
The remainder of this paper is organized as follows. We first review some related models and their limitations in Section 2. The proposed model is introduced in Section 3. The experimental results of the proposed model are given in Section 4. This paper is concluded in Section 5.
2. Related Models
2.1. The Mumford-Shah Model for Image Segmentation
Mumford and Shah proposed a famous image segmentation model in [24]. Let be the image domain, and be a given gray level image. Their idea is to find a contour that segments the given image into nonoverlapping regions and a piecewise smooth image that approximates . is smooth within each of the connected components in the image domain separated by the contour . The energy functional they formulated is
where and are positive constants. is the length of the contour . The Mumford-Shah model has been often used in image segmentation. However, it is difficult to minimize the functional (2.1) in practice due to the unknown contour of lower dimension and the nonconvexity of the functional.
2.2. The CV Model
For a special case of the Mumford-Shah problem when the image in the functional (2.1) is a piecewise constant function, Chan and Vese [2] formulated a piecewise constant model called the CV model without using the image gradient. For an image , they proposed to minimize the following energy: where , , and are positive constants. and represent the regions outside and inside the contour , respectively, and and are two constants that approximate the image intensity in and . This energy can be represented by a level set formulation, and then the energy minimization problem can be converted to solve a level set evolution equation [2]. One of the most attractive properties of the CV model is that it is much less sensitive to the initialization.
The optimal constants and that minimize the above energy are the averages of the intensities in the entire regions and , respectively. Such optimal constants and will not be accurate if the intensities within or are not homogeneous. Local intensity information which is crucial for inhomogeneous image segmentation is not considered in this model. This is the reason why the CV model can not handle image inhomogeneity. Similarly, more general piecewise constant models in a multiphase level set framework [9, 13] are still not suitable for images with intensity inhomogeneity.
2.3. The LBF Model
Li et al. proposed the LBF model in [14, 15] to segment images with intensity inhomogeneity by using the local intensity information efficiently. The energy functional they proposed is where and . and are two local fitting functions that approximate the intensities outside and inside the contour . is a Gaussian kernel with the standard deviation . The localization property of this kernel function plays a key role in segmenting images with intensity inhomogeneity. However, such localization property may also introduce many local minimums of the energy functional. Consequently, the result is more dependent on the initialization of the contour. This has been deeply explained in [16].
2.4. The LGIF Model
Wang et al. proposed the LGIF model in [16]. The LGIF model combines the advantages of the CV model and the LBF model by taking the local and global intensity information into account. There, they define the local and global intensity fitting energy as follows: where is a constant ().
The local intensity fitting (LIF) energy is defined as the first two terms of the LBF model [14, 15]: where is the level set function, is the Heaviside function. and .
The global intensity fitting (GIF) energy is defined as the first two terms of the CV model [2]:
Then the other two terms and are also needed as in [2, 7, 13] to regularize the level set function and the contour , respectively. In practice, the Heaviside function is approximated by a smooth function defined by where is a positive constant. The derivative of is the smoothed Dirac delta function: Thus the energy functional they proposed is where and are positive constants.
3. The Proposed Model
3.1. The New Proposed Convex Model
We propose a new and convex region-based image segmentation model to consider both the local and the global information as [16]. The energy functional (2.9) of the LGIF model in Section 2.4 is not convex. Following the idea in Chan et al. [23], we first propose a convex energy functional based on the LGIF model.
According to Wang et al. [16], the optimal local fitting functions , , constants , , and level set function that minimize the energy functional (2.9) in the LGIF model are updated using the standard gradient descent method as follows: where , . and are defined as
and are called the LIF force and the GIF force, respectively.
To apply the idea in Chan et al. [23] to the LGIF model, we consider the gradient flow equation (3.3). The last term is used to regularize the level set function to be close to a distance function. Here we drop it first, the new obtained gradient flow equation is and without loss of generality, we take .
We then apply the globally convex segmentation idea of Chan et al. [23]; the stationary solution of (3.5) coincides with the stationary solution of
We now propose a new energy functional as follows:
It can be clearly seen that the simplified flow (3.6) is just the gradient descent flow of the new proposed energy functional (3.7). Thus the minimization problem we want to solve is
To guarantee the global minimum, the solution is restricted to lie in a finite interval. In this paper we use a more general form as follows: where .
The segmented region can be found by thresholding the level set function for some if the optimal is found: where the thresholding value is chosen as in this paper.
The first term in the proposed energy functional (3.7) is in fact the traditional total variation (TV) norm:
To incorporate information from an edge detector [25], we then replace the standard TV norm (3.11) with the weighted TV norm: where is the given image and is the nonnegative edge detector function. is a parameter that determines the detail level of the segmentation.
Thus the proposed minimization problem becomes where .
3.2. Split Bregman Method for Minimization of the Proposed Model
The efficiency of the split Bregman method for image segmentation has been demonstrated in [21, 22]. We now apply the split Bregman method to solve the proposed minimization problem in a more efficient way. We introduce the auxiliary variable, . To weakly enforce the resulting equality constraint, we add a quadratic penalty function which results in the following unconstrained problem:
The Bregman iteration is then applied to strictly enforce the constraint . The optimization problem becomes
Keeping fixed, the Euler-Lagrange equation of the optimization problem (3.15) with respect to is
For (3.17), a central difference is used for the Laplace operator and a backward difference is used for the divergence operator; the numerical scheme for (3.17) is
For a fixed level set function , we minimize (3.15) with respect to and obtain where is the shrinkage operator [18, 21] defined as
Thus the split Bregman algorithm for the proposed minimization problem (3.13) can be summarized in Algorithm 1.
| ||||||||||||||||
In this algorithm we use to denote one sweep of the Gauss-Seidel formula (3.18). This algorithm is different from the one in our previous work [22] when updating .
Note here that we say the proposed energy functional in the minimization problem (3.13) is convex; in fact it means that it is convex with respect to for a fixed . From Algorithm 1, it can be seen that is computed before updating . Thus each time when we update using the Gauss-Seidel formula, the value of is in fact fixed.
3.3. The Choosing for the Parameter
The parameter controls the influence of the LIF force and GIF force which can be seen clearly from (3.4). When the intensity inhomogeneity is severe, the accuracy of the segmentation relies on the LIF force. Thus a smaller value of should be chosen as the weight of the GIF force. For images with minor inhomogeneity, the GIF force alone is able to attract the contour to a location near the object boundaries. In this case, relatively larger should be chosen as the weight of the GIF force. In [16], is chosen as a constant for a given image. Wang et al. need to choose an appropriate value for according to the degree of inhomogeneity.
In our paper, we choose in a different way as [26]. Instead of a constant value for , a weight function that varies dynamically with location of the image is chosen in this paper. The weight function is defined as follows where is a fixed parameter and represents the local contrast ratio of the given image, which is defined as where denotes the size of the local window, and are the maximum and minimum of the intensities within this local window, respectively. represents the intensity level of the image, for gray level images, it is usually 255. varies between 0 and 1. It reflects how rapidly the intensity changes in a local region. It is smaller in smooth regions and larger in regions close to the object boundaries.
In the above weight function (3.21), is the average value of over the whole image and it reflects the overall contrast information of the image. For an image with a strong overall contrast, we believe that the image has much more obvious background and foreground, so we increase the weight of the global term on the whole. adjusts the weight of the global term dynamically in all regions, making it smaller in regions with high local contrast and larger in regions with low local contrast. Thus the weight value can vary dynamically with different locations. It is determined by the intensity of the given image.
4. Experimental Results
We have tested the proposed model with synthetic and real images in this section. As in [22], we simply initialize the level set function as a binary step function which takes a constant value inside a region and another constant value outside. The advantage of using a binary step function as the initial level set function is that new contours can emerge easily and the curve evolution is significantly faster than the evolution from an initial function as a signed distance map. We use , , , , , and for all images shown in this paper. We choose for all gray images and for all color images. The values chosen for the parameters and are specified in each figure.
Figure 1 shows the results for an inhomogeneous image from [26] with different methods. (a) and (b) show the original image and the initial contour, while the final contours using the CV model, the LBF model, the LGIF model, and the proposed model are shown in (c)–(f). From this example, we can see that the CV model fails to get the correct segmentation result. The LBF model traps into the local minimum. The LGIF model also gets an incorrect result by using a constant weight . The proposed model gives the right segmentation result. We choose and for this image. This example demonstrates the superiority of our proposed model over other models. Furthermore, our result is even better than the result gotten from [26]; the upper part of the object was missed there.
(a)
(b)
(c)
(d)
(e)
(f)
The results for an airplane image using different methods are shown in Figure 2. and are also used for this image. We can see that the result obtained by applying the proposed model is the best one among all these four models. This can also be seen clearly from the final level set function shown in Figure 3.
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
(c)
(d)
Figure 4 shows the results for two X-ray images of vessels with different methods. We use for these two vessel images. We can see from (b) and (g) that the CV model fails to segment the vessels correctly by only using the global intensity information. (d) and (i) show that the results with the LGIF model are either incorrect, because a constant weight of the global term is used and thus the segmenting curve is influenced too much by the global fitting energy. Column 3 and Column 5 show that both the LBF model and the proposed model can segment these two images correctly.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
To compare the results of the two vessel images in Figure 4 with the LBF model and the proposed model, we show the final level set function with the two models in Figure 5. It can be seen that the proposed model can get better final than the LBF model.
(a)
(b)
(c)
(d)
In Figure 6 we give a synthetic inhomogeneous image that the proposed model can handle while the other models fail to segment it correctly. We use for this image. From (b)–(d) we can see that there are unwanted curves in the final segmentation results when using the other three models. (e) shows that the proposed model can get the correct final contour. In fact by observing the curve evolution processes of these four models, unwanted curves also grow when using the proposed model. However, these unwanted curves will finally disappear with our model because they can move quickly by using the split Brgman method.
(a)
(b)
(c)
(d)
(e)
Figure 7 shows the corresponding mesh figures of the final level set function with different models for the same synthetic image from Figure 6. Comparing these four mesh figures of the final , we can observe that the proposed model can obtain the best final level set function shown in (d) of Figure 7.
(a)
(b)
(c)
(d)
The proposed model can be easily extended to be applied for color images. Figure 8 shows an application of our model to a color image of elephants. and are used for this image. The first row shows the active contours on the original image from its initial to converged state. The proposed model can segment this image correctly which can be seen from (d). The second and third rows show the corresponding two fitting images and . The whole fitting images at different iterations are shown in the last row. The final fitting image shown in (p) can fit the original image well.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
(p)
Figure 9 shows the curve evolution process from the initial contour to the final contour for a color image of starfish with the proposed model. This image is very inhomogeneous both in the background and foreground and is difficult to segment. We choose for this image. The proposed model can segment it correctly.
(a)
(b)
(c)
(d)
In Figure 10, we show the results of several other color images with the proposed model. We use and for the first banana image, while for the other three images. (a)–(d) show the original images with the initial contours. (e)–(h) show the original images with the final contours. Experimental results show that our model can segment color images well.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
In our previous work [22], we have proposed a convex model by applying the split Bregman method to the RSF model. We call it the SBRSF model here. In the SBRSF model we only consider the local information without considering the global information. Thus the SBRSF model may get many local minimums, which can be seen in Figure 11. In Figure 11, we give the results of some color images from Figures 8, 9, and 10. The same initial contours and parameters have been used for the SBRSF model as the proposed model. However, local minimums will occur in the leg part of the back elephant, in the right part of the starfish image, in the middle part of the banana image and in the green pepper. The proposed model considers both the local and the global information and thus can get better results than the SBRSF model.
(a)
(b)
(c)
(d)
5. Conclusion
A new convex region-based image segmentation model is proposed in this paper. We consider the local and global intensity fitting terms together and propose a convex energy functional using the globally convex segmentation method. By applying a weight function that varies dynamically with location of the image, the proposed model can adjust the weight of the global intensity fitting term by itself. The split Bregman method is then used to minimize the proposed energy functional more efficiently. We have compared the proposed model with the CV model, the LBF model, the LGIF model, and our previous SBRSF model with synthetic and real images. Experimental results have shown the advantages of the proposed model in image segmentation. The proposed model is a little sensitive to the parameters and . In fact, the SBRSF model also has this problem. It may be caused by the application of the split Bregman method. This is what we should study more in the future work.
Acknowledgment
This work is supported by Natural Sciences Foundation of Heilongjiang Province (A200909).