A Novel Histogram-Based Multi-Threshold Searching Algorithm for Multilevel Colour Thresholding

Image segmentation is an important preliminary process required in object tracking applications. This paper addresses the issue of unsupervised multi-colour thresholding design for colour-based multiple objects segmentation. Most of the current unsupervised colour thresholding techniques require adopting a supervised training algorithm or a cluster-number decision algorithm to obtain optimal threshold values of each colour channel for a colour-of-interest. In this paper, a novel unsupervised multi-threshold searching algorithm is proposed to automatically search the optimal threshold values for segmenting multiple colour objects. To achieve this, a novel ratio-map image computation method is proposed to efficiently enhance the contrast between colour and non-colour pixels. The Otsu's method is then applied to the ratio-map image to extract all colour objects from the image. Finally, a new histogram-based multi-threshold searching algorithm is developed to search the optimal upper-bound and lower-bound threshold values of hue, saturation and brightness components for each colour object. Experimental results show that the proposed method not only succeeds in separating all colour objects-of-interest in colour images, but also provides satisfactory colour thresholding results compared with an existing multilevel thresholding method.


Introduction
Object tracking in video streams is essential to many intelligent vision systems. In such systems, image segmentation plays a necessary preliminary processing role in a variety of image processing applications, especially in object detection and recognition. Accurately segmenting foreground objects from a background image is a significant issue in many image analysis fields. Several sophisticated image segmentation techniques have been reported in the literature, however, nature image segmentation is still a challenging task due to an inherently ambiguous problem [1]. To handle such an inherent complexity, some researchers have tried to extend the traditional bilevel thresholding techniques to a multilevel process since histogram thresholding is one of the most widely used techniques in image segmentation. This is commonly known as multilevel thresholding.
Various multilevel thresholding techniques have been proposed for grey images. They roughly can be divided into parametric and nonparametric approaches [2]. In parametric approaches [3,4], the probability density function (pdf) of each object's grey-level distribution is required. But, to estimate the pdf of the grey-level distribution is typically a nonlinear optimization problem. This usually leads to an inefficient algorithm with high computational complexity and poor performance. By contrast, nonparametric approaches directly determine the threshold values by optimizing some certain cost functions [5][6][7][8]. For instance, Otsu proposed a thresholding method that determines the optimal thresholds by maximizing a between-class variance criterion using an exhaustive search [5]. To speed up the maximization process in the conventional Otsu's method, Liao et al. proposed a fast multilevel global thresholding algorithm that maximizes a modified between-class variance with a look-up table acceleration approach [6]. However, this method still takes too much time for multilevel threshold selection since it requires evaluating all possible solutions. The authors in [7] proposed a new criterion for automatic multilevel thresholding with low computational complexity. Recently, Gao et al. used a quantum-behaved particle swarm optimization technique to improve the convergence rate of the Otsu's method [8]. These nonparametric multilevel thresholding approaches produce satisfactory segmentation results for grey images, however, the papers [5][6][7][8] considers only grey scale images (i.e., single channel) and cannot be directly applicable to multi channel images.
As to colour image thresholding, some researchers have employed neural-network techniques to achieve unsupervised colour segmentation. For instance, Wu et al. proposed an automatic self-organizing map (SOM) based multilevel thresholding algorithm for colour segmentation and human hand localization [9]. They developed a SOM transduction algorithm to learn the non-stationary colour distribution in HSI colour space that overcomes the issue of dynamic lighting condition. Haghighatdoost et al. proposed an automatic multilevel thresholding algorithm for colour image quantization [10]. They first used a growing time adaptive SOM network to find the 3-dimensional RGB histogram. Then a peak finding algorithm was employed to find the thresholds automatically. Deshmukh et al. proposed an adaptive colour image segmentation approach based on a multilayer neural-network, which detects multiple objects in an input image by independently segmenting saturation and intensity planes of the image [11]. Although the neural-network-based approaches usually work well in colour image segmentation, they have to undergo a supervised training process to obtain the best segmentation performance. This requirement restricts the applicability of the neural-network-based segmentation approaches.
Some researchers have attempted to use fuzzy set theory to deal with colour image segmentation [12][13][14][15]. The authors in [12] proposed a coarse-to-fine segmentation strategy based on the thresholding and fuzzy c-means (FCM) techniques. This method first coarsely segments the image using a thresholding technique. The fine segmentation process then assigns the unclassified pixels to the closest class using the FCM technique. Chaabane et al. also adopted the concept of the coarse-to-fine strategy to segment colour images, but utilized an automatic histogram thresholding method in the coarse segmentation stage [13]. Kurugollu et al. developed a colour image segmentation technique using fuzzy thresholding and Dempster-Shafer's fusion rule [14]. This method first uses a fuzzy thresholding method to fuzzify each colour band into mass functions of three classes (object class, background class and uncertain class). These mass functions are then merged using the Dempster-Shafer's orthogonal sum rule. Finally, the optimal thresholds are taken by comparing the mid point of belief intervals of the object and background classes. Recently, Yu et al. introduced an adaptive unsupervised clustering algorithm based on the FCM technique and ant colony optimization [15]. This method is able to adaptively detect the cluster centroid distribution and centroid number, which are two critical initialization problems in FCM approach. The fuzzy-logic segmentation approaches usually require the knowledge of cluster number, otherwise they will become quite sensitive to the initialization condition. This requirement increases the complexity of image segmentation, leading to an inefficient solution to the unsupervised colour image segmentation problem.
From the literature survey, how to develop a new unsupervised colour image segmentation method efficiently extracting multiple colours-of-interest from foreground pixels in the image remains a challenge. This problem motivates us to develop a computationally efficient unsupervised multilevel thresholding algorithm for colour image segmentation. The proposed algorithm consists of a colour-pixel extraction process and a histogram-based multi-threshold searching process. In the colour-pixel extraction process, a novel ratio-map image computation method is proposed to enhance the contrast between colour and non-colour pixels. This helps leading to an efficient colour-object extraction algorithm using a simple bilevel thresholding method, such as the Otsu's method. On the other hand, the histogram-based multi-threshold searching process automatically searches the optimal upper-bound and lower-bound threshold values of hue, saturation and brightness components for each colour object. Experimental results validate the performance of the proposed method by comparing with an existing multilevel thresholding method using nature colour images.
The rest of this paper is organized as follows. Section 2 defines the problem statement of this work. Section 3 introduces an existing method related to the development of the proposed algorithm. Section 4 presents the proposed multi-threshold searching algorithm in details. Experimental results are carried out in Section 5 to evaluate the performance of the proposed colour image segmentation approach. Section 6 concludes the contributions of this paper.

Problem statement
Image segmentation techniques have been widely used in various machine vision applications. To obtain the information of an object-of-interest from images, one of the most important tasks is to accurately extract all foreground objects from the background. Traditionally, a simple but efficient multilevel thresholding method that distinguishes pixels belonging to foreground and background based on colour distribution of a specific object-of-interest is used for this purpose. Let denote the input colour values of each pixel in HSV colour space. Suppose that the objects-of-interest are belonging to monotone colour objects, so that colour distribution in the HSV space is continuous with a bounded range and can be characterized by a rectangle region for each colour channel. Then, it is possible to find three threshold pairs (hl, hu), (sl, su), and (vl, vu) respectively thresholding each colour channel to create a binary image from an input HSV colour image such that where B(x,y) is the output binary image allowing to segment the object-of-interest from the background. Expression (1) means that if the hue, saturation and brightness values of an input colour pixel are within the range of lower-bound to upper-bound threshold values, then the corresponding output pixel is assigned to the object class labelled 1, otherwise assigned to a null class. Figure 1 shows an example of image threshold segmentation using the image-thresholding method (1). In Fig. 1(a), the original colour image contains several colour objects. In this example, the threshold values are chosen as (hl, hu)=(190, 235), (sl, su)=(0.21, 1), and (vl, vu)=(0.1, 1) for extracting the blue-colour object ( Fig. 1(b)).
Although the expression (1) provides a simple way to deal with colour image segmentation, the main challenge is how to determine the optimal threshold values of each colour channel for segmenting multiple colour objects. That is, for N object classes, 1  N , we try to find 3N optimal threshold pairs where LN(x,y) is the output labelled image separating Nobject and background pixels with minimum error. Expression (2) is the type of multilevel thresholding. According to [16], multilevel thresholding is generally less reliable since it is difficult to establish multiple thresholds effectively isolating each region-of-interest. Therefore, searching for optimal multiple thresholds typically is performed in a supervised manner (i.e., manual adjusting the thresholds to optimize thresholding) in order to improve the reliability of multilevel thresholding. This problem highlights the importance of developing a reliable multi-threshold searching algorithm to assist searching for optimal thresholds. This study thus proposes a novel histogrambased multi-threshold searching algorithm to achieve this purpose efficiently and effectively.

The existing Otsu's image thresholding method
This section briefly describes the Otsu's method [5] that is a well-known image thresholding technique used in grey image segmentation. Suppose that a grey image consists of one or more foreground objects, each having different grey-level values. The purpose of image thresholding is to separate these objects from the background using one or more threshold values obtained from the histogram of the image. More specifically, if the image histogram contains 2  M dominant modes (including one background mode), the image thresholding algorithm aims to find 2  M threshold values to extract all foreground objects in the image. Let Np denote the number of total pixels in the image, L the maximum greylevel value of the pixel and ni the number of pixels with grey level i. Given the pixel-level probabilities  In the case of M=2 (bilevel thresholding), Otsu first defined a between-class variance of the thresholded image with respect to a threshold value t such that p are the probabilities of class occurrence; ip are the class mean values. Then, the optimal threshold value t can be determined by maximizing the between-class variance so that In the case of M>2 (multilevel thresholding), the expression (3) can be easily extended with respect to Similarly, the optimal threshold values 1 2 1ˆ{ , , , }   M t t t are determined by maximizing the criteria (5) so that The optimal threshold values are then obtained by maximizing the criteria (6) instead of (5) so that The main advantage of using the modified criteria (6) is that the computation of maximization process (7) can be reduced by employing two look-up tables. Let 0 ( , ) M u v and 1 ( , ) M u v denote, respectively, the u-v interval zeroorder and first-order moments of a class with grey levels from u to v defined as For index 1  u , the above expression can be rewritten in a recursive form: , and (9) From the expressions (8)-(10), it follows that According to (11), ( ) Substituting (12) and (13) into (6) yields where 0 t and M t are previously defined in (5). Expression (14) shows that the modified between-class variance 2  B can be simplified as a function of 0 ( , ) To speed up the computation of (14), an efficient way is to pre-compute the two moments 0 ( , ) M u v and 1 ( , ) M u v for all possible intensities from u to v using the expressions (9)- (11). By doing so, the computations required for (14) are significantly reduced. Based on this observation, one can define the following function.  (16) are reduced to addition and LUT indexing operations that drastically speeds up the maximization process given in (7). In this paper, we term this LUT-accelerated multilevel thresholding method as fast Otsu's method. Although the fast Otsu's method is useful in achieving multilevel thresholding applied to t defined in th section will the multileve

The propo
This section searching al extraction pr searching pro and non-colo an input colo multiple thre the image wi

Colour-pix
Colour is an extract the co colour-based representing hue compon perceived co example, pr shown in Fig   Figure 2.  Figure 2(a) ent of the im own in Fig. 1 alue to avoid t term aims to m to highlight esults a highnamic range ce a ratio-map (18) e highlighting β is a non-zero ol the contrast 3(a) and 3(b) applying (17) ectively. From to control the perty helps to n the original image via a oach, i.e., the ction.   Fig. 4(a) t s can be sim ence, the colou iently extract el thresholding obtained from e colour regio ixels in the o correspondin ( Fig. 5(b)). Vis ds that Fig. 5  ur and non-co ished by a gl the original im the conventi hreshold the r 8) (Fig. 4(b)). A ge, one can ob ge ( Fig. 5(a)) gram of the co ring Fig. 2 Histogram-base ure 6 shows th ed multi-thres a peak-point d rching process our regions of el extraction p peak-point nts in the inpu ue of the hist ectly on the or ge noise, the ration on the ct during peak hing algorithm.  of the propose g algorithm, w cess and a m at the hue hist obtained from bed in the pre ocess aims to am depending searching the stogram is ver rst performs stogram to red on ( Fig. 7(a)).  h . Observing Figs. 9(a) and 9(b) we find that the merged result produces the same thresholding range to the original one, but using a smaller number of threshold pairs. Therefore, the computational complexity of the following colour segmentation process can be reduced efficiently.
In the process of searching saturation and brightness thresholds, we only find the lower-bound thresholds ( )

Experimental results
This section contains two parts. The first part presents the experimental results of the proposed method compared with an existing method and the second part presents the extension of the proposed method to skin colour segmentation.

Performance evaluation of the proposed method
In the first experiments, two test images, as shown in Figs. 11(a) and 12(a), were employed to evaluate the performance of the proposed method.
The fast Otsu's multilevel thresholding method described in Section 3 was used as a competing method to evaluate the performance of the proposed method. Table 1 tabulates the parameter settings of the competing and proposed methods used in the experiments. In order to quantify the performance of the proposed method, the PSNR metric (in dB) was adopted in the experiments. The PSNR metric in this paper is defined as x y x y UV , where U and V are respectively the total row and column number of the image, Ts is the multilevel thresholding result obtained by a manually supervised thresholding, and Tp is the corresponding result by using an unsupervised thresholding method. Figure 11(b) presents the multilevel thresholding result obtained by the supervised method that is manually adjusting each threshold to get the best segmentation result by visual observation.
Figures 11(c) and 11(d) present respectively the multilevel thresholding result and the multiple thresholds of hue channel obtained from the fast Otsu's method. It is clear from these two figures that the fast Otsu's method cannot provide satisfactory segmentation results since the multiple threshold values are not able to threshold the two colour classes in the hue histogram properly. This observation also can be verified in Table 2, which records the multiple threshold values obtained by each compared method and the corresponding PSNR measures. From Table 2, one can see that the PSNR metric between Figs. 11(b) and 11(c) is 17.1552 dB, which means a lowsimilarity between the results produced by the fast Otsu's and manually supervised methods. On the other hand, Fig. 11(e) shows the multilevel thresholding result obtained from the proposed method. Visually comparing Figs. 11(b) and 11(e) finds that the proposed method produces similar segmentation results to the manually supervised method. Figure 11(f) illustrates the multiple thresholds of hue channel obtained from the proposed method. It is clear from Fig.  11(f) that the multiple threshold values succeed to threshold the two colour classes in the hue histogram, producing a high PSNR of 38.0751 dB as shown in Table  2. These results explain why the proposed method produces a similar result to the manually supervised method. Similar observations can also be found in the case shown in Figs. 12(a)-12(f). Therefore, the above experimental studies validate the performance of the proposed method.       Table 3. Multiple threshold values obtained by the proposed method and the previous work [19] for skin colour thresholding.

Extension to skin colour segmentation
The proposed method can be extended to other colour image segmentation applications, such as skin colour segmentation. Figure 13(a) illustrates a skin colour image, which challenges many existing skin colour segmentation methods since it contains a variety of ethnic skin tones. Figure 13(b) shows the colour thresholding result obtained from the proposed method. It can be observed from Fig. 13(b) that the proposed method efficiently extracts most skin pixels in the original image, except some white skin pixels. Table 3 tabulates the multiple threshold values found by the proposed method in this experiment. From Table 3, one can see that the skin colour distribution ranges from 1 to 41 in hue channel. This observation is consistent with the result of a previous work [19] which shows that the skin colour distribution in the Hue-Saturation plane typically ranges from 0 to 50 in hue channel for Asian and Caucasian ethnicities. On the other hand, the reported result of skin colour in saturation channel is characterized by values between 0.23 and 0.68, but our finding is between 0.1843 and 1.0. This is caused by that the test image used here additionally containing the Ethiopian ethnicity, leading to a smaller lower-bound threshold value on saturation. Furthermore, as mentioned in the end of Section 4.2, the upper-bound threshold of saturation is fixed as 1.0 in our design. Figure 13(c) shows the colour thresholding result obtained by the previous work [19]. Visually comparing Figs. 13(b) and 13(c) we can observe that the skin colour thresholding results obtained by the proposed method and in the previous work are very similar to each other. This result validates the performance of the proposed method extended to skin colour segmentation as we expected.

Conclusions and future work
In this study, a novel histogram-based multi-threshold searching algorithm has been developed to achieve automatic multilevel colour image thresholding for segmenting multiple objects in colour images. The proposed method comprises a new colour-pixel extraction algorithm, which computes a ratio-map image efficiently enhancing the contrast between colour and non-colour pixels. This property helps to extract all foreground colour objects from background image via a conventional single-level image thresholding method. Moreover, a new histogram-based multi-threshold searching algorithm is also proposed to search the optimal upper-bound and lower-bound thresholds of hue, saturation and brightness channels for each colour object. Experimental results validate the performance of the proposed method by comparing with the fast Otsu's multilevel thresholding method, both visually and quantitatively. It is worth noting that the proposed algorithm can be extended to other colour image segmentation applications, such as nature colour image segmentation or skin colour segmentation. This advantage greatly increases the applicability of the proposed method. The extension to multilevel video thresholding is left to our future study.

Acknowledgments
This work was supported by the National Science Council of Taiwan, ROC under grant NSC 101-2221-E-032-022.