Superpixel segmentation based on image density

Superpixel segmentation can get the middle features in image processing, effectively reduce the dimensionality of the image, and is widely used in image processing fields. To get the regular and compact superpixels in real-time, a superpixel segmentation algorithm based on image density is proposed in this paper. Firstly, the image is uniformly divided according to the number of superpixels to be obtained. Secondly, to get the clustering ability of the pixels, the density image is produced. Thirdly, the seed is chosen in each sub-region block according to the density and then the superpixels are obtained by clustering. During the clustering process, the pixel around the seed should be added into the superpixel if it meets the conditions, and the small supeipixels are merged into the big superpixels around them. Finally, the result shows that the proposed algorithm has the best segmentation effect, and a good balance in accuracy, regularity, and time cost.


Introduction
With the gradual increase of image resolution, the application of computer vision based on pixels is difficult to meet the requirement of the real-time image processing.And the low dimensional information can improve the processing speed, so it is desirable to reduce the dimensionality of the image.As the middle layer features of the image, superpixels can effectively reduce the dimension of the image and the complexity of image processing, and are widely used in image segmentation (Turkmenli et al., 2009), target detection (Yan & Zhu, 2019), target tracking (Wang et al., 2018), image classification (Jia et al., 2019), reconstruction (Suryansh et al., 2021) and so on.
The superpixels segmentation can be divided into graph-based and clustering-based.The graph-based methods regard each pixel in the image as a vertex, and the adjacent relationships between pixels are regarded as edges, the similarities or differences between pixels are represented by the weights of the edges.Thus, the image is transformed into an undirected weighted graph.Ncut (Shi & Malik, 2000) is of this type of segmentation, whose boundary is significantly cohesive, but the time cost is high.TurboPixels (Levinshtein et al., 2009) obtains segmentation boundary by distributing the seeds uniformly and constructs an energy function to get the optimized parameters.LRW (Shen, Du, et al., 2014) obtains the superpixel by probability and optimized energy function, but the computation complexity is high.ER (Liu Clustering-based algorithms mainly obtain superpixels by a certain clustering algorithm, such as K-means, which is widely used in image processing (Gai et al., 2021).SLIC (Achanta et al., 2012) uses K-means to get the superpixels, whose clustering centres are updated by iteration.In addition, there are also many algorithms to improve SLIC.FLIC (Zhao et al., 2018) improves the convergence speed of clustering by adjusting the search method of the clustering process.CAS (Xiao et al., 2018) adds texture features and contour features into the feature space, at the same time, the proportions of colour features, spatial features, texture features and contour features in the clustering process are determined by certain rules, which can optimize the classification method and improve the convergence speed.LSC (Li & Chen, 2015) obtains superpixels by linear spectral clustering, reduces the time cost compared with Ncut algorithms, while in the clustering process, it needs to build the ten-dimensional features, it is more time costing compared with SLIC.DBSCAN is a fast clustering method, some scholars use it for superpixel segmentation (Hou et al., 2014;Manavalan & Thangavel, 2011;Shen, Hao, et al., 2016).Compared with algorithms based on K-means, this kind of algorithm is faster due to the good clustering centre selection, and can meet the need for realtime processing.Medoid shift is also used for superpixel segmentation (Vedaldi & Soatto, 2008).A Gaussian kernel function is constructed to estimate the image density, but the number of superpixels can not be decided as needed.Additionally, some scholars regard the pixels in the superpixel as the samples that obey a distribution (Ban et al., 2018), then get the estimated parameters by the expectation maximization(EM) algorithm.Based on the idea of density peak clustering (Rodriguez & Laio, 2014), some scholars use the method of building a tree to segment superpixels (Zhang et al., 2020), it is fast and real-time, but the irregular superpixels can not adhere well near the boundary.In general, some algorithms require many iterations to get the clustering centres or parameters, which cost more time, but the superpixels are irregular and can not adhere well to the boundary.A good superpixel segmentation algorithm should meet the accuracy, regularity, and realtime need.
In this paper, our purpose is to construct a superpixels segmentation method that can keep a balance in accuracy, regularity and time cost.The main contribution of this paper can be highlighted as follows: (1) density image is calculated to evaluate the clustering ability of each pixel in the image by the peaks density clustering method.(2)seeds(cluster centres) are obtained by the density image rather than iteration as the most algorithms use in the clustering process.The paper is organized as follows.The superpixels segmentation method is introduced in Section 2. In Section 3, the experiment and analysis are presented.Finally, the conclusion is given in Section 4.

The superpixels segmentation based on density image
The main idea of the proposed method in this paper is mainly derived from density peak clustering (Rodriguez & Laio, 2014), which provides a new clustering algorithm and gets a good clustering result on nonspherical data.In general, the density peak clustering is that the seeds(clustering centres) must be the data that have high density and are far from other data with high density.It is used to evaluate the clustering ability of the pixel and select the seeds in this paper.

Calculating the density image
Let p i stand for the pixel of the input image to be segmented, W and H denote the width and the height of the image, respectively, (x i , y i ) stands for the position of the pixel p i on the image plane, (l i , a i , b i ) denotes the colour information of the pixel p i in the Lab colour space, K is the desired number of superpixels.When K is specified, the average size of superpixels is achieved by (W • H)/K, then s denotes the side length of the square area and can be obtained by where * denotes the integer rounded up of * .The input image can be approximately segmented as uniform squared subblocks with side length s.To obtain the density image, we choose the local square centred at p i with a side length of s, the distance between two pixels is calculated by where m stands for the weight of space distance, it satisfies m = c/s, c is a constant, then the indicator function is obtained by where T denotes the constraint constant, then the density of the pixel p i can be obtained by where R denotes the square centred at the pixel p i , N denotes the number of pixels in the local square centred at the pixel p i .It is similar to the method of obtaining the density of the data in density peak clustering, but some changes are made by normalizing the density in this paper.A higher density means the pixel is more likely locates in a flat area.On the contrary, a lower density means the pixel is near the boundary.Additionally, it is found that the constant T has a great influence on the density image.The smaller T is, the more black the density image is, which means a more ambiguous boundary.And vice versa, a big T makes it difficult to distinguish the clustering ability of pixels.Figure 1 is the density image with different T values.(a) is the input image, (b), (c), (d) are the density images at T of 4,8,12, respectively.In summary, a bigger or smaller T is neither suitable for further clustering.In this paper, the T value is adaptively chosen according to the K. Furthermore, it is related to s, T is set 0.4s in the experiment.A density image in high quality contributes to the accurate segmentation and low complexity.

Choosing the seeds
The seeds need be chosen as the clustering centres.In each pre-segmented square block, the density of each pixel is sorted in ascending order, and the pixels in a certain density range are selected as the candidate seed rather than the pixel that has the maximum density in density peak clustering.It is found that the density in the flat block varies little, so choosing the maximum is insignificant.In the experiment, the pixels whose density in the middle range are selected as candidate seeds.
For each candidate seed, the distance is summarized between the candidate seeds and the seeds in surrounding subblocks.The seeds which have been decided in surrounding subblocks are chosen as the surrounding seeds, then one of the candidate seeds is chosen as the seed in the present subblock if its summation of the distance has the maximum value.Thus, it can make the seeds distribute more uniformly, furthermore, and contribute to the regular superpixels.A k , B k stand for the set of pixels and the candidate seeds in the kth subblock of the image, respectively.
where x k t , x k d , y k l , y k r denote the upper boundary, lower boundary, left boundary and right boundary of the kth subblock, respectively.t 1 (k),t 2 (k) are constraint parameters, which are fourth deciles and sixth deciles of the ordered density in ascending order in the kth subblock.

ds(p
where p i ∈ B k , z j i denotes the seed in the jth subblock, and N k denotes the set of the seeds in surrounding subblocks.Then, the seed of the kth block is chosen as Thus, the seeds are obtained in the simple way rather than the iteration.It can greatly reduce the time cost.

Clustering for the superpixels
After the seed is selected, each pixel of the image should be given a unique label.In each subblock, the clustering area is extended to guarantee the superpixels more accurate.The clustering area of the kth superpixel is denoted by where stands for the location of the seed z k i in the image plane.A larger initial value w(p i ) is given to p i as the dist.And then the distance between the pixels and the seed is calculated by where d c , d s , d g represent the colour distance, space distance, and gradient distance, respectively.n is the weight of the gradient distance, g i is the gradient of the pixel p i , (l k i , a k i , b k i ), g k i are the colour information in Lab colour space and the gradient of the seed z k i , respectively.A sixdimensional feature is introduced into the clustering.If ), the label of the pixel p i is updated as k and w(p i ) is update by d 2 (p i , z k i ) at the same time, where t 0 is a constraint constant, it is set t 0 = 3 + 0.4s in the experiments.In this way, the pixel can be assigned to the nearest superpixel, which can make the superpixels adhere to the boundary better.The space and gradient distances can ensure the regularity and accuracy of the segmentation.

Merging the small superpixels
After the clustering is completed, some pixels can not be labelled, including some noise and the ones near the boundary.Then the connected and unlabelled pixels are assigned to the same label.Thus, new superpixels are constructed.Besides, some small superpixels should be merged into the surrounding superpixels.To estimate the small superpixel that should be merged, it is important to estimate the size of the superpixel.So the size of superpixels is sorted in descending order at first.Then, the superpixels are classified as small superpixels when whose size is smaller than the qth tercile or much less than the average size in superpixels.Finally, the small superpixels are merged into the most similar superpixels around them.The proposed algorithm can be summarized as follows: Input : The image to be segmented, the desired superpixel number K.
Output: The label of each pixel.
(1) Segmenting the image uniformly according to the number K; (2) Calculating the density image of the input image by the formula (1), ( 2), ( 3), ( 4); (3) Seeds are obtained by the density image; (4) A unique label is given to each pixel by clustering; (5) Giving the label for the pixels which are not labelled in the previous clustering; (6) Merging the small superpixels into the neighbour superpixels.

Experiment and analysis
The visual effects, accuracy, and computation complexity of the algorithm are estimated.TP (Levinshtein et al., 2009), DBSCAN (Shen, Hao, et al., 2016), and density peak clustering(DPC) (Zhang et al., 2020) are used to compare with the proposed algorithm.They have the same computation cost O(W * H).The codes of DBSCAN and TP are public and used in the comparison, and all the parameters of them are recommended by the author.The images of all experiments in this paper are from Berkeley Segmentation Database (Martin et al., 2001).

Visual effect comparison
The visual segmentation results of the algorithms are shown in the Figure 2.And the proposed algorithm can reach a better segmentation effect compared with other three algorithms on boundary cohesion and regularity, which are important for further processing.Besides, the results show that DPC behaves worst on regularity and TP behaves worst on boundary cohesion.Some details of the superpixels of the proposed algorithms are shown when K = 200.In Figure 3, it shows good segmentations near the boundary.In fact, K can be taken any value as needed.
A smaller K can make the segmentation less detailed, In Figure 2, K is chosen at 300 for visual comparison.In Figure 3, to show the accuracy better, a smaller value is chosen.

Accuracy evaluation
BR(Boundary Recall), UE(Under Segmentation Error) and ASA(Achievable Segmentation Error) are used to evaluate the segmentation accuracy in this paper.They are widely used to estimate superpixels segmentation accuracy.

BR =
p i ∈B(gr j ) I( min where B(su i ) and B(gr j ) represent the sets of pixels for boundaries of superpixels and ground truth segment respectively, I() is the indicator function, is a constant.

Complexity analysis
The computation complexity is also an indictor that should be taken into consideration for superpixel segmentation.DBSCAN, TP, and DPC have the same computation complexity O(W * H), which is the lowest until now.The computation complexity of the proposed method is also O(W * H).For each pixel, the density can be calculated by the formula (2), ( 3), (4).N R is the number of pixels in the area R centred at the pixel p i , and the complexity is O(N R * W * H).Next,each superpixel needs to select seed by formula ( 7), ( 8), the complexity is O(K).
Then, each pixel is given a label by clustering into the nearest superpixel, and complexity is O(W * H).Because W * H is far greater than K. Finally, the total complexity is O(W * H).

Conclusion
An alternative superpixel segmentation method is proposed in this paper, it can efficiently segment an image into accurate and regular superpixels.The main idea of this method is to construct a density image and to find the best seeds(cluster centres) for clustering instead of iteration as some algorithms have been used.Thus, the computation complexity can be greatly reduced.Besides, it solves the problem of poor segmentation effect caused by random selection of seeds like DBSCAN, and improves the segmentation accuracy.In other words, the regularity, accuracy, and time complexity are taken into consideration in the proposed method.Additionally, the proposed algorithm can obtain the desired number of superpixels, which is of great significance in later processing.The proposed algorithm is mainly suitable for the segmentation of colour image, the gray image is not considered, which may be the next research direction.

Figure 3 .
Figure 3.The segmentation effect with 200 superpixels and the details of local area.
| su i ∩ gr j |> v. | su i |).| su i | − n j=1 | gr j | n j=1 | gr j | (15) where | su i |, | gr j | stands for the size of the superpixel and ground truth, respectively, v is a constant.evaluate the boundary coincidence of the segmentation.UE measures the segmentation that exceeds the boundary.ASA compares the largest overlapping area of superpixels with the ground truth segments.The average BR, ASA, and UE of the images in BSD500 are shown in Figure 4, respectively.All indictors of the proposed algorithms have outstanding advantages over other algorithms, which means the best segmentation effect in the four algorithms.

Figure 4 .
Figure 4.The accuracy evaluation of different index.(a)The average BR; (b)The average ASA and (c)The average UE.