Salient region detection for complex background images using integrated features
Introduction
One common thread in the recent literature is the increasing in methods that simulate human biological systems, including neural network model [8], decision model [3], visual attention model [20], and so on. Visual attention results both from fast, unintentionally bottom-up visual saliency of the retinal input, as well as from slower, top-down memory and volition based processing that is task-dependent [2]. Since the bottom-up saliency does not require any prior knowledge and runs faster than the top-down saliency, it is widely applied in image segmentation [9], adaptive image display, image/video compression [25], [14], sketch retrieval, image retrieval [23], etc. We focus on the bottom-up saliency module in this paper.
In general, bottom-up saliency module can be classified into local and global methods. Local methods determine saliency value of pixels based on local contrast (i.e. center-to-surrounding difference) while global methods consider center-to-global differences or suppress frequently occurred features. For salient objects in simple background images, local contrast is equivalent to global contrast due to the equivalence relation between the center-to-surrounding difference and the center-to-global difference. As a result, both local and global methods work well on simple background images. As for the images with complex background, however, neither of them performs well. Moreover, several issues have been observed with the application of two methods (see Fig. 1):
- •
Problem 1: Local methods highlight the boundaries more than the content of salient regions (red boxes in Fig. 1).
- •
Problem 2: Global methods tend to undervalue big salient region or treat big salient region as background (blue boxes in Fig. 1).
- •
Problem 3: In complex background image, the background usually includes various areas that are visually different. Local methods offer high saliency value to pixels on the border of these areas (green boxes in Fig. 1).
- •
Problem 4: In complex background image, the background usually contains tiny patches with excessively infrequent visual features. As human perspective, they are too sparse to attract attention. Nonetheless, global methods may produce high saliency value to these noises (yellow boxes in Fig. 1).
To overcome these issues, we conclude five characteristics of salient regions in complex background images (see Table 1).
According to these principles, we propose a novel saliency detection method to solve the above problems. Our method consists of three parts:
- 1.
Local part: we detect pixels with high local visual contrast as source set based on Principle 1, then extend this set to candidate set based on Principle 2.
- 2.
Global part: we propose a novel measure to assign visual saliency values to candidates based on Principles 3 and 4.
- 3.
Spatial part: we propose a simple, fast algorithm which computes spatial weights based on the estimation of the smallest circumscribed circle containing all salient regions. This part follows Principle 5.
The final saliency value of pixels is considered to be the combination of visual saliency value and spatial weight.
The rest of the paper is organized as follows. Details of the proposed algorithm is presented and analyzed in Section 2. In Section 3, we perform extensive experiments to evaluate our method on two benchmark image sets. One image set consists of natural scene images with complex texture background. The other is one of the largest image sets with relatively simple background. Applications of our method in sketch retrieval are also introduced in Section 3. The related work is reviewed in Section 4 and concluding remarks are presented in Section 5.
Section snippets
Salient region detection using integrated features
In order to distinguish salient regions from complex background images, we propose a novel saliency detection method satisfying Principles (1)–(5). The framework of our method is shown in Fig. 2, and details of the framework are presented as below:
- •
Local part (details in Section 2.1): In this part, we first generate source set which consists of pixels with high local visual contrast. Visual features of the source set are then used to generate candidate set, aiming at finding content pixels as
Experiment
In this section, we evaluate the effectiveness of the proposed approach. The experiment contains three parts. The first part evaluates the performance of the proposed approach by comparing with seven state-of-the-art saliency detection methods. The selection of comparing methods is refer to [7], [2]: citation (IT [17] is widely cited), variety (IT, AC [1], NS [22] are local methods, LC [28], RC [7], FT [2], SR [15] are global methods), recency (LC, SR, AC, FT, NS, RC are recent methods). The
Related work
In general, the bottom-up saliency detection approaches normally select saliency region on a basis of the unconscious stimulation by visual contrast.
Methods on the local contrast basis determine saliency regions by comparing them with surroundings. Itti et al. [17] used center-surrounding difference of intensity, color and orientation to yield feature maps, and then combined them into the final saliency map. It was the first practice with feature of integration to yield saliency map
Conclusion
This paper proposes five principles for saliency detection in complex background images. Based on these principles, a novel method using integrated features is proposed for detecting salient regions in complex background images. The integrated method computes saliency value of pixels by simultaneously considering pixel-to-neighborhood contrast, low-level feature frequencies and spatial coherence. The integrated method is simple, fast, and yields the same size saliency map with investigated
Acknowledgments
The work described in this paper was fully supported by the Grants from the National Nature Science Foundation of China (61271428, 61273247, 61303159); National Key Technology Research and Development Program of China (2012BAH39B02). This work was supported in part to Dr. Qi Tian by ARO grant W911NF-12-1-0057, NSF IIS 1052851, Faculty Research Awards by Google, FXPAL, and NEC Laboratories of America, and 2012 UTSA START-R Research Award respectively. This work was supported in part by NSFC
References (28)
- et al.
Fuzzy logic-based generalized decision theory with imperfect information
Inform. Sci.
(2012) - et al.
Pasar: an integrated model of prediction, anticipation, sensation, attention and response for artificial sensorimotor systems
Inform. Sci.
(2012) - et al.
Salient region detection and segmentation
- et al.
Frequency-tuned salient region detection
- et al.
Shape matching and object recognition using shape contexts
IEEE Trans. Pattern Anal. Mach. Intell.
(2002) - et al.
Saliency based on information maximization
Adv. Neural Inform. Process. Syst.
(2005) A computational approach to edge detection
IEEE Trans. Pattern Anal. Mach. Intell.
(1986)- et al.
Global contrast based salient region detection
- et al.
Genetic interval neural networks for granular data regression
Inform. Sci.
(2013) - et al.
A matrix-based approach to unsupervised human action categorization
IEEE Trans. Multimedia
(2012)
Efficient graph-based image segmentation
Int. J. Comput. Vis.
Context-aware saliency detection
IEEE Trans. Pattern Anal. Mach. Intell.
Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform
Cited by (18)
MCCH: A novel convex hull prior based solution for saliency detection
2019, Information SciencesCitation Excerpt :Visual saliency [13] is one of the classical ways to find regions of interest in the image. In the past many years, abundant efforts have been made on salient object detection [10,22,26,29,32,40,41,48,50]. The essence of the salient object detection is to highlight object(s) of interest in the image [22].
Discrete stationary wavelet transform based saliency information fusion from frequency and spatial domain in low contrast images
2018, Pattern Recognition LettersAutomated 3D segmentation of brain tumor using visual saliency
2018, Information SciencesCitation Excerpt :Such models seem to predict based on certain aspects of human eye movement [14]. Visual saliency can be defined as the outcome of comparing a region with its surrounding, with respect to unpredictability, contrast and rarity [1,25,26,38,43]. The comparison can be done in terms of low level feature similarity.
Learning visual saliency from human fixations for stereoscopic images
2017, NeurocomputingComplex networks driven salient region detection based on superpixel segmentation
2017, Pattern RecognitionCitation Excerpt :The implementation of region based methods can be realized in various ways, such as multi-dimension, histogram based contrast, color distribution, etc. By the way, some methods work on both local and global consideration, e.g., Goferman et al. [36], Zhang et al. [37]. Peng et al. [38] propose a saliency cut based on high-order energy term for stereo images.
Saliency-based stereoscopic image retargeting
2016, Information Sciences