Local mesh quantized extrema patterns for image retrieval

In this paper, we propose a new feature descriptor, named local mesh quantized extrema patterns (LMeQEP) for image indexing and retrieval. The standard local quantized patterns collect the spatial relationship in the form of larger or deeper texture pattern based on the relative variations in the gray values of center pixel and its neighbors. Directional local extrema patterns explore the directional information in 0°, 90°, 45° and 135° for a pixel positioned at the center. A mesh structure is created from a quantized extrema to derive significant textural information. Initially, the directional quantized data from the mesh structure is extracted to form LMeQEP of given image. Then, RGB color histogram is built and integrated with the LMeQEP to enhance the performance of the system. In order to test the impact of proposed method, experimentation is done with bench mark image repositories such as MIT VisTex and Corel-1k. Avg. retrieval rate and avg. retrieval precision are considered as the evaluation metrics to record the performance level. The results from experiments show a considerable improvement when compared to other recent techniques in the image retrieval.

based on color have been proposed by researchers, a few of these are outlined here in this section. A method based on color histogram and histogram intersection has been proposed by Swain and Ballard (1991). Stricker and Orengo (1995) introduced two novel color indexing methods using cumulative color histogram and color moments. A vector quantization method has been proposed by Idris and Panchanathan (1997) to obtain a code word, which becomes a feature vector. Similarly, Lu and Burkhardt (2005) introduced an integrated method to color image retrieval by combining vector quantization method and discrete cosine transform. A popular image compression technique called block truncation coding (BTC) has been introduced in Qiu (2003) to explore two main features i.e. block pattern histogram and block color co-occurrence matrix. Hauang et al. (1997) introduced color correlogram method for image retrieval. By incorporating spatial information, Pass and Zabih (1996) devised the color coherence vectors (CCV). In order to extract spatial information, Rao et al. (1999) made modification to color histogram. As part of it, three color histograms annular, angular and hybrid color histograms have been used. By taking care of the issues like change of illumination, orientation and view point geometry with reduced length of feature vector, a modified color co-occurrence matrix has been proposed by Chang et al. (2005) Another prominent feature in CBIR is texture. Variance and mean values of the wavelet coefficients have been used to describe texture features in image retrieval by Smith and Chang (1996). Ahmadian and Mostafa (2003) devised the wavelet transform to classify the texture. Subrahmanyam et al. (2011) modified the concept of correlogram using wavelets and rotated wavelets (WC + RWC). A spatial computation method called local binary pattern (LBP) was introduced by Ojala et al. (1996) and later local binary patterns have been enhanced for rotational invariant texture classification (Ojala et al. 2002). Pietikainen et al. (2000) proposed the rotational invariant texture classification using feature distributions. Zhao and Pietikainen (2007) applied LBP in face recognition and analysis. Heikkil and Pietikainen (2006) used LBP for background modeling and detection by using LBP. Li and Staunton (2008) proposed a combination of LBP and Gabor filter for texture segmentation. By considering LBP as a non-directional 1st order spatial pattern, Zhang et al. (2010) presented local derivative pattern for face recognition areas. A modified version of LBP called center-symmetric LBP, combined with scale invariant feature transform (SIFT) has been used to describe interest region in Heikkil et al. (2009). An extension of LBP concept based on edge responses named Local directional pattern was proposed by Jabid et al. (2010). Subrahmanyam et al. (2012) designed different types of spatial pattern based feature descriptors, local tetra patterns (LTrP) local maximum edge patterns (LMEBP) (Subrahmanyam et al. 2012), directional local extrema patterns (DLEP) (Subrahmanyam et al. 2012) for natural or texture image retrieval. Directional binary wavelet patterns (DBWP) (Subrahmanyam et al. 2012), local mesh patterns (LMeP) (Subrahmanyam and Wu 2014) and local ternary co-occurrence patterns (LTCoP) (Subrahmanyam and Wu 2013) for biomedical image retrieval. Reddy and Reddy (2014) added magnitude information to DLEP to enhance the performance of retrieval system. To address few issues of LBP, Hussain and trigges (2012) introduced local quantized patterns (LQP) for visual recognition. Verma et al. (2015) proposed an integrated approach based on local patterns to natural, biomedical images (Koteswara Rao and Venkata Rao 2015). Vipparthi et al. integrated Local patterns and Gabor feature to propose a feature descriptor (Verma et al. 2015).
In recent times, combination of texture and color features has been proved effective in image retrieval. Jhanwar et al. (2004) introduced the motif co-occurrence matrix (MCM) for content-based image retrieval. They applied the MCM on blue (B), red (R) and green (G) color planes. In Vadivel et al. (2007) combined color and intensity co-occurrence matrix. After analyzing the features of HSV space, an appropriate weight values have been suggested to extract relative share of color and intensity levels of a pixel.
Recent methods on spatial patterns LQP (Hussain and Triggs 2012) and DLEP (Subrahmanyam et al. 2012) have motivated us to propose the local mesh quantized extrema patterns (LMeQEP) for image indexing and retrieval systems. Primary contributions of the work are briefed as follows.
(1) Proposed method extracts a mesh quantized HVDA structure from an image (2) Directional extrema information is collected from the set of mesh patterns to create LMeQEP (3) To achieve the improvement in the performance, LMeQEP and RGB color histogram are integrated (4) Experiments are conducted on standard datasets of images for natural and texture image retrieval.
The paper is organized as follows: a brief review of image retrieval and related work is provided in Sect. "A review of existing spatial local patterns". Section "Local quantized extrema patterns (LQEP)" provides a detailed review of existing feature extractors based on local patterns. A framework for image retrieval and similarity measures are depicted in Sect. "Local mesh quantized extrema patterns (LMeQEP)". The results of the experiments and discussions are given in Sect. "Experimental results and discussion". Conclusions and future directions are given in Sect. "Conclusions".

Local binary patterns (LBP)
A modified form of texture unit called LBP operator was proposed by Ojala et al. (1996) for texture analysis. Some of the specific features such as speed (as there is no requirement to set parameters), simplicity made LBP prominent in many research directions. The performance is observed in multiple areas of research such as face recognition and analysis (Pietikainen et al. 2000;Zhao and Pietikainen 2007), object tracking (Subrahmanyam et al. 2012), texture classification (Ojala et al. 1996(Ojala et al. , 2002Pietikainen et al. 2000;Zhao and Pietikainen 2007), content based retrieval systems (Subrahmanyam et al. 2012a, b, c, d;Wu 2014, 2013;Reddy and Reddy 2014) and finger print recognition. A pixel at the center becomes the threshold to yield a local binary pattern in a small 3 × 3 arrangement of spatial structure. The computation of LBP of a center pixel w.r.t its neighbors is done according to Eqs. (1) and (2): where I(n c ) is intensity value of center pixel, I(n p ) is gray level of its surrounding elements, X represents the no. of neighbors where as Y is the length of the neighborhood. (1) Subsequent to derivation of LBP for each pixel (j, k), a histogram is built to represent the whole image as per the Eq. (3).
where the size of input image is X 1 × X 2 .
Calculation of LBP for a 3 × 3 neighborhood is given in Fig. 1. The occurrence of edges in the image is depicted by histograms and the histograms show the information pertaining to edge distribution.

Center-symmetric LBP (CS_LBP)
A center symmetric sets of pixels are considered instead of the existing center pixelneighbor comparison by Heikkil and Pietikainen (2006). The computation of CS_LBP is done as per the Eq. (5): Subsequent to calculation of CS_LBP for each pixel (j, k), a histogram is built to represent the extracted data in a similar way to LBP. The histogram is considered as the feature vector for retrieval. Jabid et al. (2010) have presented LDP for human face recognition. Relative edge response of a pixel in different directions is the key idea behind LDP. Kirsch masks are used to derive this spatial pattern. Eight masks are employed to extract responses in eight directions. High response values in particular direction indicate the presence of edge or corner. To get the information about the spatial location along with the pattern, the image is divided into regions. Subsequent to derivation of local pattern, a histogram with 56 bins is built to represent various values of the encoded image.

Local quantized patterns (LQP)
Hussain and Trigges (2012) have introduced LQP for visual recognition. The directional geometric information is extracted in vertical, horizontal, anti-diagonal, diagonal strips 2 (x−1) × g 1 I(n X ) − I(n x+(X/2) ) Fig. 1 Calculation of LBP of pixels along conventional circular and disk-shaped areas. Figure 2 depicts the possible arrangement of quantized geometric structures for LQP calculation. The more details about LQP are available in Vipparthi et al. (2015).

Directional local extrema patterns (DLEP)
To derive the spatial structure of local texture, Subrahmanyam et al. (2012) proposed directional local extrema patterns (DLEP) for image retrieval system. Key idea behind DLEP is the extraction of local extrema of a center pixel g c .
In DLEP, local extrema values in 0°, 90°, 45° and 135° directions are taken out by considering the local variation in the values of center pixel and its neighbors as given below: Local extremas are calculated according the Eq. (7) Subsequently, DLEP is defined in β = 0°, 45°, 90°, and 135° as given below:

Local quantized extrema patterns (LQEP)
The operators DLEP (Jhanwar et al. 2004) and LQP (Vipparthi et al. 2015) are integrated to propose the LQEP (Koteswara Rao and Venkata Rao 2015) for image retrieval. In the first step, the possible structures are extracted from the considered pattern of an image. Then, local extrema method is applied on the extracted geometric structures in four major directions. As shown in the Fig. 3, pixels in 5 × 5 pattern are indexed with arbitrary values to enable user understanding. HVDA 5 geometric structure is used for feature extraction in the presented work.

Local mesh quantized extrema patterns (LMeQEP)
The proposed method (LMeQEP) collects the spatial texture information from an image based on mesh and LQEP concepts. More discriminative information can be collected by forming the mesh with the pixels at alternate positions still without losing the connectivity in pixel information. The LMeQEP calculation is given in Fig. 4. A method to collect HVDA 5 geometric structure for a given center pixel (P) in an image I is given in Fig. 4. Directional extremas in four directions i.e. 0°, 90°, 45°, and 135° are derived as follows.

Proposed image retrieval system
In this paper, we propose a novel feature descriptor for image retrieval by applying the mesh concept on the extracted geometric structure. First, the image is loaded and converted into gray scale in case if it is RGB. The four directional HVDA 5 structure is collected using the LQP method. Calculation of extremas in 0°, 90°, 45° and 135° directions is done on the mesh as given in Eqs. (10) to (13). Finally, the LMeQEP feature is generated by constructing the histogram. In order to enhance the performance of proposed method, LMeQEP is integrated with color RGB histogram.

Query matching
Followed by feature extraction, feature vector for query image is represented as In the similar way, feature vector of every image in database is represented as f DS j = (f DS j1 , f DS j2 , . . . f DS jLg ); j = 1, 2, . . . , |DS|. The primary requirement is to identify n best images those have close resemblance to the query image. This is achieved by measuring the distance between query and database image|DS|. We have used four different distance metrics as given in Eqs. (17)-(20).
Here, f DS ji represents ith feature of jth image in database |DS|.

Experimental results and discussion
To analyze the performance of LMeQEP method, experimentation is done with standard image databases. It is specified further that Corel-1k (Corel 1000), and MIT VisTex databases are used in this process (http://vismod.media.mit.edu/pub/). In the process of experimentation, each image in the database is treated as query image. The system returns 'n' database images M = (m 1 , m 2 …m n ) for each query as per the distance measured as per the Eq. (17). When the resultant image m i = 1, 2…n is related to the category of query image, it can be established that the retrieval system is correct, otherwise it is treated as a failure.
Avg. retrieval precision and avg. retrieval rate are considered to evaluate performance of the proposed LMeQEP as shown below: The precision for a query image is specified as follows:

Experiment no. 1
MIT VisTex database that contains images of forty varying textures (http://vismod. media.mit.edu/pub/) is used in the present work. As specified earlier, each image in database is treated as query image. Avg. retrieval precision, avg. retrieval rate mentioned earlier are used to compare the results. Figure 6 depicts the performance of different methods against ARR on MIT VisTex database. It is obvious that proposed LMeQEP shows a substantial increase in the performance when compared to related approaches in terms of ARR on MIT VisTex image database. Figure 7 provides the query results of proposed LMeQEP on MIT VisTex dataset.

Experiment no. 2
Corel-1k database that contains large number of images with varying content Verma et al. (2015) is considered for the experiment. All the images have been pre-categorized into various classes each of size 100 by domain experts. 1000 images are collected from 10 different domains containing 100 images per domain. Avg. retrieval precision as provided in Eq. (22) is considered to evaluate the performance of our method. Figure 8 shows sample images of Corel-1k database. Figure 9 illustrates the retrieval results of proposed LMeQEP and other existing methods w.r.t ARP on Corel-1k database which depicts that the LMeQEP method shows a (21) Pre − P(I q ) = No. of relevant images retrieved Total no. of images retrieved No. of relevant images retrieved Total no. of relevant images in the database . 6 a, b, c Comparison of proposed approach and other related methods in terms of ARR with MIT VisTex image data base substantial increase in ARP values as compared to other recent methods. Figure 10, depicts comparison of the precision Vs recall of various methods on Corel-1k database. From Fig. 10, it is obvious that the proposed LMeQEP method outperforms the other existing methods on Corel-1k database. Figure 11 shows retrieved images for a query image using Corel-1k database.

Conclusions
A novel image feature descriptor named local mesh quantized extrema patterns (LMeQEP) for texture and natural image retrieval is presented in this paper. A mesh is created out of the quantized geometric structure. The directional extrema information in specified directions is extracted from the quantized mesh of the pattern. By creating a mesh from a pattern, it has been found that more discriminative information associated with each pixel can be extracted. In order to enhance the performance of proposed method further, color feature in the form of RGB histogram is added to form the feature vector for retrieval system. The effectiveness of proposed method is tested by