SIFT Matching with CNN Evidences for Particular Object Retrieval

Abstract

Many object instance retrieval systems are typically based on matching of local features, such as SIFT. However, these local descriptors serve as low-level clues, which are not sufficiently distinctive to prevent false matches. Recently, deep convolutional neural networks (CNN) have shown their promise as a semantic-aware representation for many computer vision tasks. In this paper, we propose a novel approach to employ CNN evidences to improve the SIFT matching accuracy, which plays a critical role in improving the object retrieval performance. To weaken the interference of noise, we extract compact CNN representations from a number of generic object regions. Then a query-adaptive method is proposed to choose appropriate CNN evidence to verify each pre-matched SIFT pair. Two different visual matching verification functions are introduced and evaluated. Moreover, we investigate the suitability of fine-tuning the CNN for our proposed approach. Extensive experiments on benchmark datasets demonstrate the effectiveness of our method for particular object retrieval. Our results compare favorably to the state-of-the-art methods with acceptable memory usage and query time.

Introduction

This paper considers the task of particular object retrieval. Given a query image in which a particular object has been selected, the retrieval system should return from its corpus a set of relevant images in which that object appears. This is a harder problem than whole-image retrieval, since the query object may be cluttered or occluded with diverse backgrounds in the returned images. Some examples are shown in Fig. 1.

Most object retrieval systems are based on matching of local features. There are two steps for extracting local features. The first step detects keypoints of interest. This step also delineates a local patch around every keypoint and normalizes the local patch into fixed-size. The second step describes the normalized patches based on algorithms such as SIFT [1]. However, matching the 128-dimensional SIFTs directly costs a lot of time. To speed up the matching process, bag-of-words (BoW) model is widely used [2], [3]. BoW model defines a visual dictionary and quantizes the local features to visual words. An image can be represented by a frequency histogram of visual words. Local features are matched if they are quantized to the same visual word, so the similarity between two images can be expressed by the inner product of their BoW representations. Inverted index which exploits the sparsity of BoW representation also makes the search efficient. Since matching the SIFTs based on visual words may be too coarse, Jégou et al. [4] propose Hamming embedding (HE) to improve the matching accuracy. HE adds a compact binary signature to each SIFT when quantizing it. Then two coarsely matched SIFTs will be filtered out if their Hamming distance between binary signatures is larger than a threshold.

However, SIFT only describes the local gradient distribution and serves as a low-level representation, which is often not sufficiently distinctive to prevent false matches. As shown in Fig. 2, some local patches are very similar in the SIFT feature space, but they depict different contents. This challenging problem is mainly due to the semantic gap. So seeing the big picture and adopting semantic clues may be a good solution to bridge the semantic gap.

Recently, deep convolutional neural networks (CNN) have been proven to achieve state-of-the-art performance in many computer vision tasks such as image classification [6], [7], object detection [8] or semantic segmentation [9]. With the deep architectures, semantic abstractions that are close to human cognition can be learned. A number of recent works show that CNN features trained on large and diverse datasets such as ImageNet [10] can be used to solve tasks for which they have not been trained. Particularly for image retrieval, many works have adopted solutions based on off-the-shelf features extracted from a pre-trained CNN, achieving promising performance on popular benchmarks [11], [12], [13], [14].

In this paper, we propose to adopt the semantic-aware CNN features to improve the SIFT matching accuracy, which helps to improve the particular object retrieval performance. Considering that global CNN representation is sensitive to background clutter, which is common in object retrieval (see Fig. 1), we extract CNN feature at the object-level. We detect a number of potential objects in the images and extract CNN feature on each object. For each pre-matched SIFT pair, we choose appropriate semantic evidence from the candidate CNN features in a query-adaptive manner, and use them to verify the SIFT match quality. By fusing low-level and high-level clues, high visual matching accuracy can be achieved.

The major contributions of this paper are summarized in the following aspects. First, we adopt the object-level CNN features to improve the SIFT matching accuracy. A query-adaptive method is proposed to choose appropriate CNN evidences to verify the SIFT match quality. Second, two different visual matching verification functions are introduced. We evaluate these two functions and show that they have different effectiveness for different types of datasets. Third, we explore the suitability of fine-tuning the CNN to obtain better semantic evidences for our proposed method. Extensive experiments on benchmark datasets demonstrate that our results compare favorably to the state-of-the-art methods with acceptable memory usage and query time.

The remainder of the paper is organized as follows. Related works are reviewed in Section 2. We show our approach in details in Section 3. After that, we provide the experimental results and comparisons in Section 4. Final conclusions are in Section 5.