Fine-residual VLAD for image retrieval

Abstract

This paper revisits the vector of locally aggregated descriptors (VLAD), which aggregates the residuals of local descriptors to their cluster centers. Since VLAD usually adopts a small-size codebook, the clusters are coarse and residuals not discriminative. To address this problem, this paper proposes to generate a number of residual codebooks descended from the original clusters. After quantizing local descriptors with these codebooks, we pool the resulting secondary residuals as well as the primary ones to obtain the fine residuals. We show that, with two-step aggregation, the fine-residual VLAD has the same dimension as the original. Experiments on two image search benchmarks confirm the improved discriminative power of our method: we observe consistent superiority to the baseline and competitive performance to the state-of-the-arts.

Introduction

We are interested in the problem of near duplicate image retrieval in large scale databases. Specifically, given a query image of object or scene, we aim to retrieve similar images from a large scale database with high search accuracy, efficiency and small memory usage.

Many state-of-the-art image retrieval systems adopt the Bag-of-Words (BoW) relying on local invariant features such as SIFT [1], [2]. In the BoW model, codebook is generated by clustering local features with unsupervised k-means algorithms [3], [4]. Then each local descriptor is quantized to its closest visual word. Based on the quantization results, an image is represented by a high-dimensional sparse histogram. Each visual word is weighted using the TF-IDF scheme [5], [6] and inverted lists can be employed to implement fast retrieval.

Nevertheless, when considering very large databases, search time and memory requirement limit the number of images that can be indexed in practice. Aggregated vectors such as Fisher Vector [7] and VLAD [8] address this problem by encoding an image into a single vector, achieving reasonable trade-offs on both search accuracy and efficiency. In both representations, a small-size codebook is used and the accumulation of residuals on each visual word is concatenated into a single vector. Residual is the difference vector between local descriptor and its assigned visual word. In the following, we use “residual” and “difference vector” interchangeably. These methods transform a set of local descriptors into a fixed-length vector and obtain better performance than the BoW with the same dimension. Usually, the dimension of VLAD vector can be reduced to 128-D by Principal Component Analysis (PCA) for fast search. Since VLAD can be seen as a simplified version of the Fisher Vector, we only focus on the former in this paper.

Due to the large quantization error of small-size codebook, the residual is not very discriminative, limiting the accuracy of VLAD descriptor. Quantization error can be reduced by increasing the codebook size. However, the VLAD descriptor with large-size codebook is high-dimensional and needs more search time and memory usage. It sacrifices efficiency for improving accuracy. Besides, high-dimensional representation suffers more from dimensionality reduction. Considering this, we aim at improving the search accuracy while preserving the search time and the memory usage.

In our approach, we propose a number of residual codebooks descended from the original clusters. Then local descriptors assigned to the same cluster are distinguished by their residuals and divided into finer clusters. Through these codebooks, we calculate the difference vector between primary residual and its closest visual word in the residual codebooks, denoted as the secondary residual. By pooling them with the primary ones, the fine residuals with more discriminative information are obtained. Furthermore, the fine residuals are aggregated into one vector through the two-step aggregation, keeping the same dimension as the original. The examples of image retrieval using fine-residual VLAD are showed in Fig. 1.

The remainder of the paper is organized as follows. In Section 2, we briefly review the related works. In Section 3, we describe our method in detail. Experiments are shown in Section 4. Finally, we conclude in Section 5.