Robust image retrieval with hidden classes

Highlights

• We address a new robust problem, named hidden classes, in content-based image retrieval (CBIR).

• We propose a new robust CBIR scheme using multi-image queries to tackle the difficule problem.

• We develop a novel query detection technique to separate queries as common or novel according to their underlying classes.

• We apply a self-adaptive retrieval strategy to handle different types of queries automatically.

• We design and carry out a number of experiments to demonstrate the effectiveness of our scheme.

Abstract

For the purpose of content-based image retrieval (CBIR), image classification is important to help improve the retrieval accuracy and speed of the retrieval process. However, the CBIR systems that employ image classification suffer from the problem of hidden classes. The queries associated with hidden classes cannot be accurately answered using a traditional CBIR system. To address this problem, a robust CBIR scheme is proposed that incorporates a novel query detection technique and a self-adaptive retrieval strategy. A number of experiments carried out on the two popular image datasets demonstrate the effectiveness of the proposed scheme.

Introduction

Content-based image retrieval (CBIR) is an active research area. The aim of various CBIR systems is to search images by analyzing their content. Images are normally described by their low-level features such as color, texture and shape [1], [2]. In the literature, a significant amount of research has been conducted relating to CBIR [3], [4]. However, the robustness of CBIR systems has not been sufficiently investigated even though the topic of robustness has been explored extensively in traditional information retrieval [5]. We have already identified and addressed unclean queries as a problem of robustness [6], however in this paper, we will study the hidden class problem of CBIR systems employing image classification as preprocessing.

The application of image classification techniques into a CBIR system results in a user’s queries being answered with images in predefined classes, thus helping to improve retrieval accuracy and speed. However, in a large-scale image collection, some image classes may be unseen [4]. We call these hidden classes as opposed predefined classes. The existence of hidden classes severely affects the retrieval accuracy of image classification based CBIR systems. There are two approaches that can address the problem of robustness. One approach is detecting hidden classes at the stage of preprocessing in order to avoid the problem of hidden classes when answering a query. The second approach is to take hidden classes into account when answering a query because different retrieval strategies can be adopted for different queries. We decided upon the second approach because it is too difficult to detect hidden classes during preprocessing without extra information.

Under the query-by-example (QBE) paradigm, there are three problems that arise due to hidden classes. When considering hidden classes, a user’s queries can be divided into two categories; a common query and a novel query. Fig. 1 illustrates a hidden class, common query and novel query. A common query can be answered using a predefined image class because relevant images of the common query have been gathered in this class. A novel query is associated with a hidden class and it cannot be answered using any predefined image classes. The first problem is how to identify whether a query is a common or novel query. This determination will influence the retrieval strategy. The second problem is how to predict a relevant predefined image class for a common query. The third problem is how to perform image retrieval for a novel query if it is not associated with any predefined image class. The solutions to these problems will result in a new retrieval scheme that can manage the problem of hidden classes.

In this paper, we aim to address the critical problem of hidden classes in CBIR systems. Our major contributions are summarized as follows.

• We propose a robust CBIR scheme that can incorporate multi-image queries and a support vector machine (SVM) to effectively deal with hidden classes.

• We develop a novel query detection technique to determine whether a user’s query is a common or novel query, therefore making it feasible to consider hidden classes in the retrieval process.

• We develop a self-adaptive retrieval strategy. For a common query, a relevant predefined image class will be predicted and the within images are ranked. For a novel query, a new method is proposed to filter out the irrelevant images before image ranking.

Finally, a number of experiments that were carried out on a Corel image dataset and the NUS-WIDE-LITE dataset [23] demonstrate the effectiveness of the proposed scheme. In particular, the improvement on precision depends on the number of hidden classes, with over 10% achieved.

The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 presents the novel CBIR scheme and discussion is provided in Section 4. In Section 5, the experimental evaluation and results are reported and the conclusion to this paper is presented in Section 6.