Unsupervised Deep Pairwise Hashing

Abstract

Although unsupervised deep hashing is potentially very useful for tackling many large-scale tasks, its performance is still far below satisfactory. Additionally, its performance might be significantly improved by effectively exploiting the pair similarity relationship among training data, but the attained similarity matrix usually contains noisy information, which often largely decreases the model performance. To alleviate this issue, in this paper, we propose a novel unsupervised deep pairwise hashing method to effectively and robustly exploit the similarity information between training samples and multiple anchors. We first create an ensemble anchor-based pairwise similarity matrix to enhance the robustness of similarity and dissimilarity relations between training samples and anchors. Afterwards, we propose a novel loss function to directly and robustly take advantage of the similarity and dissimilarity information via a weighted cross-entropy loss, and make use of a square loss to reduce the gap between latent binary vectors and binary codes, and another square loss to form consensus predictions of latent binary vectors. Extensive experiments on large-scale benchmark databases demonstrate the effectiveness of the proposed method, which outperforms recent state-of-the-art unsupervised hashing methods with significantly better ranking performance.

1. Introduction

Hashing has attracted considerable attention for tackling large-scale tasks because it can encode originally high-dimensional data into short binary codes while maintaining the similarity of neighbors, thereby leading to significant gains in computation and storage costs [1,2]. Hashing can be roughly categorized into two main classes, supervised and unsupervised, based on whether semantic labels are used. Supervised hashing [3,4] usually requires a large amount of labels to achieve satisfactory performance; however, label annotation is usually time-consuming and expensive. By contrast, unsupervised hashing [5] does not need semantic labels and aims to discover and, meanwhile, encode the significant intrinsic patterns or structures hidden in data into binary codes. Thus, unsupervised hashing has the potential for large-scale applications.

Early efforts focus on data-independent hashing methods [1,6], which utilize random projections or permutations to construct hash functions, and they usually require long bits to attain high precision per hash table and multiple tables to improve the recall. Data-dependent hashing usually produces more compact binary codes with higher precision and recall. Although numerous data-dependent hashing methods have been proposed and achieved promising performance on various similarity measures, such as Euclidean distance and ${\ell }_1$-norm distance [7], they are still far from being satisfactory for many tasks via the semantic similarity measure. Most of them [5,8,9] learn hash functions using hand-crafted features, which might not be able to represent the image content [10] optimally.

Recently, because convolutional neural networks (CNNs) exhibit the powerful capability of automatically learning feature representations, several unsupervised deep hashing methods [11,12,13,14,15] adopt CNNs to learn image representations and hash functions. Most of these methods [11,12,13] utilize either the quantization loss or data reconstruction error to train models without considering the similarity relationship among data, thereby decreasing their retrieval performance on some applications. To address this problem, similarity-adaptive deep hashing (SADH) [15] takes into account the pair similarity among training data and alternately proceeds over three major modules: training deep hash models, updating a similarity graph, and learning binary codes. Although SADH achieves better performance than most previous hashing algorithms, its performance might still be restricted by the noisy information in the similarity graph, i.e., positive values are often given to some dissimilar pairs. In addition, anchor-based models have made significant advances in semi-supervised deep hashing [16] and a scalable optimization mechanism has been proposed [17].

Motivated by the observations mentioned earlier, in this paper, we propose a novel, robust, yet straightforward method, unsupervised deep pairwise hashing (UDPH), to effectively and robustly utilize the pair similarity between training data and unlabeled anchors. The framework of the proposed UDPH is presented in Figure 1. The major contributions of this paper are listed as follows:

• Different from existing anchor-based methods, we enhance the robustness of similarity relations between training data and unlabeled anchors by creating an anchor-based pairwise similarity matrix to preserve their similarity and build a robust ensemble matrix with weighted average;
• We propose a novel loss function composed of three terms: a weighted cross-entropy loss to exploit the similarity information between training data and multiple anchors, a mean square loss to reduce the gap between latent binary vectors and desired codes, and another mean square loss to form consensus predictions of latent binary vectors;
• Extensive experiments on large-scale benchmark databases illustrate the superior performance of UDPH over recent state-of-the-art methods. Additionally, ablation experiments also demonstrate the effectiveness of the three terms in the proposed loss function.

Figure 1. The framework of the proposed method, UDPH, which utilizes the VGG-16 model as our backbone network. MSE denotes the mean square error/loss, and WCE means the weighted cross-entropy loss. WCE utilizes the similarity obtained from the ensemble pairwise similarity matrix as the weight, MSE with ${\gamma }_1$ is to calculate the loss between absolute values of latent binary vectors and vectors of ones, and MSE with ${\gamma }_2$ is to calculate the loss between latent binary vectors and target latent binary vectors. $f_{\theta }(·)$ represents a convolutional neural network and $\mathbf{H}$ denotes latent binary vectors. ${\alpha }_1$ and ${\alpha }_2$ are non-negative values to adjust the weight of $\mathbf{S}$ and $\mathbf{H}$, respectively.

Figure 1. The framework of the proposed method, UDPH, which utilizes the VGG-16 model as our backbone network. MSE denotes the mean square error/loss, and WCE means the weighted cross-entropy loss. WCE utilizes the similarity obtained from the ensemble pairwise similarity matrix as the weight, MSE with ${\gamma }_1$ is to calculate the loss between absolute values of latent binary vectors and vectors of ones, and MSE with ${\gamma }_2$ is to calculate the loss between latent binary vectors and target latent binary vectors. $f_{\theta }(·)$ represents a convolutional neural network and $\mathbf{H}$ denotes latent binary vectors. ${\alpha }_1$ and ${\alpha }_2$ are non-negative values to adjust the weight of $\mathbf{S}$ and $\mathbf{H}$, respectively.

2. Related Work

In this section, we briefly review some popular unsupervised non-deep and deep hashing algorithms, and introduce their differences to the proposed method, UDPH.

Unsupervised non-deep hashing usually adopts hand-crafted features to learn hash functions. The popular hashing algorithms [5,8,18,19] learn binary codes via the strategy of “relaxation + thresholding”, which might degrade their performance due to the accumulated quantization error between binary codes and its relaxed matrix. To alleviate this issue, numerous discrete hashing algorithms [9,20,21,22,23] have been proposed to directly learn binary codes.

Unsupervised deep hashing usually utilizes CNNs to learn image features and hash functions. Deep hashing (DH) [11], DeepBit [13], and unsupervised deep binary descriptors (UDBD) [24] mainly adopt the quantization loss to learn image representations and hash functions. Unsupervised hashing with a binary deep neural network (UH-BDNN) [12] utilizes the reconstruction loss to encourage the similarity among samples. Discriminative attributes representations (DAR) [25] firstly trains a CNN coupled with unsupervised discriminative clustering and then utilizes the cluster membership as a soft supervision to learn hash functions. Unsupervised triplet hashing (UTH) [26] exploits an unsupervised triplet loss to minimize the distance between an anchor image and its rotated version, while maximize the distance between the anchor image and a random image. HashGAN [14] adopts three networks including a generator, a discriminator, and an encoder to learn hash functions. Similarity-adaptive deep hashing (SADH) [15] constructs a similarity graph using the pair similarity between real training data and anchors, and learns hash functions via alternately proceeding over three major modules: training deep hash models, updating a similarity graph, and learning binary codes. Because of its effectively exploration of similarity information among training data, SADH has achieved state-of-the-art retrieval performance on CIFAR-10 [27] and NUS-WIDE [28] databases. However, its performance might be still restricted by the noisy information in the similarity graph. Compared to SADH, UDPH takes advantage of more robust similarity information by creating a strong ensemble anchor-based pairwise similarity matrix. Additionally, unlike SADH, which approximately solves an NP-hard problem to learn binary codes for model training, the proposed UDPH directly utilizes a novel loss function to robustly train models for exploring the semantic similarity information among training data and anchors.

3. Methodology

In this section, we firstly define an anchor-based pairwise similarity matrix and its ensemble version, and then propose a novel loss function to effectively and robustly exploit the similarity information between training samples and anchors for model training.

3.1. Anchor-Based Pairwise Similarity Matrix

Given training data $\mathbf{X}={\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$ and an L-layer deep hashing network $f_{\theta }(·)$ (please see Figure 1), n is the number of samples and $\theta$ denotes the network parameters. Note that we utilize $f_{\theta }^l\left({\mathbf{x}}_i\right)$ ($1\le l\le L$) to represent the output of the l-th layer for the sample ${\mathbf{x}}_i$. Because n is usually very large, it is inefficient or even impractical to calculate the pair similarity of any two training samples. Fortunately, the similarity relationship among training data can be propagated through multiple anchors [8,29]. We randomly select m$(m<<n)$ samples from $\mathbf{X}$ as anchors $\mathbf{A}={\left\{{\mathbf{a}}_j\right\}}_{j=1}^m$ ($\mathbf{A}\subset \mathbf{X}$) to construct a scalable anchor-based pairwise similarity matrix $\mathbf{S}\in {\mathbb{R}}^{n\times m}$. Additionally, because samples that are close in the feature space should be close in the output space (local consistency) [30], we select the $p\left(t\right)$ closest neighbors of each training sample from anchors and then calculate their similarities, where t is the current number of training epochs and $p\left(t\right)$ is a piecewise linear function dependent on t to gradually exploit more useful information. However, when only utilizing the similarity information to train models, the features will easily collapse together. To avoid this issue and, meanwhile, exploit more useful information, we select the $p\left(t\right)$ farthest anchors of each training sample as non-neighbors and then calculate their dissimilarities.

Specifically, let ${\mathbf{x}}^f=f_{\theta }^{(L-1)}\left(\mathbf{x}\right)\in {\mathbb{R}}^d$ and ${\mathbf{A}}^f=f_{\theta }^{(L-1)}\left(\mathbf{A}\right)\in {\mathbb{R}}^{m\times d}$ denote feature representations of one training sample $\mathbf{x}$ and anchors $\mathbf{A}$ at the ($L-1$)-th layer, respectively. By [9,15], their similarities can be calculated by leveraging a nonlinear data-to-anchor mapping $({\mathbb{R}}^d\to {\mathbb{R}}^m)$:

$$s_s\left(\mathbf{x}\right)={\left[{\delta }_{s1}{e}^{-\frac{D^2({\mathbf{x}}^f,{\mathbf{a}}_1^f)}{{\varrho }_s}},{\delta }_{s2}{e}^{-\frac{D^2({\mathbf{x}}^f,{\mathbf{a}}_2^f)}{{\varrho }_s}},\cdots ,{\delta }_{sm}{e}^{-\frac{D^2({\mathbf{x}}^f,{\mathbf{a}}_m^f)}{{\varrho }_s}}\right]}^T/M_s,$$

where ${\delta }_{sj}\in \left\{0,1\right\}$, and ${\delta }_{sj}=1$ if ${\mathbf{a}}_j$ is one of the $p\left(t\right)$ closest anchors of $\mathbf{x}$ based on the distance function $D(·)$ (e.g., Euclidean distance), ${\varrho }_s$ is a bandwidth parameter for similarity calculation, and $M_s={\sum }_{j=1}^m{\delta }_{sj}{e}^{-\frac{D^2(\mathbf{x},{\mathbf{a}}_j)}{{\varrho }_s}}$ so that ${∥s_s\left(\mathbf{x}\right)∥}_1=1$. Similarly, we calculate their dissimilarities by:

$$s_d\left(\mathbf{x}\right)={\left[{\delta }_{d1}{e}^{-\frac{D^2({\mathbf{x}}^f,{\mathbf{a}}_1^f)}{{\varrho }_d}},{\delta }_{d2}{e}^{-\frac{D^2({\mathbf{x}}^f,{\mathbf{a}}_2^f)}{{\varrho }_d}},\cdots ,{\delta }_{dm}{e}^{-\frac{D^2({\mathbf{x}}^f,{\mathbf{a}}_m^f)}{{\varrho }_d}}\right]}^T/M_d,$$

where ${\delta }_{dj}\in \left\{0,1\right\}$ and ${\delta }_{dj}=1$ if ${\mathbf{a}}_j$ is one of the $p\left(t\right)$ farthest anchors of $\mathbf{x}$ according to the distance function $D(·)$, ${\varrho }_d$ is a bandwidth parameter for dissimilarity calculation, and $M_d={\sum }_{j=1}^m{\delta }_{dj}{e}^{-\frac{D^2(\mathbf{x},{\mathbf{a}}_j)}{{\varrho }_d}}$ leads to ${∥s_d\left(\mathbf{x}\right)∥}_1=1$. Then, we can obtain the anchor-based pairwise similarity matrix $\mathbf{S}={\left[s\left({\mathbf{x}}_1\right),s\left({\mathbf{x}}_2\right),\cdots ,s\left({\mathbf{x}}_n\right)\right]}^T\in {\mathbb{R}}^{n\times m}$, where $s\left(\mathbf{x}\right)=s_s\left(\mathbf{x}\right)-s_d\left(\mathbf{x}\right)$. Let $\mathcal{M}$ represent the neighbor set and $\mathcal{C}$ denote the non-neighbor set. For clarity, $\mathbf{S}$ can be calculated by:

$$s_{ij}=\left\{\begin{array}{cc}\frac{e^{-\frac{D^2({\mathbf{x}}_i^f,{\mathbf{a}}_j^f)}{{\varrho }_s}}}{M_s}& ({\mathbf{x}}_i,{\mathbf{a}}_j)\in \mathcal{M}\\ -\frac{e^{-\frac{D^2({\mathbf{x}}_i^f,{\mathbf{a}}_j^f)}{{\varrho }_d}}}{M_d}& ({\mathbf{x}}_i,{\mathbf{a}}_j)\in \mathcal{C}\\ 0& otherwise.\end{array}\right.$$

(1)

To attain a robust relationship between training data and anchors, we create a strong ensemble anchor-based pairwise similarity matrix $\tilde{\mathbf{S}}$ by applying a weight average to $\mathbf{S}$ within multiple previous training epochs, e.g., $\tilde{\mathbf{S}}={\alpha }_1\tilde{\mathbf{S}}+(1-{\alpha }_1)\mathbf{S}$, where ${\alpha }_1\in \left(0,1\right)$ is a momentum term to determine how far the ensemble reaches into the training history.

3.2. Formulation and Procedure

Hashing is to project original data from a high-dimensional space into a low-dimensional binary space while preserving their similarity relations. Specifically, given the L-layer hashing network $f_{\theta }(·)$, for any sample $\mathbf{x}$, suppose $f_{\theta }^L\left(\mathbf{x}\right)\in {\mathbb{R}}^r$ to be the output of the L-th layer in the network, where r is the number of hash bits. Its hash function is defined as: $h\left(\mathbf{x}\right)=sgn\left(f_{\theta }^L\left(\mathbf{x}\right)\right)$, where $sgn(·)$ is a non-linear function with the definition $sgn\left(z\right)=1$ for $z>0$, otherwise $sgn\left(z\right)=-1$.

For one training sample ${\mathbf{x}}_i$ and one anchor ${\mathbf{a}}_j$, there exists $-r\le h\left({\mathbf{x}}_i\right)\circ h\left({\mathbf{a}}_j\right)\le r$, where ∘ denotes the inner product. In order to make $h\left({\mathbf{x}}_i\right)\circ h\left({\mathbf{a}}_j\right)>0$ if $({\mathbf{x}}_i,{\mathbf{a}}_j)\in \mathcal{M}$ and $h\left({\mathbf{x}}_i\right)\circ h\left({\mathbf{a}}_j\right)<0$ when $({\mathbf{x}}_i,{\mathbf{a}}_j)\in \mathcal{C}$, we define $u_{ij}=\sigma (\lambda h\left({\mathbf{x}}_i\right)\circ h\left({\mathbf{a}}_j\right))=\frac{1}{1+e^{-\lambda h\left({\mathbf{x}}_i\right)\circ h\left({\mathbf{a}}_j\right)}}$ to represent the similarity between ${\mathbf{x}}_i$ and ${\mathbf{a}}_j$ in the low-dimensional binary space, and $u_{ij}\to 1$ if $({\mathbf{x}}_i,{\mathbf{a}}_j)\in \mathcal{M}$, and $u_{ij}\to 0$ when $({\mathbf{x}}_i,{\mathbf{a}}_j)\in \mathcal{C}$, where $\sigma (·)$ represents the sigmoid function and $\lambda >0$ is a positive constant to regularize the value of $h\left({\mathbf{x}}_i\right)\circ h\left({\mathbf{a}}_j\right)$.

Because the function $sgn(·)$ is non-differential, it is usually replaced by a relaxed differential function for model training. There are many choices for relaxed differential functions; for simplicity, here, we choose a differential hyperbolic tangent function $tanh(·)$ in the hashing network. Then, we can obtain a latent binary vector $\mathbf{h}=tanh\left(f_{\theta }^L\left(\mathbf{x}\right)\right)\in {\left(-1,1\right)}^r$ of $\mathbf{x}$. Let B and $B^a$ be the index set of mini-batch data randomly selected from training data $\mathbf{X}$ and anchors $\mathbf{A}$, respectively. To exploit the similarity information contained in the ensemble pairwise similarity matrix $\tilde{\mathbf{S}}$, one common strategy is to first learn binary codes by solving a non-differential optimization problem and to then utilize them for model training. However, it is usually difficult and time-consuming to solve the non-differential problem. To avoid this issue, we propose a novel strategy by using a weighted cross-entropy loss function to directly train networks. Because $\tilde{\mathbf{S}}$ denotes the similar weight of pairs, we introduce a matrix $\mathbf{W}$ for the cross-entropy loss function in order to directly represent whether the pair is similar, where $w_{ij}$ is dependent on ${\tilde{s}}_{ij}$ to determine whether ${\mathbf{x}}_i$ and ${\mathbf{a}}_j$ are similar, i.e., $w_{ij}=1$ when ${\tilde{s}}_{ij}>0$ and $w_{ij}=0$ when ${\tilde{s}}_{ij}<0$. Then, the weighted cross-entropy loss is:

$$\begin{array}{c}{J}_{wce}=\frac{1}{{\sum }_{i\in B,j\in B^a}\left|{\tilde{s}}_{ij}\right|}\sum _{i\in B,j\in B^a}{\tilde{s}}_{ij}(-w_{ij}log{u}_{ij}+(1-w_{ij})log(1-u_{ij}))\\ s.t.{\mathbf{h}}_i=tanh\left(f_{\theta }^L\left({\mathbf{x}}_i\right)\right),{\mathbf{h}}_j=tanh\left(f_{\theta }^L\left({\mathbf{a}}_j\right)\right),u_{ij}=\frac{1}{1+e^{-\lambda {\mathbf{h}}_i{\mathbf{h}}_j^T}},\end{array}$$

(2)

where $\left|·\right|$ is an absolute value function. Note that when ${\tilde{s}}_{ij}\ne 0$ ($j\in B^a$), we do not set ${\tilde{s}}_{ij}=1$ or $-1$ in order to decrease the effect of noisy similarity information.

However, there exists a gap between the latent binary vector $\mathbf{h}$ and desired binary codes $h\left(\mathbf{x}\right)$, thereby potentially decreasing the model performance [15]. To reduce this gap, we utilize a square loss as follows:

$$J_{hmse}={∥\left|{\mathbf{h}}_i\right|-{\mathbf{1}}_r∥}_2^2,$$

(3)

where ${\mathbf{1}}_r\in {\mathbb{R}}^r$ is a row vector with all entries being ones.

Recent semi-supervised methods [31,32,33] illustrate that forming consensus predictions under different configurations (such as training epochs, dropout, and augmentation conditions) for each training sample can improve the model performance when exploring the semantic information of unlabeled data. Inspired by these methods, we aim to form a consensus prediction of the latent binary vector for each training sample in order to boost the model performance. Specifically, similar to [32], we create a target latent binary vector ${\tilde{\mathbf{h}}}_i$ for ${\mathbf{x}}_i$ by applying exponential moving average (EMA) to ${\mathbf{h}}_i$ of multiple previous training epochs, i.e., we first accumulate ${\mathbf{h}}_i$ into an ensemble vector ${\mathbf{h}}_i^e$ by ${\mathbf{h}}_i^e={\alpha }_2{\mathbf{h}}_i^e+(1-{\alpha }_2){\mathbf{h}}_i$, and then calculate ${\tilde{\mathbf{h}}}_i={\mathbf{h}}_i^e/(1-{\alpha }_2^t)$, where ${\alpha }_2\in \left(0,1\right)$ is a momentum term to determine how much of ${\mathbf{h}}_i^e$ is affected by previous training epochs, $1-{\alpha }_2^t$ is to correct the startup bias, and t is the current number of training epochs. Then, we minimize the difference between ${\mathbf{h}}_i$ and ${\tilde{\mathbf{h}}}_i$ with the following square loss:

$$J_{smse}={∥{\mathbf{h}}_i-{\tilde{\mathbf{h}}}_i∥}_2^2.$$

(4)

To simultaneously exploit the pairwise similarity information, reduce the gap between $\mathbf{h}$ and $h\left(\mathbf{x}\right)$, and form consensus prediction for each training sample, we integrate the three terms (Equations (2)–(4)) to learn the model parameters $\theta$ as follows:

$$\begin{array}{c}J\left(\theta \right)=J_{wce}+\frac{1}{l_{B}r}\sum _{i\in B}({\gamma }_1{J}_{hmse}+{\gamma }_2{J}_{smse})\\ =\frac{1}{{\sum }_{i\in B,j\in L}\left|{\tilde{s}}_{ij}\right|}\sum _{i\in B,j\in L}{\tilde{s}}_{ij}(-w_{ij}log{u}_{ij}+(1-w_{ij})log(1-u_{ij}))\\ +\frac{1}{l_{B}r}\sum _{i\in B}({\gamma }_1{∥\left|{\mathbf{h}}_i\right|-{\mathbf{1}}_r∥}_2^2+{\gamma }_2{∥{\mathbf{h}}_i-{\tilde{\mathbf{h}}}_i∥}_2^2)\\ s.t.{\mathbf{h}}_i=tanh\left(f_{\theta }^L\left({\mathbf{x}}_i\right)\right),{\mathbf{h}}_j=tanh\left(f_{\theta }^L\left({\mathbf{a}}_j\right)\right),u_{ij}=\frac{1}{1+e^{-\lambda {\mathbf{h}}_i{\mathbf{h}}_j^T}},\end{array}$$

(5)

where $l_B$ is the length of B, and ${\gamma }_1\ge 0$ and ${\gamma }_2\ge 0$ are to weight the corresponding two regularization terms, respectively.

Based on Equation (5), we can adopt any optimizer, e.g., Adam [34], to learn the model parameters $\theta$. For clarity, we present the detailed procedure of solving Equation (5) to learn $\theta$ in Algorithm 1: UDPH. Note that, to boost the model performance, training samples are usually augmented; we utilize $g(·)$ to denote the augmentation function. Additionally, $f_{\theta }(g\left({\mathbf{x}}_{i\in B}\right),g\left({\mathbf{a}}_{j\in B^a}\right))$ denotes the output of the L-th layer followed by the function $tanh(·)$ for the sample ${\mathbf{x}}_i$ and anchor ${\mathbf{a}}_j$. After obtaining $\theta$, we can attain binary codes of each training or query data $\mathbf{x}$ by: $h\left(\mathbf{x}\right)=sgn\left(f_{\theta }^L\left(\mathbf{x}\right)\right)$.

Algorithm 1: UDPH

Input: Data $\mathbf{X}={\left\{{\mathbf{x}}_i\right\}}_{i=1}^n$, anchors $\mathbf{A}={\left\{{\mathbf{a}}_j\right\}}_{j=1}^m$, bit number r, parameters $\lambda$, ${\gamma }_1$, and ${\gamma }_2$, piecewise linear function $p\left(t\right)$, ensembling momentums ${\alpha }_1$, and ${\alpha }_2$, training epoch number T, network with parameters $\theta$: $f_{\theta }(·)$, stochastic input augmentation function: $g(·)$ Output: Parameters $\theta$   1: Initialization:       Initialize parameters $\theta$ by the pre-trained VGG-16       model on ImageNet;       Construct $\mathbf{S}$ by Equation (1);       $\tilde{\mathbf{S}}\leftarrow \mathbf{S}$,        ▹ ensemble pairwise similarity matrix;       ${\mathbf{H}}^e\leftarrow {\mathbf{0}}_{n\times m}$, ▹ ensemble latent binary vectors;       $\tilde{\mathbf{H}}\leftarrow {\mathbf{0}}_{n\times m}$, ▹ target latent binary vectors;   2: fortin$(1,T)$do:   3:     for each mini-batch B and $B^a$ do:   4:          ${\mathbf{h}}_{i\in B},{\mathbf{h}}_{j\in B^a}\leftarrow f_{\theta }(g\left({\mathbf{x}}_{i\in B}\right),g\left({\mathbf{a}}_{j\in B^a}\right))$;   5:          loss ← Equation (5);   6:          updating $\theta$ using optimizers,             e.g., Adam [34];   7:     end for;   8:     Construct $\mathbf{S}$ by Equation (1);   9:     $\tilde{\mathbf{S}}\leftarrow {\alpha }_1\tilde{\mathbf{S}}+(1-{\alpha }_1)\mathbf{S}$; 10:    ${\mathbf{H}}^e\leftarrow {\alpha }_2{\mathbf{H}}^e+(1-{\alpha }_2)\mathbf{H}$; 11:    $\tilde{\mathbf{H}}\leftarrow {\mathbf{H}}^e/(1-{\alpha }_2^t)$; 12: end for.

4. Experiments

To evaluate the proposed UDPH, we conduct extensive experiments on two large-scale benchmark databases: CIFAR-10 [27] and NUS-WIDE [28]. CIFAR-10 has 60,000 color $32\times 32$ images belonging to 10 classes on average. These images are split into one training set with 50,000 images and one testing set containing 10,000 images. NUS-WIDE is composed of 269,648 images collected from Flickr [28]. There are, totally, 81 semantic concepts, with each image containing multiple labels. Similar to [8,15], we select the 21 most frequent labels for evaluation and, in total, obtain around 195,834 color images. Following [15], for these two databases, we randomly select 100 images from each class to construct the query set and use the rest as a training/gallery set.

4.1. Experimental Settings

We compare UDPH against six popular non-deep unsupervised hashing algorithms (LSH [1], SH [5], AGH [8], ITQ [21], SpH [18], and SGH [19]) and eight state-of-the-art unsupervised deep hashing algorithms (DH [11], UAR [25], UH-BDNN [12], DeepBit [13], HashGAN [14], UTH [26], UDBD [24], and SADH [15]). For the non-deep hashing algorithms, we evaluate them by using hand-crafted features. Specifically, each image in CIFAR-10 is represented by a 512-dimensional GIST vector [35], and each one in NUS-WIDE is represented as a 500-dimensional bag of words (BoW) feature vector. Additionally, we also show their performance on deep features extracted from the $fc7$ layer of the VGG-16 model pre-trained on the ImageNet database. Among the eight deep hashing algorithms, DeepBit, UTH, UDBD, and SADH utilize the VGG-16 model as their backbone networks. For the proposed UDPH, we empirically set the function and adopt the parameters ${\gamma }_1=0.01$, ${\gamma }_2=0.1$, ${\alpha }_1=0.9$, ${\alpha }_2=0.6$, $\lambda =0.8$ on both CIFAR-10 and NUS-WIDE, except $\lambda =1.6$ on CIFAR-10 at 16-bit. We randomly select 500 images from the training data of the two databases as anchors, i.e., $m=500$.

Following [12,15], we adopt semantic similarity as the ground truth in our experiments. For NUS-WIDE, two images are neighbors if they share at least one common label. We evaluate the performance of the aforementioned hashing algorithms by using mean average precision (MAP), MAP@1000, Precision@5000, and Precision@1000. Here, MAP denotes the mean of the average precision of query images over all images in the gallery set; MAP@1000 is the MAP calculated over the top 1000 returned images from the gallery set. Precision@5000 means the rate of correctly retrieved samples from the top 5000 ranked images. A similar definition is applied to Precision@1000. We run the experiments five times and report the average results.

4.2. Experimental Results and Analysis

Table 1. Retrieval results (%) in terms of MAP, MAP@1000, Precision@5000, and Precision@1000 on CIFAR-10 and NUS-WIDE. † denotes the implemented results based on the provided codes, * means the results copied from Shen et al., 2018 [15], and the other results are directly copied from the corresponding publications. The best accuracy are in bold and the second-best results of each database are underlined.

Method	MAP			MAP@1000			Precision@5000			Precision@1000
	16-bit	32-bit	64-bit	16-bit	32-bit	64-bit	16-bit	32-bit	64-bit	16-bit	32-bit	64-bit
CIFAR-10
LSH${ }^{†}$	13.18	14.00	14.92	19.05	21.10	23.83	14.30	15.82	17.07	15.82	18.21	20.53
SH${ }^{†}$	12.85	12.65	12.51	20.69	21.03	20.52	14.24	14.06	14.12	16.66	16.98	16.97
AGH${ }^{†}$	14.31	13.52	13.44	22.74	22.12	23.54	16.11	15.57	15.58	19.02	19.30	20.48
ITQ${ }^{†}$	15.52	15.94	16.49	24.13	26.00	27.56	17.41	18.17	18.89	20.06	22.12	23.39
SpH${ }^{†}$	14.28	14.53	15.27	21.68	23.31	26.38	15.94	16.80	17.92	18.34	20.02	22.55
SGH${ }^{†}$	14.51	15.05	15.37	22.97	24.92	26.38	15.82	17.67	17.86	19.79	21.34	22.34
LSH+VGG${ }^{†}$	13.71	15.81	19.54	20.45	26.13	34.03	15.09	18.37	23.09	17.64	22.45	29.54
SH+VGG${ }^{†}$	22.14	19.65	18.18	40.26	38.89	38.48	25.84	23.41	22.07	34.05	32.58	31.76
AGH+VGG${ }^{†}$	31.43	28.26	26.55	45.05	47.24	48.79	34.02	32.21	30.79	42.77	43.75	44.90
ITQ+VGG${ }^{†}$	31.93	32.21	33.76	45.58	50.53	53.86	35.24	35.75	37.17	42.31	45.94	48.93
SpH+VGG${ }^{†}$	19.84	24.23	26.00	33.48	41.34	45.02	22.88	28.13	30.09	28.72	36.78	40.26
SGH+VGG${ }^{†}$	23.93	24.30	27.15	42.01	44.12	48.48	27.40	28.52	31.48	36.49	38.74	43.14
DH	16.17	16.62	16.96	-	-	-	-	-	-	23.79	26.00	27.70
DAR	16.82	17.01	17.21	-	-	-	-	-	-	24.54	26.62	28.06
UH-BDNN${ }^{*}$	30.10	30.89	31.18	-	-	-	33.97	34.48	35.00	-	-	-
HashGAN	29.94	31.47	32.53	44.65	46.34	48.12	-	-	-	41.76	43.62	45.51
DeepBit${ }^{*}$	15.95	19.16	20.96	-	-	-	18.02	22.27	24.36	-	-	-
UTH	-	-	-	28.66	30.66	32.41	-	-	-	-	-	-
UDBD	21.70	20.64	23.07	26.36	27.92	34.05	-	-	-	-	-	-
SADH${ }^{*}$	38.70	38.49	37.68	-	-	-	41.80	41.56	41.15	-	-	-
UDPH	39.81	40.68	$\mathbf{43}.\mathbf{25}$	$\mathbf{46}.\mathbf{11}$	$\mathbf{52}.\mathbf{52}$	$\mathbf{58}.\mathbf{17}$	$\mathbf{42}.\mathbf{08}$	$\mathbf{43}.\mathbf{06}$	$\mathbf{44}.\mathbf{31}$	$\mathbf{42}.\mathbf{95}$	$\mathbf{49}.\mathbf{72}$	$\mathbf{54}.\mathbf{23}$
NUS-WIDE
LSH${ }^{†}$	36.06	36.19	37.16	39.95	40.98	43.01	39.17	39.63	41.43	39.48	40.32	42.28
SH${ }^{†}$	34.47	34.96	34.97	41.78	43.00	41.94	37.80	38.77	38.07	40.22	41.40	40.41
AGH${ }^{†}$	35.74	35.87	35.75	42.30	43.26	44.41	40.29	41.44	41.71	41.56	42.36	43.46
ITQ${ }^{†}$	38.25	38.64	38.75	45.35	46.36	47.05	42.97	43.85	44.20	44.25	45.23	45.71
SpH${ }^{†}$	37.26	37.42	37.83	43.58	44.82	46.30	41.82	42.69	43.77	42.76	43.93	45.25
SGH${ }^{†}$	37.39	37.33	37.41	45.66	45.79	45.82	43.11	42.91	43.02	44.60	44.56	44.62
LSH+VGG${ }^{†}$	38.95	39.56	44.53	46.55	50.57	62.82	43.42	45.64	55.69	45.28	48.65	60.32
SH+VGG${ }^{†}$	44.74	42.59	41.54	67.21	66.40	66.69	58.65	55.52	53.93	64.84	63.01	62.54
AGH+VGG${ }^{†}$	49.91	49.77	48.38	70.59	72.67	73.94	66.48	67.79	67.64	69.65	71.49	72.63
ITQ+VGG${ }^{†}$	54.76	55.20	55.55	70.21	74.14	76.32	68.92	70.43	71.55	70.22	74.01	74.69
SpH+VGG${ }^{†}$	47.40	50.44	51.33	64.28	70.18	72.94	58.97	63.83	66.10	62.64	68.24	70.90
SGH+VGG${ }^{†}$	46.98	47.71	50.01	69.43	71.49	74.89	61.81	63.07	66.67	67.30	69.22	72.70
UH-BDNN${ }^{*}$	39.22	40.32	42.06	-	-	-	45.54	51.34	57.72	-	-	-
DeepBit${ }^{*}$	54.26	51.72	54.74	-	-	-	70.18	69.60	72.74	-	-	-
SADH${ }^{*}$	60.14	57.99	56.33	-	-	-	71.45	73.88	75.04	-	-	-
UDPH	60.75	61.29	63.35	$\mathbf{71}.\mathbf{89}$	76.60	77.87	70.49	74.52	75.56	71.31	75.65	76.88

presents retrieval results of the non-deep and deep hashing algorithms at 16-, 32- and 64-bit, selecting 100 query images per class and using the remaining ones as a training/gallery set from CIFAR-10 and NUS-WIDE databases, respectively. Note that we do not present the results of DH, DAR, Hash-GAN, UTH, and UDPD on NUS-WIDE, because there is no publicly reported results. As we can see from Table 1, non-deep hashing algorithms with deep features extracted from the pre-trained VGG-16 model achieve better performance than that using hand-crafted features, probably because the data distribution of ImageNet is similar to that of CIFAR-10 and NUS-WIDE. Additionally, SADH outperforms the other non-deep and deep hashing algorithms except UDPH, because it effectively exploits the adaptive pairwise similarity among data. Moreover, UDPH obtains superior performance over all the non-deep and deep hashing algorithms, especially in terms of the metric MAP. Specifically, on CIFAR-10, the gain of UDPH in MAP is from a relative 2.87% to a relative 14.78% over the best competitor, SADH; on NUS-WIDE, the gain of UDPH in MAP ranges from 1.01% to 12.46%, relatively, over the best competitor; it also obtains better performance in terms of MAP@1000, Precision@5000, and Precision@1000 on the two databases except the Precision@5000 at 16-bit on NUS-WIDE. These results demonstrate the effectiveness and strength of the proposed UDPH. Note that SADH usually achieves its best MAP with short binary codes, e.g., 16-bit, while UDPH obtains better performance with an increasing number of bits. This might be because SADH can effectively preserve the similarity information of the low-rank graph matrix by using short binary codes, but UDPH with longer binary codes can better preserve the similarity relationship between training data and anchors.

On CIFAR-10, some popular methods, including DBD-MQ [36] and GraphBit [37], utilize 50,000 images as a training/gallery set and 10,000 images as a query set, and they adopt relatively shallow CNNs. Following the experimental protocols in DBD-MQ and GraphBit, to better illustrate the strength of the proposed UDPH, we adopt a shallow network AlexNet [38] as the backbone network, which is pre-trained on the ImageNet database. Additionally, UDPH with AlexNet adopts the same parameter settings as that with VGG-16.

Table 2. Retrieval results (%) in terms of MAP@1000 on CIFAR-10 with AlexNet as the backbone. The results of DBD-MQ and GraphBit are directly copied from the publications. The best accuracy results are in bold and the second-best results are underlined.

Method	MAP@1000
	16-bit	32-bit	64-bit
DBD-MQ	21.53	26.50	31.85
GraphBit	32.15	36.74	39.90
UDPH	$\mathbf{37}.\mathbf{81}$	$\mathbf{43}.\mathbf{66}$	$\mathbf{47}.\mathbf{22}$

presents their raking performance in terms of MAP@1000. This further demonstrates the superior performance of UDPH. Specifically, its gain is 17.40%, 18.84%, and 18.35%, relatively, over the best competitor GraphBit at 16-, 32-, and 64-bit, respectively.

4.3. Ablation Studies

Here, we analyze the influence of the ensemble anchor-based pairwise similarity matrix $\tilde{\mathbf{S}}$ and the two terms, $J_{hmse}$ and $J_{smse}$, on UDPH using VGG-16 as the backbone network. We randomly select 100 images per class from the CIFAR-10 database to construct a query set, and 500 images per class from the remaining ones to construct a set for training and retrieval. Figure 2a shows the MAP of UDPH using four different settings of $\tilde{\mathbf{S}}$. Specifically, (i) “${\alpha }_1=0.9$, weight” means using the entries in $\tilde{\mathbf{S}}$ as the weight of $J_{wce}$; (ii) “${\alpha }_1=0.9$, equal” means ${\tilde{s}}_{ij}=1$ if ${\tilde{s}}_{ij}>0$ and ${\tilde{s}}_{ij}=-1$ when ${\tilde{s}}_{ij}<0$; (iii) “${\alpha }_1=0$, update $\mathbf{S}$” denotes $\tilde{\mathbf{S}}=\mathbf{S}$ with updating $\mathbf{S}$ every training epoch; (iv) “${\alpha }_1=0$, fix $\mathbf{S}$” represents $\tilde{\mathbf{S}}=\mathbf{S}$ without updating $\mathbf{S}$ during the training process. Figure 2a suggests that $\tilde{\mathbf{S}}$ can boost the model performance and smooth the training process, i.e., the best or sub-best MAP is achieved at the last several training epochs. Figure 2b presents the MAP of UDPH without $J_{hmse}$, i.e., ${\gamma }_1=0$. It illustrates the effectiveness $J_{hmse}$ and the significance of reducing the gap between latent binary vector and binary codes. Figure 2c displays the result of UDPH without $J_{smse}$, i.e., ${\gamma }_2=0$. It demonstrates that forming consensus predictions of latent binary vectors can also improve model performance.

5. Conclusions

In this paper, we propose a novel unsupervised deep pairwise hashing method, which effectively and robustly takes advantage of the similarity information between training samples and anchors. We first construct an anchor-based pairwise similarity matrix, upon which we create a strong and robust ensemble pairwise similarity matrix to preserve their similarity and dissimilarity relations. Then, we propose a novel loss function consisting of a weighted cross-entropy loss, which utilizes the similarity and dissimilarity between training samples and anchors as the weight to explore their semantic similarity relationship, a square loss to reduce the gap between latent binary vectors and binary codes, and another square loss to form consensus predictions of latent binary vectors for boosting model performance. Experiments on benchmark databases demonstrate the strength of the proposed method and the effectiveness of each term in the proposed loss function. In the future, it is very promising to apply the robust ensemble pairwise similarity matrix and the weighted cross-entropy loss on unsupervised or semi-supervised deep methods because they can effectively explore the semantic similarity information hidden in unlabeled data. Exploring advanced backbones, such as ResNet, to further improve performance is another research direction.