Classifier and Exemplar Synthesis for Zero-Shot Learning

Abstract

Zero-shot learning (ZSL) enables solving a task without the need to see its examples. In this paper, we propose two ZSL frameworks that learn to synthesize parameters for novel unseen classes. First, we propose to cast the problem of ZSL as learning manifold embeddings from graphs composed of object classes, leading to a flexible approach that synthesizes “classifiers” for the unseen classes. Then, we define an auxiliary task of synthesizing “exemplars” for the unseen classes to be used as an automatic denoising mechanism for any existing ZSL approaches or as an effective ZSL model by itself. On five visual recognition benchmark datasets, we demonstrate the superior performances of our proposed frameworks in various scenarios of both conventional and generalized ZSL. Finally, we provide valuable insights through a series of empirical analyses, among which are a comparison of semantic representations on the full ImageNet benchmark as well as a comparison of metrics used in generalized ZSL. Our code and data are publicly available at https://github.com/pujols/Zero-shot-learning-journal.

Appendices

Appendix A: Details on How to Obtain Word Vectors on ImageNet

We use the word2vec package.¹⁰ We preprocess the input corpus with the word2phrase function so that we can directly obtain word vectors for both single-word and multiple-word terms, including those terms in the ImageNet synsets; each class of ImageNet is a synset: a set of synonymous terms, where each term is a word or a phrase. We impose no restriction on the vocabulary size. Following Frome et al. (2013), we use a window size of 20, apply the hierarchical softmax for predicting adjacent terms, and train the model for a single epoch. As one class may correspond to multiple word vectors by the nature of synsets, we simply average them to form a single word vector for each class.

Fig. 8

Data splitting for different cross-validation (CV) strategies: a the seen-unseen class splitting for (conventional) zero-shot learning, b the sample-wise CV, c the class-wise CV

Appendix B: Hyper-parameter Tuning

1.1 B.1 For Conventional Zero-Shot Learning

The standard approach for cross-validation (CV) in a classification task splits training data into several folds such that they share the same set of class labels. This strategy is less sensible in zero-shot learning as it does not imitate what actually happens at the test stage. We thus adopt the strategy in Elhoseiny et al. (2013), Akata et al. (2015), Romera-Paredes and Torr (2015), Zhang and Saligrama (2015), Romera-Paredes and Torr (2015). In this scheme, we split training data into several folds such that the class labels of these folds are disjoint. We then hold out data from one fold as pseudo-unseen classes, train our models on the remaining folds (which belong to the remaining classes), and tune hyper-parameters based on a certain performance metric on the held-out fold. For clarity, we denote the standard CV as sample-wise CV and the zero-shot CV scheme as class-wise CV. Figure 8 illustrates the two scenarios.

We use this strategy to tune hyper-parameters in both our approaches (SynC and EXEM) and the baselines. In SynC, the main hyper-parameters are the regularization parameter $\lambda$ in Eq. (6) and the scaling parameter $\sigma$ in Eq. (3). When learning semantic representations [Eq. (9)], we also tune $\eta$ and $\gamma$. To reduce the search space during CV, we first fix $\varvec{b}_r = \varvec{a}_r$ for $r = 1,\ldots ,\mathsf {R}$ and tune $\lambda , \sigma$. Then we fix $\lambda$ and $\sigma$ and tune $\eta$ and $\gamma$. The metric is the classification accuracy.

Table 10 Expanded results of Table 4. The metric is “per-sample” accuracy for F@K to aid comparison with previous published results

In EXEM, we tune (a) projected dimensionality $\mathsf {d}$ for PCA and (b)$\lambda$, $\nu$, and the RBF-kernel bandwidth in SVR.¹¹ Since EXEM is a two-stage approach, we consider the following two performance metrics. The first one minimizes the distance between the predicted exemplars and the ground-truth (average of the hold-out data of each class after the PCA projection) in $\mathbb {R}^{\mathsf {d}}$. We use the Euclidean distance in this case. We term this measure “CV-distance.” This approach does not assume the downstream task at training and aims to measure the quality of predicted exemplars by its faithfulness. The other approach “CV-accuracy” maximizes the per-class classification accuracy on the hold-out fold. This measure can easily be obtained for EXEM (1NN) and EXEM (1NNs), which use simple decision rules that have no further hyper-parameters to tune. Empirically, we found that CV-accuracy generally leads to slightly better performance. The results reported in the main text for these two approaches are thus based on this measure. On the other hand, EXEM (ZSL method) (where ZSL method = SynC, ConSE, ESZSL) requires further hyper-parameter tuning. For computational purposes, we use CV-distance¹² for tuning hyper-parameters of the regressors, followed by the hyper-parameter tuning for ZSL methods using the predicted exemplars. As SynC and ConSE construct their classifiers based on the distance values between class semantic representations, we do not expect a significant performance drop in this case.

Table 11 Expanded results of the third section of Table 7

1.2 B.2 For Generalized Zero-shot Learning

To perform class-wise CV in the generalized zero-shot learning (GZSL) setting, we further separate each fold into two splits, each with either 80% or 20% of data. We then hold out one fold, train models on the $80\%$ splits of the remaining folds, and tune hyper-parameters based on a certain performance metric on (i) the $80\%$ split of the hold-out fold and (ii) the $20\%$ splits of the training (i.e., remaining) folds. In this way we can mimic the GZSL setting in hyper-parameter tuning. Specifically, for metrics with calibration (cf. Table 6), we first compute AUSUC using (i) and (ii) to tune the hyper-parameters mentioned in Sect. B.1, and select the calibration factor $\gamma$ that maximizes the harmonic mean. For the uncalibrated harmonic mean, we follow Xian et al. (2018a) to tune hyper-parameters in the same way as in the conventional ZSL setting.

Table 12 Expanded results of Table 7 with “per-sample” accuracy used to differentiate this accuracy from the “per-class” one in Table 7

Appendix C: Experimental Results on ImageNet with Previous Experimental Setups

The first ZSL work on ImageNet and much of its follow-up considers only 2-hop, 3-hop, and All test sets and other evaluation metrics. We include our results here in Tables 10, 11, and 12 to aid comparison with such work. As mentioned in Sect. 3.4.1, we also consider Flat hit@K (F@K) and Hierarchical precision@K (HP@K). F@K is defined as the percentage of test images for which the model returns the true label in its top K predictions. HP@K is defined as the percentage of overlapping (i.e., precision) between the model’s top K predictions and the ground-truth list. For each class, the ground-truth list of its K closest categories is generated based on the ImageNet hierarchy. Note that F@1 is the per-sample multi-way classification accuracy.

When computing Hierarchical precision@K (HP@K), we use the algorithm in the “Appendix” of Frome et al. (2013) to compute the ground-truth list, a set of at least K classes that are considered to be correct. This set is called hCorrectSet and it is computed for each K and class c. See Algorithm 1 for more details. The main idea is to expand the radius around the true class c until the set has at least K classes.

Note that validRadiusSet depends on which classes are in the label space to be predicted (i.e., depending on whether we consider 2-hop, 3-hop, or All. We obtain the label sets for 2-hop and 3-hop from the authors of Frome et al. (2013), Norouzi et al. (2014). We implement Algorithm 1 to derive hCorrectSet ourselves.

Appendix D: Analysis on SynC

In this section, we focus on SynC$^\text {o-vs-o}$ together with GoogLeNet features and the standard split (SS). We look at the effect of modifying the regularization term, learning base semantic representations, and varying the number of base classes and their correlations.

Table 13 Comparison between regularization with $\varvec{w}_c$ and $\varvec{v}_c$ on SynC$^\text {o-vs-o}$

Table 14 Effect of learning semantic representations

Fig. 9

We vary the number of phantom classes $\mathsf {R}$ as a percentage of the number of seen classes $\mathsf {S}$ and investigate how much that will affect classification accuracy (the vertical axis corresponds to the ratio with respect to the accuracy when $\mathsf {R}= \mathsf {S}$). The base classifiers are learned with SynC$^\text {o-vs-o}$ (Color figure online)

Fig. 10

Percentages of basis components required to capture 95% of variance in classifier matrices for AwA and CUB (Color figure online)

Table 15 We compute the Euclidean distance matrix between the unseen classes based on semantic representations ($\varvec{D}_{\varvec{a}_u}$), predicted exemplars ($\varvec{D}_{\varvec{\psi }(\varvec{a}_u)}$), and real exemplars ($\varvec{D}_{\varvec{v}_u}$)

1.1 D.1 Different Forms of Regularization

In Eqs. (6) and (9), $\left\| \varvec{w}_c\right\| _2^2$ is the regularization term. Here we consider modifying that term to $\left\| \varvec{v}_r\right\| _2^2$—regularizing the bases directly. Table 13 shows that $\left\| \varvec{v}_r\right\| _2^2$ leads to better results. However, we find that learning with $\left\| \varvec{v}_r\right\| _2^2$ converges much slower than with $\left\| \varvec{w}_c\right\| _2^2$. Thus, we use $\left\| \varvec{w}_c\right\| _2^2$ in our main experiments (though it puts our methods at a disadvantage).

1.2 D.2 Learning Phantom Classes’ Semantic Representations

So far we adopt the version of SynC that sets the number of base classifiers to be the number of seen classes $\mathsf {S}$, and sets $\varvec{b}_r = \varvec{a}_c$ for $r=c$. Here we study whether we can learn optimally the semantic representations for the phantom classes that correspond to base classifiers. The results in Table 14 suggest that learning representations could have a positive effect.

1.3 D.3 How Many Base Classifiers are Necessary?

In Fig. 9, we investigate how many base classifiers are needed—so far, we have set that number to be the number of seen classes out of convenience. The plot shows that in fact, a smaller number ($\sim 60\%$) is enough for our algorithm to reach the plateau of the performance curve. Moreover, increasing the number of base classifiers does not seem to have an overwhelming effect.

Note that the semantic representations $\varvec{b}_r$ of the phantom classes are set equal to $\varvec{a}_r, \forall r\in \{1, \ldots , \mathsf {R}\}$ at 100% (i.e., $\mathsf {R}=\mathsf {S}$). For percentages smaller than 100%, we perform K-means and set $\varvec{b}_r$ to be the cluster centroids after $\ell _2$ normalization (in this case, $\mathsf {R}= K$). For percentages larger than 100%, we set the first $\mathsf {S}$$\varvec{b}_r$ to be $\varvec{a}_r$, and the remaining $\varvec{b}_r$ as the random combinations of $\varvec{a}_r$ (also with $\ell _2$ normalization on $\varvec{b}_r$).

We have shown that even by using fewer base (phantom) classifiers than the number of seen classes (e.g., around 60 %), we get comparable or even better results, especially for CUB. We surmise that this is because CUB is a fine-grained recognition benchmark and has higher correlations among classes, and provide analysis in Fig. 10 to justify this.

Table 16 Overlap of k-nearest classes (in %) on AwA, CUB, SUN. We measure the overlap between those searched by real exemplars and those searched by semantic representations (i.e., attributes) or predicted exemplars

Fig. 11

t-SNE (Van der Maaten and Hinton 2008) visualization of randomly selected real images (crosses) and predicted visual exemplars (circles) for the unseen classes on (from left to right, then from top to down) AwA, CUB, SUN, and ImageNet. Different colors of symbols denote different unseen classes. Perfect predictions of visual features would result in well-aligned crosses and circles of the same color. Plots for CUB and SUN are based on their first splits of SS. Plots for ImageNet are based on randomly selected 48 unseen classes from 2-hop and word vectors as semantic representations. Best viewed in color (Color figure online)

Table 17 Comparison between EXEM (1NN) with support vector regressors (SVR) and with 2-layer multi-layer perceptron (MLP) for predicting visual exemplars

Table 18 Accuracy of EXEM (1NN) on AwA, CUB, and SUN when predicted exemplars are from original visual features (No PCA) and PCA-projected features (PCA with $\mathsf {d}$ = 1024, 500, 200, 100, 50, 10)

We train one-versus-other classifiers for each value of the regularization parameter on both AwA and CUB, and then perform PCA on the resulting classifier matrices. We then plot the required number (in percentage) of PCA components to capture 95% of variance in the classifiers. Clearly, AwA requires more. This explains why we see the drop in accuracy for AwA but not CUB when using even fewer base classifiers. Particularly, the low percentage for CUB in Fig. 10 implies that fewer base classifiers are possible. Given that CUB is a fine-grained recognition benchmark, this result is not surprising in retrospection as the classes are highly correlated.

Appendix E: Analysis on EXEM

In this section, we provide more analysis on EXEM. We focus on GoogLeNet features and the standard split (SS). We provide both qualitative and quantitative measures of predicted exemplars. We also investigate neural networks for exemplar prediction functions and the effect of PCA.

1.1 E.1 Quality of Predicted Exemplars

We first show that predicted visual exemplars better reflect visual similarities between classes than semantic representations. Let $\varvec{D}_{\varvec{a}_u}$ be the pairwise Euclidean distance matrix between unseen classes computed from semantic representations (i.e., $\mathsf {U}$ by $\mathsf {U}$), $\varvec{D}_{\varvec{\psi }(\varvec{a}_u)}$ the distance matrix computed from predicted exemplars, and $\varvec{D}_{\varvec{v}_u}$ the distance matrix computed from real exemplars (which we do not have access to). Table 15 shows that the Pearson correlation coefficient¹³ between $\varvec{D}_{\varvec{\psi }(\varvec{a}_u)}$ and $\varvec{D}_{\varvec{v}_u}$ is much higher than that between $\varvec{D}_{\varvec{a}_u}$ and $\varvec{D}_{\varvec{v}_u}$. Importantly, we improve this correlation without access to any data of the unseen classes.

Besides the correlation used in Table 15, we can also use %kNN-overlap defined in Sect. 4.5.1 as another evidence that predicted exemplars better reflect visual similarities (as defined by real exemplars) than semantic representations. Recall that %kNN-overlap(A1, A2) is the average (over all unseen classes u) of the percentages of overlap between two sets of k-nearest neighbors kNN$_{A1}(u)$ and kNN$_{A2}(u)$. In Table 16, we report %kNN-overlap (semantic representations, real exemplars) and %kNN-overlap (predicted exemplar, real exemplars). We set K to be 40% of the number of unseen classes, but we note that the trends are consistent for different values of K.

We then show some t-SNE (Van der Maaten and Hinton 2008) visualization of predicted visual exemplars of the unseen classes. Ideally, we would like them to be as close to their corresponding real images as possible. In Fig. 11, we demonstrate that this is indeed the case for many of the unseen classes. For those unseen classes (each of which denoted by a color), their real images (crosses) and our predicted visual exemplars (circles) are well-aligned.

The quality of predicted exemplars (here based on the distance to the real images) depends on two main factors: the predictive capability of semantic representations and the number of semantic representation-visual exemplar pairs available for training, which in this case is equal to the number of seen classes $\mathsf {S}$. On AwA where we have only 40 training pairs, the predicted exemplars are surprisingly accurate, mostly either placed in their corresponding clusters or at least closer to their clusters than predicted exemplars of the other unseen classes. Thus, we expect them to be useful for discriminating among the unseen classes. On ImageNet, the predicted exemplars are not as accurate as we would have hoped, but this is expected since the word vectors are purely learned from text.

We also observe relatively well-separated clusters in the semantic embedding space (in our case, also the visual feature space since we only apply PCA projections to the visual features), confirming our assumption about the existence of clustering structures. On CUB, we observe that these clusters are more mixed than on other datasets. This is not surprising given that it is a fine-grained classification dataset of bird species.

1.2 E.2 Exemplar Prediction Function

We compare two approaches for predicting visual exemplars: kernel-based support vector regressors (SVR) and 2-layer multi-layer perceptron (MLP) with ReLU nonlinearity. MLP weights are $\ell _2$ regularized, and we cross-validate the regularization constant.

Similar to Zhang et al. (2017), our multi-layer perceptron is of the form:

$$\begin{aligned} \frac{1}{\mathsf {S}} \sum _{c=1}^{\mathsf {S}} \left\| \varvec{v}_c - \varvec{W}_2 \cdot \text {ReLU}(\varvec{W}_1 \cdot \varvec{a}_c)\right\| _2^2 + \lambda \cdot R(\varvec{W}_1, \varvec{W}_2), \end{aligned}$$

(19)

where R denotes the $\ell _2$ regularization, $\mathsf {S}$ is the number of seen classes, $\varvec{v}_c$ is the visual exemplar of class c, $\varvec{a}_c$ is the semantic representation of class c, and the weights $\varvec{W}_1$ and $\varvec{W}_2$ are parameters to be optimized.

Following Zhang et al. (2017), we randomly initialize the weights $\varvec{W}_1$ and $\varvec{W}_2$, and set the number of hidden units for AwA and CUB to be 300 and 700, respectively. We use Adam optimizer with a learning rate 0.0001 and minibatch size of $\mathsf {S}$. We tune $\lambda$ on the same splits of data as in other experiments with class-wise CV (Sect. B). Our code is implemented in TensorFlow (Abadi et al. 2016).

Table 17 shows that SVR performs more robustly than MLP. One explanation is that MLP is prone to overfitting due to the small training set size (the number of seen classes) as well as the model selection challenge imposed by ZSL scenarios. SVR also comes with other benefits; it is more efficient and less susceptible to initialization.

1.3 E.3 Effect of PCA

Table 18 investigates the effect of PCA. In general, EXEM (1NN) performs comparably with and without PCA. Moreover, we see that our approach is extremely robust, working reasonably over a wide range of (large enough) $\mathsf {d}$ on all datasets. Clearly, a smaller PCA dimension leads to faster computation due to fewer regressors to be trained.