Dependence maximization based label space dimension reduction for multi-label classification

Abstract

High dimensionality of label space poses crucial challenge to efficient multi-label classification. Therefore, it is needed to reduce the dimensionality of label space. In this paper, we propose a new algorithm, called dependence maximization based label space reduction (DMLR), which maximizes the dependence between feature vectors and code vectors via Hilbert–Schmidt independence criterion while minimizing the encoding loss of labels. Two different kinds of instance kernel are discussed. The global kernel for DMLR_G and the local kernel for DM_LR_L take global information and locality information into consideration respectively. Experimental results over six categorization problems validate the superiority of the proposed algorithm to state-of-art label space dimension reduction methods in improving performance at the cost of a very short time.

Introduction

During the last decade, multi-label classification has aroused the interest of researchers from engineering and academic areas because of its wide applications in real world. In multi-label setting, a document may be associated with multiple categories (Ji et al., 2010, Ueda and Saito, 2003); an image may be annotated with several concepts (Boutell et al., 2004). It is rather different from the traditional single-label (binary or multi-class) classification where each document is only allowed to be associated with one category.

A lot of algorithms have been proposed for multi-label classification (Zhang and Zhou, 2014). Currently a consensus on multi-label classification is that label correlations play an important role and should be utilized for performance improvement (Dembczyński et al., 2010, Zhang and Zhang, 2010, Zhang and Zhou, 2014). Most algorithms usually build classification model based on some label correlation assumption, such as ensemble of classifier chains (ECC) (Read et al., 2011) and calibrated label ranking (CLR) (Fürankranz et al., 2008).

Although these algorithms achieve satisfactory results, they suffer from computational inefficiency in both training and testing even for the most intuitive approach binary relevance (BR) (Boutell et al., 2004), which decomposes a multi-label classification problem into several independent binary classification problems, one for each label, based on one-versus-all (OVA) strategy (Hastie et al., 2009). This problem poses a rather crucial challenge to classification especially when there are a large number of possible labels. Therefore, it is necessary to explore ways that balances the classification performance and computational efforts.

Several algorithms on label space dimension reduction (LSDR) have been proposed along this avenue, which can be categorized into two groups: learning methods and reduction methods. The former group reduces the label space while jointly learning a classifier from the instances to the code vectors as well, for example multi-label prediction via compressed sensing (CS) (Hsu et al., 2009). We can obtain a classifier finally and use it for predicting directly. However, in order to get a promising classifier, these methods often employ complicated algorithms in the learning part, which again costs too much time. Therefore, the latter group is the mainstream in this avenue.

The latter group focuses on how to efficiently compress the label space and does not consider what learning algorithm to apply after compression. An exemplar is principal label space transformation (PLST) (Tai and Lin, 2012), which only reduces the dimensionality of label space by analyzing the principal components. A key problem of this group is on how to utilize the instances, which is still an open question. Since the ultimate objective is to make classification, some methods only use a simple model from instances to code vectors, for instance, conditional PLST (CPLST) (Chen and Lin, 2012). Nevertheless, this strategy might be suboptimal as it may over-fit the learnt model, which has a negative impact on the learning process later.

In this paper, we propose a new LSDR method, called dependence maximization based label space reduction (DMLR), which can be categorized into the latter group as a reduction method. Different from previous reduction methods, it assumes that the objective function should consist of two components: encoding loss and dependence loss. The former one measures the loss of label compression while the latter one measures the dependence between instances and code vectors. Specifically, it measures the encoding loss using least square loss function as used in PLST and measures the dependence loss based on Hilbert–Schmidt independence criterion (HSIC) (Gretton et al., 2005). Two different instance kernels are applied and we obtain two methods: DMLR_G with the instance kernel exploiting global information and DMLR_L with the instance kernel exploiting local information. Experimental results across six data sets from various application domains validate the superiority of two proposed algorithms to two state-of-art LSDR methods, PLST and CPLST, in performance and save a lot of training and testing time compared with a simple representative multi-label classification method – BR. Moreover, DMLR_L outperforms DMLR_G in performance in most cases and costs similar or less training time due to the sparsity of instance kernel used in DMLR_L.

The rest of this paper is organized as follows. Section 2 presents a brief literature review on multi-label classification algorithms and pays more attention on LSDR methods and the HSIC. We describe the two proposed algorithms, DMLR_G and DMLR_L, in detail in Section 3 and. Experimental results and discussion are given in Section 4 Finally, Section 5 concludes this paper and presents some clues for future work.