Group discriminative least square regression for multicategory classification

Highlights

• A new regularization of the classifier is given to guarantee group discrimination.

• A group discriminative LSR model is given for multicategory classification.

• Extended experiments show superior performance of the proposed method.

Abstract

The least square regression (LSR) is a popular framework for multicategory classification because it has simple mathematical formulation and efficient solution. The classification performance of LSR based methods depends heavily on the discriminative capability of the label transformation. In this work, we aim to enhance the discriminative capability of the label transformation by imposing a new class-induced structure constraint. Specifically, we propose to regularize the label transformation matrix by the difference of $l_{2,1}$ norm and $l_{2,2}$ norm of the predicted labels of each class. The major advantage of the new regularity is that, it can guarantee the ideal discrimination of the label transformation matrix and make the classification more stable. For better generalization capability, we adopt the existing $\epsilon$-dragging technique to relax the binary label. Leveraging the new regularity term and the label relaxation, we give a group discriminative least square regression (GDLSR) training model to learn the label transformation for multicategory classification. To solve the proposed model, we present an ADMM-like iteration algorithm, which we can guarantee a weak convergency. Experiments on several commonly used datasets show that our method outperforms both related LSR-based methods and some traditional methods.

Introduction

The least square regression (LSR) has wide applications in multi-category pattern classification, because it is mathematically tractable and computationally efficient. In the past decades, many LSR-based methods, such as local LSR [1], weight LSR [2], partial LSR [3], support vector machine (SVM) [4] have been proposed. In addition, several representation-based classification methods, such as sparse representation-based classification (SRC) [5] and linear-regression-based classification (LRC) [6], can also be regarded as LSR based methods, because they learn the representation coefficient by using the LSR technique. Among these methods, linear regression (LR) is widely used because it is simple and highly efficient.

LR [7], [8], [9] based multi-category classification usually follows a two-stage procedure. In the training stage, under supervision of the training data and their known labels, a label transformation matrix is learnt with an LR-based training model; in the test stage, the label transformation matrix is used to predict the class identity of a test sample. A commonly used method for predicting the class identity of a test sample is that, the transformation matrix is used to map a test sample into a label vector, and the class identity is defined as the index corresponding to the maximum entry of the label vector. The classification performance of such methods mainly depends on the discriminative and the generalization capability of the label transformation.

Many LR-based methods have been developed to ensure the discriminative and generalization capability of the label transformation. The discriminative LSR (DLSR) [10] relaxes the one-hot label by using the $\epsilon$-dragging technique. The retargeted LSR (ReLSR) [11] enlarges the margin between the true and false classes more intuitively. Despite their great success by relaxing the labels, these methods often suffer from overfitting [12], [13], [14]. To solve this problem, various regularization techniques have been proposed to impose certain constraint on the transformation matrix of the LR-based training model. The regularized label relaxation linear regression (RLRR) [15] relaxes the strict binary label matrix into a slack variable matrix by introducing a nonnegative label relaxation matrix and avoids overfitting by constructing a class compactness graph. The inter-class sparsity based discriminative least square regression (ICS$\_$DLSR) [16] pursues a common sparsity structure shared by samples within class by using an inter-class sparsity constraint: the $l_{2,1}$ norm. Once the transformation matrix is obtained in the training stage, the nearest neighbor classifier is performed to obtain the final classification results. The group low-rank representation-based discriminative linear regression (GLRRDLR) [17] introduces an inter-class low-rank constraint on the transformed data such that the intrinsic structure of data can be naturally captured and exploited during regression.

By using relaxation and various regularization, the previous methods can ensure the discrimination and generalization capability to some extent. However, the structure regularization in these methods are not sufficient to guarantee the ideal structure or discriminative capability of the predicted labels. Ideally, the predicted binary labels of data from the same class should have such a property: they have one single common entry valued 1 while other entries valued 0. Arranging all the label vectors column-wise into a label matrix, it has one single row of 1s and other rows are all zeros. Analysis in our motivation section suggests that such structure ensures good discriminative capability and stability of the classification. Motivated by this observation, we propose a new regularization on the transformation matrix to guarantee this structure of the predicted labels. Specifically, we regularize the label transformation matrix by minimizing the difference of $l_{2,1}$ norm and $l_{2,2}$ norm of the predicted label matrix of each class. We also adopt the $\epsilon$-dragging technique of DLSR to relax the binary label for better generalization capability. We call it group discriminative least square regression (GDLSR). Experiments on several commonly used datasets show that our method outperforms some state-of-the-art LSR-based methods.