Elsevier

Information Sciences

Volume 573, September 2021, Pages 1-19
Information Sciences

Laplacian pair-weight vector projection for semi-supervised learning

https://doi.org/10.1016/j.ins.2021.05.039Get rights and content

Abstract

Semi-supervised learning is a new challenge that exploits the information of unlabeled instances. In this paper, we propose a novel Laplacian pair-weight vector projection (LapPVP) algorithm for semi-supervised classification. LapPVP consists of two stages: projection and classification. In the projection stage, LapPVP integrates a within-class scatter, a between-class scatter and a Laplacian regularization together and formulates a pair of optimization problems. The goal of LapPVP is to maximize the between-class scatter and minimize the within-class scatter on the basis of labeled data, and simultaneously maintain the geometric structure of labeled and unlabeled data in the projected subspace. The optimization problems of LapPVP are identical to generalized eigenvalue ones. Thus, it is easy to obtain the optimal solutions to LapPVP. In the classification stage, LapPVP adopts a minimal distance classifier to implement tasks in the projected subspace. The proposed LapPVP can find a better separability using the geometric information embedded in labeled and unlabeled instances and the discriminative information embedded in labeled instance. Experiments on artificial datasets and UCI datasets validate the feasibility and effectiveness of the proposed algorithm.

Introduction

In classification tasks, the labeled instances have great contributions to predict unknown instances, since the label information can help build a prediction model to achieve good classification performance. The way that only makes use of labeled instances is called supervised learning. In supervised learning, support vector machine (SVM) is a famous learner, which implements the structural risk minimization principle to get good performance and introduces the kernel trick to deal with nonlinear cases [1], [2], [3]. On the basis of SVM, many related classification methods have been derived [4], [5], [6], [7], [8]. These SVM-type methods are powerful tools and have many successful application [9], [10], [11], [12].

It is well known that the standard SVM constructs a separating hyperplane that can separate two-class data points well [2]. Generalized eigenvalue proximal support vector machine (GEPSVM) is different from the standard SVM, which seeks two nonparallel hyperplanes for the complex exclusive or (XOR) problem [13], [14]. The nonparallel hyperplanes are generated by generalized eigenvalue problems instead of solving the quadratic programming problem (QPP) with numerous constraints [15], [16]. Each hyperplane of GEPSVM is required be as close as possible to one class data and meanwhile as far as possible to the other class data [17]. The issue of unstable solution restricts the application of GEPSVM to more complex data. On the basis of GEPSVM, multi-weight vector projection support vector machine (MVSVM) was proposed [18]. MVSVM seeks two optimal weight vector projections instead of two hyperplanes. Each weight vector projection is obtained as the eigenvector corresponding to the maximum eigenvalue of a standard eigenvalue problem [19], such that one class data are close to their class mean and are separated from the other class mean [20]. MVSVM exhibits a better generalization than GEPSVM. Since MVSVM finds only a weight-vector projection for each class that may be not enough for the acquisition of the optimal classification performance, an enhanced multi-weight vector projection support vector machine (EMVSVM) was proposed [21]. EMVSVM redefines the between-class matrix used for measuring the separability between classes.

In practical applications, it is time-consuming and difficult to acquire many labeled instances for training. Oppositely, it is very easy to collect a large amount of unlabeled instances that can also be beneficial to construct a classifier. Methods with a small number of labeled instances and a large amount of unlabeled instances are called semi-supervised learning ones [22], [23], [24]. Recently, semi-supervised learning have attracted much attention. Graph-based semi-supervised learning methods are a kind of popular technologies for utilizing the unlabeled data [25], [26], [27]. Moreover, the Laplacian matrix has been proved as a reasonable and effective representation of graphs [28], [29], [30], [31]. Laplacian support vector machine (LapSVM) is a classic semi-supervised method [32], which introduces the manifold regularization (or Laplacian regularization) into SVM for encoding the geometric information of instances and making the smoothness of the classifier along the intrinsic manifold. On the basis of LapSVM, some variants with a single hyperplane have been proposed, such as cost-sensitive Laplacian support vector machine (Cos-LapSVM) [33] and total margin support vector machine based on Laplacian within-class scatter (LapWCS-TSVM) [34]. Laplacian twin support vector machine (LapTSVM) with two hyperplanes is an extension of nonparallel classifier (twin support vector machine (TSVM)) for the semi-supervised classification tasks [35]. LapTSVM exploits the geometry of the marginal distribution embedded in unlabeled data. To enhance the performance of LapTSVM, Laplacian smooth twin support vector machine (LapSTSVM) was proposed by introducing smooth technique [36]. In order to reduce the computational cost of LapTSVM, Laplacian least squares twin support vector machine (LapLSTSVM) was formulated. LapLSTSVM finds its solutions only by solving two systems of linear equations, which appears to speed up the training procedure [37]. Laplacian twin parametric-margin support vector machine (LapTPMSVM), as an extension of twin parametric-margin support vector machine (TPMSVM), constructs a semi-supervised classifier by solving two smaller-sized SVM-type QPPs [38]. Further, a safe sample screening rule (SSSR) for LapTPMSVM was proposed to reduce the computational cost and handle large-scale data [39].

To extend EMVSVM to deal with semi-supervised classification tasks, this paper proposes a novel Laplacian pair-weight vector projection (LapPVP) algorithm. For labeled instances, LapPVP employs the Fisher discriminant criterion to measure the discriminative information by maximizing the between-class scatter and simultaneously minimizing the within-class scatter. The good discriminative information can make the separating hyperplane close to the same class and stay away from the other class. For both labeled and unlabeled instances, LapPVP exploits the geometric information using the Laplacian regularization term based on the graph. The geometric information is the information contained in data structure. LapPVP makes full use of both labeled and unlabeled instances and can achieve better classification performance. Furthermore, the proposed LapPVP avoids the possible issue of matrix singularity that exists in the traditional projection support vector machine inherently, such as GEPSVM. Moreover, LapPVP implements its training procedure by solving eigenvalue problems instead of QPP, which would be much easier to solve.

The remainder of this paper is organized as follows. We briefly review the related algorithms including EMVSVM in Section 2. Section 3 introduces the Laplacian regularization and describes the proposed algorithm LapPVP for linear and nonlinear cases in detail. Experiments conducted on artificial and UCI datasets are analyzed in Section 4. Finally, the conclusion is drawn in Section 5.

Section snippets

Related work

In this section, we briefly review three related supervised algorithms: GEPSVM, MVSVM and EMVSVM. Suppose that we are given a set of training samples {(xi,yi)}i=1, where xiRm is the i-th instance with the label yi{-1,+1},m and are the number of features and instances, respectively. We denote the training instances with yi=-1 as the data matrix X1=[x11,x12,,x11]TR1×m and those with yi=+1 as the data matrix X2=[x21,x22,,x22]TR2×m, where 1 is the number of instances in class -1,2 is

LapPVP

Although EMVSVM has shown its good generalization for complex data, it is a supervised algorithm that considers only the labeled data for its model training. In the case that it is hard to obtain the label information, the performance of EMVSVM usually goes worse because of an insufficient volume of supervised information. To deal with the issue of lacking labels and further improve the generalization performance of EMVSVM, we elaborate the formulations of Laplacian pair-weight vector

Experiments

In this section, we verify the effectiveness of the proposed LapPVP through a series of experiments based on artificial datasets and benchmark ones [45]. In experiments, we compare our proposed LapPVP with both supervised and semi-supervised methods. The goal of comparison with supervised methods is to demonstrate the validity of using unlabeled data for classification. Compared supervised methods include SVM [1] that is the standard baseline, and GEPSVM [13], MVSVM [18] and EMVSVM [21] that

Conclusion

In this paper, we propose a novel semi-supervised classification method, called LapPVP, which is to maximize the between-class scatter of labeled data, minimize the within-class scatter of labeled data, and maintain the geometric structure of both labeled and unlabeled data in the projected subspace. Once we obtain the pair of projection vectors, we project the unknown instance into two different subspaces and then compute the perpendicular distance between the class mean and its images in

CRediT authorship contribution statement

Yangtao Xue: Investigation, Methodology, Software, Validation, Writing - original draft, Writing - review & editing. Li Zhang: Conceptualization, Methodology, Software, Writing - review & editing, Validation, Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This study was funded by the National Natural Science Foundation of China (grants number 61373093, 61572339), by the Soochow Scholar Project, by the Six Talent Peak Project of Jiangsu Province of China, by the Collaborative Innovation Center of Novel Software Technology and Industrialization, and by Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.

References (49)

  • R. Merris

    Laplacian matrices of graphs: a survey

    Linear Algebra Appl.

    (1994)
  • K. Sato

    Optimal graph laplacian

    Automatica

    (2019)
  • H. Pei et al.

    Laplacian total margin support vector machine based on within-class scatter

    Knowl.-Based Syst.

    (2017)
  • Z. Qi et al.

    Laplacian twin support vector machine for semi-supervised classification

    Neural Networks

    (2012)
  • W. Chen et al.

    Laplacian least squares twin support vector machine for semi-supervised classification

    Neurocomputing

    (2014)
  • Z. Yang et al.

    Laplacian twin parametric-margin support vector machine for semi-supervised classification

    Neurocomputing

    (2016)
  • Z. Yang et al.

    A safe sample screening rule for Laplacian twin parametric-margin support vector machine

    Pattern Recogn.

    (2018)
  • H. Gan et al.

    Using clustering analysis to improve semi-supervised classification

    Neurocomputing

    (2013)
  • C.J.C. Burges

    A tutorial on support vector machines for pattern recognition

    Data Min. Knowl. Discov.

    (1998)
  • V. Vapnik

    Statistical Learning Theory

    Wiley NewYork

    (1995)
  • J.A.K. Suykens et al.

    Least squares support vector machine classifiers

    Neural Process. Lett.

    (1999)
  • L. Zhang et al.

    Wavelet support vector machine

    IEEE Trans. Syst. Man Cybern. Part B

    (2004)
  • Y. Zhang et al.

    Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation

    IEEE Access

    (2017)
  • O.L. Mangasarian et al.

    Multisurface proximal support vector machine classification via generalized eigenvalues

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2006)
  • Cited by (9)

    • A semisupervised classification algorithm combining noise learning theory and a disagreement cotraining framework

      2023, Information Sciences
      Citation Excerpt :

      In this method, the learner automatically modifies the statistical model constructed based on a small amount of labelled data with unlabelled data. Currently, there are four mainstream paradigms of semisupervised learning methods, i.e., generative model-based methods [12–14], graph-based methods [15–17], semisupervised support vector machine methods [18–20], and disagreement-based methods [21–23]. The disagreement-based method is the main form discussed in this paper.

    • Hessian scatter regularized twin support vector machine for semi-supervised classification

      2023, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      To further validate the classification performance of our proposed HSR-TSVM and HSR-LSTSVM, in this section we perform a series of experiments on ten UCI datasets to systematically compare HSR-TSVM and HSR-LSTSVM with LapSVM, LapTSVM, LapLSTSVM, LapTPMSVM, LapPVP and other semi-supervised methods. We used the same experimental setup as in the literature (Xue and Zhang, 2021), here we fixed 10% of the training data as labeled samples and changed the size of the unlabeled samples. The proportion of unlabeled samples to the total training data is 30%, 50% and 70%, respectively.

    • Supervised learning of explicit maps with ability to correct distortions in the target output for manifold learning

      2022, Information Sciences
      Citation Excerpt :

      As regularizers, we have chosen the LE loss function [6] called the Laplacian with three variants of edge weighting functions of the neighborhood graph or Laplacian graph to rectify the structural distortions and restore the disturbed neighborhood relations, if any. The use of the Laplacian matrix to characterize local structures for manifold learning is quite well established [27–29]. The literature on the use of hypergraph Laplacian for various tasks that exploit neighborhood relation is quite rich.

    • Hypergraph regularized semi-supervised support vector machine

      2022, Information Sciences
      Citation Excerpt :

      However, a lot of data lacks labels in real-world and manual labeling is extremely difficult and expensive. Unlabeled data contains abundant useful information which can enhance the performance of a method. [4]. Therefore, how to combine limited labeled data and massive unlabeled data to improve the algorithm performance is a focus of current machine learning researches.

    View all citing articles on Scopus
    View full text