Laplacian pair-weight vector projection for semi-supervised learning
Introduction
In classification tasks, the labeled instances have great contributions to predict unknown instances, since the label information can help build a prediction model to achieve good classification performance. The way that only makes use of labeled instances is called supervised learning. In supervised learning, support vector machine (SVM) is a famous learner, which implements the structural risk minimization principle to get good performance and introduces the kernel trick to deal with nonlinear cases [1], [2], [3]. On the basis of SVM, many related classification methods have been derived [4], [5], [6], [7], [8]. These SVM-type methods are powerful tools and have many successful application [9], [10], [11], [12].
It is well known that the standard SVM constructs a separating hyperplane that can separate two-class data points well [2]. Generalized eigenvalue proximal support vector machine (GEPSVM) is different from the standard SVM, which seeks two nonparallel hyperplanes for the complex exclusive or (XOR) problem [13], [14]. The nonparallel hyperplanes are generated by generalized eigenvalue problems instead of solving the quadratic programming problem (QPP) with numerous constraints [15], [16]. Each hyperplane of GEPSVM is required be as close as possible to one class data and meanwhile as far as possible to the other class data [17]. The issue of unstable solution restricts the application of GEPSVM to more complex data. On the basis of GEPSVM, multi-weight vector projection support vector machine (MVSVM) was proposed [18]. MVSVM seeks two optimal weight vector projections instead of two hyperplanes. Each weight vector projection is obtained as the eigenvector corresponding to the maximum eigenvalue of a standard eigenvalue problem [19], such that one class data are close to their class mean and are separated from the other class mean [20]. MVSVM exhibits a better generalization than GEPSVM. Since MVSVM finds only a weight-vector projection for each class that may be not enough for the acquisition of the optimal classification performance, an enhanced multi-weight vector projection support vector machine (EMVSVM) was proposed [21]. EMVSVM redefines the between-class matrix used for measuring the separability between classes.
In practical applications, it is time-consuming and difficult to acquire many labeled instances for training. Oppositely, it is very easy to collect a large amount of unlabeled instances that can also be beneficial to construct a classifier. Methods with a small number of labeled instances and a large amount of unlabeled instances are called semi-supervised learning ones [22], [23], [24]. Recently, semi-supervised learning have attracted much attention. Graph-based semi-supervised learning methods are a kind of popular technologies for utilizing the unlabeled data [25], [26], [27]. Moreover, the Laplacian matrix has been proved as a reasonable and effective representation of graphs [28], [29], [30], [31]. Laplacian support vector machine (LapSVM) is a classic semi-supervised method [32], which introduces the manifold regularization (or Laplacian regularization) into SVM for encoding the geometric information of instances and making the smoothness of the classifier along the intrinsic manifold. On the basis of LapSVM, some variants with a single hyperplane have been proposed, such as cost-sensitive Laplacian support vector machine (Cos-LapSVM) [33] and total margin support vector machine based on Laplacian within-class scatter (LapWCS-TSVM) [34]. Laplacian twin support vector machine (LapTSVM) with two hyperplanes is an extension of nonparallel classifier (twin support vector machine (TSVM)) for the semi-supervised classification tasks [35]. LapTSVM exploits the geometry of the marginal distribution embedded in unlabeled data. To enhance the performance of LapTSVM, Laplacian smooth twin support vector machine (LapSTSVM) was proposed by introducing smooth technique [36]. In order to reduce the computational cost of LapTSVM, Laplacian least squares twin support vector machine (LapLSTSVM) was formulated. LapLSTSVM finds its solutions only by solving two systems of linear equations, which appears to speed up the training procedure [37]. Laplacian twin parametric-margin support vector machine (LapTPMSVM), as an extension of twin parametric-margin support vector machine (TPMSVM), constructs a semi-supervised classifier by solving two smaller-sized SVM-type QPPs [38]. Further, a safe sample screening rule (SSSR) for LapTPMSVM was proposed to reduce the computational cost and handle large-scale data [39].
To extend EMVSVM to deal with semi-supervised classification tasks, this paper proposes a novel Laplacian pair-weight vector projection (LapPVP) algorithm. For labeled instances, LapPVP employs the Fisher discriminant criterion to measure the discriminative information by maximizing the between-class scatter and simultaneously minimizing the within-class scatter. The good discriminative information can make the separating hyperplane close to the same class and stay away from the other class. For both labeled and unlabeled instances, LapPVP exploits the geometric information using the Laplacian regularization term based on the graph. The geometric information is the information contained in data structure. LapPVP makes full use of both labeled and unlabeled instances and can achieve better classification performance. Furthermore, the proposed LapPVP avoids the possible issue of matrix singularity that exists in the traditional projection support vector machine inherently, such as GEPSVM. Moreover, LapPVP implements its training procedure by solving eigenvalue problems instead of QPP, which would be much easier to solve.
The remainder of this paper is organized as follows. We briefly review the related algorithms including EMVSVM in Section 2. Section 3 introduces the Laplacian regularization and describes the proposed algorithm LapPVP for linear and nonlinear cases in detail. Experiments conducted on artificial and UCI datasets are analyzed in Section 4. Finally, the conclusion is drawn in Section 5.
Section snippets
Related work
In this section, we briefly review three related supervised algorithms: GEPSVM, MVSVM and EMVSVM. Suppose that we are given a set of training samples , where is the i-th instance with the label and are the number of features and instances, respectively. We denote the training instances with as the data matrix and those with as the data matrix , where is the number of instances in class is
LapPVP
Although EMVSVM has shown its good generalization for complex data, it is a supervised algorithm that considers only the labeled data for its model training. In the case that it is hard to obtain the label information, the performance of EMVSVM usually goes worse because of an insufficient volume of supervised information. To deal with the issue of lacking labels and further improve the generalization performance of EMVSVM, we elaborate the formulations of Laplacian pair-weight vector
Experiments
In this section, we verify the effectiveness of the proposed LapPVP through a series of experiments based on artificial datasets and benchmark ones [45]. In experiments, we compare our proposed LapPVP with both supervised and semi-supervised methods. The goal of comparison with supervised methods is to demonstrate the validity of using unlabeled data for classification. Compared supervised methods include SVM [1] that is the standard baseline, and GEPSVM [13], MVSVM [18] and EMVSVM [21] that
Conclusion
In this paper, we propose a novel semi-supervised classification method, called LapPVP, which is to maximize the between-class scatter of labeled data, minimize the within-class scatter of labeled data, and maintain the geometric structure of both labeled and unlabeled data in the projected subspace. Once we obtain the pair of projection vectors, we project the unknown instance into two different subspaces and then compute the perpendicular distance between the class mean and its images in
CRediT authorship contribution statement
Yangtao Xue: Investigation, Methodology, Software, Validation, Writing - original draft, Writing - review & editing. Li Zhang: Conceptualization, Methodology, Software, Writing - review & editing, Validation, Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
This study was funded by the National Natural Science Foundation of China (grants number 61373093, 61572339), by the Soochow Scholar Project, by the Six Talent Peak Project of Jiangsu Province of China, by the Collaborative Innovation Center of Novel Software Technology and Industrialization, and by Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.
References (49)
- et al.
Complex-valued support vector classifiers
Digital Signal Process.
(2010) - et al.
Density-induced margin support vector machines
Pattern Recogn.
(2011) Tpmsvm: a novel twin parametric-margin support vector machine for pattern recognition
Pattern Recogn.
(2011)- et al.
Fisher-regularized support vector machine
Inf. Sci.
(2016) - et al.
Credit rating analysis with support vector machines and neural networks: a market comparative study
Decis. Support Syst.
(2004) - et al.
Dynamic financial distress prediction with concept drift based on time weighting combined with adaboost support vector machine ensemble
Knowl.-Based Syst.
(2017) - et al.
Spatial and class structure regularized sparse representation graph for semi-supervised hyperspectral image classification
Pattern Recogn.
(2018) - et al.
Multi-weight vector projection support vector machines
Pattern Recogn. Lett.
(2010) - et al.
A study on three linear discriminant analysis based methods in small sample size problem
Pattern Recogn.
(2008) - et al.
Enhanced multi-weight vector projection support vector machine
Pattern Recogn. Lett.
(2014)
Laplacian matrices of graphs: a survey
Linear Algebra Appl.
Optimal graph laplacian
Automatica
Laplacian total margin support vector machine based on within-class scatter
Knowl.-Based Syst.
Laplacian twin support vector machine for semi-supervised classification
Neural Networks
Laplacian least squares twin support vector machine for semi-supervised classification
Neurocomputing
Laplacian twin parametric-margin support vector machine for semi-supervised classification
Neurocomputing
A safe sample screening rule for Laplacian twin parametric-margin support vector machine
Pattern Recogn.
Using clustering analysis to improve semi-supervised classification
Neurocomputing
A tutorial on support vector machines for pattern recognition
Data Min. Knowl. Discov.
Statistical Learning Theory
Wiley NewYork
Least squares support vector machine classifiers
Neural Process. Lett.
Wavelet support vector machine
IEEE Trans. Syst. Man Cybern. Part B
Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation
IEEE Access
Multisurface proximal support vector machine classification via generalized eigenvalues
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (9)
A semisupervised classification algorithm combining noise learning theory and a disagreement cotraining framework
2023, Information SciencesCitation Excerpt :In this method, the learner automatically modifies the statistical model constructed based on a small amount of labelled data with unlabelled data. Currently, there are four mainstream paradigms of semisupervised learning methods, i.e., generative model-based methods [12–14], graph-based methods [15–17], semisupervised support vector machine methods [18–20], and disagreement-based methods [21–23]. The disagreement-based method is the main form discussed in this paper.
Hessian scatter regularized twin support vector machine for semi-supervised classification
2023, Engineering Applications of Artificial IntelligenceCitation Excerpt :To further validate the classification performance of our proposed HSR-TSVM and HSR-LSTSVM, in this section we perform a series of experiments on ten UCI datasets to systematically compare HSR-TSVM and HSR-LSTSVM with LapSVM, LapTSVM, LapLSTSVM, LapTPMSVM, LapPVP and other semi-supervised methods. We used the same experimental setup as in the literature (Xue and Zhang, 2021), here we fixed 10% of the training data as labeled samples and changed the size of the unlabeled samples. The proportion of unlabeled samples to the total training data is 30%, 50% and 70%, respectively.
Supervised learning of explicit maps with ability to correct distortions in the target output for manifold learning
2022, Information SciencesCitation Excerpt :As regularizers, we have chosen the LE loss function [6] called the Laplacian with three variants of edge weighting functions of the neighborhood graph or Laplacian graph to rectify the structural distortions and restore the disturbed neighborhood relations, if any. The use of the Laplacian matrix to characterize local structures for manifold learning is quite well established [27–29]. The literature on the use of hypergraph Laplacian for various tasks that exploit neighborhood relation is quite rich.
Hypergraph regularized semi-supervised support vector machine
2022, Information SciencesCitation Excerpt :However, a lot of data lacks labels in real-world and manual labeling is extremely difficult and expensive. Unlabeled data contains abundant useful information which can enhance the performance of a method. [4]. Therefore, how to combine limited labeled data and massive unlabeled data to improve the algorithm performance is a focus of current machine learning researches.