On-line twin independent support vector machines
Introduction
The classification problem involves constructing a rule for grouping data into a set of categories based on the training set whose categories are previously known. Support vector machine (SVM) [1], [2], [3] is one of the used machine learning approaches for solving pattern recognition problems and is one of the most popular and promising classification algorithms. It was first proposed by Vapnik in 1995 [1] and is based on statistical learning theory and structural risk minimization principle. In this method at first, the data is mapped into a higher dimensional input space and then an optimal separating hyperplane is constructed in this space.
This essentially involves solving a quadratic programming problem. In this approach, kernel functions are selected to maximize the decision boundary in the mapped space.
The quality and complexity of the SVM solution do not depend directly on the input dimensionality because it is based on the structural risk minimization principle and capacity concept with pure combinatorial definitions [1], [2], [3].
SVMs which are computationally powerful tools for supervised learning [4], outperformed many existing systems in a wide range of applications [5], [6], [7]. They are known for their robustness, good generalization ability, and unique global optimum solution in the case of convex problem [8].
The training phase in SVM method is based on a quadratic programming problem, when training samples are small, it can be solved conveniently. However as the number of training samples increases, it becomes harder to use traditional quadratic programming method for solving the problem [9], owing to the scale of the quadratic programming matrix grows quickly. For this reason, finding a way to solve a large-scale quadratic programming problem is a hot topic in SVM researches.
Among a variety of improvements and applications that have been proposed in SVM approaches, incremental improvements are the considered case in this paper, because the power of SVM has not yet been fully developed in incremental or online learning.
Online learning is a classical learning scenario in which training is done via providing one sample at a time, as opposed to the batch mode in which all samples are available at once [10]. An important practical privilege of online algorithms is that they allow incorporating additional training, whenever data is available, without retraining from beginning.
As system should deal with a never-ending data stream and knowledge updating is required after receiving a new sample in online learning, a method to limit the size of the solution and a trade-off between accuracy and speed is desired.
In this paper we proposed a new on-line algorithm, On-line Twin Independent SVM (OTWISVM). Foundation of our algorithm is based on two impressive SVM algorithms, On-line Independent SVM (OISVM) [11] and Twin SVM (TWSVM) [3].
OISVM algorithm constructs a hypothesis via a subset of the samples seen so far for each class separately called basis. When a new sample is received, if they are linearly independent in the feature space from the current basis, it will put it in the relevant basis. This algorithm approximately converges to the batch SVM solution. In this algorithm, a dramatic reduction of time and space requirement is claimed against a negligible loss of accuracy. The size of the basis is limited and does not grow linearly with the training set.
TWSVM finds two nonparallel proximal hyperplanes such that each hyperplane is closer to one of the two classes and positioned as far as possible from the other. It is implemented by solving two smaller SVM-type problems which increases the training speed significantly and has good generalization.
Twin Bounded SVM (TBSVM) [12] is an improved version of TWSVM which its formulation optimization problems are changed slightly by adding a regularization term with the idea of maximizing some margin.
Our proposed method inherits OISVM and TWSVM properties and profits from advantages of both. In OTWISVM, incremental training is done via defining two optimization problems. The problems are solved without introducing the Lagrangian and dual formulation in such a way two nonparallel hyperplanes are found. In this method like TWSVM, each generated hyperplane is closer to one of the two classes and is as far as possible from the other. In OTWISVM, classification is performed based on linearly independent positive and negative basis and size of basis vectors are limited, while retaining the good generalization power of standard SVM.
The significant advantages of our OTWISVM over OISVM are that it performs considerable faster and is more accurate with bounded number of basis.
These statements are supported by an extensive evaluation on standard benchmarks.
This paper is organized as follows. In Section 2, we briefly introduce the related works of applying on-line and incremental learning to SVM. Section 3 reviews primarilies. The proposed algorithm, OTWISVM, is discussed in Section 4. Section 5 describes the experiments and results, and Section 6 concludes the paper.
Section snippets
Related works
Attempts for applying on-line learning or incremental learning to SVM has been done, which can be divided into three categories according to the size of found solution: 1) unbounded [13], [14], [15], [17], 2) amender [18], [20], 3) preventative [21], [22], [23], [24], [25], [26].
Unbounded methods do not make any attempts for reducing the growth of the soloution. In both references [13] and [14], an incremental decremental algorithm of SVM is proposed. Reference [13] expresses single incremental
Primarilies
In this section, we give a brief outline of SVM, TWSVM, TBSVM and OISVM.
Before these overviews, formally, let be the training data set which is a set of column vectors of data points in the real dimensional space . k-th input pattern is and k-th output pattern is and determines the label of k-th instance.
Proposed on-line twin independent support vector machine (OTWISVM)
We proposed a new on-line algorithm, OTWISVM. This algorithm properly makes a twin version of OISVM. OTWISVM has merits of both TWSVM and OISVM.
In OTWISVM, two optimization problems are defined and incremental training is done based on them. The optimization problems can be expressed as follows:where
Experiments and results
In this section at first, the used data set is introduced and then evaluation criteria for measuring the efficiency of algorithms are expressed. Finally the compared methods and results are given.
Conclusion
In this paper, we have developed and examined a new algorithm for efficiently training SVM on-line. The proposed method, OTWISVM, has been inspired by two effectual methods OISVM and TWSVM. It solves two optimization problems without defining the Lagrangian and dual formulation via an incremental approach to find two nonparallel hyperplanes as decision functions with more discriminative ability. Whenever a new sample is available, its linearly independence is checked from two subsets of
Acknowledgments
We would like to thank the High Performance Computing Research Center (HPCRC) of Amirkabir University of Technology for providing computer facilities.
Fatemeh Alamdar received her Master׳s degree in Computer Engineering-Artificial Intelligence from Alzahra University, Iran in 2011. She is currently pursuing the Ph.D. degree at University of Zanjan, Iran. Her research interests include image processing, machine learning, classification and clustering techniques.
References (37)
- et al.
Forecasting stock market movement direction with support vector machine
Comput. Oper. Res.
(2005) - et al.
An application of support vector machines in bankruptcy prediction model
Expert. Syst. Appl.
(2005) - et al.
On-line independent support vector machines
Pattern Recognit.
(2010) The Nature of Statistical Learning Theory
(1995)A tutorial on support vector machines for pattern recognition
Data Min. Knowl. Discov. 2.2
(1998)- et al.
Twin support vector machines for pattern classification
IEEE Trans. Pattern Anal. Mach. Intell.
(2007) Support Vector Machines for Pattern Classification
(2005)- et al.
Support vector channel selection in BCI
IEEE Trans. Biomed. Eng.
(2004) A tutorial on support vector machines for pattern recognition
Data Min. Knowl. Discov.
(1998)- et al.
A new incremental support vector machine algorithm
TELKOMNIKA Indones. J. Electr. Eng.
(2012)
Incremental support vector learning: analysis, implementation and applications
J. Mach. Learn. Res.
Improvements on twin support vector machines
IEEE Trans. Neural Netw.
Multiple incremental decremental learning of support vector machines
IEEE Trans. Neural Netw.
Fast kernel classifiers with online and active learning
J. Mach. Learn. Res.
Exact simplification of support vectors solutions
J. Mach. Learn. Res.
Cited by (13)
Incremental and decremental fuzzy bounded twin support vector machine
2020, Information SciencesCitation Excerpt :In each iteration, it retrains the model considering the previous SVs and the new data points (there is no decremental step). The on-line Twin Independent SVM (OTWISVM) [18] uses a modified Newton method to build a decision function via a subset of data points seen so far for each class separately (called basis). The basis vectors are found (or added during the on-line procedure) during iterative minimization by checking if a new data point is linearly independent in the feature space from the current basis.
Factors affecting households' decisions in biogas technology adoption, the case of Ofla and Mecha Districts, northern Ethiopia
2016, Renewable EnergyCitation Excerpt :Furthermore, logistic regression was employed to find out the underlying factors determining households' decisions on adoption of biogas technology. In the analysis of discrete binary responses, different statistical models like support vector machines [25–29], discriminant analysis, log-linear regression, probit regression or logistic regression [30,31] can be utilized. Support vector machines work with the principle of selecting certain data points from the existing dataset, known as support vectors that help to define an optimal hyperplane separating two sets of observations [28].
Support Vector Machines in Big Data Classification: A Systematic Literature Review
2021, Research SquareSupport vector machines for oil classification link with polyaromatic hydrocarbon contamination in the environment
2021, Water Science and Technology
Fatemeh Alamdar received her Master׳s degree in Computer Engineering-Artificial Intelligence from Alzahra University, Iran in 2011. She is currently pursuing the Ph.D. degree at University of Zanjan, Iran. Her research interests include image processing, machine learning, classification and clustering techniques.
Sara Ghane received her bachelor׳s degree in Computer Engineering in 2012 at University of Guilan, Iran. She is currently a M.Sc. student in Computer Engineering at University of Zanjan, Iran. Her main research interests are machine learning, classification and their intersection in intrusion detection systems.
Ali Amiri was born in Zanjan, Iran. He received the B.Sc. degree in Computer Software Engineering from Zanjan Branch, Islamic Azad University, Zanjan, Iran. He received the M.Sc. degree from Iran University of Science and Technology, Tehran, Iran, and the Ph.D. degree from Iran University of Science and Technology, Tehran, Iran in Computer Engineering, Artificial Intelligence and Robotic, in 2006 and 2011, respectively. Currently, he is an Assistant professor of Computer Engineering Department at University of Zanjan, Iran.