Elsevier

Neurocomputing

Volume 186, 19 April 2016, Pages 8-21
Neurocomputing

On-line twin independent support vector machines

https://doi.org/10.1016/j.neucom.2015.12.062Get rights and content

Highlights

  • On-line twin independent support vector machine (OTWISVM) is proposed.

  • OTWISWM uses way two nonparallel hyper planes as decision functions.

  • The hyperplanes are described by SVs and are updated when new sample is received.

  • The SVs are constructed as a set of linearly independent samples iteratively.

  • Assessments confirm effectiveness of OTWISVM on conventional and noisy data.

Abstract

The success of SVM in solving pattern recognition problems has encouraged researcher to extend the development of different versions. They are well-known for their robustness and good generalization performance. In many real-world applications, the data to be trained are available on-line in a sequential fashion and because of space and time requirements, batch training methods are not suitable.

This paper proposes a new fast on-line algorithm called OTWISVM. It defines two optimization problems and incremental learning is done based of them. Two hyperplanes are generated as decision functions thus each of them is closer to one of the two classes and is as far as possible from the other. The solution is constructed via two subsets of linearly independent samples seen so far, and is always bounded. Good accuracy and notable speed of the method was tested and affirmed both on ordinary and noisy data sets as opposed to similar algorithms.

Introduction

The classification problem involves constructing a rule for grouping data into a set of categories based on the training set whose categories are previously known. Support vector machine (SVM) [1], [2], [3] is one of the used machine learning approaches for solving pattern recognition problems and is one of the most popular and promising classification algorithms. It was first proposed by Vapnik in 1995 [1] and is based on statistical learning theory and structural risk minimization principle. In this method at first, the data is mapped into a higher dimensional input space and then an optimal separating hyperplane is constructed in this space.

This essentially involves solving a quadratic programming problem. In this approach, kernel functions are selected to maximize the decision boundary in the mapped space.

The quality and complexity of the SVM solution do not depend directly on the input dimensionality because it is based on the structural risk minimization principle and capacity concept with pure combinatorial definitions [1], [2], [3].

SVMs which are computationally powerful tools for supervised learning [4], outperformed many existing systems in a wide range of applications [5], [6], [7]. They are known for their robustness, good generalization ability, and unique global optimum solution in the case of convex problem [8].

The training phase in SVM method is based on a quadratic programming problem, when training samples are small, it can be solved conveniently. However as the number of training samples increases, it becomes harder to use traditional quadratic programming method for solving the problem [9], owing to the scale of the quadratic programming matrix grows quickly. For this reason, finding a way to solve a large-scale quadratic programming problem is a hot topic in SVM researches.

Among a variety of improvements and applications that have been proposed in SVM approaches, incremental improvements are the considered case in this paper, because the power of SVM has not yet been fully developed in incremental or online learning.

Online learning is a classical learning scenario in which training is done via providing one sample at a time, as opposed to the batch mode in which all samples are available at once [10]. An important practical privilege of online algorithms is that they allow incorporating additional training, whenever data is available, without retraining from beginning.

As system should deal with a never-ending data stream and knowledge updating is required after receiving a new sample in online learning, a method to limit the size of the solution and a trade-off between accuracy and speed is desired.

In this paper we proposed a new on-line algorithm, On-line Twin Independent SVM (OTWISVM). Foundation of our algorithm is based on two impressive SVM algorithms, On-line Independent SVM (OISVM) [11] and Twin SVM (TWSVM) [3].

OISVM algorithm constructs a hypothesis via a subset of the samples seen so far for each class separately called basis. When a new sample is received, if they are linearly independent in the feature space from the current basis, it will put it in the relevant basis. This algorithm approximately converges to the batch SVM solution. In this algorithm, a dramatic reduction of time and space requirement is claimed against a negligible loss of accuracy. The size of the basis is limited and does not grow linearly with the training set.

TWSVM finds two nonparallel proximal hyperplanes such that each hyperplane is closer to one of the two classes and positioned as far as possible from the other. It is implemented by solving two smaller SVM-type problems which increases the training speed significantly and has good generalization.

Twin Bounded SVM (TBSVM) [12] is an improved version of TWSVM which its formulation optimization problems are changed slightly by adding a regularization term with the idea of maximizing some margin.

Our proposed method inherits OISVM and TWSVM properties and profits from advantages of both. In OTWISVM, incremental training is done via defining two optimization problems. The problems are solved without introducing the Lagrangian and dual formulation in such a way two nonparallel hyperplanes are found. In this method like TWSVM, each generated hyperplane is closer to one of the two classes and is as far as possible from the other. In OTWISVM, classification is performed based on linearly independent positive and negative basis and size of basis vectors are limited, while retaining the good generalization power of standard SVM.

The significant advantages of our OTWISVM over OISVM are that it performs considerable faster and is more accurate with bounded number of basis.

These statements are supported by an extensive evaluation on standard benchmarks.

This paper is organized as follows. In Section 2, we briefly introduce the related works of applying on-line and incremental learning to SVM. Section 3 reviews primarilies. The proposed algorithm, OTWISVM, is discussed in Section 4. Section 5 describes the experiments and results, and Section 6 concludes the paper.

Section snippets

Related works

Attempts for applying on-line learning or incremental learning to SVM has been done, which can be divided into three categories according to the size of found solution: 1) unbounded [13], [14], [15], [17], 2) amender [18], [20], 3) preventative [21], [22], [23], [24], [25], [26].

Unbounded methods do not make any attempts for reducing the growth of the soloution. In both references [13] and [14], an incremental decremental algorithm of SVM is proposed. Reference [13] expresses single incremental

Primarilies

In this section, we give a brief outline of SVM, TWSVM, TBSVM and OISVM.

Before these overviews, formally, let {xk,yk}k=1 be the training data set which is a set of column vectors of data points in the real n dimensional space n. k-th input pattern is xk and k-th output pattern is yk and yk{1,+1} determines the label of k-th instance.

Proposed on-line twin independent support vector machine (OTWISVM)

We proposed a new on-line algorithm, OTWISVM. This algorithm properly makes a twin version of OISVM. OTWISVM has merits of both TWSVM and OISVM.

In OTWISVM, two optimization problems are defined and incremental training is done based on them. The optimization problems can be expressed as follows:Minw+,b+,ξJ(w+,b+,ξ)=12(w+22+b+2)+c12w+Tφ(x+)+e+b+22+c2eTξs.t.(w+Tφ(x)+eb+)eξ,ξ0x+AMinw,b,ηJ(w,b,η)=12(w22+b2)+c32wTφ(x)+eb22+c4e+Tηs.t.(wTφ(x+)+e+b)e+η,η0xBwhere c1,c2

Experiments and results

In this section at first, the used data set is introduced and then evaluation criteria for measuring the efficiency of algorithms are expressed. Finally the compared methods and results are given.

Conclusion

In this paper, we have developed and examined a new algorithm for efficiently training SVM on-line. The proposed method, OTWISVM, has been inspired by two effectual methods OISVM and TWSVM. It solves two optimization problems without defining the Lagrangian and dual formulation via an incremental approach to find two nonparallel hyperplanes as decision functions with more discriminative ability. Whenever a new sample is available, its linearly independence is checked from two subsets of

Acknowledgments

We would like to thank the High Performance Computing Research Center (HPCRC) of Amirkabir University of Technology for providing computer facilities.

Fatemeh Alamdar received her Master׳s degree in Computer Engineering-Artificial Intelligence from Alzahra University, Iran in 2011. She is currently pursuing the Ph.D. degree at University of Zanjan, Iran. Her research interests include image processing, machine learning, classification and clustering techniques.

References (37)

  • W. Huang et al.

    Forecasting stock market movement direction with support vector machine

    Comput. Oper. Res.

    (2005)
  • K.-S. Shin et al.

    An application of support vector machines in bankruptcy prediction model

    Expert. Syst. Appl.

    (2005)
  • F. Orabona et al.

    On-line independent support vector machines

    Pattern Recognit.

    (2010)
  • V. Vapnik

    The Nature of Statistical Learning Theory

    (1995)
  • C.J.C. Burges

    A tutorial on support vector machines for pattern recognition

    Data Min. Knowl. Discov. 2.2

    (1998)
  • R. Khemchandani et al.

    Twin support vector machines for pattern classification

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2007)
  • S. Abe

    Support Vector Machines for Pattern Classification

    (2005)
  • T.N. Lal et al.

    Support vector channel selection in BCI

    IEEE Trans. Biomed. Eng.

    (2004)
  • C.J.C. Burges

    A tutorial on support vector machines for pattern recognition

    Data Min. Knowl. Discov.

    (1998)
  • F. Han et al.

    A new incremental support vector machine algorithm

    TELKOMNIKA Indones. J. Electr. Eng.

    (2012)
  • P. Laskov et al.

    Incremental support vector learning: analysis, implementation and applications

    J. Mach. Learn. Res.

    (2006)
  • Y.H. Shao et al.

    Improvements on twin support vector machines

    IEEE Trans. Neural Netw.

    (2011)
  • G. Cauwenberghs, T. Poggio, Incremental and decremental support vector machine learning. In: Advances in Neural...
  • M. Karasuyama et al.

    Multiple incremental decremental learning of support vector machines

    IEEE Trans. Neural Netw.

    (2010)
  • C. Domeniconi, D. Gunopulos, Incremental support vector machine construction, In: Proceedings of IEEE International...
  • N. Syed, H. Liu, K. Sung, Incremental learning with support vector machines, In: Proceedings of the International Joint...
  • A. Bordes et al.

    Fast kernel classifiers with online and active learning

    J. Mach. Learn. Res.

    (2005)
  • T. Downs et al.

    Exact simplification of support vectors solutions

    J. Mach. Learn. Res.

    (2001)
  • Cited by (13)

    • Incremental and decremental fuzzy bounded twin support vector machine

      2020, Information Sciences
      Citation Excerpt :

      In each iteration, it retrains the model considering the previous SVs and the new data points (there is no decremental step). The on-line Twin Independent SVM (OTWISVM) [18] uses a modified Newton method to build a decision function via a subset of data points seen so far for each class separately (called basis). The basis vectors are found (or added during the on-line procedure) during iterative minimization by checking if a new data point is linearly independent in the feature space from the current basis.

    • Factors affecting households' decisions in biogas technology adoption, the case of Ofla and Mecha Districts, northern Ethiopia

      2016, Renewable Energy
      Citation Excerpt :

      Furthermore, logistic regression was employed to find out the underlying factors determining households' decisions on adoption of biogas technology. In the analysis of discrete binary responses, different statistical models like support vector machines [25–29], discriminant analysis, log-linear regression, probit regression or logistic regression [30,31] can be utilized. Support vector machines work with the principle of selecting certain data points from the existing dataset, known as support vectors that help to define an optimal hyperplane separating two sets of observations [28].

    View all citing articles on Scopus

    Fatemeh Alamdar received her Master׳s degree in Computer Engineering-Artificial Intelligence from Alzahra University, Iran in 2011. She is currently pursuing the Ph.D. degree at University of Zanjan, Iran. Her research interests include image processing, machine learning, classification and clustering techniques.

    Sara Ghane received her bachelor׳s degree in Computer Engineering in 2012 at University of Guilan, Iran. She is currently a M.Sc. student in Computer Engineering at University of Zanjan, Iran. Her main research interests are machine learning, classification and their intersection in intrusion detection systems.

    Ali Amiri was born in Zanjan, Iran. He received the B.Sc. degree in Computer Software Engineering from Zanjan Branch, Islamic Azad University, Zanjan, Iran. He received the M.Sc. degree from Iran University of Science and Technology, Tehran, Iran, and the Ph.D. degree from Iran University of Science and Technology, Tehran, Iran in Computer Engineering, Artificial Intelligence and Robotic, in 2006 and 2011, respectively. Currently, he is an Assistant professor of Computer Engineering Department at University of Zanjan, Iran.

    View full text