Skip to main content
SpringerLink
Log in

Sparse L1-norm quadratic surface support vector machine with Universum data

  • Data analytics and machine learning
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In binary classification, kernel-free quadratic support vector machines are proposed to avoid difficulties such as finding appropriate kernel functions or tuning their hyper-parameters. Furthermore, Universum data points, which do not belong to any class, can be exploited to embed prior knowledge into the corresponding models to improve the general performance. This paper designs novel kernel-free Universum quadratic surface support vector machine models. Further, this paper proposes the \(\ell _1\) norm regularized version that is beneficial for detecting potential sparsity patterns in the Hessian of the quadratic surface and reducing to the standard linear models if the data points are (almost) linearly separable. The proposed models are convex, so standard numerical solvers can be utilized to solve them. Moreover, a least squares version of the \(\ell _1\) norm regularized model is proposed. We also design an effective tailored algorithm that only requires solving one linear system. Several theoretical properties of these models are then reported and proved as well. The numerical results show that the least squares version of the proposed model achieves the highest mean accuracy scores with promising computational efficiency on some artificial and public benchmark data sets. Some statistical tests are conducted to show the competitiveness of the proposed models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

Enquiries about data availability should be directed to the authors.

Notes

  1. The sources of data sets can be found here: https://github.com/tonygaobasketball/Sparse-UQSSVM-Models-for-Binary-Classification

References

  • Akyildirim E, Goncu A, Sensoy A (2021) Prediction of cryptocurrency returns using machine learning. Ann Oper Res 297(1):3–36

    Article  MathSciNet  MATH  Google Scholar 

  • Arabasadi Z, Alizadehsani R, Roshanzamir M, Moosaei H, Yarifard AA (2017) Computer aided decision making for heart disease detection using hybrid neural network-Genetic algorithm. Comput Methods Programs Biomed 141:19–26

    Article  Google Scholar 

  • Bai Y, Han X, Chen T, Yu H (2015) Quadratic kernel-free least squares support vector machine for target diseases classification. J Comb Optim 30(4):850–870

    Article  MathSciNet  MATH  Google Scholar 

  • Bazikar F, Ketabchi S, Moosaei H (2020) DC programming and DCA for parametric-margin \(\nu \)-support vector machine. Appl Intell 50(6):1763–1774

    Article  Google Scholar 

  • Calvo-Zaragoza J, Valero-Mas JJ, Rico-Juan JR (2017) Selecting promising classes from generated data for an efficient multi-class nearest neighbor classification. Soft Comput 21(20):6183–6189

    Article  Google Scholar 

  • Cherkassky V, Mulier FM (2007) Learning from data: concepts, theory, and methods. Wiley, London

    Book  MATH  Google Scholar 

  • Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    Article  MATH  Google Scholar 

  • Dagher I (2008) Quadratic kernel-free non-linear support vector machine. J Global Optim 41(1):15–30

    Article  MathSciNet  MATH  Google Scholar 

  • Devi RM, Seenivasagam V (2020) Automatic segmentation and classification of liver tumor from CT image using feature difference and SVM based classifier-soft computing technique. Soft Comput 24(24):18591–18598

    Article  Google Scholar 

  • Fayed HA, Atiya AF (2021) Decision boundary clustering for efficient local SVM. Appl Soft Comput 110:107628

    Article  Google Scholar 

  • Gao QQ, Bai YQ, Zhan YR (2019) Quadratic kernel-free least square twin support vector machine for binary classification problems. J Oper Res Soc China 7(4):539–559

    Article  MathSciNet  MATH  Google Scholar 

  • Gao Z, Fang SC, Gao X, Luo J, Medhin N (2021) A novel kernel-free least squares twin support vector machine for fast and accurate multi-class classification. Knowl-Based Syst 226:107123

    Article  Google Scholar 

  • Gao Z, Fang SC, Luo J, Medhin N (2021) A kernel-free double well potential support vector machine with applications. Eur J Oper Res 290(1):248–262

    Article  MathSciNet  MATH  Google Scholar 

  • Gao Z, Wang Y, Huang M, Luo J, Tang S (2022) A kernel-free fuzzy reduced quadratic surface \(\nu \)-support vector machine with applications. Appl Soft Comput 127:109390

    Article  Google Scholar 

  • Javadi SH, Moosaei H, Ciuonzo D (2019) Learning wireless sensor networks for source localization. Sensors 19(3):635

    Article  Google Scholar 

  • Javed AR, Shahzad F, Rehman S, Zikria YB, Razzak I, Jalil Z, Xu G (2022) Future smart cities requirements, emerging technologies, applications, challenges, and future aspects. Cities 129:103794

    Article  Google Scholar 

  • Ketabchi S, Moosaei H, Razzaghi M, Pardalos PM (2019) An improvement on parametric \(\nu \)-support vector algorithm for classification. Ann Oper Res 276(1–2):155–168

    Article  MathSciNet  MATH  Google Scholar 

  • Luo J, Fang SC, Deng Z, Guo X (2016) Soft quadratic surface support vector machine for binary classification. Asia-Pacific J Oper Res 33(06):1650046

    Article  MathSciNet  MATH  Google Scholar 

  • Luo J, Hong T, Fang SC (2018) Benchmarking robustness of load forecasting models under data integrity attacks. Int J Forecast 34(1):89–104

    Article  Google Scholar 

  • Magnus JR, Neudecker H (1980) The elimination matrix: some lemmas and applications. SIAM J Algebr Discrete Methods 1(4):422–449

    Article  MathSciNet  MATH  Google Scholar 

  • Mohammadi N, Doyley MM, Cetin M (2020) A statistical framework for model-based inverse problems in ultrasound elastography. arXiv preprint arXiv:2010.10729

  • Mousavi A, Shen J (2019) Solution uniqueness of convex piecewise affine functions based optimization with applications to constrained \(\ell _1\) minimization. ESAIM Control Optim Calc Variat 25:56

  • Mousavi A, Rezaee M, Ayanzadeh R (2020) A survey on compressive sensing: classical results and recent advancements. J Math Model 8(3):309–344

    MathSciNet  MATH  Google Scholar 

  • Mousavi A, Gao Z, Han L, Lim A (2022) Quadratic surface support vector machine with L1 norm regularization. J Ind Manag Optim 18(3):1835–1861

    Article  MathSciNet  MATH  Google Scholar 

  • Noble WS et al (2004) Support vector machine applications in computational biology. In: Vert JP, B Schoelkopf, K Tsuda (eds) Kernel methods in computational biology, computational molecular biology, MIT Press: London, pp 71–92

  • Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  • Qi Z, Tian Y, Shi Y (2012) Twin support vector machine with universum data. Neural Netw 36:112–119

    Article  MATH  Google Scholar 

  • Shen J, Mousavi A (2018) Least sparsity of \(p\)-norm based optimization problems with \(p>1\). SIAM J Optim 28(3):2721–2751

  • Shen J, Mousavi A (2019) Exact support and vector recovery of constrained sparse vectors via constrained matching pursuit. arXiv preprint arXiv:1903.07236

  • Singh SP, Dhiman G, Tiwari P, Jhaveri RH (2021) A soft computing based multi-objective optimization approach for automatic prediction of software cost models. Appl Soft Comput 113:107981

    Article  Google Scholar 

  • Sinz FH, Chapelle O, Agarwal A, Schölkopf B (2007) An analysis of inference with the universum. In: Proceedings of the 20th international conference on neural information processing systems, Curran Associates Inc., Red Hook, NY, USA, NIPS’07, pp 1369–1376

  • Stone M (1974) Cross-validatory choice and assessment of statistical predictions. J R Stat Soc Ser B (Methodol) 36(2):111–133

    MathSciNet  MATH  Google Scholar 

  • Vapnik VN (1998) Statistical learning theory. Wiley, Chichester

    MATH  Google Scholar 

  • Vapnik VN, Chervonenkis AJ (1974) Theory of pattern recognition. Nauka, Moscow

    MATH  Google Scholar 

  • Wang X, Fan N, Pardalos PM (2018) Robust chance-constrained support vector machines with second-order moment information. Ann Oper Res 263(1):45–68

    Article  MathSciNet  MATH  Google Scholar 

  • Wang XY, Wang T, Bu J (2011) Color image segmentation using pixel wise support vector machine classification. Pattern Recogn 44(4):777–787

    Article  MATH  Google Scholar 

  • Weston J, Collobert R, Sinz F, Bottou L, Vapnik V (2006) Inference with the universum. In: Proceedings of the 23rd international conference on machine learning, pp 1009–1016

  • Xiao Y, Wen J, Liu B (2021) A new multi-task learning method with universum data. Appl Intell 51:3421–3434

    Article  Google Scholar 

  • Yan X, Zhu H (2023) A kernel-free fuzzy support vector machine with universum. J Ind Manag Optim 19(1):282

    Article  MathSciNet  MATH  Google Scholar 

  • Yan X, Bai Y, Fang SC, Luo J (2018) A proximal quadratic surface support vector machine for semi-supervised binary classification. Soft Comput 22(20):6905–6919

    Article  MATH  Google Scholar 

  • Zhou ZH (2021) Machine learning. Springer, Cham

    Book  MATH  Google Scholar 

Download references

Funding

The work of H. Moosaei was supported by the Czech Science Foundation Grant P403-22-11117S and Center for Foundations of Modern Computer Science (Charles Univ. project UNCE/SCI/004). The work of M. Hladík was supported by the Czech Science Foundation Grant P403-22-11117S. The work of Z. Gao was supported by the Fundamental Research Funds for the Central Universities under Grant N2204017 and by the National Natural Science Foundation of China under Grant 72201052.

Author information

Authors and Affiliations

  1. Department of Informatics, Faculty of Science, Jan Evangelista Purkyně University, Ústí nad Labem, Czech Republic

    Hossein Moosaei

  2. Department of Applied Mathematics, School of Computer Science, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

    Hossein Moosaei

  3. Informatics Institute, University of Florida, Gainesville, FL, 32611, USA

    Ahmad Mousavi

  4. Department of Applied Mathematics, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

    Milan Hladík

  5. College of Information Science and Engineering, Northeastern University, Shenyang, 110819, Liaoning, China

    Zheming Gao

Authors
  1. Hossein Moosaei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmad Mousavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Milan Hladík
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zheming Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HM was involved in conceptualization, supervision and methodology; AM was participated in conceptualization and methodology; MH took part in conceptualization; ZG contributed to software and validation; and all the authors are responsible for writing, reviewing and editing the manuscript.

Corresponding author

Correspondence to Zheming Gao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consents

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moosaei, H., Mousavi, A., Hladík, M. et al. Sparse L1-norm quadratic surface support vector machine with Universum data. Soft Comput 27, 5567–5586 (2023). https://doi.org/10.1007/s00500-023-07860-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-07860-3

Keywords

Search

Navigation