Skip to main content
SpringerLink
Log in

A new multi-task learning method with universum data

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Multi-task learning (MTL) obtains a better classifier than single-task learning (STL) by sharing information between tasks within the multi-task models. Most existing multi-task learning models only focus on the data of the target tasks during training, and ignore the data of non-target tasks that may be contained in the target tasks. In this way, Universum data can be added to classifier training as prior knowledge, and these data do not belong to any indicated categories. In this paper, we address the problem of multi-task learning with Universum data, which improves utilization of non-target task data. We introduce Universum learning to make non-target task data act as prior knowledge and propose a novel multi-task support vector machine with Universum data (U-MTLSVM). Based on the characteristics of MTL, each task have corresponding Universum data to provide prior knowledge. We then utilize the Lagrange method to solve the optimization problem so as to obtain the multi-task classifiers. Then, conduct experiments to compare the performance of the proposed method with several baslines on different data sets. The experimental results demonstrate the effectiveness of the proposed methods for multi-task classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://people.csail.mit.edu/jrennie/20Newsgroups/

  2. http://www.daviddlewis.com/resources/testcollections/

  3. http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/webkb-data.gtar.gz

  4. http://people.ee.duke.edu/ lcarin/LandmineData.zip

References

  1. Smith FW (1968) Pattern classifier design by linear programming. IEEE Transactions on Computers C-17 4:367–372

    Article  Google Scholar 

  2. Yuan R, Li Z, Guan X, Xu L (2010) An svm-based machine learning method for accurate internet traffic classification. Inf Syst Front 12(2):149–156

    Article  Google Scholar 

  3. Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29 (5):1189–1232

    Article  MathSciNet  Google Scholar 

  4. Su C, Yang F, Zhang S, Tian Q, Davis LS, Gao W (2015) Multi-task learning with low rank attribute embedding for person re-identification pp 3739–3747

  5. Xue Y, Liao X, Carin L, Krishnapuram B (2007) Multi-task learning for classification with dirichlet process priors. J Mach Learn Res 8:35–63

    MathSciNet  MATH  Google Scholar 

  6. Tang Z, Li L, Wang D, Vipperla R (2017) Collaborative joint training with multitask recurrent model for speech and speaker recognition. IEEE Transactions on Audio Speech, and Language Processing 25 (3):493–504

    Article  Google Scholar 

  7. Do VH, Chen NF, Lim BP, Hasegawajohnson M (2018) Multitask learning for phone recognition of underresourced languages using mismatched transcription. IEEE Transactions on Audio Speech, and Language Processing 26(3):501–514

    Article  Google Scholar 

  8. Xie X, Zhai X, Hou F, Hao A, Qin H (2019) Multitask learning on monocular water images: surface reconstruction and image synthesis. computer animation and virtual worlds

  9. Yan C, Zhou L, Wan Y (2019) A multi-task learning model for better representation of clothing images. IEEE Access 7:34499–34507

    Article  Google Scholar 

  10. Ji Y, Sun S (2013) Multitask multiclass support vector machines: Model and experiments. Pattern Recogn 46(3):914–924

    Article  Google Scholar 

  11. Chen J, Tang L, Liu J, Ye J (2009) A convex formulation for learning shared structures from multiple tasks pp 137–144

  12. Blackwood G, Ballesteros M, Ward T (2018) Multilingual neural machine translation with task-specific attention pp 3112–3122.

  13. Das A, Dantcheva A, Bremond F (2018) Mitigating bias in gender, age and ethnicity classification: A multi-task convolution neural network approach pp 573–585

  14. Salucci M, Poli L, Oliveri G (2019) Full-vectorial 3d microwave imaging of sparse scatterers through a multi-task bayesian compressive sensing approach

  15. Huang Y, Beck JL, Li H (2019) Multitask sparse bayesian learning with applications in structural health monitoring. Computer-aided Civil and Infrastructure Engineering 34(9):732–754

    Article  Google Scholar 

  16. Wang S, Chang X, Li X, Sheng QZ, Chen W (2016) Multi-task support vector machines for feature selection with shared knowledge discovery. Signal Process 120:746–753

    Article  Google Scholar 

  17. Chandra R, Gupta A, Ong Y, Goh C (2018) Evolutionary multi-task learning for modular knowledge representation in neural networks. Neural Process Lett 47(3):993–1009

    Article  Google Scholar 

  18. Pearce M, Branke J (2018) Continuous multi-task bayesian optimisation with correlation. European Journal of Operational Research p 270(3):1074–1085

    Article  MathSciNet  Google Scholar 

  19. Vapnik V (1998a) Statistical Learning Theory, dblp

  20. Vapnik V (1998b) Estimation of Dependence Based on Empirical Data, springer

  21. Qi Z, Tian Y, Shi Y (2012) Twin support vector machine with universum data. Neural Netw 36:112–119

    Article  Google Scholar 

  22. Liu CL, Lee CH (2016) Enhancing text classification with the universum. In: 2016 12th International Conference on Natural Computation and 13th Fuzzy Systems and Knowledge Discovery (ICNC-FSKD)

  23. Pan S, Wu J, Zhu X, Long G, Zhang C (2017) Boosting for graph classification with universum. Knowledge & Information Systems 50(1):1–25

    Article  Google Scholar 

  24. Richhariya B, Gupta D (2018) Facial expression recognition using iterative universum twin support vector machine, applied soft computing

  25. Weston J, Collobert R, Sinz FH, Bottou L, Vapnik V (2006) Inference with the universum pp 1009–1016

  26. Sinz F, Chapelle O, Agarwal A, Scholkopf B, Platt CJ, Koller D, Singer Y, Roweis S (2007) An analysis of inference with the universum. Advances in Neural Information Processing Systems 20 (2008):1369–1376

    Google Scholar 

  27. Qi Z, Tian Y, Shi Y (2014) A nonparallel support vector machine for a classification problem with universum learning. J Comput Appl Math 263(1):288–298

    Article  MathSciNet  Google Scholar 

  28. Xu Y, Chen M, Yang Z, Li G (2016) V-twin support vector machine with universum data for classification. Applied Intelligence 44(4):956–968

    Article  Google Scholar 

  29. Liu D, Tian Y, Bie R, Shi Y (2014) Self-universum support vector machine. Personal and ubiquitous computing 18(8):1813–1819

    Article  Google Scholar 

  30. Chen S, Zhang C (2009) Selecting informative universum sample for semi-supervised learning. In: International Jont Conference on Artifical Intelligence

  31. Dhar S, Cherkassky V (2015) Development and evaluation of cost-sensitive universum-svm. IEEE Transactions on Systems Man, and Cybernetics 45(4):806–818

    Google Scholar 

  32. Zhao J, Xu Y (2017) A safe sample screening rule for universum support vector machines. Knowledge Based Systems 138:46–57

    Article  Google Scholar 

  33. Zhu C, Miao D, Zhou R, Wei L (2020) Weight-and-universum-based semi-supervised multi-view learning machine. Soft Comput 24(14):10657–10679

    Article  Google Scholar 

  34. Chen X, Yin H, Jiang F, Wang L (2018) Multi-view dimensionality reduction based on universum learning. Neurocomputing 275:2279–2286

    Article  Google Scholar 

  35. Wang Z, Zhu Y, Liu W, Chen Z, Gao D (2014) Multi-view learning with universum. Knowledge Based Systems 70:376–391

    Article  Google Scholar 

  36. Deng J, Xu X, Zhang Z, Fruhholz S, Schuller B (2017) Universum autoencoder-based domain adaptation for speech emotion recognition. IEEE Signal Processing Letters 24(4):500–504

    Article  Google Scholar 

  37. Songsiri P, Cherkassky V, Kijsirikul B (2018) Universum selection for boosting the performance of multiclass support vector machines based on one-versus-one strategy. Knowledge Based Systems 159:9–19

    Article  Google Scholar 

  38. Ruder S (2017) An overview of multi-task learning in deep neural networks. arXiv: Learning

  39. Fawzi A, Sinn M, Frossard P (2017) Multitask additive models with shared transfer functions based on dictionary learning. International conference on machine learning 65(5):1352–1365

    MathSciNet  MATH  Google Scholar 

  40. Palmieri N, Yang X, De Rango F, Santamaria AF (2018) Self-adaptive decision-making mechanisms to balance the execution of multiple tasks for a multi-robots team. Neurocomputing 306:17–36

    Article  Google Scholar 

  41. Khosravan N, Bagci U (2018) Semi-supervised multi-task learning for lung cancer diagnosis pp 710–713

  42. Wen Z, Li K, Huang Z, Lee C, Tao J, Wen Z (2017) Improving deep neural network based speech synthesis through contextual feature parametrization and multi-task learning. Journal of Signal Processing Systems 90:0–1037

    Google Scholar 

  43. Chikontwe P, Lee HJ (2018) Deep multi-task network for learning person identity and attributes. IEEE Access 6:1–1

    Article  Google Scholar 

  44. Su C, Yang F, Zhang S, Tian Q, Davis LS, Gao W (2017) Multi-task learning with low rank attribute embedding for multi-camera person re-identification. IEEE Transactions on Pattern Analysis & Machine Intelligence 40:1167–1181

    Article  Google Scholar 

  45. Ma AJ, Yuen PC, Li J (2013) Domain transfer support vector ranking for person re-identification without target camera label information pp 3567–3574

  46. Zhi S, Liu Y, Li X, Guo Y (2017) Toward real-time 3d object recognition: a lightweight volumetric cnn framework using multitask learning. Computers & Graphics 71(APR.):199–207

    Google Scholar 

  47. Zhang K, Zheng L, Liu Z, Jia N (2019) A deep learning based multitask model for network-wide traffic speed predication. Neurocomputing

  48. Huang Y, Beck JL, Li H (2019) Multitask sparse bayesian learning with applications in structural health monitoring. Computer-Aided Civil and Infrastructure Engineering 34(9):732–754

    Article  Google Scholar 

  49. Zhang Y, Yang Y, Li t, Fujita H (2018) A multitask multiview clustering algorithm in heterogeneous situations based on lle and le. knowledge based systems

  50. Chen J, Liu J, Ye J (2012) Learning incoherent sparse and low-rank patterns from multiple tasks. Acm Transactions on Knowledge Discovery from Data 5(4):1–31

    Article  Google Scholar 

  51. Gong P, Ye J, Zhang C (2013) Multi-stage multi-task feature learning pp 2979–3010

  52. Li Y, Tian X, Liu T (2017) On better exploring and exploiting task relationships in multi-task learning: joint model and feature learning pp 1–11

  53. Yousefi F, Smith M, Alvarez MA (2019) Multi-task learning for aggregated data using gaussian processes. arXiv: machine learning

  54. Richhariya B, Sharma A, Tanveer M (2018) Improved universum twin support vector machine. In: IEEE Symposium Series on Computational Intelligence SSCI

  55. Mei B, Xu Y (2019) Multi-task least squares twin support vector machine for classification. neurocomputing

  56. Ping L i, Chen S (2018) Hierarchical gaussian processes model for multi-task learning, pattern recognition the journal of the pattern recognition society

  57. Alzaidy RA, Caragea C, Giles CL (2019) Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents pp 2551–2557

  58. Joachims T (1998) Text categorization with support vector machines: Learning with many relevant features, ecml

  59. Liu B, Xiao Y, Hao Z (2018) A selective multiple instance transfer learning method for text categorization problems. Knowl-Based Syst 141(1):178–187

    Article  Google Scholar 

  60. Evgeniou T, Pontil M (2004) Regularized multi–task learning. In: Tenth Acm Sigkdd International Conference on Knowledge Discovery & Data Mining

Download references

Acknowledgments

The authors would like to thank the reviewers for their very useful comments and suggestions. This work was supported in part by the Natural Science Foundation of China under Grant 61876044, 62076074 and Grant 61672169, in part by Guangdong Natural Science Foundation under Grant 2020A1515010670 and 2020A1515011501, in part by the Science and Technology Planning Project of Guangzhou under Grant 202002030141.

Author information

Authors and Affiliations

  1. Department of Computer Science, Guangdong University of Technology, Guangzhou, Guangdong, China

    Yanshan Xiao & Jing Wen

  2. Department of Automation, Guangdong University of Technology, Guangzhou, Guangdong, China

    Bo Liu

Authors
  1. Yanshan Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiao, Y., Wen, J. & Liu, B. A new multi-task learning method with universum data. Appl Intell 51, 3421–3434 (2021). https://doi.org/10.1007/s10489-020-01954-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01954-3

Keywords

Search

Navigation