Skip to main content
SpringerLink
Log in

BSRU: boosting semi-supervised regressor through ramp-up unsupervised loss

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Semi-supervised regression aims to improve the performance of the learner with the help of unlabeled data. Popular approaches select some unlabeled data with high-quality pseudo labels to enrich the training set. In this paper, we propose a new approach with a semi-supervised regressor, a learner, and a respective loss function. First, an off-the-shelf semi-supervised regressor is trained to provide pseudo labels for all unlabeled data. These labels are often reliable enough to guide the learning process. Second, we design a neural network with dropout to train data with Gaussian noise added. In this way, the robustness of our learners is enhanced. Third, we design a weighted sum combining the supervised and unsupervised loss. The weight for pseudo-labels ramp-up over time, indicating more attention to the pseudo-labels. Six state-of-the-art algorithms are employed as the base model of our framework. Results on 15 real-world data sets show that our model has a significant improvement over the respective base regressor on most data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.lamda.nju.edu.cn/code_COREG.ashx.

  2. http://www.lamda.nju.edu.cn/code_SAFER.ashx.

  3. http://ml.upatras.gr/mssregression.

  4. https://github.com/timilsinamohan/SSR.

References

  1. Zhu X-J, Goldberg AB (2009) Introduction to semi-supervised learning. Synthe Lect Artif Intell Mach Learn 3(1):1–130. https://doi.org/10.2200/S00196ED1V01Y200906AIM006

    Article  Google Scholar 

  2. Engelen JEV, Hoos HH (2020) A survey on semi-supervised learning. Mach Learn 109(2):373–440. https://doi.org/10.1007/s10994-019-05855-6

    Article  MathSciNet  Google Scholar 

  3. Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: COLT, pp 92–100

  4. Zhou Z-H, Li M (2005) Semi-supervised regression with co-training. In: IJCAI, pp 908–913

  5. Zhou Z-H, Li M (2005) Tri-training: exploiting unlabeled data using three classifiers. IEEE Trans Knowl Data Eng 17(11):1529–1541. https://doi.org/10.1109/TKDE.2005.186

    Article  Google Scholar 

  6. Hady MFA, Schwenker F, Palm G (2009) Semi-supervised learning for regression with co-training by committee. In: ICANN, pp 121–130

  7. Qiao S, Shen W, Zhang Z, Wang B, Yuille A (2018) Deep co-training for semi-supervised image recognition. In: ECCV, pp 142–159

  8. Chen D-D, Wang W, Gao W, Zhou Z-H (2018) Tri-Net for semi-supervised deep learning. In: IJCAI, pp 2014–2020

  9. Zhang J, Li M, Gao K, Meng S, Zhou C (2021) Word and graph attention networks for semi-supervised classification. Knowl Inf Syst 63(11):2841–2859. https://doi.org/10.1007/s10115-021-01610-3

    Article  Google Scholar 

  10. Zhang T, Zhu T, Han M, Chen F, Li J, Zhou W, Yu PS (2022) Fairness in graph-based semi-supervised learning. Knowl Inf Syst 65(2):543–570. https://doi.org/10.1007/s10115-022-01738-w

    Article  Google Scholar 

  11. Lebichot B, Saerens M (2020) An experimental study of graph-based semi-supervised classification with additional node information. Knowl Inf Syst 62(11):4337–4371. https://doi.org/10.1007/s10115-020-01500-0

    Article  Google Scholar 

  12. Jean N, Xie SM, Ermon S (2018) Semi-supervised deep kernel learning: regression with unlabeled data by minimizing predictive variance. In: NeurIPS, pp 5327–5338

  13. Reed SE, Lee H, Anguelov D, Szegedy C, Erhan D, Rabinovich A (2015) Training deep neural networks on noisy labels with bootstrapping. In: ICLRW, pp 1–11

  14. Wei H, Feng L, Chen X, An B (2020) Combating noisy labels by agreement: a joint training method with co-regularization. In: CVPR, pp 13726–13735

  15. Tan C, Xia J, Wu L, Li SZ (2021) Co-learning: Learning from noisy labels with self-supervision. In: ACM Int. Conf. Multimedia, pp 1405–1413

  16. Khan FH, Qamar U, Bashir S (2017) A semi-supervised approach to sentiment analysis using revised sentiment strength based on sentiwordnet. Knowl Inf Syst 51(3):851–872. https://doi.org/10.1007/s10115-016-0993-1

    Article  Google Scholar 

  17. Dai W, Li X, Cheng K-T (2023) Semi-supervised deep regression with uncertainty consistency and variational model ensembling via bayesian neural networks. In: AAAI, pp 1–10

  18. Berthelot D, Carlini N, Cubuk ED, Kurakin A, Sohn K, Zhang H, Raffel C (2020) Remixmatch: semi-supervised learning with distribution alignment and augmentation anchoring. In: ICLR, pp 1–10

  19. Kuo C-W, Ma C-Y, Huang J-B, Kira Z (2020) Featmatch: Feature-based augmentation for semi-supervised learning. In: ECCV, pp 479–495

  20. Laine S, Aila T (2017) Temporal ensembling for semi-supervised learning. In: ICLR, pp 1–13

  21. Miyato T, Maeda S-I, Koyama M, Ishii S (2019) Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans Pattern Anal Mach Intell 41(8):1979–1993. https://doi.org/10.1109/TPAMI.2018.2858821

    Article  Google Scholar 

  22. Tarvainen A, Valpola H (2007) Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: NeurIPS, pp 1–16

  23. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  Google Scholar 

  24. Ke Z, Wang D, Yan Q, Ren J, Lau RWH (2019) Dual student: breaking the limits of the teacher in semi-supervised learning. In: ICCV, pp 1–12

  25. Sohn K, Berthelot D, Carlini N, Zhang Z, Zhang H, Raffel CA, Cubuk ED, Kurakin A, Li C-L (2020) Fixmatch: simplifying semi-supervised learning with consistency and confidence. In: NeurIPS, pp 596–608

  26. Zhang B, Wang Y, Hou W, WU H, Wang J, Okumura M, Shinozaki T (2021) Flexmatch: boosting semi-supervised learning with curriculum pseudo labeling. In: NeurIPS, pp 18408–18419

  27. Xu Y, Wei F, Sun X, Yang C, Shen Y, Dai B, Zhou B, Lin S (2022) Cross-model pseudo-labeling for semi-supervised action recognition. In: CVPR, pp 2959–2968

  28. Bodla N, Hua G, Chellappa R (2018) Semi-supervised fusedGAN for conditional image generation. In: ECCV, pp 689–704

  29. You C, Zhao R, Staib LH, Duncan JS (2022) Momentum contrastive voxel-wise representation learning for semi-supervised volumetric medical image segmentation. In: MICCAI, pp 639–652

  30. Lin Y, Yao H, Li Z, Zheng G, Li X (2022) Calibrating label distribution for class-imbalanced barely-supervised knee segmentation. In: MICCAI, pp 109–118

  31. Zheng M, You S, Huang L, Wang F, Qian C, Xu C (2022) Simmatch: semi-supervised learning with similarity matching. In: CVPR, pp 14451–14461

  32. Yang L, Qi L, Feng L, Zhang W, Shi Y (2023) Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. In: CVPR, pp 1–13

  33. Lee D-H (2013) Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: ICML Workshop, pp 1–20

  34. Fazakis N, Karlos S, Kotsiantis S, Sgarbas K (2019) A multi-scheme semi-supervised regression approach. Pattern Recognit Lett 125:758–765. https://doi.org/10.1016/j.patrec.2019.07.022

    Article  Google Scholar 

  35. Nigam K, Ghani R (2000) Analyzing the effectiveness and applicability of co-training. In: CIKM, pp 86–93

  36. Wang W, Zhou Z-H (2013) Co-training with insufficient views. In: ACML, pp 467–482

  37. Brefeld U, Gärtner T, Scheffer T, Wrobel S (2006) Efficient co-regularised least squares regression. In: ICML, pp 137–144

  38. Wang X, Fu L, Ma L (2011) Semi-supervised support vector regression model for remote sensing water quality retrieving. Chin Geogr Sci 21:57–64

    Article  Google Scholar 

  39. Bao L, Yuan X, Ge Z (2015) Co-training partial least squares model for semi-supervised soft sensor development. Chemom Intell Lab Syst 147:75–85. https://doi.org/10.1016/j.chemolab.2015.08.002

    Article  Google Scholar 

  40. Liu Y, Xu Z, Li C (2018) Online semi-supervised support vector machine. Inf Sci 439–440:125–141. https://doi.org/10.1016/j.ins.2018.01.048

    Article  MathSciNet  Google Scholar 

  41. Chen X, Cao W, Gan C, Ohyama Y, She J, Wu M (2021) Semi-supervised support vector regression based on data similarity and its application to rock-mechanics parameters estimation. Eng Appl Artif Intell 104:104317. https://doi.org/10.1016/j.engappai.2021.104317

    Article  Google Scholar 

  42. Kostopoulos G, Karlos S, Kotsiantis S, Ragos O, Tiwari S, Trivedi M, Kohle ML (2018) Semi-supervised regression: a recent review. J Intell Fuzzy Syst 35:1483–1500. https://doi.org/10.3233/JIFS-169689

    Article  Google Scholar 

  43. Nigam K, Ghani R (2000) Analyzing the effectiveness and applicability of co-training. In: CIKM, pp 86–93

  44. Brefeld U, Scheffer T (2004) Co-EM support vector learning. In: ICML, p 16

  45. Zhou Z-H, Li M (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3):415–439. https://doi.org/10.1007/s10115-009-0209-z

    Article  Google Scholar 

  46. Sun X, Gong D, Zhang W (2012) Interactive genetic algorithms with large population and semi-supervised learning. Appl Soft Comput 12(9):3004–3013. https://doi.org/10.1016/j.asoc.2012.04.021

    Article  Google Scholar 

  47. Zhou Y, Goldman S (2004) Democratic co-learning. In: ICTAI, pp 594–602

  48. Min F, Li Y, Liu L (2022) Self-paced safe co-training for regression. In: PAKDD, pp 71–82

  49. Verma V, Kawaguchi K, Lamb A, Kannala J, Solin A, Bengio Y, Lopez-Paz D (2022) Interpolation consistency training for semi-supervised learning. Neural Netw 145:90–106. https://doi.org/10.1016/j.neunet.2021.10.008

    Article  Google Scholar 

  50. Owen AB (2007) A robust hybrid of lasso and ridge regression. Contemp Math 443(7):59–72

    Article  MathSciNet  Google Scholar 

  51. Dong X, Yu Z, Cao W, Shi Y, Ma Q (2020) A survey on ensemble learning. Front Comput Sci 14:241–258. https://doi.org/10.1007/s11704-019-8208-z

    Article  Google Scholar 

  52. Mobahi H, Farajtabar M, Bartlett PL (2020) Self-distillation amplifies regularization in Hilbert space. In: NeurIPS, pp 3351–3361

  53. Zhang Z, Sabuncu M (2020) Self-distillation as instance-specific label smoothing. In: NeurIPS, pp 2184–2195

  54. Gou J, Yu B, Maybank SJ, Tao D (2021) Knowledge distillation: a survey. Int J Comput Vision 129(6):1789–1819. https://doi.org/10.1007/s11263-021-01453-z

    Article  Google Scholar 

  55. Wang Y, Chen H, Heng Q, Hou W, Fan Y, Wu Z, Wang J, Savvides M, Shinozaki T, Raj B, Schiele B, Xie X (2023) Freematch: self-adaptive thresholding for semi-supervised learning. In: ICLR, pp 1–20

  56. Chen H, Tao R, Fan Y, Wang Y, Wang J, Schiele B, Xie X, Raj B, Savvides M (2023) Softmatch: addressing the quantity-quality tradeoff in semi-supervised learning. In: ICLR, pp 1–21

  57. Rizve MN, Duarte K, Rawat YS, Shah M (2021) In defense of pseudo-labeling: an uncertainty-aware pseudo-label selection framework for semi-supervised learning. In: ICLR, pp 1–20

  58. Xu Y, Shang L, Ye J, Qian Q, Li Y-F, Sun B, Li H, Jin R (2021) Dash: semi-supervised learning with dynamic thresholding. In: ICML, pp 11525–11536

  59. Yarowsky D (1995) Unsupervised word sense disambiguation rivaling supervised methods. In: ACL, pp 189–196

  60. Cover T, Hart P (1967) Nearest neighbor pattern classification. IEEE Trans Inf Theory 13(1):21–27. https://doi.org/10.1109/TIT.1967.1053964

    Article  Google Scholar 

  61. Li Y-F, Zha H-W, Zhou Z-H (2017) Learning safe prediction for semi-supervised regression. In: AAAI, pp 2217–2223

  62. Shevade SK, Keerthi SS, Bhattacharyya C, Murthy KRK (2000) Improvements to the SMO algorithm for SVM regression. IEEE Trans Neural Netw 11(5):1188–1193. https://doi.org/10.1109/72.870050

    Article  Google Scholar 

  63. Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324

    Article  Google Scholar 

  64. Barros RC, Ruiz DD, Basgalupp MP (2011) Evolutionary model trees for handling continuous classes in machine learning. Inf Sci 181(5):954–971. https://doi.org/10.1016/j.ins.2010.11.010

    Article  Google Scholar 

  65. Timilsina M, Figueroa A, d’Aquin M, Yang H (2021) Semi-supervised regression using diffusion on graphs. Appl Soft Comput 104:107188. https://doi.org/10.1016/j.asoc.2021.107188

    Article  Google Scholar 

  66. Seok K (2014) Semi-supervised regression based on support vector machine. J Korean Data Inf Sci Soc 25(2):447–454

    Google Scholar 

  67. Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32(200):675–701. https://doi.org/10.1080/01621459.1937.10503522

    Article  Google Scholar 

  68. Nemenyi PB (1963) Distribution-free multiple comparisons. PhD thesis, Princeton University

Download references

Acknowledgements

This work was supported by the National Social Science Foundation of China under Grant No. 22FZXB092. We thank Yan-Xue Wu for his valuable suggestions.

Author information

Author notes

  1. H. Zuo and F. Min contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science and Software Engineering, Southwest Petroleum University, Chengdu, 610500, China

    Liyan Liu, Haimin Zuo & Fan Min

  2. Lab of Machine Learning, Southwest Petroleum University, Chengdu, 610500, China

    Liyan Liu, Haimin Zuo & Fan Min

  3. Institute for Artificial Intelligence, Southwest Petroleum University, Chengdu, 610500, China

    Fan Min

Authors
  1. Liyan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haimin Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fan Min
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CRediT authorship contribution statement Liyan Liu did methodology, software, writing—original draft; Haimin Zuo done formal analysis, writing—review & editing; Fan Min contributed to conceptualization, supervision, funding acquisition, writing—review & editing.

Corresponding author

Correspondence to Fan Min.

Ethics declarations

Conflict interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, L., Zuo, H. & Min, F. BSRU: boosting semi-supervised regressor through ramp-up unsupervised loss. Knowl Inf Syst (2024). https://doi.org/10.1007/s10115-023-02044-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10115-023-02044-9

Keywords

Search

Navigation