Skip to main content
SpringerLink
Log in

CSLSEP: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognition

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Applying ensemble learning to facial expression recognition is an important research field nowadays, but all may not be better than many, the redundant learners in the classifier pool may hinder the ensemble system’s performance, so ensemble pruning is needed. Ensemble pruning selects the most suitable subset of classifiers to classify test samples according to the classifier competence. However, the noisy and redundant samples in the validation set will often adversely affect the evaluation of the classifier, making it impossible to select the most suitable classifier. In this paper, a novel ensemble pruning algorithm based on clustering soft label optimization and sorting for facial expression recognition is proposed. First, to increase classifier evaluation objectivity, the novel method uses the clustering optimization model to perform prototype selection and classifier clustering simultaneously. Then the accuracy-based ordering is employed to remove the redundant or poor quality learners, and keep a balance between diversity and accuracy of the ensemble system. Experimental results show that the proposed method outperforms or competes with some state-of-the-art methods on several typical facial expression datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

The datasets used during the current study are reasonably available from the corresponding authors.

Code availability

Some codes for this experiment are available from the corresponding author.

References

  1. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020). https://doi.org/10.1109/taffc.2020.2981446

    Article  Google Scholar 

  2. Wu, S.H., Wang, B.: Facial expression recognition based on computer deep learning algorithm: taking cognitive acceptance of college students as an example. J. Ambient Intell. Humaniz. Comput. (2021). https://doi.org/10.1007/s12652-021-03113-z

    Article  Google Scholar 

  3. Zhang, H.F., Su, W., Yu, J., Wang, Z.F.: Identity-expression dual branch network for facial expression recognition. IEEE Trans. Cogn. Dev. Syst. 13(4), 898–911 (2021). https://doi.org/10.1109/tcds.2020.3034807

    Article  Google Scholar 

  4. Ge, H., Zhu, Z., Dai, Y., Wang, B., Wu, X.: Facial expression recognition based on deep learning. Comput. Methods Prog. Biomed. 215, 106621 (2022)

    Article  Google Scholar 

  5. Zou, W., Zhang, D., Lee, D.-J.: A new multi-feature fusion based convolutional neural network for facial expression recognition. Appl. Intell. 52(3), 2918–2929 (2021). https://doi.org/10.1007/s10489-021-02575-0

    Article  Google Scholar 

  6. Sun, Z., Chiong, R., Hu, Z.-p: An extended dictionary representation approach with deep subspace learning for facial expression recognition. Neurocomputing 316, 1–9 (2018). https://doi.org/10.1016/j.neucom.2018.07.045

    Article  Google Scholar 

  7. Li, D., Wen, G., Hou, Z., Huan, E., Hu, Y., Li, H.: RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl. Inf. Syst. 59(1), 219–250 (2018). https://doi.org/10.1007/s10115-018-1176-z

    Article  Google Scholar 

  8. Liang, Z.F., Wang, H., Yang, K.X., Shi, Y.F.: Adaptive fusion based method for imbalanced data classification. Front. Neurorobotics 16, 8 (2022). https://doi.org/10.3389/fnbot.2022.827913

    Article  Google Scholar 

  9. Martinez, W.G.: Ensemble pruning via quadratic margin maximization. IEEE Access 9, 48931–48951 (2021). https://doi.org/10.1109/access.2021.3062867

    Article  Google Scholar 

  10. Zhang, C., Wu, Y., Zhu, M.: Pruning variable selection ensembles. Stat. Analy. Data Min. mASA Data Sci. J. 12(3), 168–184 (2019). https://doi.org/10.1002/sam.11410

    Article  MathSciNet  MATH  Google Scholar 

  11. Zyblewski, P., Woźniak, M.: Novel clustering-based pruning algorithms. Patt. Anal. Appl. 23(3), 1049–1058 (2020). https://doi.org/10.1007/s10044-020-00867-8

    Article  MathSciNet  Google Scholar 

  12. Diao, R., Chao, F., Peng, T.X., Snooke, N., Shen, Q.: Feature selection inspired classifier ensemble reduction. IEEE T. Cybern. 44(8), 1259–1268 (2014). https://doi.org/10.1109/tcyb.2013.2281820

    Article  Google Scholar 

  13. Alberghini, G., Barbon Junior, S., Cano, A.: Adaptive ensemble of self-adjusting nearest neighbor subspaces for multi-label drifting data streams. Neurocomputing 481, 228–248 (2022). https://doi.org/10.1016/j.neucom.2022.01.075

    Article  Google Scholar 

  14. Wang, Z., Wang, K., Wang, X., Pan, S., Qiao, X.: Dynamic ensemble selection of convolutional neural networks and its application in flower classification. Int. J. Agri. Biol. Eng. 14(6), 216–223 (2021). https://doi.org/10.25165/j.ijabe.20221501.6313

    Article  Google Scholar 

  15. Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C.: Prototype selection for dynamic classifier and ensemble selection. Neural Comput. Appl. 29(2), 447–457 (2016). https://doi.org/10.1007/s00521-016-2458-6

    Article  Google Scholar 

  16. Ben, X., Ren, Y., Zhang, J., Wang, S.J., Kpalma, K., Meng, W., Liu, Y.J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Patt. Anal. Mach. Intell. 44(9), 5826–5846 (2022). https://doi.org/10.1109/TPAMI.2021.3067464

    Article  Google Scholar 

  17. Cheng, J., Liu, G.Y.: Affective detection based on an imbalanced fuzzy support vector machine. Biomed. Sig. Process. Control 18, 118–126 (2015). https://doi.org/10.1016/j.bspc.2014.12.006

    Article  Google Scholar 

  18. Ben, X., Gong, C., Huang, T., Li, C., Yan, R., Li, Y.: Tackling micro-expression data shortage via dataset alignment and active learning. IEEE Trans. Multimed. (2022). https://doi.org/10.1109/TMM.2022.3192727

    Article  Google Scholar 

  19. Xu, C.E., Cui, Y., Zhang, Y.H., Gao, P., Xu, J.Y.: Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware. Multimed. Syst. 26(1), 53–61 (2020). https://doi.org/10.1007/s00530-019-00628-6

    Article  Google Scholar 

  20. Kumar, Y., Verma, S.K., Sharma, S.: Multi-pose facial expression recognition using hybrid deep learning model with improved variant of gravitational search algorithm. Int. Arab. J Inf. Technol. 19(2), 281–287 (2022). https://doi.org/10.3402/iajit/19/2/15

    Article  Google Scholar 

  21. Zhu, X.L., Ye, S.H., Zhao, L., Dai, Z.C.: Hybrid attention cascade network for facial expression recognition. Sensors 21(6), 15 (2021). https://doi.org/10.3390/s21062003

    Article  Google Scholar 

  22. Li, B., Zhou, Y., Xiao, R., Wang, J., Ben, X., Kpalma, K., Zhou, H.: Unsupervised cross-database micro-expression recognition based on distribution adaptation. Multimed. Syst. 28(3), 1099–1116 (2022). https://doi.org/10.1007/s00530-022-00896-9

    Article  Google Scholar 

  23. Xie, S., Hu, H.: Facial expression recognition with FRR-CNN. Electron. Lett. 53(4), 235–237 (2017). https://doi.org/10.1049/el.2016.4328

    Article  Google Scholar 

  24. Agrawal, A., Mittal, N.: Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36(2), 405–412 (2019). https://doi.org/10.1007/s00371-019-01630-9

    Article  Google Scholar 

  25. Wen, G.H., Li, H.H., Li, D.Y.: Ieee, An Ensemble Convolutional Echo State Networks for Facial Expression Recognition, 6th AAAC Affective Computing and Intelligent Interaction International Conference (ACII), pp. 873–878. Xian, Peoples R China, Ieee (2015)

    Google Scholar 

  26. Yu Z,Zhang C, Image based Static Facial Expression Recognition with Multiple Deep Network Learning, Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015, pp. 435–442.

  27. Pons, G., Masip, D.: Supervised committee of convolutional neural networks in automated facial expression analysis. Affect. Comput. IEEE Trans. 9(3), 343–350 (2018)

    Article  Google Scholar 

  28. Sultan Zia, M., Hussain, M., Arfan Jaffar, M.: A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier. Multimed. Tools Appl. 77(19), 25537–25567 (2018). https://doi.org/10.1007/s11042-018-5806-y

    Article  Google Scholar 

  29. Zavaschi T, Koerich AL,Oliveira L, Facial expression recognition using ensemble of classifiers, IEEE International Conference on Acoustics, 2011.

  30. Li, D., Wen, G., Li, X., Cai, X.: Graph-based dynamic ensemble pruning for facial expression recognition. Appl. Intell. 49(9), 3188–3206 (2019). https://doi.org/10.1007/s10489-019-01435-2

    Article  Google Scholar 

  31. Zhang, H., Cao, L.: A spectral clustering based ensemble pruning approach. Neurocomputing 139, 289–297 (2014). https://doi.org/10.1016/j.neucom.2014.02.030

    Article  Google Scholar 

  32. Guo, L., Boukir, S.: Margin-based ordered aggregation for ensemble pruning. Pattern Recogn. Lett. 34(6), 603–609 (2013). https://doi.org/10.1016/j.patrec.2013.01.003

    Article  Google Scholar 

  33. Dai, Q., Han, X.: An efficient ordering-based ensemble pruning algorithm via dynamic programming. Appl. Intell. 44(4), 816–830 (2015). https://doi.org/10.1007/s10489-015-0729-z

    Article  Google Scholar 

  34. Nan L, Yang Y,Zhou ZH, Diversity Regularized Ensemble Pruning, Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2012.

  35. Liu, Z., Dai, Q., Liu, N.: Ensemble selection by GRASP. Appl. Intell. 41(1), 128–144 (2014). https://doi.org/10.1007/s10489-013-0510-0

    Article  Google Scholar 

  36. Fatemifar S, Awais M, Akbari A,Kittler J, A Stacking Ensemble for Anomaly Based Client-Specific Face Spoofing Detection, 2020 IEEE International Conference on Image Processing (ICIP), 2020.

  37. Fatemifar, S., Asadi, S., Awais, M., Akbari, A., Kittler, J.: Face spoofing detection ensemble via multistage optimisation and pruning. Patt. Recogn. Lett. 158, 1–8 (2022). https://doi.org/10.1016/j.patrec.2022.04.006

    Article  Google Scholar 

  38. Luong, A.V., Nguyen, T.T., Liew, A.W.C., Wang, S.L.: Heterogeneous ensemble selection for evolving data streams. Patt. Recogn. 112, 16 (2021). https://doi.org/10.1016/j.patcog.2020.107743

    Article  Google Scholar 

  39. Elmi, J., Eftekhari, M.: Multi-Layer Selector(MLS): dynamic selection based on filtering some competence measures. Appl. Soft Comput. (2021). https://doi.org/10.1016/j.asoc.2021.107257

    Article  Google Scholar 

  40. Li, D., Wen, G.: MRMR-based ensemble pruning for facial expression recognition. Multimed. Tools Appl. 77(12), 15251–15272 (2018). https://doi.org/10.1007/s11042-017-5105-z

    Article  Google Scholar 

  41. Markatopoulou, F., Tsoumakas, G., Vlahavas, I.: Dynamic ensemble pruning based on multi-label classification. Neurocomputing 150, 501–512 (2015). https://doi.org/10.1016/j.neucom.2014.07.063

    Article  Google Scholar 

  42. Lin, X., Mao, S., Jiao, L.: Selective ensemble based on transformation of classifiers used SPCA. Int. J. Patt. Recogn. Artif. Intell. 29(1), 1550005 (2015)

    Article  Google Scholar 

  43. Jan, M.Z., Verma, B.: A novel diversity measure and classifier selection approach for generating ensemble classifiers. Ieee Access 7, 156360–156373 (2019). https://doi.org/10.1109/access.2019.2949059

    Article  Google Scholar 

  44. Baek, B., Euh, S., Baek, D., Kim, D., Hwang, D.: Histogram entropy representation and prototype based machine learning approach for malware family classification. Ieee Access 9, 152098–152114 (2021). https://doi.org/10.1109/access.2021.3127195

    Article  Google Scholar 

  45. Zhang, X.X., Zhu, Z.F., Zhao, Y., Chang, D.X., Liu, J.: Seeing all from a few: l(1)-norm-induced discriminative prototype selection. IEEE Trans. Neural Netw. Learn. Syst. 30(7), 1954–1966 (2019). https://doi.org/10.1109/tnnls.2018.2875347

    Article  Google Scholar 

  46. Cruz, R.M.O., Oliveira, D.V.R., Cavalcanti, G.D.C., Sabourin, R.: FIRE-DES plus plus : Enhanced online pruning of base classifiers for dynamic ensemble selection. Patt. Recogn. 85, 149–160 (2019). https://doi.org/10.1016/j.patcog.2018.07.037

    Article  Google Scholar 

  47. Sajjanhar A, Wu ZQ,Wen Q, Deep Learning Models for Facial Expression Recognition, 2018 Digital Image Computing: Techniques and Applications (DICTA), 2018.

  48. Kumari, N., Bhatia, R.: Efficient facial emotion recognition model using deep convolutional neural network and modified joint trilateral filter. Soft. Comput. (2022). https://doi.org/10.1007/s00500-022-06804-7

    Article  Google Scholar 

  49. Conlin, R., Erickson, K., Abbate, J., Kolemen, E.: Keras2c: A library for converting Keras neural networks to real-time compatible C. Eng. Appl. Artif. Intell. (2021). https://doi.org/10.1016/j.engappai.2021.104182

    Article  Google Scholar 

  50. Gan YJ,Acm, Facial Expression Recognition Using Convolutional Neural Network, 2nd International Conference on Vision, Image and Signal Processing (ICVISP), Assoc Computing Machinery, Las Vegas, NV, 2018.

  51. Shafira SS, Ulfa N, Wibawa HA, Rismiyati,Ieee, Facial Expression Recognition Using Extreme Learning Machine, 3rd International Conference on Informatics and Computational Sciences (ICICoS), Ieee, Univ Diponegoro, Dept Informat, Semarang, INDONESIA, 2019.

  52. Videla LS, Kumar PMA,Ieee, Facial Expression Classification Using Vanilla Convolution Neural Network, 7th IEEE International Conference on Smart Structures and Systems (ICSSS), Ieee, Saveetha Engn Coll, Chennai, INDIA, 2020, pp. 107–111.

  53. Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST,Knippenberg AV (2010) Presentation and validation of the Radboud Faces Database.

  54. Kuncheva, L.I.: A bound on kappa-error diagrams for analysis of classifier ensembles. IEEE Trans. Knowl. Data Eng. 25(3), 494–501 (2013). https://doi.org/10.1109/tkde.2011.234

    Article  Google Scholar 

  55. Hou C, Xia Y, Xu Z,Sun J, Learning classifier competence based on graph for dynamic classifier selection, 2016 12th International Conference on Natural Computation and 13th Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), 2016.

  56. Ko, A.H.R., Sabourin, R., Britto, J.A.S.: From dynamic classifier selection to dynamic ensemble selection. Patt. Recogn. 41(5), 1718–1731 (2008). https://doi.org/10.1016/j.patcog.2007.10.015

    Article  MATH  Google Scholar 

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. College of Big Data and Information Engineering, Guizhou University, Guiyang, 550025, China

    Shisong Huang, Danyang Li, Zhuhong Zhang, Yating Wu, Yumei Tang, Xing Chen & Yiqing Wu

Authors
  1. Shisong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhuhong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yating Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yumei Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xing Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yiqing Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Shisong Huang carried out the experimental design and completed the writing of the paper. Danyang Li instructed the proposal of the method and reviewed and revised the article. Zhuhong Zhang put forward guiding opinions. Yumei Tang, Xing Chen and Yiqing Wu collected and processed the experimental data.All authors reviewed the manuscript.

Corresponding author

Correspondence to Danyang Li.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, S., Li, D., Zhang, Z. et al. CSLSEP: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognition. Multimedia Systems 29, 1463–1479 (2023). https://doi.org/10.1007/s00530-023-01062-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01062-5

Keywords

Search

Navigation