Skip to main content
Log in

Micro-expression recognition using local binary pattern from five intersecting planes

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Micro-expression recognition has important research value and huge research difficulties. Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) is a common and effective feature in micro-expression recognition. However, LBP-TOP only extracts the dynamic texture features in the horizontal and vertical directions and does not consider muscle movement in the oblique direction. In this paper, the feature in oblique directions is studied, and a new feature called Local Binary Pattern from Five Intersecting Planes (LBP-FIP) is proposed by analyzing the movement direction of facial muscles in the micro-expression video. LBP-FIP concatenates the proposed Eight Vertices LBP (EVLBP) with LBP-TOP extracted from three planes, where EVLBP is extracted from two planes in the oblique direction. In this way, the dynamic texture features in the oblique direction are extracted more directly. On the CASME II and SMIC database, we evaluated the proposed feature and the effectiveness of the features in the oblique direction. Extensive experiments prove that LBP-FIP provides more effective feature information than LBP-TOP, and extracting the features in oblique directions is discriminative for recognizing micro-expressions. Also, LBP-FIP has advantages comparing with other LBP based features and achieves satisfactory performance, especially on CASME II.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://github.com/WJShengD/Micro-Expression-Classification-using-Local-Binary-Pattern-from-Five-Intersecting-Planes.

References

  1. Chakraborty S, Singh SK, Chakraborty P (2015) Performance enhancement of local vector pattern with generalized distance local binary pattern for face recognition. In: 2015 IEEE UP section conference on electrical computer and electronics, pp 1–5. https://doi.org/10.1109/UPCON.2015.7456681

  2. Davison AK, Lansley C, Costen N et al (2018) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129. https://doi.org/10.1109/TAFFC.2016.2573832

    Article  Google Scholar 

  3. Ekman P (2009) Lie catching and microexpressions, The Philosophy of Deception, pp 118–136. https://doi.org/10.1093/acprof:oso/9780195327939.003.0008

  4. Ekman P Microexpression training tool (mett), San Francisco:University of California

  5. Ekman P Telling lies: Clues to deceit in the marketplace politics and marriage

  6. Fan K, Hung T (2014) A novel local pattern descriptor—local vector pattern in high-order derivative space for face recognition. IEEE Trans Image Process 23(7):2877–2891. https://doi.org/10.1109/TIP.2014.2321495

    Article  MathSciNet  MATH  Google Scholar 

  7. Han J, Ma K-K (2002) Fuzzy color histogram and its use in color image retrieval. IEEE Trans Image Process 11(8):944–952. https://doi.org/10.1109/TIP.2002.801585

    Article  Google Scholar 

  8. Happy SL, Routray A (2019) Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans Affect Comput 10(3):394–406. https://doi.org/10.1109/TAFFC.2017.2723386

    Article  Google Scholar 

  9. Huang X, Wang S, Liu X et al (2019) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47. https://doi.org/10.1109/TAFFC.2017.2713359

    Article  Google Scholar 

  10. Huang X, Wang S-J, Zhao G et al (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: 2015 IEEE international conference on computer vision workshop (ICCVW), pp 1–9. https://doi.org/10.1109/ICCVW.2015.10

  11. Huang X, Zhao G (2017) Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern. In: 2017 international conference on the frontiers and advances in data science (FADS), pp 159–164. https://doi.org/10.1109/FADS.2017.8253219

  12. Huang X, Zhao G, Hong X, et al. (2015) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578. https://doi.org/10.1016/j.neucom.2015.10.096

    Article  Google Scholar 

  13. Khor H-Q, See J, Liong S, Phan R, Lin W (2019) Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE international conference on image processing (ICIP), pp 36–40. https://doi.org/10.1109/ICIP.2019.8802965

  14. Khor H-Q, See J, Phan R et al (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE international conference on automatic face gesture recognition (FG 2018), pp 667–674. https://doi.org/10.1109/FG.2018.00105

  15. Kim DH, Baddar WJ, Ro YM (2016) Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM international conference on multimedia, pp 382–386. https://doi.org/10.1145/2964284.2967247

  16. Le Ngo AC, Oh Y-H, Phan R et al (2016) Eulerian emotion magnification for subtle expression recognition. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1243–1247. https://doi.org/10.1109/ICASSP.2016.7471875

  17. Le Ngo AC, Phan R, See J (2014) Spontaneous subtle expression recognition: Imbalanced databases and solutions. Computer Vision – ACCV 2014:33–48. https://doi.org/10.1007/978-3-319-16817-3∖_3

    Google Scholar 

  18. Le Ngo AC, See J, Phan RC (2017) Sparsity in dynamics of spontaneous subtle emotions: Analysis and application. IEEE Trans Affect Comput 8 (3):396–411. https://doi.org/10.1109/TAFFC.2016.2523996

    Article  Google Scholar 

  19. Li X, Hong X, Moilanen A, et al. (2018) Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577. https://doi.org/10.1109/TAFFC.2017.2667642

    Article  Google Scholar 

  20. Li X, Pfister T, Huang X et al (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–6. https://doi.org/10.1109/FG.2013.6553717

  21. Li J, Wang Y, See J, et al. (2019) Micro-expression recognition based on 3d flow convolutional neural network. Pattern Anal Applic 22:1331–1339. https://doi.org/10.1007/s10044-018-0757-5

    Article  MathSciNet  Google Scholar 

  22. Liong S-T, Phan R, See J, et al. (2014) Optical strain based recognition of subtle emotions. 2014 International Symposium on Intelligent Signal Processing and Communication Systems, ISPACS 2014:180–184. https://doi.org/10.1109/ISPACS.2014.7024448

    Article  Google Scholar 

  23. Liu Y-J, LI B-J, Lai Y-K Sparse mdmo: Learning a discriminative feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2018.2854166

  24. Liu Y-J, Zhang J-K, Yan W-J, et al. (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7:299–310. https://doi.org/10.1109/TAFFC.2015.2485205

    Article  Google Scholar 

  25. Martin O, Kotsia I, Macq B et al (2006) The enterface’ 05 audio-visual emotion database. In: 22nd international conference on data engineering workshops (ICDEW’06), pp 8–8. https://doi.org/10.1109/ICDEW.2006.145

  26. Matsumoto D, Hwang H (2011) Evidence for training the ability to read microexpressions of emotion. Motiv Emot 35:181–191. https://doi.org/10.1007/s11031-011-9212-2

    Article  Google Scholar 

  27. Mendez-Vazquez H, Martinez-Diaz Y, Chai Z (2013) Volume structured ordinal features with background similarity measure for video face recognition. In: 2013 international conference on biometrics (ICB), pp 1–6. https://doi.org/10.1109/ICB.2013.6612990

  28. Oh Y-H, See J, Le Ngo AC et al A survey of automatic facial micro-expression analysis: Databases, methods, and challenges, Frontiers in Psychology 9. https://doi.org/10.3389/fpsyg.2018.01128

  29. O’Sullivan M, Frank M, Hurley C, Tiwana J (2009) Police lie detection accuracy: The effect of lie scenario. Law Hum Behav 33:530–538. https://doi.org/10.1007/s10979-008-9166-4

    Article  Google Scholar 

  30. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recogn 29:51–59. https://doi.org/10.1016/0031-3203(95)00067-4

    Article  Google Scholar 

  31. Porter S, Brinke L (2008) Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions. Psychol Sci 19:508–514. https://doi.org/10.1111/j.1467-9280.2008.02116.x

    Article  Google Scholar 

  32. Song B, Li K, Zong Y, Jie Z, Zheng W, Shi J, Zhao L (2019) Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access 7:184537–184551. https://doi.org/10.1109/ACCESS.2019.2960629

    Article  Google Scholar 

  33. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650. https://doi.org/10.1109/TIP.2010.2042645

    Article  MathSciNet  MATH  Google Scholar 

  34. Verma M, Vipparthi S, Singh G, Murala S (2019) Learnet dynamic imaging network for micro expression recognition. IEEE Trans Image Process 29:1618–1627. https://doi.org/10.1109/TIP.2019.2912358

    Article  MathSciNet  Google Scholar 

  35. Wang Y, See J, Oh Y-H et al (2016) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76:21665–21690. https://doi.org/10.1007/s11042-016-4079-6

    Article  Google Scholar 

  36. Wang Y, See J, Phan R et al (2015) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. 2014 ACCV 9003:525–537. https://doi.org/10.1007/978-3-319-16865-4∖_34

    Google Scholar 

  37. Wang Y, See J, Phan R et al (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS One 10:e0124674. https://doi.org/10.1371/journal.pone.0124674

    Article  Google Scholar 

  38. Wang S, Yan W, Li X et al (2015) Micro-expression recognition using color spaces. IEEE Trans Image Process 24(12):6034–6047. https://doi.org/10.1109/TIP.2015.2496314

    Article  MathSciNet  MATH  Google Scholar 

  39. Weinberger S (2010) Airport security: Intent to deceive?. Nature 465:412–415. https://doi.org/10.1038/465412a

    Article  Google Scholar 

  40. Wu H-Y, Rubinstein M, Shih E et al (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics - TOG 31:1–8. https://doi.org/10.1145/2185520.2185561

    Article  Google Scholar 

  41. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267. https://doi.org/10.1109/TAFFC.2016.2518162

    Article  Google Scholar 

  42. Yan W-J, Li X, Wang S-J, et al. (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS One 9:1–8. https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  43. Yan W, Wu Q, Chen Y et al (2013) How fast are the leaked facial expressions: The duration of micro-expressions. J Nonverbal Behav 37:217–230. https://doi.org/10.1007/s10919-013-0159-8

    Article  Google Scholar 

  44. Yang B, Cheng J, Yang Y et al Merta: micro-expression recognition with ternary attentions, Multimedia Tools and Applications. https://doi.org/10.1007/s11042-019-07896-4

  45. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928. https://doi.org/10.1109/TPAMI.2007.1110

    Article  Google Scholar 

  46. Zheng H, Geng X, Yang Z (2016) A relaxed k-svd algorithm for spontaneous micro-expression recognition. PRICAI 2016: Trends in Artificial Intelligence 9810:692–699. https://doi.org/10.1007/978-3-319-42911-3∖_58

    Google Scholar 

  47. Zhou Z, Zhao G, Pietikäinen M (2011) Towards a practical lipreading system. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 137–144. https://doi.org/10.1109/CVPR.2011.5995345

  48. Zong Y, Huang X, Zheng W et al (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia 20(11):3160–3172. https://doi.org/10.1109/TMM.2018.2820321

    Article  Google Scholar 

Download references

Acknowledgements

This work was partly supported by the Postgraduate Research and Practice Innovation Program of Jiangsu Province under Grant KYCX19_0899, partly by the National Natural Science Foundation of China (NSFC) under Grants 72074038, partly by the Key Research and Development Program of Jiangsu Province under Grant BE2016775, partly by the National Natural Science Foundation of China (NSFC) under Grants 61971236, partly by China Postdoctoral Science Foundation under Grant 2018M632348.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Guanming Lu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 10 Accuracy rate of LBP-FIP not using EVM, and the bold entries denote the highest accuracy rate under each database.(Unit: %)
Table 11 Accuracy rate of LBP-FIP with different Alpha on the CASME II database, and the bold entries denote the highest accuracy rate under each Alpha.(Unit: %)
Fig. 11
figure 11

The confusion matrices of different features on the SMIC-HS database

Fig. 12
figure 12

The confusion matrices of different features on the CASME II database

Table 12 Accuracy rates of LBP-FIP with different Alpha on the SMIC-HS database, and the bold entries denote the highest accuracy rate under each Alpha.(Unit: %)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wei, J., Lu, G., Yan, J. et al. Micro-expression recognition using local binary pattern from five intersecting planes. Multimed Tools Appl 81, 20643–20668 (2022). https://doi.org/10.1007/s11042-022-12360-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12360-x

Keywords

Navigation