Abstract
Micro-expression recognition has important research value and huge research difficulties. Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) is a common and effective feature in micro-expression recognition. However, LBP-TOP only extracts the dynamic texture features in the horizontal and vertical directions and does not consider muscle movement in the oblique direction. In this paper, the feature in oblique directions is studied, and a new feature called Local Binary Pattern from Five Intersecting Planes (LBP-FIP) is proposed by analyzing the movement direction of facial muscles in the micro-expression video. LBP-FIP concatenates the proposed Eight Vertices LBP (EVLBP) with LBP-TOP extracted from three planes, where EVLBP is extracted from two planes in the oblique direction. In this way, the dynamic texture features in the oblique direction are extracted more directly. On the CASME II and SMIC database, we evaluated the proposed feature and the effectiveness of the features in the oblique direction. Extensive experiments prove that LBP-FIP provides more effective feature information than LBP-TOP, and extracting the features in oblique directions is discriminative for recognizing micro-expressions. Also, LBP-FIP has advantages comparing with other LBP based features and achieves satisfactory performance, especially on CASME II.
Similar content being viewed by others
References
Chakraborty S, Singh SK, Chakraborty P (2015) Performance enhancement of local vector pattern with generalized distance local binary pattern for face recognition. In: 2015 IEEE UP section conference on electrical computer and electronics, pp 1–5. https://doi.org/10.1109/UPCON.2015.7456681
Davison AK, Lansley C, Costen N et al (2018) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129. https://doi.org/10.1109/TAFFC.2016.2573832
Ekman P (2009) Lie catching and microexpressions, The Philosophy of Deception, pp 118–136. https://doi.org/10.1093/acprof:oso/9780195327939.003.0008
Ekman P Microexpression training tool (mett), San Francisco:University of California
Ekman P Telling lies: Clues to deceit in the marketplace politics and marriage
Fan K, Hung T (2014) A novel local pattern descriptor—local vector pattern in high-order derivative space for face recognition. IEEE Trans Image Process 23(7):2877–2891. https://doi.org/10.1109/TIP.2014.2321495
Han J, Ma K-K (2002) Fuzzy color histogram and its use in color image retrieval. IEEE Trans Image Process 11(8):944–952. https://doi.org/10.1109/TIP.2002.801585
Happy SL, Routray A (2019) Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans Affect Comput 10(3):394–406. https://doi.org/10.1109/TAFFC.2017.2723386
Huang X, Wang S, Liu X et al (2019) Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47. https://doi.org/10.1109/TAFFC.2017.2713359
Huang X, Wang S-J, Zhao G et al (2015) Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: 2015 IEEE international conference on computer vision workshop (ICCVW), pp 1–9. https://doi.org/10.1109/ICCVW.2015.10
Huang X, Zhao G (2017) Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern. In: 2017 international conference on the frontiers and advances in data science (FADS), pp 159–164. https://doi.org/10.1109/FADS.2017.8253219
Huang X, Zhao G, Hong X, et al. (2015) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578. https://doi.org/10.1016/j.neucom.2015.10.096
Khor H-Q, See J, Liong S, Phan R, Lin W (2019) Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE international conference on image processing (ICIP), pp 36–40. https://doi.org/10.1109/ICIP.2019.8802965
Khor H-Q, See J, Phan R et al (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE international conference on automatic face gesture recognition (FG 2018), pp 667–674. https://doi.org/10.1109/FG.2018.00105
Kim DH, Baddar WJ, Ro YM (2016) Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM international conference on multimedia, pp 382–386. https://doi.org/10.1145/2964284.2967247
Le Ngo AC, Oh Y-H, Phan R et al (2016) Eulerian emotion magnification for subtle expression recognition. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1243–1247. https://doi.org/10.1109/ICASSP.2016.7471875
Le Ngo AC, Phan R, See J (2014) Spontaneous subtle expression recognition: Imbalanced databases and solutions. Computer Vision – ACCV 2014:33–48. https://doi.org/10.1007/978-3-319-16817-3∖_3
Le Ngo AC, See J, Phan RC (2017) Sparsity in dynamics of spontaneous subtle emotions: Analysis and application. IEEE Trans Affect Comput 8 (3):396–411. https://doi.org/10.1109/TAFFC.2016.2523996
Li X, Hong X, Moilanen A, et al. (2018) Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577. https://doi.org/10.1109/TAFFC.2017.2667642
Li X, Pfister T, Huang X et al (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–6. https://doi.org/10.1109/FG.2013.6553717
Li J, Wang Y, See J, et al. (2019) Micro-expression recognition based on 3d flow convolutional neural network. Pattern Anal Applic 22:1331–1339. https://doi.org/10.1007/s10044-018-0757-5
Liong S-T, Phan R, See J, et al. (2014) Optical strain based recognition of subtle emotions. 2014 International Symposium on Intelligent Signal Processing and Communication Systems, ISPACS 2014:180–184. https://doi.org/10.1109/ISPACS.2014.7024448
Liu Y-J, LI B-J, Lai Y-K Sparse mdmo: Learning a discriminative feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2018.2854166
Liu Y-J, Zhang J-K, Yan W-J, et al. (2015) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7:299–310. https://doi.org/10.1109/TAFFC.2015.2485205
Martin O, Kotsia I, Macq B et al (2006) The enterface’ 05 audio-visual emotion database. In: 22nd international conference on data engineering workshops (ICDEW’06), pp 8–8. https://doi.org/10.1109/ICDEW.2006.145
Matsumoto D, Hwang H (2011) Evidence for training the ability to read microexpressions of emotion. Motiv Emot 35:181–191. https://doi.org/10.1007/s11031-011-9212-2
Mendez-Vazquez H, Martinez-Diaz Y, Chai Z (2013) Volume structured ordinal features with background similarity measure for video face recognition. In: 2013 international conference on biometrics (ICB), pp 1–6. https://doi.org/10.1109/ICB.2013.6612990
Oh Y-H, See J, Le Ngo AC et al A survey of automatic facial micro-expression analysis: Databases, methods, and challenges, Frontiers in Psychology 9. https://doi.org/10.3389/fpsyg.2018.01128
O’Sullivan M, Frank M, Hurley C, Tiwana J (2009) Police lie detection accuracy: The effect of lie scenario. Law Hum Behav 33:530–538. https://doi.org/10.1007/s10979-008-9166-4
Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recogn 29:51–59. https://doi.org/10.1016/0031-3203(95)00067-4
Porter S, Brinke L (2008) Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions. Psychol Sci 19:508–514. https://doi.org/10.1111/j.1467-9280.2008.02116.x
Song B, Li K, Zong Y, Jie Z, Zheng W, Shi J, Zhao L (2019) Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access 7:184537–184551. https://doi.org/10.1109/ACCESS.2019.2960629
Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650. https://doi.org/10.1109/TIP.2010.2042645
Verma M, Vipparthi S, Singh G, Murala S (2019) Learnet dynamic imaging network for micro expression recognition. IEEE Trans Image Process 29:1618–1627. https://doi.org/10.1109/TIP.2019.2912358
Wang Y, See J, Oh Y-H et al (2016) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76:21665–21690. https://doi.org/10.1007/s11042-016-4079-6
Wang Y, See J, Phan R et al (2015) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. 2014 ACCV 9003:525–537. https://doi.org/10.1007/978-3-319-16865-4∖_34
Wang Y, See J, Phan R et al (2015) Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS One 10:e0124674. https://doi.org/10.1371/journal.pone.0124674
Wang S, Yan W, Li X et al (2015) Micro-expression recognition using color spaces. IEEE Trans Image Process 24(12):6034–6047. https://doi.org/10.1109/TIP.2015.2496314
Weinberger S (2010) Airport security: Intent to deceive?. Nature 465:412–415. https://doi.org/10.1038/465412a
Wu H-Y, Rubinstein M, Shih E et al (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics - TOG 31:1–8. https://doi.org/10.1145/2185520.2185561
Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267. https://doi.org/10.1109/TAFFC.2016.2518162
Yan W-J, Li X, Wang S-J, et al. (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS One 9:1–8. https://doi.org/10.1371/journal.pone.0086041
Yan W, Wu Q, Chen Y et al (2013) How fast are the leaked facial expressions: The duration of micro-expressions. J Nonverbal Behav 37:217–230. https://doi.org/10.1007/s10919-013-0159-8
Yang B, Cheng J, Yang Y et al Merta: micro-expression recognition with ternary attentions, Multimedia Tools and Applications. https://doi.org/10.1007/s11042-019-07896-4
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928. https://doi.org/10.1109/TPAMI.2007.1110
Zheng H, Geng X, Yang Z (2016) A relaxed k-svd algorithm for spontaneous micro-expression recognition. PRICAI 2016: Trends in Artificial Intelligence 9810:692–699. https://doi.org/10.1007/978-3-319-42911-3∖_58
Zhou Z, Zhao G, Pietikäinen M (2011) Towards a practical lipreading system. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 137–144. https://doi.org/10.1109/CVPR.2011.5995345
Zong Y, Huang X, Zheng W et al (2018) Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia 20(11):3160–3172. https://doi.org/10.1109/TMM.2018.2820321
Acknowledgements
This work was partly supported by the Postgraduate Research and Practice Innovation Program of Jiangsu Province under Grant KYCX19_0899, partly by the National Natural Science Foundation of China (NSFC) under Grants 72074038, partly by the Key Research and Development Program of Jiangsu Province under Grant BE2016775, partly by the National Natural Science Foundation of China (NSFC) under Grants 61971236, partly by China Postdoctoral Science Foundation under Grant 2018M632348.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Wei, J., Lu, G., Yan, J. et al. Micro-expression recognition using local binary pattern from five intersecting planes. Multimed Tools Appl 81, 20643–20668 (2022). https://doi.org/10.1007/s11042-022-12360-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-12360-x