Abstract
For facial expression recognition, we selected three images: (i) just before speaking, (ii) speaking the first vowel, and (iii) speaking the last vowel in an utterance. In this study, as a pre-processing module, we added a judgment function to distinguish a front-view face for facial expression recognition. A frame of the front-view face in a dynamic image is selected by estimating the face direction. The judgment function measures four feature parameters using thermal image processing, and selects the thermal images that have all the values of the feature parameters within limited ranges which were decided on the basis of training thermal images of front-view faces. As an initial investigation, we adopted the utterance of the Japanese name “Taro,” which is semantically neutral. The mean judgment accuracy of the front-view face was 99.5% for six subjects who changed their face direction freely. Using the proposed method, the facial expressions of six subjects were distinguishable with 84.0% accuracy when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” We expect the proposed method to be applicable for recognizing facial expressions in daily conversation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Yoshitomi Y, Kimura S, Hira E, et al (1996) Facial expression recognition using infrared rays image processing. Proceedings of the Annual Convention IPS Japan, Osaka, Japan, September 4–6, 1996, 2:339–340
Yoshitomi Y, Kimura S, Hira E, et al (1997) Facial expression recognition using thermal image processing. IPSJ SIG Notes, CVIM103-3, Kyoto, Japan, January 23–24, 1997, pp 17–24
Yoshitomi Y, Miyawaki N, Tomita S, et al (1997) Facial expression recognition using thermal image processing and neural network. Proceedings of the 6th IEEE International Workshop on Robot and Human Communication, Sendai, Japan, September 29–October 1, 1997, pp 380–385
Sugimoto Y, Yoshitomi Y, Tomita S (2000) A method for detecting transitions of emotional states using a thermal face image based on a synthesis of facial expressions. J Robotics Auton Syst 31(3): 147–160
Yoshitomi Y, Kim SIll, Kawano T, et al (2000) Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. Proceedings of the 6th IEEE International Workshop on Robot and Human Interactive Communication, Osaka, Japan, September 27–29, 2000, pp 178–183
Ikezoe F, Ko R, Tanijiri T, et al (2004) Facial expression recognition for speaker using thermal image processing (in Japanese). Trans Human Interface Soc 6(1):19–27
Nakano M, Ikezoe F, Tabuse M, et al (2009) A study on the efficient facial expression using thermal face image in speaking and the influence of individual variations on its performance (in Japanese). J IEEJ 38(2):156–163
Koda Y, Yoshitomi Y, Nakano M, et al (2009) Facial expression recognition for a speaker of a phoneme of vowel using thermal image processing and a speech recognition system. Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication, Toyama, Japan, September 29–Octber 1, 2009, pp 955–960
Yoshitomi Y (2010) Facial expression recognition for speaker using thermal image processing and speech recognition system. Proceedings of the 10th WSEAS International Conference on Applied Computer Science, Appi Kogen, Iwate, Japan, October 4–6, 2010, pp 182–186
Kuno H (1994) Infrared rays engineering (in Japanese). Tokyo, IEICE, pp 22
Kuno H (1994) Infrared rays engineering (in Japanese). Tokyo, IEICE, pp 45
Yoshitomi Y, Tsuchiya A, Tomita S (1998) Face recognition using dynamic thermal image processing. Proceedings of the 7th IEEE International Workshop on Robot and Human Communication, Takamatsu, Kagawa, Japan, September 30–October 2, 1998, pp 443–448
Yamazaki S, Kamakura H, Tanijiri T, et al (2004) Three-dimensional CG expression of face rotation using fuzzy algorithm and thermal face image (in Japanese). Trans Human Interface Soc 6(3): 321–331
Yoshitomi Y, Asada T, Shimada K, et al (2011) Facial expression recognition of a speaker using vowel judgment and thermal image processing. Proceedings of the 16th International Symposium on Artificial Life and Robotics, Beppu, Oita, Japan, January 27–29, 2011, pp 225–230
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was presented in part at the 16th International Symposium on Artificial Life and Robotics, Oita, Japan, January 27–29, 2011
About this article
Cite this article
Fujimura, T., Yoshitomi, Y., Asada, T. et al. Facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing. Artif Life Robotics 16, 411–417 (2011). https://doi.org/10.1007/s10015-011-0967-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10015-011-0967-z