Browse > Article
http://dx.doi.org/10.4218/etrij.15.0114.0523

Improved Two-Phase Framework for Facial Emotion Recognition  

Yoon, Hyunjin (IT Convergence Technology Research Laboratory, ETRI)
Park, Sangwook (IT Convergence Technology Research Laboratory, ETRI)
Lee, Yongkwi (IT Convergence Technology Research Laboratory, ETRI)
Han, Mikyong (IT Convergence Technology Research Laboratory, ETRI)
Jang, Jong-Hyun (IT Convergence Technology Research Laboratory, ETRI)
Publication Information
ETRI Journal / v.37, no.6, 2015 , pp. 1199-1210 More about this Journal
Abstract
Automatic emotion recognition based on facial cues, such as facial action units (AUs), has received huge attention in the last decade due to its wide variety of applications. Current computer-based automated two-phase facial emotion recognition procedures first detect AUs from input images and then infer target emotions from the detected AUs. However, more robust AU detection and AU-to-emotion mapping methods are required to deal with the error accumulation problem inherent in the multiphase scheme. Motivated by our key observation that a single AU detector does not perform equally well for all AUs, we propose a novel two-phase facial emotion recognition framework, where the presence of AUs is detected by group decisions of multiple AU detectors and a target emotion is inferred from the combined AU detection decisions. Our emotion recognition framework consists of three major components - multiple AU detection, AU detection fusion, and AU-to-emotion mapping. The experimental results on two real-world face databases demonstrate an improved performance over the previous two-phase method using a single AU detector in terms of both AU detection accuracy and correct emotion recognition rate.
Keywords
Automatic emotion recognition; facial action unit detection; decision fusion; facial expression analysis;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 P. Ekman and W. Friesen, "The Facial Action Coding System: A Technique for the Measurement of Facial Movement," Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
2 W. Yun et al., "Hybrid Facial Representations for Emotion Recognition," ETRI J., vol. 35, no. 6, Dec. 2013, pp. 1021-1028.   DOI
3 Y. Li et al., "Data-Free Prior Model for Facial Action Unit Recognition," IEEE Trans. Affective Comput., vol. 4, no. 2, Apr. 2013, pp. 127-141.   DOI
4 G. Donato et al., "Classifying Facial Actions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 21, no. 10, 1999, pp. 974-989.   DOI
5 Y. Zhu et al., "Dynamic Cascades with Bidirectional Bootstrapping for Action Unit Detection in Spontaneous Facial Behavior," IEEE Trans. Affective Comput., vol. 2, no. 2, Apr.-June 2011, pp. 79-91.   DOI
6 M.F. Valstar and M. Pantic, "Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics," IEEE Int. Conf. Human-Comput. Interaction, Rio de Janeiro, Brazil, Oct. 20, 2007, pp. 118-127.
7 B. Jiang, M.F. Valstar, and M. Pantic, "Action Unit Detection Using Sparse Appearance Descriptors in Space-Time Video Volumes," IEEE Int. Conf. Automat. Face Gesture Recogn., Santa Barbara, CA, USA, Mar. 21-25, 2011, pp. 314-321.
8 Z. Zeng et al., "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, Jan. 2009, pp. 39-58.   DOI
9 I. Cohen et al., "Facial Expression Recognition from Video Sequences: Temporal and Static Modeling," Comput. Vis. Image Understanding, vol. 91, no. 1-2, July 2003, pp. 160-187.   DOI
10 M.S. Bartlett et al., "Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior," IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn., San Diego, CA, USA, vol. 2, June 20-25, 2005, pp. 568-573.
11 K.-Y. Chang, T.-L. Liu, and S.-H. Lai, "Learning Partially-Observed Hidden Conditional Random Fields for Facial Expression Recognition," IEEE Conf. Comput. Soc. Vis. Pattern Recogn., Miami, FL, USA, June 20-25, 2009, pp. 533-540.
12 Y. Zhang and Q. Ji, "Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, May 2005, pp. 699-714.   DOI
13 P. Lucey et al., "The Extended Cohn-Kanade Dataset (CK+): A Complete Dataset for Action Unit and Emotion-Specified Expression," IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. Workshop, San Francisco, CA, USA, June 13-18, 2010, pp. 94-101.
14 M. Pantic and L.J.M. Rothkrantz, "An Expert System for Multiple Emotional Classification of Facial Expressions," IEEE Int. Conf. Tools Artif. Intell., Chicago, IL, USA, 1999, pp. 113-120.
15 M.F. Valstar and M. Pantic, "Biologically vs. Logic Inspired Encoding of Facial Actions and Emotions in Video," IEEE Int. Conf. Multimedia Expo, Toronto, Canada, 2006, pp. 325-328.
16 S. Velusamy et al., "A Method to Infer Emotions from Facial Action Units," IEEE Int. Conf. Acoust., Speech Signal Process., Prague, Czech Republic, May 22-27, 2011, pp. 2028-2031.
17 Q. Ji, ISL Facial Expression Databases, Intelligent Systems Lab, Rensselear Polytechnic Institute. Accessed Feb. 26, 2014. http://www.ecse.rpi.edu/-cvrl/database/database.html
18 D. Heckerman, "A Tutorial on Learning with Bayesian Networks," in Learning in Graphical Models, Cambridge, MA, USA: MIT Press, 1999, pp. 301-354.
19 N. Littlestone and M.K. Warmuth, "The Weighted Majority Algorithm," Inf. Comput., vol. 108, no. 2, Feb. 1994, pp. 212-261.   DOI
20 Y.S. Huang and C.Y. Suen, "The Behavior-Knowledge Space Method for Combination of Multiple Classifiers," IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn., New York, USA, June 15-17, 1993, pp. 347-352.
21 M.F. Valstar and M. Pantic, "Induced Disgust, Happiness and Surprise: An Addition to the MMI Facial Expression Database," Int. Conf. Language Resources Evaluation, Istanbul, Turkey, May 21-27, 2010, pp. 65-70.