Browse > Article
http://dx.doi.org/10.9717/kmms.2017.20.8.1175

Weighted Soft Voting Classification for Emotion Recognition from Facial Expressions on Image Sequences  

Kim, Kyeong Tae (Dept. of Computer and Electronic Systems Eng., Hankuk University of Foreign Studies)
Choi, Jae Young (Dept. of Computer and Electronic Systems Eng., Hankuk University of Foreign Studies)
Publication Information
Journal of Korea Multimedia Society / v.20, no.8, 2017 , pp. 1175-1186 More about this Journal
Abstract
Human emotion recognition is one of the promising applications in the era of artificial super intelligence. Thus far, facial expression traits are considered to be the most widely used information cues for realizing automated emotion recognition. This paper proposes a novel facial expression recognition (FER) method that works well for recognizing emotion from image sequences. To this end, we develop the so-called weighted soft voting classification (WSVC) algorithm. In the proposed WSVC, a number of classifiers are first constructed using different and multiple feature representations. In next, multiple classifiers are used for generating the recognition result (namely, soft voting) of each face image within a face sequence, yielding multiple soft voting outputs. Finally, these soft voting outputs are combined through using a weighted combination to decide the emotion class (e.g., anger) of a given face sequence. The weights for combination are effectively determined by measuring the quality of each face image, namely "peak expression intensity" and "frontal-pose degree". To test the proposed WSVC, CK+ FER database was used to perform extensive and comparative experimentations. The feasibility of our WSVC algorithm has been successfully demonstrated by comparing recently developed FER algorithms.
Keywords
Emotion Recognition; Facial Expression Recognition; Weighted Soft Voting; Face Sequences; Combination Weights;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Y. Chang, C. Hu, R. Feris, and M Turk, “Manifold Based Analysis of Facial Expression,” Image Vision Computing, Vol. 24, No. 10, pp. 605-614, 2006.   DOI
2 G. Zhao and M. Pietikäinen, “Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 29, No. 6, pp. 915-928, 2007.   DOI
3 B. Jiang, F. Valstar, and M. Pantic, "Action Unit Detection Using Sparse Appearance Descriptors in Space-Time Video Volumes," Proceeding of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 314-321, 2011.
4 B. Jiang, M. Valstar, B. Martinez, and M. Pantic, “A Dynamic Appearance Descriptor Approach to Facial Actions Temporal Modeling,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, Vol. 44, No. 2, pp. 161-174, 2014.
5 D. Ruta and B. Gabrys, “Classifier Selection for Majority Voting,” Information Fusion, Vol. 6, No. 1, pp. 63-81, 2005.   DOI
6 P. Lucey, J.F. Cohn, T. Kanade, J. Saragih, and Z. Ambadar, "The Extended Cohn-Kanade Dataset (CK+): A Complete Dataset for Action Unit and Emotion-specified Expression," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 94-101, 2010.
7 P. Viola and M. Jones, "Rapid Object Detection Using a Boosted Cascade of Simple Features," Proceeding of IEEE International Conference on Computer Vision Pattern Recognition, pp. I-511-I-518, 2001.
8 C. Tomasi and T. Kanade, "Detection and Tracking of Point Features," Carnegie Mellon University, Pittsburgh, PA, Technical Report CMU-CS-91-132, 1991.
9 J.A. Davis, D.E. McNamara, D.M. Cottrell, and J. Campos, “Image Processing with the Radial Hilbert Transform: Theory and Experiments,” Optics Letters, Vol. 25, No. 2, pp. 99-101, 2000.   DOI
10 M.S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 568-573, 2005.
11 A. Jain, K. Nandakumar, and A. Ross, “Score Normalization in Multimodal Biometric Systems,” Journal of Pattern Recognition, Vol. 38, No. 12, pp. 2270-2285, 2005.   DOI
12 J.Y. Choi, Y.M. Ro, and K.N. Plataniotis, “Color Local Texture Features for Color Face Recognition,” IEEE Transactions on Image Processing, Vol. 21, No. 3, pp. 1366-1380, 2012.   DOI
13 T. Ahonen, A. Hadid, and M. Pietikainen, “Face Description with Local Binary Pattern: Application to Face Recognition,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 28, No. 12, pp. 2037-2041, 2006.   DOI
14 C. Liu and H. Wechsler, “Gabor Feature Based Classification Using the Enhanced Fisher Linear Discriminant Model for Face Recognition,” IEEE Transactions on Image Processing, Vol. 11, No. 4, pp. 467-476, 2002.   DOI
15 J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas, “On Combining Classifiers,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 20, No. 3, pp. 226-239, 1998.   DOI
16 R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-pie,” Image and Vision Computing, Vol. 28, No. 5, pp. 807-813, 2010.   DOI
17 M. Suk and P. Balakrishnan, "Real-time Mobile Facial Expression Recognition System-A Case Study," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 132-137, 2014.
18 B. Moghaddam, “Principal Manifolds and Probabilistic Subspaces for Visual Recognition,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 24, No. 6, pp. 780-788, 2002.   DOI
19 P.L. Carrier, A. Courville, I.J. Goodfellow, M. Mirza, and Y. Bengio, FER-2013 Face Database, Technical Report, 2013.
20 P. Michel and R.E.l. Kaliouby, "Real Time Facial Expression Recognition in Video Using Support Vector Machines," Proceedings of the 5th Association for Computing Machinery International Conference on Multimodal Interfaces, pp. 258-264, 2003.
21 B. Moghaddam, “Principal Manifolds and Probabilistic Subspaces for Visual Recognition,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 24, No. 6, pp. 780-788, 2002.   DOI
22 T. Sim, S. Baker, and M. Bsat, “The CMU Pose, Illumination, and Expression Database,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 25, No. 12, pp. 1615-1618, 2003.   DOI
23 P.J. Phillips, H. Moon, S.A. Rizvi, and P.J. Rauss, “The FERET Evaluation Methodology for Face Recognition Algorithms,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 22, No. 10, pp. 1090-1104, 2000.   DOI
24 Intra-Face, http://humansensing.cs.cmu.edu/intraface (accessed June, 23, 2017).
25 R.V. Yampolskiy, Artificial Superintelligence: a Futuristic Approach, Chemical Rubber Company Press, London, 2015.
26 Y. Tian, "Evaluation of Face Resolution for Expression Analysis," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition Workshop, pp. 82-88, 2004.
27 P. Michel and R. El Kaliouby, "Real Time Facial Expression Recognition in Video Using Support Vector Machines," Proceedings of the 5th International Conference on Multimodal Interfaces (Association for Computing Machinery) , pp. 258-264, 2003.
28 G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin, “Static and Dynamic 3D Facial Expression Recognition: A Comprehensive Survey,” Image and Vision Computing, Vol. 30, No. 10, pp. 683-697, 2012.   DOI
29 S. Taheri, Q. Qiu, and R. Chellappa, "Structure-Preserving Sparse Decomposition for Facial Expression Analysis," IEEE Transactions on Image Processing, Vol. 23, No. 8, pp. 3590-3603, 2014.   DOI
30 Y.L. Tian, T. Kanade, and J.F. Cohn, Facial Expression Analysis, Handbook of Face Recognition, Springer, New York, pp. 247-276, 2005.
31 W. Zhen and Y. Zilu, "Facial Expression Recognition Based on Local Phase Quantization and Sparse Representation," Proceeding of IEEE International Conference on Natural Computation, pp. 222-225, 2012.
32 H.W. Kang, K.T. Lim, and C.H. Won, "Learning Directional LBP Features and Discriminative Feature Regions for Facial Expression Recognition," Journal of Korea Multimedia Society, Vol. 20, No. 5, pp. 748-757, 2017.   DOI
33 X. Huang, G. Zhao, W. Zheng, and M. Pietikainen, "Spatiotemporal Local Monogenic Binary Patterns for Facial Expression Recognition," IEEE Signal Processing Letters, Vol. 19, No. 5, pp. 243-246, 2012.   DOI
34 M. Huang, Z. Wang, and Z. Ying, "A New Method for Facial Expression Recognition Based on Sparse Representation Plus LBP," IEEE International Congress on Image and Signal Processing, pp. 1750-1754, 2010.
35 S. Zafeiriou and M. Petrou, "Sparse Representation for Facial Expression Recognition via l1 Optimization," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 32-39, 2010.
36 M.F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, "The First Expression Recognition and Analysis Challenge," Proceeding of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 921-926, 2011.
37 A.R. Rivera, J.R. Castillo, and O. Chae, “Local Directional Number Pattern for Face Analysis,” IEEE Transactions on Image Processing, Vol. 22, No. 5, pp. 1740-1752, 2013.   DOI
38 J. Stallkamp, H.K. Ekenel, and R. Stiefelhagen, "Video-based Face Recognition on Real-World Data on Real-world Dataset," Proceeding of IEEE International Conference on Computer Vision, pp. 1-8, 2007.
39 Microsoft Azure, https://www.projectoxford.ai/demo/Emotion#detection (accessed July, 10, 2017).
40 Affectiva, https://www.affectiva.com (accessed July, 10, 2017).
41 Y. Zhang and A.M. Martinez, “A Weighted Probabilistic Approach to Face Recognition from Multiple Images and Video Sequences,” Image Vision Computing, Vol. 24, No. 6, pp. 626-638, 2006.   DOI
42 J.J. Lien, T. Kanade, J.F. Cohn, and C. Li, “Detection, Tracking, and Classification of Action Units in Facial Expression,” Journal of Robotics and Autonomous Systems, Vol. 31, No. 3, pp. 131-146, 2000.   DOI
43 A. Saeed, A. Al-Hamadi, and R. Niese, "Neutral-independent Geometric Features for Facial Expression Recognition," Proceeding of IEEE International Conference on Intelligent Systems Design and Applications, pp. 842-846, 2012.
44 S. Taheri, V.M. Patel, and R. Chellappa, “Component-based Recognition of Faces and Facial Expressions,” IEEE Transactions on Affective Computing, Vol. 23, No. 8, pp. 360-371, 2013.
45 S.H. Lee, K.N. Plataniotis, and Y.M. Ro, "Intra-Class Variation Reduction Using Training Expression Images for Sparse Representation Based Facial Expression Recognition," IEEE Transactions on Affective Computing, Vol. 5, No. 3, pp. 340-351, 2014.   DOI
46 A. Hamid and M. Pietikainen, "From still Image to Video-based Face Recognition: An Experimental Analysis," Proceeding of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 813-818, 2004.
47 Y. Li, S. Wang, Y. Zhao, and Q. Ji, “Simultaneous Facial Feature Tracking and Facial Expression Recognition,” IEEE Transactions on Image Processing, Vol. 22, No. 7, pp. 2559-2573, 2013.   DOI