Journal of Korea Multimedia Society (한국멀티미디어학회논문지)

Korea Multimedia Society (한국멀티미디어학회)
권호 표지
DOI QR코드

DOI QR Code

Weighted Soft Voting Classification for Emotion Recognition from Facial Expressions on Image Sequences

이미지 시퀀스 얼굴표정 기반 감정인식을 위한 가중 소프트 투표 분류 방법

  • Kim, Kyeong Tae (Dept. of Computer and Electronic Systems Eng., Hankuk University of Foreign Studies) ;
  • Choi, Jae Young (Dept. of Computer and Electronic Systems Eng., Hankuk University of Foreign Studies)
  • Received : 2017.07.17
  • Accepted : 2017.07.24
  • Published : 2017.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Human emotion recognition is one of the promising applications in the era of artificial super intelligence. Thus far, facial expression traits are considered to be the most widely used information cues for realizing automated emotion recognition. This paper proposes a novel facial expression recognition (FER) method that works well for recognizing emotion from image sequences. To this end, we develop the so-called weighted soft voting classification (WSVC) algorithm. In the proposed WSVC, a number of classifiers are first constructed using different and multiple feature representations. In next, multiple classifiers are used for generating the recognition result (namely, soft voting) of each face image within a face sequence, yielding multiple soft voting outputs. Finally, these soft voting outputs are combined through using a weighted combination to decide the emotion class (e.g., anger) of a given face sequence. The weights for combination are effectively determined by measuring the quality of each face image, namely "peak expression intensity" and "frontal-pose degree". To test the proposed WSVC, CK+ FER database was used to perform extensive and comparative experimentations. The feasibility of our WSVC algorithm has been successfully demonstrated by comparing recently developed FER algorithms.

Keywords

References

  1. P. Michel and R. El Kaliouby, "Real Time Facial Expression Recognition in Video Using Support Vector Machines," Proceedings of the 5th International Conference on Multimodal Interfaces (Association for Computing Machinery) , pp. 258-264, 2003.
  2. G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin, “Static and Dynamic 3D Facial Expression Recognition: A Comprehensive Survey,” Image and Vision Computing, Vol. 30, No. 10, pp. 683-697, 2012. https://doi.org/10.1016/j.imavis.2012.06.005
  3. S. Taheri, Q. Qiu, and R. Chellappa, "Structure-Preserving Sparse Decomposition for Facial Expression Analysis," IEEE Transactions on Image Processing, Vol. 23, No. 8, pp. 3590-3603, 2014. https://doi.org/10.1109/TIP.2014.2331141
  4. Y.L. Tian, T. Kanade, and J.F. Cohn, Facial Expression Analysis, Handbook of Face Recognition, Springer, New York, pp. 247-276, 2005.
  5. R.V. Yampolskiy, Artificial Superintelligence: a Futuristic Approach, Chemical Rubber Company Press, London, 2015.
  6. Microsoft Azure, https://www.projectoxford.ai/demo/Emotion#detection (accessed July, 10, 2017).
  7. Affectiva, https://www.affectiva.com (accessed July, 10, 2017).
  8. J. Stallkamp, H.K. Ekenel, and R. Stiefelhagen, "Video-based Face Recognition on Real-World Data on Real-world Dataset," Proceeding of IEEE International Conference on Computer Vision, pp. 1-8, 2007.
  9. Y. Zhang and A.M. Martinez, “A Weighted Probabilistic Approach to Face Recognition from Multiple Images and Video Sequences,” Image Vision Computing, Vol. 24, No. 6, pp. 626-638, 2006. https://doi.org/10.1016/j.imavis.2005.08.004
  10. A. Hamid and M. Pietikainen, "From still Image to Video-based Face Recognition: An Experimental Analysis," Proceeding of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 813-818, 2004.
  11. Y. Li, S. Wang, Y. Zhao, and Q. Ji, “Simultaneous Facial Feature Tracking and Facial Expression Recognition,” IEEE Transactions on Image Processing, Vol. 22, No. 7, pp. 2559-2573, 2013. https://doi.org/10.1109/TIP.2013.2253477
  12. J.J. Lien, T. Kanade, J.F. Cohn, and C. Li, “Detection, Tracking, and Classification of Action Units in Facial Expression,” Journal of Robotics and Autonomous Systems, Vol. 31, No. 3, pp. 131-146, 2000. https://doi.org/10.1016/S0921-8890(99)00103-7
  13. Y. Chang, C. Hu, R. Feris, and M Turk, “Manifold Based Analysis of Facial Expression,” Image Vision Computing, Vol. 24, No. 10, pp. 605-614, 2006. https://doi.org/10.1016/j.imavis.2005.08.006
  14. G. Zhao and M. Pietikäinen, “Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 29, No. 6, pp. 915-928, 2007. https://doi.org/10.1109/TPAMI.2007.1110
  15. B. Jiang, F. Valstar, and M. Pantic, "Action Unit Detection Using Sparse Appearance Descriptors in Space-Time Video Volumes," Proceeding of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 314-321, 2011.
  16. B. Jiang, M. Valstar, B. Martinez, and M. Pantic, “A Dynamic Appearance Descriptor Approach to Facial Actions Temporal Modeling,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, Vol. 44, No. 2, pp. 161-174, 2014.
  17. D. Ruta and B. Gabrys, “Classifier Selection for Majority Voting,” Information Fusion, Vol. 6, No. 1, pp. 63-81, 2005. https://doi.org/10.1016/j.inffus.2004.04.008
  18. P. Lucey, J.F. Cohn, T. Kanade, J. Saragih, and Z. Ambadar, "The Extended Cohn-Kanade Dataset (CK+): A Complete Dataset for Action Unit and Emotion-specified Expression," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 94-101, 2010.
  19. P. Viola and M. Jones, "Rapid Object Detection Using a Boosted Cascade of Simple Features," Proceeding of IEEE International Conference on Computer Vision Pattern Recognition, pp. I-511-I-518, 2001.
  20. C. Tomasi and T. Kanade, "Detection and Tracking of Point Features," Carnegie Mellon University, Pittsburgh, PA, Technical Report CMU-CS-91-132, 1991.
  21. M.S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 568-573, 2005.
  22. J.A. Davis, D.E. McNamara, D.M. Cottrell, and J. Campos, “Image Processing with the Radial Hilbert Transform: Theory and Experiments,” Optics Letters, Vol. 25, No. 2, pp. 99-101, 2000. https://doi.org/10.1364/OL.25.000099
  23. A. Jain, K. Nandakumar, and A. Ross, “Score Normalization in Multimodal Biometric Systems,” Journal of Pattern Recognition, Vol. 38, No. 12, pp. 2270-2285, 2005. https://doi.org/10.1016/j.patcog.2005.01.012
  24. J.Y. Choi, Y.M. Ro, and K.N. Plataniotis, “Color Local Texture Features for Color Face Recognition,” IEEE Transactions on Image Processing, Vol. 21, No. 3, pp. 1366-1380, 2012. https://doi.org/10.1109/TIP.2011.2168413
  25. T. Ahonen, A. Hadid, and M. Pietikainen, “Face Description with Local Binary Pattern: Application to Face Recognition,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 28, No. 12, pp. 2037-2041, 2006. https://doi.org/10.1109/TPAMI.2006.244
  26. C. Liu and H. Wechsler, “Gabor Feature Based Classification Using the Enhanced Fisher Linear Discriminant Model for Face Recognition,” IEEE Transactions on Image Processing, Vol. 11, No. 4, pp. 467-476, 2002. https://doi.org/10.1109/TIP.2002.999679
  27. J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas, “On Combining Classifiers,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 20, No. 3, pp. 226-239, 1998. https://doi.org/10.1109/34.667881
  28. M. Suk and P. Balakrishnan, "Real-time Mobile Facial Expression Recognition System-A Case Study," Proceeding of IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 132-137, 2014.
  29. B. Moghaddam, “Principal Manifolds and Probabilistic Subspaces for Visual Recognition,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 24, No. 6, pp. 780-788, 2002. https://doi.org/10.1109/TPAMI.2002.1008384
  30. P.L. Carrier, A. Courville, I.J. Goodfellow, M. Mirza, and Y. Bengio, FER-2013 Face Database, Technical Report, 2013.
  31. R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, “Multi-pie,” Image and Vision Computing, Vol. 28, No. 5, pp. 807-813, 2010. https://doi.org/10.1016/j.imavis.2009.08.002
  32. P. Michel and R.E.l. Kaliouby, "Real Time Facial Expression Recognition in Video Using Support Vector Machines," Proceedings of the 5th Association for Computing Machinery International Conference on Multimodal Interfaces, pp. 258-264, 2003.
  33. B. Moghaddam, “Principal Manifolds and Probabilistic Subspaces for Visual Recognition,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 24, No. 6, pp. 780-788, 2002. https://doi.org/10.1109/TPAMI.2002.1008384
  34. T. Sim, S. Baker, and M. Bsat, “The CMU Pose, Illumination, and Expression Database,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 25, No. 12, pp. 1615-1618, 2003. https://doi.org/10.1109/TPAMI.2003.1251154
  35. P.J. Phillips, H. Moon, S.A. Rizvi, and P.J. Rauss, “The FERET Evaluation Methodology for Face Recognition Algorithms,” IEEE Transactions on Pattern Analysis Machine Intelligence, Vol. 22, No. 10, pp. 1090-1104, 2000. https://doi.org/10.1109/34.879790
  36. Intra-Face, http://humansensing.cs.cmu.edu/intraface (accessed June, 23, 2017).
  37. Y. Tian, "Evaluation of Face Resolution for Expression Analysis," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition Workshop, pp. 82-88, 2004.
  38. H.W. Kang, K.T. Lim, and C.H. Won, "Learning Directional LBP Features and Discriminative Feature Regions for Facial Expression Recognition," Journal of Korea Multimedia Society, Vol. 20, No. 5, pp. 748-757, 2017. https://doi.org/10.9717/kmms.2017.20.5.748
  39. X. Huang, G. Zhao, W. Zheng, and M. Pietikainen, "Spatiotemporal Local Monogenic Binary Patterns for Facial Expression Recognition," IEEE Signal Processing Letters, Vol. 19, No. 5, pp. 243-246, 2012. https://doi.org/10.1109/LSP.2012.2188890
  40. M. Huang, Z. Wang, and Z. Ying, "A New Method for Facial Expression Recognition Based on Sparse Representation Plus LBP," IEEE International Congress on Image and Signal Processing, pp. 1750-1754, 2010.
  41. W. Zhen and Y. Zilu, "Facial Expression Recognition Based on Local Phase Quantization and Sparse Representation," Proceeding of IEEE International Conference on Natural Computation, pp. 222-225, 2012.
  42. S. Zafeiriou and M. Petrou, "Sparse Representation for Facial Expression Recognition via l1 Optimization," Proceeding of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 32-39, 2010.
  43. M.F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, "The First Expression Recognition and Analysis Challenge," Proceeding of IEEE International Conference on Automatic Face and Gesture Recognition, pp. 921-926, 2011.
  44. A.R. Rivera, J.R. Castillo, and O. Chae, “Local Directional Number Pattern for Face Analysis,” IEEE Transactions on Image Processing, Vol. 22, No. 5, pp. 1740-1752, 2013. https://doi.org/10.1109/TIP.2012.2235848
  45. A. Saeed, A. Al-Hamadi, and R. Niese, "Neutral-independent Geometric Features for Facial Expression Recognition," Proceeding of IEEE International Conference on Intelligent Systems Design and Applications, pp. 842-846, 2012.
  46. S. Taheri, V.M. Patel, and R. Chellappa, “Component-based Recognition of Faces and Facial Expressions,” IEEE Transactions on Affective Computing, Vol. 23, No. 8, pp. 360-371, 2013.
  47. S.H. Lee, K.N. Plataniotis, and Y.M. Ro, "Intra-Class Variation Reduction Using Training Expression Images for Sparse Representation Based Facial Expression Recognition," IEEE Transactions on Affective Computing, Vol. 5, No. 3, pp. 340-351, 2014. https://doi.org/10.1109/TAFFC.2014.2346515