Browse > Article
http://dx.doi.org/10.1633/JISTaP.2019.7.2.3

Facial Data Visualization for Improved Deep Learning Based Emotion Recognition  

Lee, Seung Ho (Department of Future Technology, Korea University of Technology and Education)
Publication Information
Journal of Information Science Theory and Practice / v.7, no.2, 2019 , pp. 32-39 More about this Journal
Abstract
A convolutional neural network (CNN) has been widely used in facial expression recognition (FER) because it can automatically learn discriminative appearance features from an expression image. To make full use of its discriminating capability, this paper suggests a simple but effective method for CNN based FER. Specifically, instead of an original expression image that contains facial appearance only, the expression image with facial geometry visualization is used as input to CNN. In this way, geometric and appearance features could be simultaneously learned, making CNN more discriminative for FER. A simple CNN extension is also presented in this paper, aiming to utilize geometric expression change derived from an expression image sequence. Experimental results on two public datasets (CK+ and MMI) show that CNN using facial geometry visualization clearly outperforms the conventional CNN using facial appearance only.
Keywords
facial expression recognition; convolutional neural network; facial landmark points; facial geometry visualization;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 210-227.
2 Yang, H., Ciftci, U., & Yin, L. (2018). Facial expression recognition by de-expression residue learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18-23 June (pp. 2168-2177).
3 Donato, G., Stewart, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10), 974-989.   DOI
4 Huang, M. W., Wang, Z. W., & Ying, Z. L. (2010). A new method for facial expression recognition based on sparse representation plus LBP. In 2010 3rd International Congress on Image and Signal Processing, Yantai, China, 16-18 October (Vol. 4, pp. 1750-1754). Piscataway: IEEE.
5 Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015). Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7-13 December (pp. 2983-2991).
6 Kotsia, I., & Pitas, I. (2007). Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing, 16(1), 172-187.   DOI
7 Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.   DOI
8 Lee, S. H., Kim, H., Ro, Y. M., & Plataniotis, K. N. (2013). Using color texture sparsity for facial expression recognition. In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China, 22-26 April (pp. 1-6). Piscataway: IEEE.
9 Lee, S. H., & Ro, Y. M. (2016). Partial matching of facial expression sequence using over-complete transition dictionary for emotion recognition. IEEE Transactions on Affective Computing, 7(4), 389-408.   DOI
10 Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13-18 June (pp. 94-101). Piscataway: IEEE.
11 Asthana, A., Zafeiriou, S., Cheng, S., & Pantic, M. (2014). Incremental face alignment in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23-28 June (pp. 1859-1866).
12 Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2005). Recognizing facial expression: Machine learning and application to spontaneous behavior. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 20-25 June (pp. 568-573). Piscataway: IEEE.
13 Chen, J., Chen, D., Gong, Y., Yu, M., Zhang, K., & Wang, L. (2012). Facial expression recognition using geometric and appearance features. In Proceedings of the 4th International Conference Internet Multimedia Computing and Service, Wuhan, China (pp. 29-33). New York: ACM.
14 Choi, J. Y., Neve, W. D., Plataniotis, K. N., & Ro, Y. M. (2011). Collaborative face recognition for improved face annotation in personal photo collections shared on online social networks. IEEE Transactions on Multimedia, 13(1), 14-28.   DOI
15 Pantic, M., Valstar, M., Rademaker, R., & Maat, L. (2005). Web-based database for facial expression analysis. In 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, Netherlands, 6 July (p. 5). Piscataway: IEEE.
16 Yang, J., Zhang, D. D., Frangi, A. F., & Yang, J. Y. (2004). Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), 131-137.   DOI
17 Zafeiriou, S., & Petrou, M. (2010). Sparse representation for facial expressions recognition via l1 optimization. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA, 13-18 June (pp. 32-39). Piscataway: IEEE.
18 Meng, Z., Liu, P., Cai, J., Han, S., & Tong, Y. (2017). Identity-aware convolutional neural network for facial expression recognition. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, Washington, DC, USA, 30 May-3 June (pp. 558-565). Piscataway: IEEE.
19 Sandbach, G., Zafeiriou, S., Pantic, M., & Yin, L. (2012). Static and dynamic 3D facial expression recognition: A comprehensive survey. Image and Vision Computing, 30(10), 683-697.   DOI
20 Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137-154.   DOI
21 Wang, Z., & Ying, Z. (2012). Facial expression recognition based on local phase quantization and sparse representation. In 2012 8th International Conference on Natural Computation, Chongqing, China, 29-31 May (pp. 222-225). Piscataway: IEEE.
22 Weng, Z., Wang, S., & Ji, Q. (2013). Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 23-28 June (pp. 3422-3429).