Browse > Article
http://dx.doi.org/10.3745/JIPS.01.0070

A Facial Expression Recognition Method Using Two-Stream Convolutional Networks in Natural Scenes  

Zhao, Lixin (Dean's Office, Sanmenxia Polytechnic)
Publication Information
Journal of Information Processing Systems / v.17, no.2, 2021 , pp. 399-410 More about this Journal
Abstract
Aiming at the problem that complex external variables in natural scenes have a greater impact on facial expression recognition results, a facial expression recognition method based on two-stream convolutional neural network is proposed. The model introduces exponentially enhanced shared input weights before each level of convolution input, and uses soft attention mechanism modules on the space-time features of the combination of static and dynamic streams. This enables the network to autonomously find areas that are more relevant to the expression category and pay more attention to these areas. Through these means, the information of irrelevant interference areas is suppressed. In order to solve the problem of poor local robustness caused by lighting and expression changes, this paper also performs lighting preprocessing with the lighting preprocessing chain algorithm to eliminate most of the lighting effects. Experimental results on AFEW6.0 and Multi-PIE datasets show that the recognition rates of this method are 95.05% and 61.40%, respectively, which are better than other comparison methods.
Keywords
Attentional Mechanism; Confrontational Learning; Double Flow Convolutional Neural Network; Image Preprocessing; Natural Scene Expression Recognition;
Citations & Related Records
연도 인용수 순위
  • Reference
1 B. K. Kim, J. Roh, S. Y. Dong, and S. Y. Lee, "Hierarchical committee of deep convolutional neural networks for robust facial expression recognition," Journal on Multimodal User Interfaces, vol. 10, no. 2, pp. 173-189, 2016.   DOI
2 G. Pons and D. Masip, "Supervised committee of convolutional neural networks in automated facial expression analysis," IEEE Transactions on Affective Computing, vol. 9, no. 3, pp. 343-350, 2018. https://doi.org/10.1109/TAFFC.2017.2753235   DOI
3 L. Chen, M. Zhou, W. Su, M. Wu, J. She, and K. Hirota, "Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction," Information Sciences, vol. 428, pp. 49- 61, 2018.   DOI
4 S. Eleftheriadis, O. Rudovic, and M. Pantic, "Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition," IEEE Transactions on Image Processing, vol. 24, no. 1, pp. 189-204, 2014. https://doi.org/10.1109/TIP.2014.2375634   DOI
5 P. Hu, D. Cai, S. Wang, A. Yao, and Y. Chen, "Learning supervised scoring ensemble for emotion recognition in the wild," in Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK, 2017, pp. 553-560.
6 J. Ueda and K. Okajima, "Face morphing using average face for subtle expression recognition," in Proceedings of 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), Dubrovnik, Croatia, 2019, pp. 187-192. https://doi.org/10.1109/ISPA.2019.8868931   DOI
7 N. P. Gopalan, S. Bellamkonda, and V. S. Chaitanya, "Facial expression recognition using geometric landmark points and convolutional neural networks," in Proceedings of 2018 International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, 2018, pp. 1149-1153.
8 Y. He and X. He, "Facial expression recognition based on multi-feature fusion and HOSVD," in Proceedings of 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chengdu, China, 2019, pp. 638-643. https://doi.org/10.1109/ITNEC.2019.8729003   DOI
9 S. Wang, B. Pan, H. Chen, and Q. Ji, "Thermal augmented expression recognition," IEEE Transactions on Cybernetics, vol. 48, no. 7, pp. 2203-2214, 2018. https://doi.org/10.1109/TCYB.2017.2786309   DOI
10 Y. Fan, J. C. Lam, and V. O. Li, "Video-based emotion recognition using deeply-supervised neural networks," in Proceedings of the 20th ACM International Conference on Multimodal Interaction, Boulder, CO, 2018, pp. 584-588.
11 P. Liu, S. Han, Z. Meng, and Y. Tong, "Facial expression recognition via a boosted deep belief network," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, 2014, pp. 1805-1812. https://doi.org/10.1109/CVPR.2014.233   DOI
12 N. Song, H. Yang, and P. Wu, "A gesture-to-emotional speech conversion by combining gesture recognition and facial expression recognition," in Proceedings of 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), Beijing, China, 2018, pp. 1-6. https://doi.org/10.1109/ACIIAsia.2018.8470350   DOI