Browse > Article
http://dx.doi.org/10.3745/JIPS.02.0156

A Video Expression Recognition Method Based on Multi-mode Convolution Neural Network and Multiplicative Feature Fusion  

Ren, Qun (Dept. of Electronic and Information Engineering, Bo Zhou University)
Publication Information
Journal of Information Processing Systems / v.17, no.3, 2021 , pp. 556-570 More about this Journal
Abstract
The existing video expression recognition methods mainly focus on the spatial feature extraction of video expression images, but tend to ignore the dynamic features of video sequences. To solve this problem, a multi-mode convolution neural network method is proposed to effectively improve the performance of facial expression recognition in video. Firstly, OpenFace 2.0 is used to detect face images in video, and two deep convolution neural networks are used to extract spatiotemporal expression features. Furthermore, spatial convolution neural network is used to extract the spatial information features of each static expression image, and the dynamic information feature is extracted from the optical flow information of multiple expression images based on temporal convolution neural network. Then, the spatiotemporal features learned by the two deep convolution neural networks are fused by multiplication. Finally, the fused features are input into support vector machine to realize the facial expression classification. Experimental results show that the recognition accuracy of the proposed method can reach 64.57% and 60.89%, respectively on RML and Baum-ls datasets. It is better than that of other contrast methods.
Keywords
Facial Expression Recognition; Multi-Mode Deep Learning; Multiplicative Fusion; Optical Flow Method; Spatial Convolutional Neural Network; Time Convolutional Neural Network;
Citations & Related Records
연도 인용수 순위
  • Reference
1 H. D. Nguyen, S. Yeom, G. S. Lee, H. J. Yang, I. S. Na, and S. H. Kim, "Facial emotion recognition using an ensemble of multi-level convolutional neural networksm," International Journal of Pattern Recognition and Artificial Intelligence, vol. 33, no. 11, article no. 1940015, 2019. https://doi.org/10.1142/S0218001419400159   DOI
2 F. Kong, "Facial expression recognition method based on deep convolutional neural network combined with improved LBP features," Personal and Ubiquitous Computing, vol. 23, no. 3, pp. 531-539, 2019.   DOI
3 F. Z. Salmam, A. Madani, and M. Kissi, "Fusing multi-stream deep neural networks for facial expression recognition," Signal, Image and Video Processing, vol. 13, no. 3, pp. 609-616, 2019.   DOI
4 N. Jain, S. Kumar, and A. Kumar, "Effective approach for facial expression recognition using hybrid square-based diagonal pattern geometric model," Multimedia Tools and Applications, vol. 78, no. 20, pp. 29555-29571, 2019.   DOI
5 X. Zhu and Z. Chen, "Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network," The Visual Computer, vol. 36, pp. 743-755, 2020.   DOI
6 Y. Luo, X. Y. Liu, X. Zhang, X. F. Chen, and Z. Chen, "Facial expression recognition based on improved completed local ternary patterns," Optoelectronics Letters, vol. 15, no. 3, pp. 224-230, 2019.   DOI
7 S. Nigam, R. Singh, and A. K. Misra, "Efficient facial expression recognition using histogram of oriented gradients in wavelet domain," Multimedia Tools and Applications, vol. 77, no. 21, pp. 28725-28747, 2018.   DOI
8 T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L. P. Morency, "Openface 2.0: Facial behavior analysis toolkit," in Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi'an, China, 2018, pp. 59-66.
9 M. S. Zia, M. Hussain, and M. A. Jaffar, "A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier," Multimedia Tools and Applications, vol. 77, no. 19, pp. 25537-25567, 2018.   DOI
10 H. Wang, S. Wei, and B. Fang, "Facial expression recognition using iterative fusion of MO-HOG and deep features," The Journal of Supercomputing, vol. 76, no. 5, pp. 3211-3221, 2020.   DOI
11 Y. Zhou and N. Chen, "The LAP under facility disruptions during early post-earthquake rescue using PSO-GA hybrid algorithm," Fresenius Environmental Bulletin, vol. 28, no. 12A, pp. 9906-9914, 2019.
12 S. Maity, M. Abdel-Mottaleb, and S. S. Asfour, "Multimodal biometrics recognition from facial video with missing modalities using deep learning," Journal of Information Processing Systems, vol. 16, no. 1, pp. 6-29, 2020.   DOI
13 O. Agbolade, A. Nazri, R. Yaakob, A. A. Ghani, and Y. K. Cheah, "3-Dimensional facial expression recognition in human using multi-points warping," BMC Bioinformatics, vol. 20, no. 1, article no. 619, 2019. https://doi.org/10.1186/s12859-019-3153-2   DOI
14 K. Talele and K. Tuckley, "Facial expression recognition using digital signature feature descriptor," Signal, Image and Video Processing, vol. 14, pp. 701-709, 2020. https://doi.org/10.1007/s11760-019-01595-1   DOI
15 U. A. Shaikh, V. J. Vishwakarma, and S. S. Mahale, "Dynamic scene multi-exposure image fusion," IETE Journal of Education, vol. 59, no. 2, pp. 53-61, 2018.   DOI
16 H. Sadeghi and A. A. Raie, "Human vision inspired feature extraction for facial expression recognition," Multimedia Tools and Applications, vol. 78, no. 21, pp. 30335-30353, 2019.   DOI
17 M. Sajjad, S. Zahir, A. Ullah, Z. Akhtar, and K. Muhammad, "Human behavior understanding in big multimedia data using CNN based facial expression recognition," Mobile Networks and Applications, vol. 25, pp. 1611-1621, 2020. https://doi.org/10.1007/s11036-019-01366-9   DOI
18 S. Nestler, "Safety-critical human computer interaction," it-Information Technology, vol. 61, no. 1, pp. 67-70, 2019.   DOI
19 Y. Jiang, K. Zhao, K. Xia, J. Xue, L. Zhou, Y. Ding, and P. Qian, "A novel distributed multitask fuzzy clustering algorithm for automatic MR brain image segmentation," Journal of Medical Systems, vol. 43, article no. 118, 2019. https://doi.org/10.1007/s10916-019-1245-1   DOI
20 J. Jian, Y. Guo, L. Jiang, Y. An, and J. Su, "A multi-objective optimization model for green supply chain considering environmental benefits," Sustainability, vol. 11, no. 21, article no. 5911, 2019. https://doi.org/10.3390/su11215911   DOI
21 R. Ramya, K. Mala, and S. S. Nidhyananthan, "3D facial expression recognition using multi-channel deep learning framework," Circuits, Systems, and Signal Processing, vol. 39, no. 2, pp. 789-804, 2020.   DOI
22 F. Ramdani, M. T. Furqon, B. D. Setiawan, and A. N. Rusydi, "Analysis of the application of an advanced classifier algorithm to ultra-high resolution unmanned aerial aircraft imagery: a neural network approach," International Journal of Remote Sensing, vol. 41, no. 9, pp. 3266-3286, 2020.   DOI
23 N. Zikiou, M. Lahdir, and D. Helbert, "Hyperspectral image classification using graph-based wavelet transform," International Journal of Remote Sensing, vol. 41, no. 7, pp. 2624-2643, 2020.   DOI
24 F. An and Z. Liu, "Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM," The Visual Computer, vol. 36, no. 3, pp. 483-498, 2020.   DOI
25 Y. Ren, T. Cheng, and Y. Zhang, "Deep spatio-temporal residual neural networks for road-network-based data modeling," International Journal of Geographical Information Science, vol. 33, no. 9, pp. 1894-1912, 2019.   DOI
26 C. Xu, Y. Cui, Y. Zhang, P. Gao, and J. Xu, "Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware," Multimedia Systems, vol. 26, no. 1, pp. 53-61, 2020.   DOI
27 S. Kumar, M. K. Bhuyan, and Y. Iwahori, "Multi-level uncorrelated discriminative shared Gaussian process for multi-view facial expression recognition," The Visual Computer, vol. 37, no. 1, pp. 143-159, 2021.   DOI
28 P. Loslever, T. Guidini Goncalves, K. M. de Oliveira, and C. Kolski, "Using fuzzy coding with qualitative data: example with subjective data in human-computer interaction," Theoretical Issues in Ergonomics Science, vol. 20, no. 4, pp. 459-488, 2019.   DOI
29 K. Li, Y. Jin, M. W. Akram, R. Han, and J. Chen, "Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy," The Visual Computer, vol. 36, no. 2, pp. 391-404, 2020.   DOI
30 X. Liu, X. Yin, M. Wang, Y. Cai, and G. Qi, "Emotion recognition based on multi-composition deep forest and transferred convolutional neural network," Journal of Advanced Computational Intelligence and Intelligent Informatics, vol. 23, no. 5, pp. 883-890, 2019.   DOI