Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Video Expression Recognition Method Based on Multi-mode Convolution Neural Network and Multiplicative Feature Fusion

  • Ren, Qun (Dept. of Electronic and Information Engineering, Bo Zhou University)
  • Received : 2020.05.14
  • Accepted : 2020.10.04
  • Published : 2021.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

The existing video expression recognition methods mainly focus on the spatial feature extraction of video expression images, but tend to ignore the dynamic features of video sequences. To solve this problem, a multi-mode convolution neural network method is proposed to effectively improve the performance of facial expression recognition in video. Firstly, OpenFace 2.0 is used to detect face images in video, and two deep convolution neural networks are used to extract spatiotemporal expression features. Furthermore, spatial convolution neural network is used to extract the spatial information features of each static expression image, and the dynamic information feature is extracted from the optical flow information of multiple expression images based on temporal convolution neural network. Then, the spatiotemporal features learned by the two deep convolution neural networks are fused by multiplication. Finally, the fused features are input into support vector machine to realize the facial expression classification. Experimental results show that the recognition accuracy of the proposed method can reach 64.57% and 60.89%, respectively on RML and Baum-ls datasets. It is better than that of other contrast methods.

Keywords

Acknowledgement

This work was supported by the Anhui Province University excellent young talents support project (No. GXYQ2019119), the Key natural science projects of Anhui Province (No. KJ2018A0820), and Bozhou University scientific research (No. BYZ2018C05).

References

  1. S. Maity, M. Abdel-Mottaleb, and S. S. Asfour, "Multimodal biometrics recognition from facial video with missing modalities using deep learning," Journal of Information Processing Systems, vol. 16, no. 1, pp. 6-29, 2020. https://doi.org/10.3745/JIPS.02.0129
  2. O. Agbolade, A. Nazri, R. Yaakob, A. A. Ghani, and Y. K. Cheah, "3-Dimensional facial expression recognition in human using multi-points warping," BMC Bioinformatics, vol. 20, no. 1, article no. 619, 2019. https://doi.org/10.1186/s12859-019-3153-2
  3. K. Talele and K. Tuckley, "Facial expression recognition using digital signature feature descriptor," Signal, Image and Video Processing, vol. 14, pp. 701-709, 2020. https://doi.org/10.1007/s11760-019-01595-1
  4. H. Sadeghi and A. A. Raie, "Human vision inspired feature extraction for facial expression recognition," Multimedia Tools and Applications, vol. 78, no. 21, pp. 30335-30353, 2019. https://doi.org/10.1007/s11042-019-07863-z
  5. M. Sajjad, S. Zahir, A. Ullah, Z. Akhtar, and K. Muhammad, "Human behavior understanding in big multimedia data using CNN based facial expression recognition," Mobile Networks and Applications, vol. 25, pp. 1611-1621, 2020. https://doi.org/10.1007/s11036-019-01366-9
  6. S. Nestler, "Safety-critical human computer interaction," it-Information Technology, vol. 61, no. 1, pp. 67-70, 2019. https://doi.org/10.1515/itit-2018-0037
  7. P. Loslever, T. Guidini Goncalves, K. M. de Oliveira, and C. Kolski, "Using fuzzy coding with qualitative data: example with subjective data in human-computer interaction," Theoretical Issues in Ergonomics Science, vol. 20, no. 4, pp. 459-488, 2019. https://doi.org/10.1080/1463922X.2019.1574932
  8. U. A. Shaikh, V. J. Vishwakarma, and S. S. Mahale, "Dynamic scene multi-exposure image fusion," IETE Journal of Education, vol. 59, no. 2, pp. 53-61, 2018. https://doi.org/10.1080/09747338.2018.1510744
  9. Y. Jiang, K. Zhao, K. Xia, J. Xue, L. Zhou, Y. Ding, and P. Qian, "A novel distributed multitask fuzzy clustering algorithm for automatic MR brain image segmentation," Journal of Medical Systems, vol. 43, article no. 118, 2019. https://doi.org/10.1007/s10916-019-1245-1
  10. F. Ramdani, M. T. Furqon, B. D. Setiawan, and A. N. Rusydi, "Analysis of the application of an advanced classifier algorithm to ultra-high resolution unmanned aerial aircraft imagery: a neural network approach," International Journal of Remote Sensing, vol. 41, no. 9, pp. 3266-3286, 2020. https://doi.org/10.1080/01431161.2019.1688413
  11. N. Zikiou, M. Lahdir, and D. Helbert, "Hyperspectral image classification using graph-based wavelet transform," International Journal of Remote Sensing, vol. 41, no. 7, pp. 2624-2643, 2020. https://doi.org/10.1080/01431161.2019.1694194
  12. F. An and Z. Liu, "Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM," The Visual Computer, vol. 36, no. 3, pp. 483-498, 2020. https://doi.org/10.1007/s00371-019-01635-4
  13. R. Ramya, K. Mala, and S. S. Nidhyananthan, "3D facial expression recognition using multi-channel deep learning framework," Circuits, Systems, and Signal Processing, vol. 39, no. 2, pp. 789-804, 2020. https://doi.org/10.1007/s00034-019-01144-8
  14. C. Xu, Y. Cui, Y. Zhang, P. Gao, and J. Xu, "Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware," Multimedia Systems, vol. 26, no. 1, pp. 53-61, 2020. https://doi.org/10.1007/s00530-019-00628-6
  15. K. Li, Y. Jin, M. W. Akram, R. Han, and J. Chen, "Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy," The Visual Computer, vol. 36, no. 2, pp. 391-404, 2020. https://doi.org/10.1007/s00371-019-01627-4
  16. S. Kumar, M. K. Bhuyan, and Y. Iwahori, "Multi-level uncorrelated discriminative shared Gaussian process for multi-view facial expression recognition," The Visual Computer, vol. 37, no. 1, pp. 143-159, 2021. https://doi.org/10.1007/s00371-019-01788-2
  17. H. D. Nguyen, S. Yeom, G. S. Lee, H. J. Yang, I. S. Na, and S. H. Kim, "Facial emotion recognition using an ensemble of multi-level convolutional neural networksm," International Journal of Pattern Recognition and Artificial Intelligence, vol. 33, no. 11, article no. 1940015, 2019. https://doi.org/10.1142/S0218001419400159
  18. X. Liu, X. Yin, M. Wang, Y. Cai, and G. Qi, "Emotion recognition based on multi-composition deep forest and transferred convolutional neural network," Journal of Advanced Computational Intelligence and Intelligent Informatics, vol. 23, no. 5, pp. 883-890, 2019. https://doi.org/10.20965/jaciii.2019.p0883
  19. N. Jain, S. Kumar, and A. Kumar, "Effective approach for facial expression recognition using hybrid square-based diagonal pattern geometric model," Multimedia Tools and Applications, vol. 78, no. 20, pp. 29555-29571, 2019. https://doi.org/10.1007/s11042-019-7325-x
  20. F. Kong, "Facial expression recognition method based on deep convolutional neural network combined with improved LBP features," Personal and Ubiquitous Computing, vol. 23, no. 3, pp. 531-539, 2019. https://doi.org/10.1007/s00779-019-01238-9
  21. F. Z. Salmam, A. Madani, and M. Kissi, "Fusing multi-stream deep neural networks for facial expression recognition," Signal, Image and Video Processing, vol. 13, no. 3, pp. 609-616, 2019. https://doi.org/10.1007/s11760-018-1388-4
  22. X. Zhu and Z. Chen, "Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network," The Visual Computer, vol. 36, pp. 743-755, 2020. https://doi.org/10.1007/s00371-019-01660-3
  23. Y. Luo, X. Y. Liu, X. Zhang, X. F. Chen, and Z. Chen, "Facial expression recognition based on improved completed local ternary patterns," Optoelectronics Letters, vol. 15, no. 3, pp. 224-230, 2019. https://doi.org/10.1007/s11801-019-8136-z
  24. S. Nigam, R. Singh, and A. K. Misra, "Efficient facial expression recognition using histogram of oriented gradients in wavelet domain," Multimedia Tools and Applications, vol. 77, no. 21, pp. 28725-28747, 2018. https://doi.org/10.1007/s11042-018-6040-3
  25. M. S. Zia, M. Hussain, and M. A. Jaffar, "A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier," Multimedia Tools and Applications, vol. 77, no. 19, pp. 25537-25567, 2018. https://doi.org/10.1007/s11042-018-5806-y
  26. H. Wang, S. Wei, and B. Fang, "Facial expression recognition using iterative fusion of MO-HOG and deep features," The Journal of Supercomputing, vol. 76, no. 5, pp. 3211-3221, 2020. https://doi.org/10.1007/s11227-018-2554-8
  27. T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L. P. Morency, "Openface 2.0: Facial behavior analysis toolkit," in Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi'an, China, 2018, pp. 59-66.
  28. Y. Zhou and N. Chen, "The LAP under facility disruptions during early post-earthquake rescue using PSO-GA hybrid algorithm," Fresenius Environmental Bulletin, vol. 28, no. 12A, pp. 9906-9914, 2019.
  29. J. Jian, Y. Guo, L. Jiang, Y. An, and J. Su, "A multi-objective optimization model for green supply chain considering environmental benefits," Sustainability, vol. 11, no. 21, article no. 5911, 2019. https://doi.org/10.3390/su11215911
  30. Y. Ren, T. Cheng, and Y. Zhang, "Deep spatio-temporal residual neural networks for road-network-based data modeling," International Journal of Geographical Information Science, vol. 33, no. 9, pp. 1894-1912, 2019. https://doi.org/10.1080/13658816.2019.1599895