Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Multi-modal Emotion Recognition using Semi-supervised Learning and Multiple Neural Networks in the Wild

준 지도학습과 여러 개의 딥 뉴럴 네트워크를 사용한 멀티 모달 기반 감정 인식 알고리즘

  • Kim, Dae Ha (Department of Electronic Engineering College of Engineering, Inha University) ;
  • Song, Byung Cheol (Department of Electronic Engineering College of Engineering, Inha University)
  • Received : 2018.03.22
  • Accepted : 2018.04.26
  • Published : 2018.05.30
Download PDF
⟨ Previous Next ⟩

Abstract

Human emotion recognition is a research topic that is receiving continuous attention in computer vision and artificial intelligence domains. This paper proposes a method for classifying human emotions through multiple neural networks based on multi-modal signals which consist of image, landmark, and audio in a wild environment. The proposed method has the following features. First, the learning performance of the image-based network is greatly improved by employing both multi-task learning and semi-supervised learning using the spatio-temporal characteristic of videos. Second, a model for converting 1-dimensional (1D) landmark information of face into two-dimensional (2D) images, is newly proposed, and a CNN-LSTM network based on the model is proposed for better emotion recognition. Third, based on an observation that audio signals are often very effective for specific emotions, we propose an audio deep learning mechanism robust to the specific emotions. Finally, so-called emotion adaptive fusion is applied to enable synergy of multiple networks. The proposed network improves emotion classification performance by appropriately integrating existing supervised learning and semi-supervised learning networks. In the fifth attempt on the given test set in the EmotiW2017 challenge, the proposed method achieved a classification accuracy of 57.12%.

인간 감정 인식은 컴퓨터 비전 및 인공 지능 영역에서 지속적인 관심을 받는 연구 주제이다. 본 논문에서는 wild 환경에서 이미지, 얼굴 특징점 및 음성신호로 구성된 multi-modal 신호를 기반으로 여러 신경망을 통해 인간의 감정을 분류하는 방법을 제안한다. 제안 방법은 다음과 같은 특징을 갖는다. 첫째, multi task learning과 비디오의 시공간 특성을 이용한 준 감독 학습을 사용함으로써 영상 기반 네트워크의 학습 성능을 크게 향상시켰다. 둘째, 얼굴의 1 차원 랜드 마크 정보를 2 차원 영상으로 변환하는 모델을 새로 제안하였고, 이를 바탕으로 한 CNN-LSTM 네트워크를 제안하여 감정 인식을 향상시켰다. 셋째, 특정 감정에 오디오 신호가 매우 효과적이라는 관측을 기반으로 특정 감정에 robust한 오디오 심층 학습 메커니즘을 제안한다. 마지막으로 소위 적응적 감정 융합 (emotion adaptive fusion)을 적용하여 여러 네트워크의 시너지 효과를 극대화한다. 제안 네트워크는 기존의 지도 학습과 반 지도학습 네트워크를 적절히 융합하여 감정 분류 성능을 향상시켰다. EmotiW2017 대회에서 주어진 테스트 셋에 대한 5번째 시도에서, 제안 방법은 57.12 %의 분류 정확도를 달성하였다.

Keywords

References

  1. Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, "EEG-based emotion recognition in music listening," Proceeding of IEEE Transactions on Biomedical Engineering, 57(7), pp.1798-1806, 2010.
  2. Y. Fan, X. Lu, D. Li, and Y. Liu, "Video-based emotion recognition using cnn-rnn and c3d hybrid networks. Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.445-450, 2016, doi:10.1145/2993148.2997632.
  3. A. Yao, D. Cai, P. Hu, S. Wang, L. Sha, and Y. Chen, "HoloNet: towards robust emotion recognition in the wild," Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.472-478, 2016, doi:10.1145/2993148.2997639.
  4. N. Dalal, and B. Triggs, "Histograms of oriented gradients for human detection," Proceeding of IEEE Computer Society Conference on (Vol. 1), pp.886-893, 2005, doi:10.1109/CVPR.2005.177.
  5. T. Ojala, M. Pietikainen, and D. Marwood, "Performance evaluation of texture measures with classification based on Kullback discrimination of distributions," Proceeding of the 12th IAPR International Conference on, pp.582-585, 1994, doi:10.1109/ICPR.1994.576366.
  6. T. Ojala, M. Pietikäinen, and D. Harwood, "A comparative study of texture measures with classification based on featured distributions," Pattern recognition, 29(1), pp.51-59, 1996, doi:10.1016/0031-3203(95)00067-4.
  7. C. Cortes, and V. Vapnik, "Support-vector networks," Machine learning, pp.273-297, 1995.
  8. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, " Imagenet: A large-scale hierarchical image database," Proceeding of the IEEE conference on Computer Vision and Pattern Recognition, pp.248-255, 2009.
  9. A. Krizhevsky, I. Sutskever, and G. Hinton, "Imagenet classification with deep convolutional neural networks," In Advances in neural information processing systems, pp.1097-1105, 2012.
  10. Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel, "Handwritten digit recognition with a back-propagation network," In Advances in neural information processing systems, pp.396-404, 1990.
  11. K. Simonyan, and A. Zisserman, "Very deep convolutional networks for large-scale image recognition". arXiv preprint arXiv:1409.1556, 2014.
  12. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceeding of the IEEE conference on computer vision and pattern recognition, pp.770-778, 2016.
  13. G. Huang, Z. Liu, K. Weinberger, and L. van der Maaten, "Densely connected convolutional networks," Proceeding of the IEEE conference on computer vision and pattern recognition, 2017.
  14. P. Ekman, "An argument for basic emotions," Cognition & emotion, pp.169-200, 1992.
  15. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, "Collecting large, richly annotated facial-expression databases from movies," 2012, doi:10.1.1. 407.4632.
  16. Y. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," Proceeding of the IEEE Transactions on pattern analysis and machine intelligence, pp.97-115, 2001.
  17. P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar, Z.,I. Matthews, "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," Proceeding of the IEEE conference on Computer Vision and Pattern Recognition Workshops, pp.94-101, 2010.
  18. D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3d convolutional networks," Proceeding of the IEEE international conference on computer vision, pp.4489- 4497, 2015.
  19. S. Bargal, E. Barsoum, C. Ferrer, and C. Zhang, "Emotion recognition in the wild from videos using images," Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.433-436, 2016, doi:10.1145/2993148.2997627.
  20. X. Zhu, "Semi-supervised learning literature survey". Computer Science, University of Wisconsin-Madison, 2(3), 4, 2006.
  21. L. Wang, C. Lee, Z. Tu, and S. Lazebnik, "Training deeper convolutional networks with deep supervision," arXiv preprint arXiv:1505.02496, 2015.
  22. H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua, "A convolutional neural network cascade for face detection," Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, pp.5325-5334, 2015.
  23. S. Ruder, "An Overview of Multi-Task Learning in Deep Neural Networks," arXiv preprint arXiv:1706.05098, 2017.
  24. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, ... and A. Rabinovich, "Going deeper with convolutions,". Proceeding of the IEEE conference on computer vision and pattern recognition, pp.1-9, 2015.
  25. A. Dhall, R. Goecke, S. Ghosh, J. Hoshi, J. Hoey, T. Gedeon, "From Individual to Group-level Emotion Recognition: EmotiW 5.0", Proceeding of the 18th ACM International Conference on Multimodal Interaction (in press), 2017.
  26. S. Zagoruyko, and N. Komodakis, "Wide residual networks," arXiv preprint arXiv:1605.07146, 2016.
  27. F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions," Proceeding of the IEEE conference on computer vision and pattern recognition, pp.1251-1258, 2017.
  28. A. Asthana, S. Zafeiriou, S. Cheng, and M. Pantic, "Incremental face alignment in the wild," Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1859-1866, 2014.
  29. H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, "Joint fine-tuning in deep neural networks for facial expression recognition," Proceeding of the IEEE International Conference on Computer Vision, pp.2983-2991, 2015.
  30. J. Yan, W. Zheng, Z. Cui, C. Tang, T. Zhang, Y. Zong, and N. Sun, "Multi-clue fusion for emotion recognition in the wild," Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.458-463, 2016.
  31. F. Eyben, M. Wöllmer, B. Schuller, "Opensmile: the munich versatile and fast open-source audio feature extractor," Proceeding of the 18th ACM international conference on Multimedia, pp.1459-1462, 2010.
  32. B. McFee, C. Raffel, D. Liang, D. Ellis, M. McVicar, E. Battenberg, and O. Nieto, "librosa: Audio and music signal analysis in python," Proceeding of the 14th python in science conference, pp.18-25, 2015.
  33. F. Chollet, Keras, http://keras.io, 2015.
  34. K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," Proceeding of the IEEE international conference on computer vision, pp.1026-1034, 2015.
  35. S. Ioffe, and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," In International Conference on Machine Learning, pp.448-456, 2015.
  36. Zhang, Kaipeng et al. "Joint face detection and alignment using multitask cascaded convolutional networks," Proceeding of IEEE Signal Processing letters, pp.1499-1503, 2016.
  37. Li, Xi, et al. "DeepSaliency: Multi-task deep neural network model for salient object detection," Proceeding of IEEE Transactions on Image Processing, pp.3919-3930, 2016.
  38. Rasmus, Antti, et al."Semi-supervised learning with ladder networks," Advances in Neural Information Processing Systems, 2015.
  39. S. Laine, and T. Aila "Temporal Ensembling for Semi-Supervised Learning," arXiv preprint arXiv: 1610.02242, 2016.
  40. V. Vielzeuf, S. Pateux, and F. Jurie. "Temporal multimodal fusion for video emotion classification in the wild." Proceeding of the 19th ACM International Conference on Multimodal Interaction, 2017.