Browse > Article
http://dx.doi.org/10.5909/JBE.2018.23.3.351

Multi-modal Emotion Recognition using Semi-supervised Learning and Multiple Neural Networks in the Wild  

Kim, Dae Ha (Department of Electronic Engineering College of Engineering, Inha University)
Song, Byung Cheol (Department of Electronic Engineering College of Engineering, Inha University)
Publication Information
Journal of Broadcast Engineering / v.23, no.3, 2018 , pp. 351-360 More about this Journal
Abstract
Human emotion recognition is a research topic that is receiving continuous attention in computer vision and artificial intelligence domains. This paper proposes a method for classifying human emotions through multiple neural networks based on multi-modal signals which consist of image, landmark, and audio in a wild environment. The proposed method has the following features. First, the learning performance of the image-based network is greatly improved by employing both multi-task learning and semi-supervised learning using the spatio-temporal characteristic of videos. Second, a model for converting 1-dimensional (1D) landmark information of face into two-dimensional (2D) images, is newly proposed, and a CNN-LSTM network based on the model is proposed for better emotion recognition. Third, based on an observation that audio signals are often very effective for specific emotions, we propose an audio deep learning mechanism robust to the specific emotions. Finally, so-called emotion adaptive fusion is applied to enable synergy of multiple networks. The proposed network improves emotion classification performance by appropriately integrating existing supervised learning and semi-supervised learning networks. In the fifth attempt on the given test set in the EmotiW2017 challenge, the proposed method achieved a classification accuracy of 57.12%.
Keywords
Emotion recognition; Multi-task learning; Semi-supervised learning; Multi modal signal; EmotiW 2017 challenge;
Citations & Related Records
연도 인용수 순위
  • Reference
1 S. Laine, and T. Aila "Temporal Ensembling for Semi-Supervised Learning," arXiv preprint arXiv: 1610.02242, 2016.
2 F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions," Proceeding of the IEEE conference on computer vision and pattern recognition, pp.1251-1258, 2017.
3 V. Vielzeuf, S. Pateux, and F. Jurie. "Temporal multimodal fusion for video emotion classification in the wild." Proceeding of the 19th ACM International Conference on Multimodal Interaction, 2017.
4 Y. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," Proceeding of the IEEE Transactions on pattern analysis and machine intelligence, pp.97-115, 2001.
5 P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar, Z.,I. Matthews, "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," Proceeding of the IEEE conference on Computer Vision and Pattern Recognition Workshops, pp.94-101, 2010.
6 D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3d convolutional networks," Proceeding of the IEEE international conference on computer vision, pp.4489- 4497, 2015.
7 S. Bargal, E. Barsoum, C. Ferrer, and C. Zhang, "Emotion recognition in the wild from videos using images," Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.433-436, 2016, doi:10.1145/2993148.2997627.   DOI
8 X. Zhu, "Semi-supervised learning literature survey". Computer Science, University of Wisconsin-Madison, 2(3), 4, 2006.
9 L. Wang, C. Lee, Z. Tu, and S. Lazebnik, "Training deeper convolutional networks with deep supervision," arXiv preprint arXiv:1505.02496, 2015.
10 H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua, "A convolutional neural network cascade for face detection," Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, pp.5325-5334, 2015.
11 S. Ruder, "An Overview of Multi-Task Learning in Deep Neural Networks," arXiv preprint arXiv:1706.05098, 2017.
12 A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, "Collecting large, richly annotated facial-expression databases from movies," 2012, doi:10.1.1. 407.4632.
13 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, ... and A. Rabinovich, "Going deeper with convolutions,". Proceeding of the IEEE conference on computer vision and pattern recognition, pp.1-9, 2015.
14 A. Dhall, R. Goecke, S. Ghosh, J. Hoshi, J. Hoey, T. Gedeon, "From Individual to Group-level Emotion Recognition: EmotiW 5.0", Proceeding of the 18th ACM International Conference on Multimodal Interaction (in press), 2017.
15 S. Zagoruyko, and N. Komodakis, "Wide residual networks," arXiv preprint arXiv:1605.07146, 2016.
16 A. Yao, D. Cai, P. Hu, S. Wang, L. Sha, and Y. Chen, "HoloNet: towards robust emotion recognition in the wild," Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.472-478, 2016, doi:10.1145/2993148.2997639.   DOI
17 N. Dalal, and B. Triggs, "Histograms of oriented gradients for human detection," Proceeding of IEEE Computer Society Conference on (Vol. 1), pp.886-893, 2005, doi:10.1109/CVPR.2005.177.   DOI
18 T. Ojala, M. Pietikainen, and D. Marwood, "Performance evaluation of texture measures with classification based on Kullback discrimination of distributions," Proceeding of the 12th IAPR International Conference on, pp.582-585, 1994, doi:10.1109/ICPR.1994.576366.   DOI
19 T. Ojala, M. Pietikäinen, and D. Harwood, "A comparative study of texture measures with classification based on featured distributions," Pattern recognition, 29(1), pp.51-59, 1996, doi:10.1016/0031-3203(95)00067-4.   DOI
20 C. Cortes, and V. Vapnik, "Support-vector networks," Machine learning, pp.273-297, 1995.
21 K. Simonyan, and A. Zisserman, "Very deep convolutional networks for large-scale image recognition". arXiv preprint arXiv:1409.1556, 2014.
22 J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, " Imagenet: A large-scale hierarchical image database," Proceeding of the IEEE conference on Computer Vision and Pattern Recognition, pp.248-255, 2009.
23 A. Krizhevsky, I. Sutskever, and G. Hinton, "Imagenet classification with deep convolutional neural networks," In Advances in neural information processing systems, pp.1097-1105, 2012.
24 Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel, "Handwritten digit recognition with a back-propagation network," In Advances in neural information processing systems, pp.396-404, 1990.
25 K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceeding of the IEEE conference on computer vision and pattern recognition, pp.770-778, 2016.
26 G. Huang, Z. Liu, K. Weinberger, and L. van der Maaten, "Densely connected convolutional networks," Proceeding of the IEEE conference on computer vision and pattern recognition, 2017.
27 P. Ekman, "An argument for basic emotions," Cognition & emotion, pp.169-200, 1992.
28 Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, "EEG-based emotion recognition in music listening," Proceeding of IEEE Transactions on Biomedical Engineering, 57(7), pp.1798-1806, 2010.
29 Y. Fan, X. Lu, D. Li, and Y. Liu, "Video-based emotion recognition using cnn-rnn and c3d hybrid networks. Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.445-450, 2016, doi:10.1145/2993148.2997632.   DOI
30 A. Asthana, S. Zafeiriou, S. Cheng, and M. Pantic, "Incremental face alignment in the wild," Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1859-1866, 2014.
31 H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, "Joint fine-tuning in deep neural networks for facial expression recognition," Proceeding of the IEEE International Conference on Computer Vision, pp.2983-2991, 2015.
32 J. Yan, W. Zheng, Z. Cui, C. Tang, T. Zhang, Y. Zong, and N. Sun, "Multi-clue fusion for emotion recognition in the wild," Proceeding of the 18th ACM International Conference on Multimodal Interaction, pp.458-463, 2016.
33 F. Eyben, M. Wöllmer, B. Schuller, "Opensmile: the munich versatile and fast open-source audio feature extractor," Proceeding of the 18th ACM international conference on Multimedia, pp.1459-1462, 2010.
34 B. McFee, C. Raffel, D. Liang, D. Ellis, M. McVicar, E. Battenberg, and O. Nieto, "librosa: Audio and music signal analysis in python," Proceeding of the 14th python in science conference, pp.18-25, 2015.
35 F. Chollet, Keras, http://keras.io, 2015.
36 K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," Proceeding of the IEEE international conference on computer vision, pp.1026-1034, 2015.
37 S. Ioffe, and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," In International Conference on Machine Learning, pp.448-456, 2015.
38 Zhang, Kaipeng et al. "Joint face detection and alignment using multitask cascaded convolutional networks," Proceeding of IEEE Signal Processing letters, pp.1499-1503, 2016.
39 Rasmus, Antti, et al."Semi-supervised learning with ladder networks," Advances in Neural Information Processing Systems, 2015.
40 Li, Xi, et al. "DeepSaliency: Multi-task deep neural network model for salient object detection," Proceeding of IEEE Transactions on Image Processing, pp.3919-3930, 2016.