Browse > Article
http://dx.doi.org/10.5370/JEET.2018.13.1.485

Facial Expression Classification Using Deep Convolutional Neural Network  

Choi, In-kyu (Dept. of Electronic Engineering, Kwangwoon Univerity)
Ahn, Ha-eun (Dept. of Electronic Engineering, Kwangwoon Univerity)
Yoo, Jisang (Dept. of Electronic Engineering, Kwangwoon Univerity)
Publication Information
Journal of Electrical Engineering and Technology / v.13, no.1, 2018 , pp. 485-492 More about this Journal
Abstract
In this paper, we propose facial expression recognition using CNN (Convolutional Neural Network), one of the deep learning technologies. The proposed structure has general classification performance for any environment or subject. For this purpose, we collect a variety of databases and organize the database into six expression classes such as 'expressionless', 'happy', 'sad', 'angry', 'surprised' and 'disgusted'. Pre-processing and data augmentation techniques are applied to improve training efficiency and classification performance. In the existing CNN structure, the optimal structure that best expresses the features of six facial expressions is found by adjusting the number of feature maps of the convolutional layer and the number of nodes of fully-connected layer. The experimental results show good classification performance compared to the state-of-the-arts in experiments of the cross validation and the cross database. Also, compared to other conventional models, it is confirmed that the proposed structure is superior in classification performance with less execution time.
Keywords
Convolutional neural network; Facial expression; Data augmentation; Database;
Citations & Related Records
연도 인용수 순위
  • Reference
1 K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
2 Y. Taigman, M. Yang, M. Ranzato and L. Wolf, "Deepface: Closing the gap to human-level performance in face verification," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
3 H. Jung, S. Lee, J. Yim, S. Park and J. Kim, "Joint fine-tuning in deep neural networks for facial expression recognition," Proceedings of the IEEE International Conference on Computer Vision, 2015.
4 AT Lopes, E de Aguiar and AF De Souza, "Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order," Pattern Recognition, vol. 61, pp.610- 628, 2017.   DOI
5 P. Burkert, F. Trier, M.Z. Afzal, A. Dengel and M. Liwichki, "Dexpression: Deep convolutional neural network for expression recognition," arXiv preprint arXiv:1509.05371, 2015.
6 A. Mollahosseini, D. Chan and M.H. Mahoor, "Going deeper in facial expression recognition using deep neural networks," Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, IEEE, 2016.
7 J. Van der Schalk, S. T. Hawk, A. H. Fischer, and B. J. Doosje, "Moving faces, looking places: validation of the Amsterdam Dynamic Facial Expression Set (ADFES)," Emotion, vol. 11, pp. 907-910, 2011.   DOI
8 D.S. Ma, J. Correll and B. Wittenbrink, "The Chicago face database: A free stimulus set of faces and norming data," Behavior research methods, vol. 47, no. 4, pp.1122-1135, 2015.   DOI
9 P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews. "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE, 2010.
10 H. O'Reilly, D. Pigat, S. Fridenson, S. Berggren, S. Tal, O. Golan, S. B"olte, S. Baron-Cohen and D. Lundqvist, "The EU-emotion stimulus set: a validation study," Behavior research methods, vol. 48, no. 2, pp. 567-576, 2016.   DOI
11 ESRC 3D Face Database. http://pics.stir.ac.uk/ESRC/
12 M. Minear and D.C. Park, "A lifespan database of adult facial stimuli," Behavior Research Methods, Instruments, &Computers, vol. 36, no. 4, pp. 630-633, 2004.   DOI
13 D. Lundqvist, A. Flykt, and A.Ohman, "The Karolinska Directed Emotional Faces(KDEF)," CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, 1998.
14 O. Langner, R. Dotsch, G. Bijlstra, D.H. Wigboldus, S.T. Hawk and A. van Knippenberg, "Presentation and validation of the Radboud Faces Database," Cognition and emotion, vol. 24, no. 8, pp.1377-1388, 2010.   DOI
15 R. Al-Rfou, G. Alain, A. Almahairi, C. Angermueller, D. Bahdanau, N. Ballas and Y. Bengio, "Theano: A Python framework for fast computation of mathematical expressions," arXiv preprint arXiv:1605.02688, 2016.
16 M. Olszanowski, G. Pochwatko, K. Kuklinski, M. Scibor-Rylski, P. Lewinski and RK. Ohme, "Warsaw set of emotional facial expression pictures: a validation study of facial display photographs," Frontiers in psychology, vol. 5, no. 1516, pp.1-8, 2015.
17 Learn facial expressions from an image. https://www.kaggle.com/c/challenges-inrepresentation-learning-facial-expressionrecognition-challenge/data
18 P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," Computer Vision and Pattern Recognition, 2001, CVPR 2001. Proceedings of the 2001, IEEE Computer Society Conference on, vol. 1, IEEE, 2001.
19 N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
20 A. Krizhevsky, I. Sutskever, and G.E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems, 2012.
21 K. Sikka, T. Wu, J. Susskind and M. Bartlett, "Exploring Bag of Words Architectures in the Facial Expression Domain," Computer Vision-ECCV 2012, Workshops and Demonstrations, Springer Berlin/Heidelberg, 2012.
22 K. Simonyan, and A. Zisserman. "Very deep convolutional networks for large-scale image recognition," in Proc. International Conference on Learning Representations, http://arxiv.org/abs/1409.1556, 2014.
23 J. Bekios-Calfa, JM. Buenaposada and L. Baumela, "Revisiting linear discriminant techniques in gender recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 4, pp.858-864, 2011.   DOI
24 M.J. Den Uyl and H Van Kuilenburg, "The FaceReader: Online facial expression recognition," Proceedings of measuring behavior, vol. 30, 2005.
25 M. Ilbeygi and H. Shah-Hosseini, "A novel fuzzy facial expression recognition system based on facial feature extraction from color face images," Engineering Applications of Artificial Intelligence, vol. 25, no. 1, pp. 130-146, 2012.   DOI
26 M. Z. Uddin, M. M. Hassan, A. Almogren, M. Zuair, G. Fortino and J. Torresen, "A facial expression recognition system using robust face features from depth videos and deep learning," Computers & Electrical Engineering, 2017.
27 P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus and Y. LeCun, "OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks," In Proc, ICLR, 2014.
28 B. Sun, L. Li, G. Zhou and J. He, "Facial expression recognition in the wild based on multimodal texture features," Journal of Electronic Imaging, vol. 25, no. 6, pp,061407-061407, 2016.   DOI
29 N. Mousavi, H. Siqueira, P. Barros, B. Fernandes and S. Wermter, "Understanding how deep neural networks learn face expressions," Neural Networks (IJCNN), 2016 International Joint Conference on, IEEE, 2016.
30 V. Mayya, R. M. Pai and M. M. Pai, "Automatic Facial Expression Recognition Using DCNN," Procedia Computer Science,k vol. 93, pp.453-461, 2016.   DOI
31 Z. Meng, P. Liu, J. Cai, S. Han and Y. Tong, "Identity-Aware Convolutional Neural Network for Facial Expression Recognition," Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on, IEEE, 2017.