Journal of Electrical Engineering and Technology

The Korean Institute of Electrical Engineers (대한전기학회)
권호 표지
DOI QR코드

DOI QR Code

Facial Expression Classification Using Deep Convolutional Neural Network

  • Choi, In-kyu (Dept. of Electronic Engineering, Kwangwoon Univerity) ;
  • Ahn, Ha-eun (Dept. of Electronic Engineering, Kwangwoon Univerity) ;
  • Yoo, Jisang (Dept. of Electronic Engineering, Kwangwoon Univerity)
  • Received : 2017.05.01
  • Accepted : 2017.10.24
  • Published : 2018.01.01
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose facial expression recognition using CNN (Convolutional Neural Network), one of the deep learning technologies. The proposed structure has general classification performance for any environment or subject. For this purpose, we collect a variety of databases and organize the database into six expression classes such as 'expressionless', 'happy', 'sad', 'angry', 'surprised' and 'disgusted'. Pre-processing and data augmentation techniques are applied to improve training efficiency and classification performance. In the existing CNN structure, the optimal structure that best expresses the features of six facial expressions is found by adjusting the number of feature maps of the convolutional layer and the number of nodes of fully-connected layer. The experimental results show good classification performance compared to the state-of-the-arts in experiments of the cross validation and the cross database. Also, compared to other conventional models, it is confirmed that the proposed structure is superior in classification performance with less execution time.

Keywords

E1EEFQ_2018_v13n1_485_f0001.png 이미지

Fig. 1. Examples of images classified as incorrect faces inFER 2013 database

E1EEFQ_2018_v13n1_485_f0002.png 이미지

Fig. 2. The result of converting cut-out face region imageinto gray image

E1EEFQ_2018_v13n1_485_f0003.png 이미지

Fig. 3. The result of applying data augmentation technique

E1EEFQ_2018_v13n1_485_f0004.png 이미지

Fig. 4. The proposed CNN architecture

E1EEFQ_2018_v13n1_485_f0005.png 이미지

Fig. 5. Computational relationship between consecutiveconvolutional layers

Table 1. Accuracy comparison of data augmentation techniques

E1EEFQ_2018_v13n1_485_t0001.png 이미지

Table 2. Average Top 1 Accuracy(%) on cross validation

E1EEFQ_2018_v13n1_485_t0002.png 이미지

Table 3. Average confusion matrix on cross validation (%)

E1EEFQ_2018_v13n1_485_t0003.png 이미지

Table 4. Average Accuracy(%) on cross database

E1EEFQ_2018_v13n1_485_t0004.png 이미지

Table 5. Training and testing time for each model (batch : 128)

E1EEFQ_2018_v13n1_485_t0005.png 이미지

Table 6. Accuracy(%) for each model

E1EEFQ_2018_v13n1_485_t0006.png 이미지

Table 7. Confusion matrix on ADFES (%)

E1EEFQ_2018_v13n1_485_t0007.png 이미지

Table 8. Confusion matrix on CFD (%)

E1EEFQ_2018_v13n1_485_t0008.png 이미지

Table 9. Confusion matrix on CK+ (%)

E1EEFQ_2018_v13n1_485_t0009.png 이미지

Table 10. Confusion matrix on EU-Emotion Stimulus Set (%)

E1EEFQ_2018_v13n1_485_t0010.png 이미지

Table 11. Confusion matrix on ESRC (%)

E1EEFQ_2018_v13n1_485_t0011.png 이미지

Table 12. Confusion matrix on FACE DATABASE (%)

E1EEFQ_2018_v13n1_485_t0012.png 이미지

Table 13. Confusion matrix on KDEF (%)

E1EEFQ_2018_v13n1_485_t0013.png 이미지

Table 14. Confusion matrix on RafD (%)

E1EEFQ_2018_v13n1_485_t0014.png 이미지

Table 15. Confusion matrix on Web Search (%)

E1EEFQ_2018_v13n1_485_t0015.png 이미지

Table 16. Confusion matrix on WSEFEP (%)

E1EEFQ_2018_v13n1_485_t0016.png 이미지

References

  1. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  2. Y. Taigman, M. Yang, M. Ranzato and L. Wolf, "Deepface: Closing the gap to human-level performance in face verification," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
  3. H. Jung, S. Lee, J. Yim, S. Park and J. Kim, "Joint fine-tuning in deep neural networks for facial expression recognition," Proceedings of the IEEE International Conference on Computer Vision, 2015.
  4. AT Lopes, E de Aguiar and AF De Souza, "Facial expression recognition with Convolutional Neural Networks: Coping with few data and the training sample order," Pattern Recognition, vol. 61, pp.610- 628, 2017. https://doi.org/10.1016/j.patcog.2016.07.026
  5. P. Burkert, F. Trier, M.Z. Afzal, A. Dengel and M. Liwichki, "Dexpression: Deep convolutional neural network for expression recognition," arXiv preprint arXiv:1509.05371, 2015.
  6. A. Mollahosseini, D. Chan and M.H. Mahoor, "Going deeper in facial expression recognition using deep neural networks," Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, IEEE, 2016.
  7. J. Van der Schalk, S. T. Hawk, A. H. Fischer, and B. J. Doosje, "Moving faces, looking places: validation of the Amsterdam Dynamic Facial Expression Set (ADFES)," Emotion, vol. 11, pp. 907-910, 2011. https://doi.org/10.1037/a0023853
  8. D.S. Ma, J. Correll and B. Wittenbrink, "The Chicago face database: A free stimulus set of faces and norming data," Behavior research methods, vol. 47, no. 4, pp.1122-1135, 2015. https://doi.org/10.3758/s13428-014-0532-5
  9. P. Lucey, J. Cohn, T. Kanade, J. Saragih, Z. Ambadar and I. Matthews. "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression," Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE, 2010.
  10. H. O'Reilly, D. Pigat, S. Fridenson, S. Berggren, S. Tal, O. Golan, S. B"olte, S. Baron-Cohen and D. Lundqvist, "The EU-emotion stimulus set: a validation study," Behavior research methods, vol. 48, no. 2, pp. 567-576, 2016. https://doi.org/10.3758/s13428-015-0601-4
  11. ESRC 3D Face Database. http://pics.stir.ac.uk/ESRC/
  12. M. Minear and D.C. Park, "A lifespan database of adult facial stimuli," Behavior Research Methods, Instruments, &Computers, vol. 36, no. 4, pp. 630-633, 2004. https://doi.org/10.3758/BF03206543
  13. D. Lundqvist, A. Flykt, and A.Ohman, "The Karolinska Directed Emotional Faces(KDEF)," CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, 1998.
  14. O. Langner, R. Dotsch, G. Bijlstra, D.H. Wigboldus, S.T. Hawk and A. van Knippenberg, "Presentation and validation of the Radboud Faces Database," Cognition and emotion, vol. 24, no. 8, pp.1377-1388, 2010. https://doi.org/10.1080/02699930903485076
  15. M. Olszanowski, G. Pochwatko, K. Kuklinski, M. Scibor-Rylski, P. Lewinski and RK. Ohme, "Warsaw set of emotional facial expression pictures: a validation study of facial display photographs," Frontiers in psychology, vol. 5, no. 1516, pp.1-8, 2015.
  16. Learn facial expressions from an image. https://www.kaggle.com/c/challenges-inrepresentation-learning-facial-expressionrecognition-challenge/data
  17. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," Computer Vision and Pattern Recognition, 2001, CVPR 2001. Proceedings of the 2001, IEEE Computer Society Conference on, vol. 1, IEEE, 2001.
  18. N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, 2014.
  19. A. Krizhevsky, I. Sutskever, and G.E. Hinton, "Imagenet classification with deep convolutional neural networks," Advances in neural information processing systems, 2012.
  20. R. Al-Rfou, G. Alain, A. Almahairi, C. Angermueller, D. Bahdanau, N. Ballas and Y. Bengio, "Theano: A Python framework for fast computation of mathematical expressions," arXiv preprint arXiv:1605.02688, 2016.
  21. K. Sikka, T. Wu, J. Susskind and M. Bartlett, "Exploring Bag of Words Architectures in the Facial Expression Domain," Computer Vision-ECCV 2012, Workshops and Demonstrations, Springer Berlin/Heidelberg, 2012.
  22. J. Bekios-Calfa, JM. Buenaposada and L. Baumela, "Revisiting linear discriminant techniques in gender recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 4, pp.858-864, 2011. https://doi.org/10.1109/TPAMI.2010.208
  23. M.J. Den Uyl and H Van Kuilenburg, "The FaceReader: Online facial expression recognition," Proceedings of measuring behavior, vol. 30, 2005.
  24. M. Ilbeygi and H. Shah-Hosseini, "A novel fuzzy facial expression recognition system based on facial feature extraction from color face images," Engineering Applications of Artificial Intelligence, vol. 25, no. 1, pp. 130-146, 2012. https://doi.org/10.1016/j.engappai.2011.07.004
  25. K. Simonyan, and A. Zisserman. "Very deep convolutional networks for large-scale image recognition," in Proc. International Conference on Learning Representations, http://arxiv.org/abs/1409.1556, 2014.
  26. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus and Y. LeCun, "OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks," In Proc, ICLR, 2014.
  27. B. Sun, L. Li, G. Zhou and J. He, "Facial expression recognition in the wild based on multimodal texture features," Journal of Electronic Imaging, vol. 25, no. 6, pp,061407-061407, 2016. https://doi.org/10.1117/1.JEI.25.6.061407
  28. N. Mousavi, H. Siqueira, P. Barros, B. Fernandes and S. Wermter, "Understanding how deep neural networks learn face expressions," Neural Networks (IJCNN), 2016 International Joint Conference on, IEEE, 2016.
  29. M. Z. Uddin, M. M. Hassan, A. Almogren, M. Zuair, G. Fortino and J. Torresen, "A facial expression recognition system using robust face features from depth videos and deep learning," Computers & Electrical Engineering, 2017.
  30. V. Mayya, R. M. Pai and M. M. Pai, "Automatic Facial Expression Recognition Using DCNN," Procedia Computer Science,k vol. 93, pp.453-461, 2016. https://doi.org/10.1016/j.procs.2016.07.233
  31. Z. Meng, P. Liu, J. Cai, S. Han and Y. Tong, "Identity-Aware Convolutional Neural Network for Facial Expression Recognition," Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on, IEEE, 2017.