Browse > Article
http://dx.doi.org/10.9717/kmms.2020.23.12.1464

Analysis of Understanding Using Deep Learning Facial Expression Recognition for Real Time Online Lectures  

Lee, Jaayeon (Division of Mechanical and Biomedical Engineering, Ewha Womans University)
Jeong, Sohyun (Division of Mechanical and Biomedical Engineering, Ewha Womans University)
Shin, You Won (Division of Mechanical and Biomedical Engineering, Ewha Womans University)
Lee, Eunhye (Division of Mechanical and Biomedical Engineering, Ewha Womans University)
Ha, Yubin (Division of Mechanical and Biomedical Engineering, Ewha Womans University)
Choi, Jang-Hwan (Division of Mechanical and Biomedical Engineering, Ewha Womans University)
Publication Information
Journal of Korea Multimedia Society / v.23, no.12, 2020 , pp. 1464-1475 More about this Journal
Abstract
Due to the spread of COVID-19, the online lecture has become more prevalent. However, it was found that a lot of students and professors are experiencing lack of communication. This study is therefore designed to improve interactive communication between professors and students in real-time online lectures. To do so, we explore deep learning approaches for automatic recognition of students' facial expressions and classification of their understanding into 3 classes (Understand / Neutral / Not Understand). We use 'BlazeFace' model for face detection and 'ResNet-GRU' model for facial expression recognition (FER). We name this entire process 'Degree of Understanding (DoU)' algorithm. DoU algorithm can analyze a multitude of students collectively and present the result in visualized statistics. To our knowledge, this study has great significance in that this is the first study offers the statistics of understanding in lectures using FER. As a result, the algorithm achieved rapid speed of 0.098sec/frame with high accuracy of 94.3% in CPU environment, demonstrating the potential to be applied to real-time online lectures. DoU Algorithm can be extended to various fields where facial expressions play important roles in communications such as interactions with hearing impaired people.
Keywords
Degree of Understanding; Real-time Analysis; Face Detection; Facial Expression Recognition; Deep Learning;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Y.H. Jung, Y.M. Song, and Y.H. Ko, "Inclined Face Detection Using JointBoost Algorithm," Journal of Korea Multimedia Society, Vol. 15, No. 5, pp. 606-614, 2012.   DOI
2 S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," IEEE Transactions on Affective Computing, 2020. (Accepted)
3 J. Park, S. Jeong, W. Lee, and K. Song, "Analyzing Facial Expression of a Learner in ELearning System," Proceedings of the Korea Contents Association Conference, pp. 160-163, 2006.
4 M.H. Immordino-Yang and A. Damasio, "We Feel, Therefore We Learn: The Relevance of Affective and Social Neuroscience to Education," Mind, Brain, and Education, Vol. 1, No. 1, pp. 3-10, 2007.   DOI
5 M. Sathik and S.G. Jonathan, "Effect of Facial Expressions on Student's Comprehension Recognition in Virtual Educational Environments," SpringerPlus, Vol. 2, No. 1, pp. 455-463, 2013.   DOI
6 K. Otwell, Facial Expression Recognition in Educational Learning Systems, 10319249, US, 2019.
7 O. El-Hammoumi, F. Benmarrakchi, N. Ouherrou, J. El-Kafi, and A. El-Hore, "Emotion Recognition in E-learning Systems," Proceeding of International Conference on Multimedia Computing and Systems, pp. 1-6, 2018.
8 A. Sarrafzadeh, S. Alexander, F. Dadgostar, C. Fan, and A. Bigdeli, "How Do You Know that I Don't Understand?" A Look at the Future of Intelligent Tutoring Systems," Computers in Human Behavior, Vol. 24, No. 4, pp. 1342-1363, 2008.   DOI
9 O.K. Akputu, K.P. Seng, Y. Lee, and L. Ang, "Emotion Recognition Using Multiple Kernel Learning toward E-Learning Applications," ACM Transactions on Multimedia Computing, Communications, and Applications, Vol. 14, No. 1, pp. 1-20, 2018.
10 P. Viola and M.J. Jones, "Robust Real-Time Face Detection," International Journal of Computer Vision, Vol. 57, No. 2, pp. 137-154, 2004.   DOI
11 C. Wu, "New Technology for Developing Facial Expression Recognition in E-Learning," Proceeding of Portland International Conference on Management of Engineering and Technology, pp. 1719-1722, 2016.
12 U. Ayvaz, H. Guruler, and M.O. Devrim, "Use of Facial Emotion Recognition in E-Learning Systems," Information Technologies and Learning Tools, Vol. 60, No. 4, pp. 95-104, 2017.   DOI
13 C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic, "300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge," Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 397-403, 2013.
14 L. Chen, C. Zhou, and L. Shen, "Facial Expression Recognition based on SVM in E-Learning," Ieri Procedia, Vol. 2, pp. 781-787, 2012.   DOI
15 J. Sun, X. Pei, and S. Zhou, "Facial Emotion Recognition in Modern Distant Education System Using SVM," Proceeding of International Conference on Machine Learning and Cybernetics, pp. 3545-3548, 2008.
16 S.P. Deshmukh, M.S. Patwardhan, and A.R. Mahajan, "Feedback Based Real Time Facial and Head Gesture Recognition for E-Learning System," Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, pp. 360-363, 2018.
17 M. Loh, Y. Wong, and C. Wong, "Facial Expression Recognition for E-Learning Systems Using Gabor Wavelet & Neural Network," Proceeding of IEEE International Conference on Advanced Learning Technologies, pp. 523-525, 2006.
18 G. Tonguc and B.O. Ozkara, "Automatic Recognition of Student Emotions from Facial Expressions during a Lecture," Computers & Education, Vol. 148, pp. 103797, 2020.   DOI
19 M.A.A. Dewan, F. Lin, D. Wen, M. Murshed, and Z. Uddin, "A Deep Learning Approach to Detecting Engagement of Online Learners," Proceeding of IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation, pp. 1895-1902, 2018.
20 G.E. Hinton, S. Osindero, and Y. Teh, "A Fast Learning Algorithm for Deep Belief Nets," Neural Computation, Vol. 18, No. 7, pp. 1527-1554, 2006.   DOI
21 I.M. Revina and W.S. Emmanuel, "A Survey on Human Face Expression Recognition Techniques," Journal of King Saud University - Computer and Information Sciences, Vol. 1, No. 5, pp. 1-9, 2018.
22 F. Ghaffar, "Facial Emotions Recognition Using Convolutional Neural Net," ArXiv Preprint ArXiv:2001.01456, 2020.
23 M.A.A. Dewan, M. Murshed, and F. Lin, "Engagement Detection in Online Learning: A Review," Smart Learning Environments, Vol. 6, No. 1, pp. 1, 2019.   DOI
24 P. Tarnowski, M. Kolodziej, A. Majkowski, and R.J. Rak, "Emotion Recognition Using Facial Expressions," Proceeding of International Conference on Computational Science, pp. 1175-1184, 2017.
25 L. Krithika and L.P. GG, "Student Emotion Recognition System (SERS) for E-Learning Improvement based on Learner Concentration Metric," Procedia Computer Science, Vol. 85, pp. 767-776, 2016.   DOI
26 L. Linnenbrink-Garcia and R. Pekrun, "Students' Emotions and Academic Engagement: Introduction to the Special Issue," Contemporary Educational Psychology, Vol. 36, No. 1, pp. 1-3, 2011.   DOI
27 A. Dhall, A. Kaur, R. Goecke, and T. Gedeon, "Emotiw 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction," Proceedings of the ACM International Conference on Multimodal Interaction, pp. 653-656, 2018.
28 I.J. Goodfellow, D. Erhan, P.L. Carrier, A. Courville, M. Mirza, B. Hamner, et al., "Challenges in Representation Learning: A Report on Three Machine Learning Contests," Proceeding of International Conference on Neural Information Processing, pp. 117-124, 2013.
29 M.J. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, and J. Budynek, "The Japanese Female Facial Expression (JAFFE) Database," Proceedings of International Conference on Automatic Face and Gesture Recognition, pp. 14-16, 1998.
30 E. Goeleven, R. De Raedt, L. Leyman, and B. Verschuere, "The Karolinska Directed Emotional Faces: A Validation Study," Cognition and Emotion, Vol. 22, No. 6, pp. 1094-1118, 2008.   DOI
31 V. Bazarevsky, Y. Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann, "Blazeface: Sub-Millisecond Neural Face Detection on Mobile Gpus," ArXiv Preprint ArXiv:1907. 05047, 2019.
32 K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, "Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks," IEEE Signal Processing Letters, Vol. 23, No. 10, pp. 1499-1503, 2016.   DOI
33 A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, et al., "Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications," ArXiv Preprint ArXiv:1704.04861, 2017.
34 C. Szegedy, A. Toshev, and D. Erhan, "Deep Neural Networks for Object Detection," Advances in Neural Information Processing Systems, Vol. 26, pp. 2553-2561, 2013.
35 P. Viola and M. Jones, "Rapid Object Detection Using a Boosted Cascade of Simple Features," Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 511-518, 2001.
36 N. Dalal and B. Triggs, "Histograms of Oriented Gradients for Human Detection," Proceeding of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 886-893, 2005.
37 K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-scale Image Recognition," ArXiv Preprint ArXiv:1409.1556, 2014.
38 J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling," ArXiv Preprint ArXiv:1412.3555, 2014.
39 T.N. Sainath, O. Vinyals, A. Senior, and H. Sak, "Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks," Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4580-4584, 2015.
40 Emotion Classification, (2018). https://github.com/XiaoYee/emotion_classification/commits?author=XiaoYee (accessed April 12, 2020).
41 Z. Zhang, C. Wang, and Y. Wang, "Video-Based Face Recognition: State of the Art," Proceeding of Chinese Conference on Biometric Recognition, pp. 1-9, 2011.
42 K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
43 Y. Fan, X. Lu, D. Li, and Y. Liu, "Video-Based Emotion Recognition Using CNN-RNN and C3D Hybrid Networks," Proceedings of the ACM International Conference on Multimodal Interaction, pp. 445-450, 2016.
44 Y. Zhai, and D. He, "Video-Based Face Recognition Based on Deep Convolutional Neural Network," Proceedings of the International Conference on Image, Video and Signal Processing, pp. 23-27, 2019.
45 H. Wang, Y. Wang, and Y. Cao, "Video-based Face Recognition: A Survey," World Academy of Science, Engineering and Technology, Vol. 60, pp. 293-302, 2009.
46 C. Shan, "Face Recognition and Retrieval in Video," Video Search and Mining, Vol. 287, pp. 235-260, 2010.   DOI