Browse > Article
http://dx.doi.org/10.9717/kmms.2020.23.2.146

Deep Learning based Emotion Classification using Multi Modal Bio-signals  

Lee, JeeEun (Graduate Program of Biomedical Engineering, Yonsei University)
Yoo, Sun Kook (Dept. of Medical Engineering, Yonsei University College of Medicine)
Publication Information
Journal of Korea Multimedia Society / v.23, no.2, 2020 , pp. 146-154 More about this Journal
Abstract
Negative emotion causes stress and lack of attention concentration. The classification of negative emotion is important to recognize risk factors. To classify emotion status, various methods such as questionnaires and interview are used and it could be changed by personal thinking. To solve the problem, we acquire multi modal bio-signals such as electrocardiogram (ECG), skin temperature (ST), galvanic skin response (GSR) and extract features. The neural network (NN), the deep neural network (DNN), and the deep belief network (DBN) is designed using the multi modal bio-signals to analyze emotion status. As a result, the DBN based on features extracted from ECG, ST and GSR shows the highest accuracy (93.8%). It is 5.7% higher than compared to the NN and 1.4% higher than compared to the DNN. It shows 12.2% higher accuracy than using only single bio-signal (GSR). The multi modal bio-signal acquisition and the deep learning classifier play an important role to classify emotion.
Keywords
Bio-signal; Deep Learning; Emotion;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 G.Y. Liu and M. Hao, "Emotion Recognition of Physiological Signals based on Adaptive Hierarchical Genetic Algorithm," Proceeding of 2009 World Resources Institute World Congress on Computer Science and Information Engineering, pp. 670-674, 2009.
2 C. Maaoui and A. Pruski, "Emotion Recognition through Physiological Signals for Humanmachine Communication," Cutting Edge Robotics 2010, IntechOpen, London, 2010.
3 C.E. Izard, "Emotion Theory and Research: Highlights, Unanswered Questions, and Emerging Issues," Annual Review of Psychology, Vol. 60, pp. 1-25, 2009.   DOI
4 J. Zhang, M. Chen, S. Hu, Y. Cao, and R. Kozma, "PNN for EEG-based Emotion Recognition," Proceeding of 2016 IEEE International Conference on Systems, Man, and Cybernetics, pp. 002319-002323, 2016.
5 L. Mirmohamadsadeghi, A. Yazdani, and J.M. Vesin, "Using Cardio-respiratory Signals to Recognize Emotions Elicited by Watching Music Video Clips," Proceeding of 2016 IEEE 18th International Workshop on Multimedia Signal Processing, pp. 1-5, 2016.
6 B. Appelhans and L. Luecken, "Heart Rate Variability as an Index of Regulated Emotional Responding," Review of General Psychology, Vol. 10, No. 3, pp. 229-240, 2006.   DOI
7 U. Acharya, Rajendra, K.P. Joseph, N. Kannathal, L.C. Min, and J.S. Suri, Heart Rate Variability, Advances in Cardiac Signal Processing, Springer, Berlin, Heidelberg, pp. 121-165, 2007.
8 J.E. Lee and S.K. Yoo, "Correlation Analysis of Electrocardiogram Signal according to Sleep Stage," Journal of Korea Multimedia Society, Vol. 21, No. 12, pp. 1370-1378, 2018.   DOI
9 J. Zhai and A. Barreto, "Stress Detection in Computer Users based on Digital Signal Processing of Noninvasive Physiological Variables," Proceeding of 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1355-1358, 2006.
10 K.C. Berridge, Pleasure, Pain, Desire, and Dread: Hidden Core Processes of Emotion, Well-Being: The Foundations of Hedonic Psychology, Washington, DC, 1999.
11 M. Swangnetr and D.B. Kaber, "Emotional State Classification in Patient-robot Interaction using Wavelet Analysis and Statisticsbased Feature Selection," IEEE Transactions on Human-Machine Systems, Vol. 43, No. 1, pp. 63-75, 2013.   DOI
12 D.R. Bach and K.J. Friston, "Model-based Analysis of Skin Conductance Responses: Towards Causal Models in Psychophysiology," Psychophysiology, Vol. 50, No. 1, pp. 15-22, 2013.   DOI
13 D.M. Shin, D. Shin, and D.K. Shin, "Development of Emotion Recognition Interface using Complex EEG/ECG Bio- signal for Interactive Contents," Multimedia Tools and Applications, Vol. 76, No. 9, pp. 11449-11470, 2017.   DOI
14 S.E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, C. Gulcehre, R. Memisevic, et al., "Combining Modality Specific Deep Neural Networks for Emotion Recognition in Video," Proceedings of the 15th ACM on International Conference on Multimodal Interaction, 2013.
15 I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, Massachusetts, 2016.
16 H. Chen and A.F. Murray, "Continuous Restricted Boltzmann Machine with an Implementable Training Algorithm," IEEE Proceedings-Vision, Image and Signal Processing, Vol. 150, No. 3, pp. 153-158, 2003.   DOI
17 G.E. Hinton, S. Osindero, and Y.W. Teh, "A Fast Learning Algorithm for Deep Belief Nets," Neural Computation, Vol. 18, No. 7, pp. 1527-1554, 2006.   DOI