Journal of the Korea Convergence Society (한국융합학회논문지)

Korea Convergence Society (한국융합학회)
권호 표지
DOI QR코드

DOI QR Code

Convergence Implementing Emotion Prediction Neural Network Based on Heart Rate Variability (HRV)

심박변이도를 이용한 인공신경망 기반 감정예측 모형에 관한 융복합 연구

  • Received : 2017.09.24
  • Accepted : 2018.05.20
  • Published : 2018.05.28
Download PDF
⟨ Previous Next ⟩

Abstract

The purpose of this study is to develop more accurate and robust emotion prediction neural network (EPNN) model by combining heart rate variability (HRV) and neural network. For the sake of improving the prediction performance more reliably, the proposed EPNN model is based on various types of activation functions like hyperbolic tangent, linear, and Gaussian functions, all of which are embedded in hidden nodes to improve its performance. In order to verify the validity of the proposed EPNN model, a number of HRV metrics were calculated from 20 valid and qualified participants whose emotions were induced by using money game. To add more rigor to the experiment, the participants' valence and arousal were checked and used as output node of the EPNN. The experiment results reveal that the F-Measure for Valence and Arousal is 80% and 95%, respectively, proving that the EPNN yields very robust and well-balanced performance. The EPNN performance was compared with competing models like neural network, logistic regression, support vector machine, and random forest. The EPNN was more accurate and reliable than those of the competing models. The results of this study can be effectively applied to many types of wearable computing devices when ubiquitous digital health environment becomes feasible and permeating into our everyday lives.

본 연구는 심박변이도(HRV)와 인공신경망을 이용하여 강건하고 정확한 융복합 감정예측 모형인 EPNN (Emotion Prediction Neural Network)을 개발하는 것을 주요 연구목적으로 한다. 본 연구에서 제안하는 EPNN은 기존 유사연구와는 달리 은닉노드의 활성함수로서 하이퍼볼릭 탄젠트, 선형, 가우시안 함수를 융복합적으로 이용하여 모형의 정확도를 향상시킨다. 본 연구에서는 EPNN의 타당성을 검증하기 위하여 20명의 실험자를 대상으로 머니게임으로 감정을 유도한 후에 해당 실험자의 심박변이도 측정값을 입력자료로 사용하였다. 아울러 그들의 Valence와 Arousal을 EPNN의 출력값으로 사용하였다. 실험결과 Valence에 대한 F-Measure는 80%이고, Arousal의 경우 95%로 나타났다. 한편 EPNN의 타당성을 측정하기 위하여 기존 감정예측 연구에 사용된 경쟁모형인 인공신경망, 로지스틱 회귀분석, 서포트 벡터 머신, 랜덤 포레스트 모형과 성과를 비교하였다. 그 결과 본 연구에서 제안하는 EPNN이 더 우수한 감정예측 결과를 보였다. 본 연구의 결과는 향후 유비쿼터스 디지털 헬스 환경에서 사용되는 다양한 웨어러블 기기에 적용되어 사용자들의 일상생활 속에서 시시각각 변하는 감정을 정확히 예측하고 적절하게 관리하는데 적용될 수 있을 것이다.

Keywords

References

  1. N. Kandasamy, S. N. Garfinkel, L. Page, B. Hardy, H. D. Critchley, M. urnell & J. M. Coates. (2016). Interoceptive ability predicts survival on a London trading floor, Scientific Reports, 6, 32986. DOI : 10.1038/srep32986
  2. R. W. Picard, E. Vyzas & J. Healey. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE transactions on pattern analysis and machine intelligence, 23(10), 1175-1191. DOI : 10.1109/34.954607
  3. I. K. Kwon, S. Y. Lee. (2012). Personalized Service Based on Context Awareness through User Emotional Perception in Mobile Environment. Journal of Digital Convergence, 10(2), 287-292. https://doi.org/10.14400/JDPM.2012.10.2.287
  4. K. J. Kim. (2018). A conversation training program that combines reason and sensitivity - Using the P-A-C technique of Transactional analysis -. Journal of Convergence for Information Technology, 8(2), 149-155. https://doi.org/10.22156/CS4SMB.2018.8.2.149
  5. F. Agrafioti, D. Hatzinakos & A. K. Anderson. (2012). ECG pattern analysis for emotion detection. IEEE Transactions on Affective Computing, 3(1), 102-115. DOI : 10.1109/T-AFFC.2011.28
  6. J. Kim. (2016). Emotion Prediction of Paragraph using Big Data Analysis. Journal of Digital Convergence, 14(11), 267-273. DOI : 10.14400/JDC.2016.14.11.267
  7. A. Hariharan & M. T. P. Adam. (2015). Blended emotion detection for decision support. IEEE Transactions on Human-Machine Systems, 45(4,), 510-517. DOI : 10.1109/THMS.2015.2418231
  8. G. Valenza, L. Citi, A. Lanata, E. P. Scilingo & R. Barbieri. (2014). Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics. Scientific reports, 4, 4998. DOI : 10.1038/srep04998
  9. S. S. Park & K. C. Lee. (2017). Emotion prediction neural network to understand how emotion is predicted by using heart rate variability measurements. Journal of The Korea Society of Computer and Information, 22(7), 75-82. DOI: 10.9708/jksci.2017.22.07.075
  10. S. S. Park & K. C. Lee. (2018). Analysis of the Relative Importance of HRV Metrics to Predict Emotion by Using Valence-Arousal driven Neural Network. under review in Journal of Korean Institute of Information Technology.
  11. M. D. Asaduzzaman, M. Shahjahan & K. Murase. (2009). Faster training using fusion of activation functions for feed forward neural networks. International journal of neural systems, 19(6), 437-448. DOI : 10.1142/S0129065709002130
  12. M. Dorofki, A. H. Elshafie, O. Jaafar, O. A. Karim & S. Mastura. (2012). Comparison of artificial neural network transfer functions abilities to simulate extreme runoff data. International Proceedings of Chemical, Biological and Environmental Engineering, 33, 39-44.
  13. M. J. Song, M. Y. Kim, I. S. Sim & W. S. Kim. (2010). Evaluation of Horticultural Therapy on the Emotional Improvement of Depressed Patients by Using Heart Rate Variability. Korean Journal of Horticultural Science & Technology, 28(6), 1066-1071.
  14. M. S. Kim, Y. N. Kim & Y. C. Cho. (2015). Electrocardiographic characteristics of significant factors of detected atrial fibrillation using WEMS. Journal of the Korea Industrial Information Systems Research, 20(6), 37-46. DOI : 10.9723/jksiis.2015.20.6.037
  15. N. Ravaja. (2014). Contributions of psychophysiology to media research: Review and recommendations. Media Psychology, 6(2), 193-235. DOI : 10.1207/s1532785xmep0602_4
  16. T. Kageyama, N. Nishikido, T. Kobayashi, Y. Kurokawa, T. Kaneko & M. Kabuto. (1998). Self-Reported Sleep Quality, Job Stress, and Daytime Autonomic Activities Assessed in Terms of Short-Term Heart Rate Variability among Male White-Collar Workers. Industrial Health, 36(3), 263-272. DOI : 10.2486/indhealth.36.263
  17. T. G. M. Vrijkotte, L. J. P. van Doornen & E. J. C. de Geus. (2000). Effects of Work Stress on Ambulatory Blood Pressure, Heart Rate, and Heart Rate Variability. Hypertension, 35, 880-886. DOI : 10.1161/01.HYP.35.4.880
  18. C. M. Bishop. (1995). Neural networks for pattern recognition. Oxford university press.
  19. M. Sansone, R. Fusco, A. Pepino & C. Sansone. (2013). Electrocardiogram pattern recognition and analysis based on artificial neural networks and support vector machines: A review. Journal of Healthcare Engineering, 4(4), 465-504. DOI : 10.1260/2040-2295.4.4.465
  20. A. Haag, S. Goronzy, P. Schaich & J. Williams. (2004). Emotion recognition using bio-sensors: First steps towards an automatic system. Affective dialogue systems, Springer, 36-48.
  21. M. Swangnetr & D. B. Kaber. (2013). Emotional state classification in patient-robot interaction using wavelet analysis and statistics-based feature selection. IEEE Transactions on Human-Machine Systems, 43(1), 63-75. DOI : 10.1109/TSMCA.2012.2210408
  22. F. A. Russo, N. N. Vempala & G. M. Sandstrom. (2013). Predicting musically induced emotions from physiological inputs: Linear and neural network models. Frontiers in Psychology, 4-468. DOI : 10.3389/fpsyg.2013.00468
  23. D. Kukolja, S. Popovi, M. Horvat, B. Kova & K. osi. (2014). Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. International Journal of Human-Computer Studies, 72(10), 717-727. DOI : 10.1016/j.ijhcs.2014.05.006
  24. R. R. Singh, S. Conjeti & R. Banerjee. (2013). A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals. Biomedical Signal Processing and Control, 8(6), 740-754. DOI : 10.1016/j.bspc.2013.06.014
  25. M. A. Nicolaou, H. Gunes & M. Pantic. (2011). Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Transactions on Affective Computing, 2(2), 92-105. DOI : 10.1109/T-AFFC.2011.9
  26. F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J. P. Thiran, T. Ebrahimi, D. Lalanne & B. Schuller. (2015). Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data. Pattern Recognition Letters, 66, 22-30. DOI : 10.1016/j.patrec.2014.11.007
  27. H. Hwang & S. Park. (2011). The Impact of User's Psychological Experience on Online Game Addiction: Perceived Reality and Sense of Presence. Journal of communication science, 11(1), 471-505.
  28. J. Posner, J. Russell & B. Peterson, (2005). The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, 17(3), 715-734. DOI : 10.1017/S0954579405050340
  29. M. M. Bradley & P. J. Lang. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1), 49-59. DOI : 10.1016/0005-7916(94)90063-9
  30. S. M. Ahn, M. C. Whang, D K. Kim, J H. Kim & S. I. Park. (2012). Real-time emotion recognition technology using individualization processemotional technology. Korean Journal of the Science of Emotion and Sensibility, 15(1), 133-140.
  31. U. R. Acharya, K. P. Joseph, N. Kannathal, C. M. Lim & J. S. Suri. (2006). Heart rate variability: a review. Medical and biological engineering and computing, 44(12), 1031-1051. DOI : 10.1007/s11517-006-0119-0
  32. M. P. Tarvainen, J. P. Niskanen, J. A. Lipponen, P. O. Ranta-Aho & P. A. Karjalainen. (2014). Kubios HRV - heart rate variability analysis software. Computer methods and programs in biomedicine, 113(1), 210-220. DOI : 10.1016/j.cmpb.2013.07.024
  33. S. N. Yu & S. F. Chen. (2015). Emotion state identification based on heart rate variability and genetic algorithm. Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, 538-541. DOI : 10.1109/EMBC.2015.7318418
  34. R. Hecht-Nielsen. (1989). Theory of the Back Propagation Neural Network. In Proceeding of the International Joint Conference on Neural Networks(IJCNN), New York, 593-605. DOI : 10.1016/B978-0-12-741252-8.50010-8
  35. M. Sokolova & G. Lapalme. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427-437. DOI : 10.1016/j.ipm.2009.03.002