Browse > Article
http://dx.doi.org/10.9708/jksci.2017.22.05.075

Emotion prediction neural network to understand how emotion is predicted by using heart rate variability measurements  

Park, Sung Soo (SKKU Business School, Sungkyunkwan University)
Lee, Kun Chang (SKKU Business School/SAIHST (Samsung Advanced Institute for Health Sciences & Technology)/Healthcare Informatics Research Center, Sungkyunkwan University)
Publication Information
Journal of the Korea Society of Computer and Information / v.22, no.7, 2017 , pp. 75-82 More about this Journal
Abstract
Correct prediction of emotion is essential for developing advanced health devices. For this purpose, neural network has been successfully used. However, interpretation of how a certain emotion is predicted through the emotion prediction neural network is very tough. When interpreting mechanism about how emotion is predicted by using the emotion prediction neural network can be developed, such mechanism can be effectively embedded into highly advanced health-care devices. In this sense, this study proposes a novel approach to interpreting how the emotion prediction neural network yields emotion. Our proposed mechanism is based on HRV (heart rate variability) measurements, which is based on calculating physiological data out of ECG (electrocardiogram) measurements. Experiment dataset with 23 qualified participants were used to obtain the seven HRV measurement such as Mean RR, SDNN, RMSSD, VLF, LF, HF, LF/HF. Then emotion prediction neural network was modelled by using the HRV dataset. By applying the proposed mechanism, a set of explicit mathematical functions could be derived, which are clearly and explicitly interpretable. The proposed mechanism was compared with conventional neural network to show validity.
Keywords
emotion prediction; HRV(heart rate variability); neural network; inference process interpretation; healthcare devices;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 D. E. Bloom, E. T. Cafiero, E. Jane-Llopis, S. Abrahams-Gessel, L. R. Bloom, S. Fathima, A. B. Feigl, T. Gaziano, M. Mowafi, A. Pandya, K. Prettner, L. Rosenberg, B. Seligman, A. Z. Stein, and C. Weinstein, "The Global Economic Burden of Non-Communicable Diseases," Geneva, Switzerland: World Economic Forum, July 2011.
2 J. Mervis, "Agencies rally to tackle big data," Science, Vol. 336, No.6077, pp. 22-22, 2012.   DOI
3 K. R. Kim, J. G. Yun, and S. G. Yoon, "Analysis on the Status of Korean Major Firms Entering into HT Fusion New Business," Korea Health Industry Development Institute, 2014. Download https://www.khidi.or.kr /fileDownload?titleId=128633&fileId=1&fileDownType =C¶mMenuId=MENU01101
4 C. M. Tucker, T. M. Arthur, J. Roncoroni, W. Wall, and J. Sanchez, "Patient-centered, culturally sensitive health care," American Journal of Lifestyle Medicine, Vol. 9, No. 1, pp. 63-77, 2015.   DOI
5 J. A. Sacristan, "Patient-centered medicine and patient-oriented research: improving health outcomes for individual patients," BMC Medical Informatics and Decision Making, Vol. 13, No. 1, pp. 6-13, 2013.   DOI
6 M. Phelan, L. Stradins, and S. Morrison, (2001). "Towards a global definition of patient centred care," British Medical Journal, Vol. 322, pp. 444-445, 2001.   DOI
7 M. Ozkaynak, P. F. Brennan, D. A. Hanauer, S. Johnson, J. Aarts, K. Zheng, and S. N. Haque, "Patient-centered care requires a patientoriented workflow model," Journal of the American Medical Informatics Association, Vol. 20, No. e1, pp. e14-e16, 2013.   DOI
8 S. M. Yoon and M. H. Lee, "Current Status and Prospects of Life Appliance based Healthcare System," The Journal of The Korean Institute of Communication Sciences, Vol. 31, No. 12, pp. 31-37, Nov. 2014.
9 Y. L. Zheng, X. R. Ding, C. C. Y. Poon, B. P. L. Lo, H. Zhang, X. L. Zhou, ... and Y. T. Zhang, "Unobtrusive sensing and wearable devices for health informatics," IEEE Transactions on Biomedical Engineering, Vol. 61, No. 5, pp. 1538-1554, 2014.   DOI
10 A. J. Wearden, N. Tarrier, C. Barrowclough, T. R. Zastowny, and A. A. Rahill, "A review of expressed emotion research in health care," Clinical psychology review, Vol. 20, No. 5, pp. 633-666, 2000.   DOI
11 B. Freshman, and L. Rubino, "Emotional intelligence: a core competency for health care administrators," The health care manager, Vol. 20, No. 4, pp. 1-9, 2002.   DOI
12 G. Valenza, L. Citi, A. Lanata, E. P. Scilingo, and R. Barbieri, "Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics," Scientific reports, Vol. 4, No. 4998, pp. 1-13, 2014.
13 Y. F. Birks, and I. S. Watt, "Emotional intelligence and patient-centred care," Journal of the Royal Society of Medicine, Vol. 100, No. 8, pp. 368-374, 2007.   DOI
14 S. Mann, "A health-care model of emotional labour: an evaluation of the literature and development of a model," Journal of health organization and management, Vol. 19, No. 4/5, pp. 304-317, 2005.   DOI
15 C. D. Katsis, N. S. Katertsidis, and D. I. Fotiadis, "An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders," Biomedical Signal Processing and Control, Vol. 6, No. 3, pp. 261-268, 2011.   DOI
16 P. A. Kragel, and K. S. LaBar, "Multivariate pattern classification reveals autonomic and experiential representations of discrete emotions," Emotion, Vol. 13, No. 4, pp. 681-690, 2013.   DOI
17 J. Selvaraj, M. Murugappan, K. Wan, and S. Yaacob, "Classification of emotional states from electrocardiogram signals: a non-linear approach based on hurst," Biomedical engineering online, Vol. 12, No. 44, 2013.
18 M. Swangnetr, and D. B. Kaber, "Emotional state classification in patient-robot interaction using wavelet analysis and statistics-based feature selection," IEEE Transactions on Human-Machine Systems, Vol. 43, No. 1, pp. 63-75, 2013.   DOI
19 D. Kukolja, S. Popovic, M. Horvat, B. Kovac, and K. Cosic, "Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications," International Journal of Human-Computer Studies, Vol. 72, No. 10, pp. 717-727, 2014.   DOI
20 C. M. Bishop, "Neural Networks for Pattern Recognition" Clarendon Press, Oxford, 1995.
21 G. D. Garson, "Interpreting neural-network connection weights," AI Expert, Vol. 6, pp. 47-51, 1991.
22 F. Stahl and I. Jordanov, "An overview of use of neural networks for data mining tasks," WIREs: Data Mining and Knowledge Discovery, pp. 193-208, 2012.
23 J. A. Anderson, "An Introduction to Neural Networks" MIT press, 1995.
24 B. D. Ripley, "Pattern Recognition and Neural Networks" Cambridge University Press, 1996.
25 L. K. Milne, "Feature selection with neural networks with contribution measures," In Proceedings of the Australian Conference on Artificial Intelligence AI'95, Canberra, 1995.
26 J. M. Zurada, A. Malinowski, and I. Cloete, "Sensitivity analysis for minimization of input data dimension for feedforward neural network," Proceedings of IEEE International Symposium on Circuits and Systems, Vol. 6, pp. 447-450, May 1994.
27 D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning Internal Representations by Error Propagation," Parallel Distributed Processing, Vol. 1, pp. 318-362, 1986.
28 S. Haykin, "Neural Network" Prentice Hall, 1994.
29 P. De Wilde, "Neural network models: theory and projects" Springer Science & Business Media, 2013.
30 A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion recognition using bio-sensors: First steps towards an automatic system," In Tutorial and research workshop on affective dialogue systems, Springer Berlin Heidelberg, pp. 36-48, June 2004.
31 L. Ma, and K. Khorasani, "Facial expression recognition using constructive feedforward neural networks," IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 34, No. 3, pp. 1588-1595, 2004.   DOI
32 M. A. Nicolaou, H. Gunes, and M. Pantic, "Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space," IEEE Transactions on Affective Computing, Vol. 2, No. 2, pp. 92-105, 2011.   DOI
33 M. M. Monwar, and S. Rezaei, "Pain recognition using artificial neural network. In Signal Processing and Information Technology," 2006 IEEE International Symposium on Signal Processing and Information Technology, pp. 28-33, August 2006.
34 Y. S. Seol, D. J. Kim, and H. W. Kim, "Emotion recognition from text using knowledge-based ANN," 23rd International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), pp. 1569-1572, July 2008.
35 C. N. Moridis, and A. A. Economides, "Prediction of student's mood during an online test using formula-based and neural network-based method," Computers & Education, Vol. 53, No. 3, pp. 644-652, 2009.   DOI
36 F. A. Russo, N. N. Vempala, and G. M. Sandstrom, "Predicting musically induced emotions from physiological inputs: Linear and neural network models," Frontiers in Psychology, Vol. 4, No. 468, 2013.
37 M. P. Tarvainen, J. P. Niskanen, J. A. Lipponen, P. O. Ranta-Aho, and P. A. Karjalainen, "Kubios HRV-heart rate variability analysis software," Computer methods and programs in biomedicine, Vol. 113, No. 1, pp. 210-220, 2014.   DOI
38 R. Hecht-Nielsen, "Theory of the Back Propagation Neural Network," Proceeding of the International Joint Conference on Neural Networks, pp. 593-605, 1989.
39 I. H. Witten, and E. Frank, "Data Mining: Practical machine learning tools and techniques" Morgan Kaufmann, 2005.
40 S. W. Byun, S. P. Lee, "Emotion Recognition Using Tone and Tempo Based on Voice for IoT," The transactions of The Korean Institute of Electrical Engineers, Vol. 65, No. 1, pp. 116-121, 2016.   DOI
41 C. D. Katsis, N. S. Katertsidis, and D. I. Fotiadis, "An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders," Biomedical Signal Processing and Control, Vol. 6, No. 3, pp. 261-268, 2011.   DOI
42 Y. J. Kang, "Smart Affect Jewelry based on Multi-modal," Journal of the Korea Institute of Information and Communication Engineering, Vol. 20, No. 7, pp. 1317-1324, 2016.   DOI
43 P. S. Kim, "Technology and development trends related to human-friendly emotional robots," The Journal of The Korean Institute of Communication Sciences, 33(8), 19-27, 2016.