Journal of Korea Multimedia Society (한국멀티미디어학회논문지)

Korea Multimedia Society (한국멀티미디어학회)
권호 표지
DOI QR코드

DOI QR Code

2D Emotion Classification using Short-Time Fourier Transform of Pupil Size Variation Signals and Convolutional Neural Network

동공크기 변화신호의 STFT와 CNN을 이용한 2차원 감성분류

  • Lee, Hee-Jae (Dept. of Digital Media., Graduate School, The Catholic University of Korea) ;
  • Lee, David (Dept. of Digital Media., Graduate School, The Catholic University of Korea) ;
  • Lee, Sang-Goog (Dept. of Media Technology & Media Contents, The Catholic University of Korea)
  • Received : 2017.06.15
  • Accepted : 2017.09.18
  • Published : 2017.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Pupil size variation can not be controlled intentionally by the user and includes various features such as the blinking frequency and the duration of a blink, so it is suitable for understanding the user's emotional state. In addition, an ocular feature based emotion classification method should be studied for virtual and augmented reality, which is expected to be applied to various fields. In this paper, we propose a novel emotion classification based on CNN with pupil size variation signals which include not only various ocular feature information but also time information. As a result, compared to previous studies using the same database, the proposed method showed improved results of 5.99% and 12.98% respectively from arousal and valence emotion classification.

Keywords

References

  1. C. Won, "Recognition of Facial Emotion Using Multi-scale LBP," Journal of Korea Multimedia Society, Vol. 17. No. 12. pp. 1383-1392, 2014. https://doi.org/10.9717/kmms.2014.17.12.1383
  2. S. Chen, Y.L. Tian, Q. Liu, and D.N. Metaxas, "Recognizing Expressions from Face and Body Gesture by Temporal Normalized Motion and Appearance Features," Image and Vision Computing, Vol. 31, pp. 175-185, 2013. https://doi.org/10.1016/j.imavis.2012.06.014
  3. G. Trigeorgis, F. Ringeval, R. Bruckner, E. Marchi, M. Nicolaou, and B. Schulleret, et al., "Adieu Features? End-To-End Speech Emotion Recognition Using A Deep Convolutional Recurrent Network," Proceeding of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5200-5204, 2016.
  4. Y. Gao, H.J. Lee, and R.M. Mehmood, "Deep Learninig of EEG Signals for Emotion Recognition," Proceeding of IEEE International Conference on Multimedia & Expo Workshops, pp. 1-5, 2015.
  5. C. Aracena, S. Basterrech, V. Snasel, and J. Velasquez, "Neural Networks for Emotion Recognition Based on Eye Tracking Data," Proceeding of IEEE International Conference on Systems, Man, and Cybernetics, pp. 2632-2637, 2015.
  6. W. Zheng, B. Dong, and B. Lu, "Multimodal Emotion Recognition Using EEG and Eye Tracking Data," Proceeding of Annual International Conference on IEEE Engineering in Medicine and Biology Society, pp. 5040-5043, 2014.
  7. P. Ekman, R.W. Levenson, and W.V. Friesen, "Autonomic Nervous System Activity Distinguishes between Emotions," Science, Vol. 221, pp. 1208-1210, 1983. https://doi.org/10.1126/science.6612338
  8. E.H. Hess and J.M. Polt, "Pupil Size as Related to Interest Value of Visual Stimuli," Science, pp. 349-35, 1960.
  9. M.M. Bradley, L. Miccoli, M.A. Escrig, and P.J. Lang, "The Pupil as A Measure of Emotional Arousal and Autonomic Activation," Psychophysiology, 45, pp. 602-607, 2008. https://doi.org/10.1111/j.1469-8986.2008.00654.x
  10. P. Ren, A. Barreto, Y. Gao, and M. Adjouadi, "Affective Assessment by Digital Processing of The Pupil Diameter," IEEE Transactions on Affective Computing, Vol. 4, pp. 2-14, 2013. https://doi.org/10.1109/T-AFFC.2012.25
  11. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
  12. S. Tripathi, S. Acharya, R.D. Sharma, S. Mittal, and S. Bhattacharya, "Using Deep and Convolutional Neural Networks for Accurate Emotion Classification on DEAP Dataset," Innovative Applications of Artificial Intelligence, pp. 4746-4752, 2017.
  13. A. TeixeiraLopes, E.d. Aguia, A.. DeSouza, and T.O. Santos, "Facial Expression Recognition with Convolutional Neural Networks: Coping with Few Data and The Training Sample Order," Pattern Recogntion, Vol. 61, pp. 610-628, 2017. https://doi.org/10.1016/j.patcog.2016.07.026
  14. W. Lim, D. Jang, and T. Lee, "Speech Emotion Recognition using Convolutional and Recurrent Neural Networks," Proceedings of the Signal and Information Processing Association Annual Summit and Conference, pp. 1-4, December 2016.
  15. Wei Liu, W.L. Zheng, and B.L. Lu, "Multimodal Emotion Recognition Using Multimodal Deep Learning," Proceeding of IEEE Winter Conference on Applications of Computer Vision, 2016.
  16. M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, "A Multimodal Database for Affect Recognition and Implicit Tagging," IEEE Transactions on Affective Computing, pp. 42- 55, 2012.
  17. M. Soleymani, M. Pantic, and T. Pun, "Multimodal Emotion Recognition in Response to Videos," IEEE Transactions on Affective Computing, Vol. 3, pp. 211-223, 2012. https://doi.org/10.1109/T-AFFC.2011.37
  18. J. Oh and J. Jeong, “Potential Significance of Eyeblinks as a Behavior Marker of Neuropsychiatric Disorders,” Korean Society of Biological Psychiatry, Vol. 19, No. 1, pp. 9-20, 2012.
  19. A. Schaefr, F. Nils, X. Sanchez, and P. Philippot, “Assessing the Effectiveness of a Large Database of Emotion-Eliciting Films: A New Tool for Emotion Researchers,” Cognition & Emotion, Vol. 24, No. 7, pp. 1153-1172, 2010. https://doi.org/10.1080/02699930903274322
  20. V.F. Pamplona, M.M. Oliveira, and G.V.G. Baranoski, “Photorealistic Models for Pupil Light Reflex and Iridal Pattern Deformation,” ACM Transactions on Graphics, Vol. 28, No. 4, pp. 1-12, 2009.