International Journal of Fuzzy Logic and Intelligent Systems

Korean Institute of Intelligent Systems (한국지능시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Emotion Recognition Method Based on Multimodal Sensor Fusion Algorithm

  • Moon, Byung-Hyun (School of Electrical and Electronics Engineering, Chung-Ang University) ;
  • Sim, Kwee-Bo (School of Electrical and Electronics Engineering, Chung-Ang University)
  • Published : 2008.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

Human being recognizes emotion fusing information of the other speech signal, expression, gesture and bio-signal. Computer needs technologies that being recognized as human do using combined information. In this paper, we recognized five emotions (normal, happiness, anger, surprise, sadness) through speech signal and facial image, and we propose to method that fusing into emotion for emotion recognition result is applying to multimodal method. Speech signal and facial image does emotion recognition using Principal Component Analysis (PCA) method. And multimodal is fusing into emotion result applying fuzzy membership function. With our experiments, our average emotion recognition rate was 63% by using speech signals, and was 53.4% by using facial images. That is, we know that speech signal offers a better emotion recognition rate than the facial image. We proposed decision fusion method using S-type membership function to heighten the emotion recognition rate. Result of emotion recognition through proposed method, average recognized rate is 70.4%. We could know that decision fusion method offers a better emotion recognition rate than the facial image or speech signal.

Keywords

References

  1. Chang-Hyun Park, Kwee-Bo Sim "The Pattern Recognition Methods for Emotion Recognition with Speech Signal", Journal of Control, Automation, and Systems Engineering, vol. 12, no. 3, pp. 284-288, Mar. 2006 https://doi.org/10.5302/J.ICROS.2006.12.3.284
  2. Weon-Goo Kim "Comparison of feature parameters for emotion recognition using speech signal", Journal of Institute of Electronics Engineers of Korea(IEEK), vol. 40, no. 5, pp. 371-377, 2003
  3. Young-Suk Shin "The Effect of Facial Expression Recognition Based on the Dimensions of Emotion using PCA Representation and Neural Networks," Lecture Notes in Computer Science(LNCS), Vol. 3656, pp. 1133-1140, 2005
  4. Su-Jeong Han, Keun-Chang Kwak, Hyoun-Joo Go, Sung-Suk Kim, Myung-Geun Chun "Facial Expression Recognition using ICA-Factorial Representation Method", Journal of Korea Fuzzy Logic and Intelligent Systems Society(KFIS), vol.13, no.3, pp. 371-376, 2003 https://doi.org/10.5391/JKIIS.2003.13.3.371
  5. Mingli Song. Jiajun Bu, Chun Chen and Nan Li, "Audio-Visual based Emotion Recognition," Proc. of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04), Vol. 2, pp. 1020-1125, 2004
  6. Hatice Guncs and Massimo Piccardi, "Bi-Modal Emotion Recognitionfrom Expressive Face and Boby Gessture" Journal of Network and Computer Applications, pp. 1-12, 2006
  7. J De Silval, Pei chi Ng., "Bimodal Emotion Recognition", Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, pp 332-335, 2000
  8. Hyeun-Joo Go, Dae-Jong Lee, Myung-Geun Chun "An Emotion Recognition Method using Facial Expression and Speech Signal", Journal of Korea Information Science Society(KISS), vol. 31, no. 6, pp. 799-807, 2004
  9. Kwee-Bo Sim, Chang-Hyun Park, "Analyzing the element of emotion recognition from speech", Journal of Korea Fuzzy Logic and Intelligent Systems Society(KFIS), vol. 11, no. 6, pp.510-515, 2001. 12
  10. Kwee-Bo Sim and Dong-Hun Lee, "Emotion Recognition and Expression System of Mobile Robot Based on 2D Facial Image", Journal of Control, Automation, and Systems Engineering, vol. 13, no. 4, pp. 371-376, Apr. 2007 https://doi.org/10.5302/J.ICROS.2007.13.4.371
  11. Ho-Duck Kim, Hyun-Chang Yang, Chang-Hyun Park, and Kwee-Bo Sim, "Emotion Recognition Method of Facial Image using PCA", Journal of Korea Fuzzy Logic and Intelligent Systems Society(KFIS), vol.16, no.6, pp. 772-776, Dec. 2006 https://doi.org/10.5391/JKIIS.2006.16.6.772
  12. Myung-Hun Kim, Chi-Geun Lee, In-Mi So, Sung-Tae Jung "Design and Implementation of a Bio-modal user Recognition System using Face and Audio," Journal of Korea Society of Computer and information(KSCI), vol. 10, no. 5, pp. 353-361, 2005