Journal of Internet Computing and Services (인터넷정보학회논문지)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지

Realtime Facial Expression Recognition from Video Sequences Using Optical Flow and Expression HMM

광류와 표정 HMM에 의한 동영상으로부터의 실시간 얼굴표정 인식

  • Published : 2009.08.30
Download PDF
⟨ Previous Next ⟩

Abstract

Vision-based Human computer interaction is an emerging field of science and industry to provide natural way to communicate with human and computer. In that sense, inferring the emotional state of the person based on the facial expression recognition is an important issue. In this paper, we present a novel approach to recognize facial expression from a sequence of input images using emotional specific HMM (Hidden Markov Model) and facial motion tracking based on optical flow. Conventionally, in the HMM which consists of basic emotional states, it is considered natural that transitions between emotions are imposed to pass through neutral state. However, in this work we propose an enhanced transition framework model which consists of transitions between each emotional state without passing through neutral state in addition to a traditional transition model. For the localization of facial features from video sequence we exploit template matching and optical flow. The facial feature displacements traced by the optical flow are used for input parameters to HMM for facial expression recognition. From the experiment, we can prove that the proposed framework can effectively recognize the facial expression in real time.

비전기반 인간컴퓨터 상호작용은 컴퓨터와 인간의 상호소통을 자연스럽게 제공하는 측면에서 과학과 산업분야에서 주목받는 연구 분야이다. 그러한 측면에서 얼굴표정인식에 의한 인간의 심리적 상태를 추론하는 기술은 중요한 이슈이다. 본 연구에서는 감성인식 HMM 모델과 광류에 기반한 얼굴 움직임 추적 방법을 이용하여 동영상으로부터 얼굴표정을 인식하는 새로운 방법을 제시하였다. 특히, 기존의 감성상태 변환을 설명하는 HMM 모델은 특정 표정상태 간의 전환 시 항상 중립 상태를 거치도록 설계되어 있다. 그러나 본 연구에서는 기존의 표정상태 전환 모델에 중간상태를 거치는 과정 없이 특정 표정 상태간의 변환이 가능한 확장된 HMM 모델을 제시한다. 동영상으로부터 얼굴의 특성정보를 추출하기 위하여 탬플릿 매칭과 광류방법을 적용하였다. 광류에 의해 추적된 얼굴의 표정특성 정보는 얼굴표정인식을 위한 HMM의 매개변수 정보로 사용된다. 실험을 통하여 제안된 얼굴표정인식 방법이 실시간 얼굴 표정인식에 효과적임을 입증하였다.

Keywords

References

  1. P. Ekman and W.V. Friesen: Facial Action Coding System (FACS), Consulting Psychologist Press, Inc., 1978.
  2. B. Fasel, J. Luettin, "Automatic Facial Expression Analysis", A Survey, Pattern Recognition, 36(1), pp. 259-275, 2003. https://doi.org/10.1016/S0031-3203(02)00052-3
  3. C.C Chien, Y.J Chang and Y.C Chen, "Facial Expression Analysis Under Various Head Poses" Proceedings of 3rd IEEE Pacific-Rim Conf. on Multimedia, pp. 16-18, 2002.
  4. I. Cohen, N. Sebe, A. Garg, L. S. Chen,T. S. Huang,"Facial expression recognition from video sequences: temporal and static modeling", Computer Vision and Image Understanding, Vol. 91, pp. 160-187, 2003. https://doi.org/10.1016/S1077-3142(03)00081-X
  5. J.C Chun, O.R Kwon, P.Park: A Robust 3D Face Pose Estimation and Facial Expression Control for Vision-Based Animation, LNCS Vol. 4351, pp. 700-708, 2007.
  6. J. Huang, V. Blanz, B. Heisele :Face Recognition with Support Vector Machines and 3D Head Models, International Workshop on Pattern Recognition with Support Vector Machines, Niagara Falls, Canada, pp. 334-341, 2002.
  7. P. Michel, R. El Kaliouby: Real Time Facial Expression Recognition in Video using Support Vector Machines, ICMI, pp. 258-264, 2003.
  8. K. Anderson and P. W. McOwan, "A Real-Time Automated System for The Recognition of Human Facial Expressions", IEEE Transactions on Systems, Man and Cybernetics, Part B vol. 36, no. 1, pp. 96-105, 2006. https://doi.org/10.1109/TSMCB.2005.854502
  9. M. Evgeny, P. Edmond C, "Tracking Facial Features with Occlusions", Journal of Zhejiang University SCIENCE A, vol. 7, No. 7, pp. 1282-1288, 2006. https://doi.org/10.1631/jzus.2006.A1282
  10. Pardas and Bonafonte, M. Pardas, A. Bonafonte, "Facial Animation Parameters Extraction and Expression Recognition Using Hidden Markov Models", Signal Processing: Image Communication 17, pp. 675-688. 2002. https://doi.org/10.1016/S0923-5965(02)00078-4
  11. Y. Zhu, L. C. De Silva, C. C. Ko, "Using Moment Invariants and HMM in Facial Expression Recognition", Pattern Recognition Letters Archive Volume 23, 83-91, 2002. https://doi.org/10.1016/S0167-8655(01)00108-8
  12. Y. Zhu, L. C. De Silva, C. C. Ko, "A Solution for Facial Expression Representation, and Recognition", Signal Processing : Image Communication 17, 657-673, 2002. https://doi.org/10.1016/S0923-5965(02)00076-0
  13. K.P. Min, J.C Chun, G.R Park, "A Nonparametric Skin Color Model for Face Detection from Color Images", LNCS Vol. 3320, pp. 115-119, 2004.
  14. M. Black, "Robust incremental optical flow," PhD thesis, Yale University, 1992.
  15. J.L. Barron, and N.A. Thacker, "Tutorial: Computing 2D and 3D Optical Flow", Tina Memo No. 2004-012, 2005.
  16. Philipp Michel, Rana El Kaliouby, "Real Time Facial Expression Recognition in Video using Support Vector Machines", ICMI 2003: 258-264, 2003.
  17. Qiang Miao, Hong-Zhong Huang, Xianfeng Fan, "A Comparison Study of Support Vector Machines and Hidden Markov Models in Machinery Condition Monitoring", Journal of Mechanical Science and Technology Vol. 21, No.4, pp. 607-615, 2007. https://doi.org/10.1007/BF03026965
  18. Ara V. Nefian, Monson H, "Hidden Markov Models for Face Recognition", ICASSP98, pp. 2721-2724, 1998.
  19. A.V. Nefian, M.H. Hayes, "Maximum Likelihood Training of The Embedded HMM for Face Detection and Recognition", Proc. of the IEEE International Conference on Image Processing, ICIP 2000, Vol. 1, pp. 10-13, 2000.