Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

The Audience Behavior-based Emotion Prediction Model for Personalized Service

고객 맞춤형 서비스를 위한 관객 행동 기반 감정예측모형

  • Ryoo, Eun Chung (Graduate School of Business Administration, Kyung Hee University) ;
  • Ahn, Hyunchul (School of Management Information Systems, Kookmin University) ;
  • Kim, Jae Kyeong (Graduate School of Business Administration, Kyung Hee University)
  • Received : 2013.05.23
  • Accepted : 2013.06.19
  • Published : 2013.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Nowadays, in today's information society, the importance of the knowledge service using the information to creative value is getting higher day by day. In addition, depending on the development of IT technology, it is ease to collect and use information. Also, many companies actively use customer information to marketing in a variety of industries. Into the 21st century, companies have been actively using the culture arts to manage corporate image and marketing closely linked to their commercial interests. But, it is difficult that companies attract or maintain consumer's interest through their technology. For that reason, it is trend to perform cultural activities for tool of differentiation over many firms. Many firms used the customer's experience to new marketing strategy in order to effectively respond to competitive market. Accordingly, it is emerging rapidly that the necessity of personalized service to provide a new experience for people based on the personal profile information that contains the characteristics of the individual. Like this, personalized service using customer's individual profile information such as language, symbols, behavior, and emotions is very important today. Through this, we will be able to judge interaction between people and content and to maximize customer's experience and satisfaction. There are various relative works provide customer-centered service. Specially, emotion recognition research is emerging recently. Existing researches experienced emotion recognition using mostly bio-signal. Most of researches are voice and face studies that have great emotional changes. However, there are several difficulties to predict people's emotion caused by limitation of equipment and service environments. So, in this paper, we develop emotion prediction model based on vision-based interface to overcome existing limitations. Emotion recognition research based on people's gesture and posture has been processed by several researchers. This paper developed a model that recognizes people's emotional states through body gesture and posture using difference image method. And we found optimization validation model for four kinds of emotions' prediction. A proposed model purposed to automatically determine and predict 4 human emotions (Sadness, Surprise, Joy, and Disgust). To build up the model, event booth was installed in the KOCCA's lobby and we provided some proper stimulative movie to collect their body gesture and posture as the change of emotions. And then, we extracted body movements using difference image method. And we revised people data to build proposed model through neural network. The proposed model for emotion prediction used 3 type time-frame sets (20 frames, 30 frames, and 40 frames). And then, we adopted the model which has best performance compared with other models.' Before build three kinds of models, the entire 97 data set were divided into three data sets of learning, test, and validation set. The proposed model for emotion prediction was constructed using artificial neural network. In this paper, we used the back-propagation algorithm as a learning method, and set learning rate to 10%, momentum rate to 10%. The sigmoid function was used as the transform function. And we designed a three-layer perceptron neural network with one hidden layer and four output nodes. Based on the test data set, the learning for this research model was stopped when it reaches 50000 after reaching the minimum error in order to explore the point of learning. We finally processed each model's accuracy and found best model to predict each emotions. The result showed prediction accuracy 100% from sadness, and 96% from joy prediction in 20 frames set model. And 88% from surprise, and 98% from disgust in 30 frames set model. The findings of our research are expected to be useful to provide effective algorithm for personalized service in various industries such as advertisement, exhibition, performance, etc.

정보기술의 비약적 발전에 힘입어, 오늘날 기업들은 지금까지 축적한 고객 데이터를 기반으로 맞춤형 서비스를 제공하는 것에 많은 관심을 가지고 있다. 고객에게 소구하는 맞춤형 서비스를 효과적으로 제공하기 위해서는 우선 그 고객이 처한 상태나 상황을 정확하게 인지하는 것이 중요하다. 특히, 고객에게 서비스가 전달되는 이른바 진실의 순간에 해당 고객의 감정 상태를 정확히 인지할 수 있다면, 기업은 더 양질의 맞춤형 서비스를 제공할 수 있을 것이다. 이와 관련하여 사람의 얼굴과 행동을 이용하여 사람의 감정을 판단하고 개인화 서비스를 제공하기 위한 연구가 활발하게 이루어지고 있다. 얼굴 표정을 통해 사람의 감정을 판단하는 연구는 좀 더 미세하고 확실한 변화를 통해 정확하게 감정을 판단할 수 있지만, 장비와 환경의 제약으로 실제 환경에서 다수의 관객을 대상으로 사용하기에는 다소 어려움이 있다. 이에 본 연구에서는 Plutchik의 감정 분류 체계를 기반으로 사람들의 행동을 통해 감정을 추론해내는 모형을 개발하는 것을 목표로 한다. 본 연구는 콘텐츠에 의해 유발된 사람들의 감정적인 변화를 사람들의 행동 변화를 통해 판단하고 예측하는 모형을 개발하고, 4가지 감정 별 행동 특징을 추출하여 각 감정에 따라 최적화된 예측 모형을 구축하는 것을 목표로 한다. 모형 구축을 위해 사람들에게 적절한 감정 자극영상을 제공하고 그 신체 반응을 수집하였으며, 사람들의 신체 영역을 나누었다. 특히, 모션캡쳐 분야에서 널리 쓰이는 차영상 기법을 적용하여 사람들의 제스쳐를 추출 및 보정하였다. 이후 전처리 과정을 통해 데이터의 타임프레임 셋을 20, 30, 40 프레임의 3가지로 설정하고, 데이터를 학습용, 테스트용, 검증용으로 구분하여 인공신경망 모형을 통해 학습시키고 성과를 평가하였다. 다수의 일반인들을 대상으로 수집된 데이터를 이용하여 제안 모형을 구축하고 평가한 결과, 프레임셋에 따라 예측 성과가 변화함을 알 수 있었다. 감정 별 최적 예측 성과를 보이는 프레임을 확인할 수 있었는데, 이는 감정에 따라 감정의 표출 시간이 다르기 때문인 것으로 판단된다. 이는 행동에 기반한 제안된 감정예측모형이 감정에 따라 효과적으로 감정을 예측할 수 있으며, 실제 서비스 환경에서 사용할 수 있는 효과적인 알고리즘이 될 수 있을 것으로 기대할 수 있다.

Keywords

References

  1. Ambady, N. and R. Rosenthal, "Thin slices of expressive behavior as predictors of interpersonal consequences : A meta-analysis," Psychological Bulletin, Vol.111, No.2(1992), 256-274. https://doi.org/10.1037/0033-2909.111.2.256
  2. Bae, J. H., S. H. So, and L. K. Choi, "Designing Mobile Framework for Intelligent Personalized Marketing Service in Interactive Exhibition Space," Journal of Intelligence and Information Systems, Vol.18, No.1(2012), 59-69.
  3. Burgoon, J. K., M. L. Jensen, T. O. Meservy, J. Kruse, and J. F. Nunamaker, "Augmenting human identification of emotional states in video," Proceedings of the international conference on intelligent data analysis, 2005.
  4. Cho, K. E., C. S. Lee, and K. H. Um, "Design of a Character' System to Express Mixed-Emotion by Gesture and Posture," Proceeding of the conference on Korea Multimedia Society, (2009), 191-194.
  5. Coulson, M., "Attributing emotion to static body postures : recognition accuracy, confusions, and viewpoint dependence," Journal of Nonverbal Behavior, Vol.28, No.2(1992), 117-139.
  6. Davis, J. W., Appearance-Based Motion Recognition of Human Actions, Technical Report, MIT, 1996.
  7. Dempster, A. P., N. M. Laird, and D. B. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," Journal of the Royal Statistical Society, Series B, Vol.39, No.1 (1977), 1-38.
  8. Ekman, P. and W. V. Friesen, Unmasking the face : a guide to recognizing emotions from facial clues, Englewood Cliffs, NJ : Prentice-Hall, 1975.
  9. Hong, D. P. and W. T. Woo, "AR-based Tangible User Interface for I2-NEXT," Proceedings of the conference on HCI 2005, (2005), 1628-1633.
  10. Jang, N. S., Data Mining, Dae Chung Media, 1999.
  11. Joo, J. T., I. H. Jang, H. C. Yang, and K. B. Sim, "Emotion Recognition and Expression Method using Bi-Modal Sensor Fusion Algorithm," Journal of Control, Automation and Systems Engineering, Vol.13, No.8(2007), 754-759. https://doi.org/10.5302/J.ICROS.2007.13.8.754
  12. Jung, M. K. and J. K. Kim, "The Intelligent Determination Model of Audience Emotion for Implementing Personalized Exhibition," Journal of Intelligence and Information Systems, Vol.18, No.1(2012), 39-57.
  13. Kim, D. S., Neural Network Theory and Application, Hitech information, 1993.
  14. Kim, J. H., Y. S. Yun, T. Y. Kim, and C. S. Lim, "Human primitive motion recognition based on the hidden markov models," Journal of Korea Multimedia Society, Vol.12, No.4(2009), 521- 529.
  15. Kim, S. J., E. C. Ryoo, M. K. Jung, J. K. Kim, and H. C. Ahn, "Application of support vector regression for improving the performance of the emotion prediction model," Journal of Intelligence and Information Systems, Vol.18, No.3(2012), 185-202.
  16. Korea Mecenat Association, (2013), Available at http://www.mecenat.or.kr/(Accessed 14 June, 2013).
  17. Laban, R., Modern Educational Dance, Macdonald and Evans, 1968.
  18. Lee, I. H. and C. J. Park, "The status of the motion capture technology and Applications," Journal of Korea Multimedia Society, Vol.3, No.1(1999), 38-48.
  19. Lee, S. H., "The Study for Equilibrium of Emotions by Oriental Medicine Music Therapy," Journal of Oriental Medical Classics, Vol.24, No.6(2011), 39-47.
  20. McCulloch, W. S. and W. Pitts, "A logical calculus of the ideas immanent in nervous activity," The Bulletin of Mathematical Biophysics, Vol.5, No.4(1943), 15-33. https://doi.org/10.1007/BF02478115
  21. Plutchik, R., Emotion : A psycho-evolutionary synthesis, New York : Harper and Row, 1980.
  22. Plutchik, R., Emotions and life : Perspectives from psychology, biology, and evolution, Washington, DC : American Psychological Association, 2002.
  23. Ryoo, E. C., S. B. Park, and J. K. Kim, "Audience reaction judgement model for interactive performance," Proceeding of the Conference on Intelligence and Information Systems, (2012), 121-126.
  24. Watanabe, T. and M. Yachida, "Real time recognition of gesture and gesture degree information using multi input image sequences," Proceeding of the 14th International Conference on Pattern Recognition, Vol.2(1998), 1855-1858.
  25. You, H. J., "The re-experience intention affects experiential traits and emotions of consumer on the performances cultural service of companies," The e-Business Studies, Vol.11, No.3 (2010), 519-538. https://doi.org/10.15719/geba.11.3.201009.519
  26. Zhu, Y., Y. Huang, G. Xu, H. Ren, and Z. Wen, "Vision-based Interpretation of Hand Gestures by Modeling Appearance Changes in Image Sequences," Proceedings of MVA'98 IAPR Workshop on Machine Vision Applications, (1998), 573-576.