Information Systems Review (경영정보학연구)

The Korea Society of Management Information Systems (한국경영정보학회)
권호 표지
DOI QR코드

DOI QR Code

Meaning of Rating Beyond Recommendation: Explorative Study on the Meaning and Usage of Content Evaluation Based on the User Experience Stages of Personalized Recommender Service

평점의 의미: 개인화 추천 서비스에서 사용자 경험단계에 따른 콘텐츠 평가의 의미와 활용에 대한 탐색적 연구

  • Hyundong Kim (Graduate School of Management of Technology, Yonsei University) ;
  • Hae-jeong Hwang (Graduate School of Information, Yonsei University) ;
  • Kieun Park (Graduate School of Information, Yonsei University) ;
  • Mingu Kang (Graduate School of Information, Yonsei University) ;
  • Jeonghun Kim (Graduate School of Management of Technology, Yonsei University) ;
  • Inseong Lee (Quryon Korea) ;
  • Jinwoo Kim (HCI Lab, Yonsei University)
  • Received : 2016.05.08
  • Accepted : 2016.09.13
  • Published : 2016.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Research on personalized recommender service that uses big data has gained considerable attention given the increasing volume of contents being created. This development indicates the need for service providers to collect personal information and content rating data to personalize content recommendations. Previous studies on this topic proposed algorithms to offer improved recommendations using minimal rating data or service designs and increase the number of ratings. However, limited studies have been conducted on the factors that motivate the ratings input of users, as well as the factors that influence their continuous usage of recommender service. The present study explored the factors that motivate users to enter ratings by conducting in-depth interviews with users who use recommender services. The meanings of these ratings were also explored. Results show that the meaning and usage range of ratings differed based on the stage of a user's with utilization of the service. When users input an initial rating, they treat such a rating as a database to save the impression of a past experience. Such a rating is then used as a tool to reflect the current feeling and thoughts of a user. In the end, users were not only interested in their own rating system, but they also actively sought out the meaning of the rating systems of others and utilized them. Users also expressed mistrust in the recommendations of the service because they were aware of the limitation of the algorithms. This study identified a number of practical implications regarding recommender services.

방대한 콘텐츠가 생산되고 소비되면서 빅데이터를 활용한 개인 추천 서비스가 최근 주목 받고 있다. 개인 추천 서비스를 위하여 개인 정보나 콘텐츠 평가 정보를 수집하는 것은 서비스 제공자 입장에서 중요해지고 있다. 기존 연구들은 적은 평점 정보로 더 나은 추천을 제공할 수 있는 알고리즘을 제안하거나, 평점의 양을 늘리기 위한 서비스 디자인을 제시하였다. 그러나 추천서비스 사용자가 어떤 동기로 평점을 입력하고, 서비스를 지속적으로 사용하는지에 대한 연구는 거의 없었다. 본 논문에서는 추천 서비스를 사용하고 있는 사용자들을 심층 인터뷰하여 평점 입력의 동기와 평점의 의미에 대하여 탐구하였다. 그 결과, 서비스를 경험 하면서 평점의 의미와 활용 정도가 달라짐을 알 수 있었다. 초기 평점을 입력할 때에는 과거 경험에 대한 데이터베이스를 구축하는 의미로 활용하였고, 초기 평점 단계를 지나면 현재의 느낌과 생각을 반영하는 도구로 활용하였다. 이 과정에서 자신의 평점 체계를 정교하게 다듬으며 자신만의 의미를 부여하는 모습을 보였다. 마지막 단계에서는 자신의 평점 체계뿐만 아니라 다른 사람의 평점 체계나 평점의 의미를 읽어내고 적극적으로 활용하는 모습을 보인다. 서비스에서 제공하는 알고리즘의 한계를 파악하고 있기 때문에 서비스의 추천을 불신하기도 하였다. 연구 결과를 바탕으로 추천 서비스에 대한 실무적 시사점을 도출하였다.

Keywords

Acknowledgement

이 논문은 한국연구재단 이공학 개인기초 연구지원 사업 NRF-2016R1D1A1B02015987의 지원을 받아 진행되었습니다.

References

  1. 김상화, 오병화, 김문종, 양지훈, "협력적 필터링과 콘텐츠 정보를 결합한 영화 추천 알고리즘", 소프트웨어 및 응용, 제39권, 제4호, 2012, pp. 261-268.
  2. 김진화, 남기찬, 이상종, "Support Vector Machine 기법을 이용한 고객의 구매의도 예측", Information Systems Review, 제10권, 제2호, 2008, pp. 137-158.
  3. 박태훈, 이충재, "개인화된 영화 서비스 왓챠를 개발하는 프로그램스(Frograms)", 정보과학회지, 제32권, 제7호, 2014, pp. 60-62. https://doi.org/10.1002/nadc.201490021
  4. 소요환, "애니메이션 속성이 관람 후 평가에 미치는 영향", 만화애니메이션연구, 통권13호, 2008, pp. 115-131.
  5. 손지은, 김성범, 김현중, 조성준, "추천 시스템 기법 연구동향 분석", 대한산업공학회지, 제41권, 제2호, 2015, pp. 185-208. https://doi.org/10.7232/JKIIE.2015.41.2.185
  6. 엄하정, 장진규, 김민지, 김태동, 김현영, 김진우, "자기표현과 사회적 기대감이 지속적 사용에 미치는 영향", HCI 2015, 2014, pp. 371-377.
  7. 이창현, 이승룡, 정태충, 윤석환, "스마트폰에서 사용자 감성정보를 이용한 개인화된 협업필터링 기반 애플리케이션 추천 시스템", 한국정보과학회 2012 한국컴퓨터종합학술대회 논문집, 제39권, 제1호(A), 2012, pp. 224-226.
  8. 임 일, "빅데이터에서 개인화 자료부터 확보 소규모로 시작해 최적 솔루션 찾아라", Dong-a Business Review, 제146권, 제1호, 2014, pp. 99-103.
  9. 조승연, 최지은, 이규현, 김희웅, "고객 온라인 구매후기를 활용한 추천시스템 개발 및 적용", 한국경영정보학회 학술대회논문집, 2015, pp. 157-164.
  10. Amatriain, X., J. M. Pujol, N. Tintarev, and N. Oliver, "Rate it again: Increasing recommendation accuracy by user re-rating", Paper Presented at the Proceedings of the Third ACM Conference on Recommender Systems, 2009.
  11. Beer, D. and R. Burrows, "Popular culture, digital archives and the new social life of data", Theory, Culture & Society, Vol.30, No.4, 2013, pp. 47-71. https://doi.org/10.1177/0263276413476542
  12. Belk, R., Possessions and Self, Wiley Online Library, 1988.
  13. Berntsen, D., "Involuntary autobiographical memories", Applied Cognitive Psychology, Vol.10, No.5, 1996, pp. 435-454. https://doi.org/10.1002/(SICI)1099-0720(199610)10:5<435::AID-ACP408>3.0.CO;2-L
  14. Bhattacherjee, A., "Science and scientific research", in Bhattacherjee, A. (eds.), Social Science Research: Principles, Methods, and Practices, Textbooks Collection. Book 3, 2012. pp. 1-8.
  15. Birks, M. and J. Mills, "Five data generation and collection", in Birks, M. and J. Mills (eds.), Grounded Theory: A Practical Guide, Sage Publications, 2011. pp. 91-118.
  16. Bollen, D., M. Graus, and M. C. Willemsen, "Remembering the stars?: Effect of time on preference retrieval from memory", Paper Presented at the Proceedings of the Sixth ACM Conference on Recommender Systems, 2012.
  17. Conway, M. A. and C. W. Pleydell-Pearce, "The construction of autobiographical memories in the self-memory system", Psychological Review, Vol.107, No.2, 2000, p. 261.
  18. Corbin, J. and A. Strauss, "Context", in Corbin, J. and A. Strauss (eds.), Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, Sage Publications, 2014. pp. 219-223.
  19. Cosley, D., S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl, "Is seeing believing?: How recommender system interfaces affect users' opinions", Paperpresented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2003.
  20. Cushing, A. L., "Self extension and the desire to preserve digital possessions", Proceedings of the American Society for Information Science and Technology, Vol.48, No.1, 2011, pp. 1-3. https://doi.org/10.1002/meet.2011.14504801304
  21. Denzin, N. K., "Triangulation 2.0.", Journal of Mixed Methods Research 6.2, 2012, pp. 80-88. https://doi.org/10.1177/1558689812437186
  22. Featherstone, M., "Archiving cultures", The British Journal of Sociology, Vol.51, No.1, 2000, pp. 161-184. https://doi.org/10.1080/000713100358480
  23. Frijda, N. H., "The laws of emotion", American Psychologist, Vol.43, No.5, 1988, p. 349.
  24. Harper, F. Maxwell, et al., "An economic model of user rating in an online recommender system", International Conference on User Modeling, Springer Berlin Heidelberg, 2005, pp. 307-316.
  25. Harper, Y. C., F. Maxwell, J. Konstan, and S. X. Li, "Social comparisons and contributions to online communities: A field experiment on movielens", The American Economic Review, Vol.100, No.4, 2010, pp. 1358-1398. https://doi.org/10.1257/aer.100.4.1358
  26. Herlocker, J. L., J. A. Konstan, L. G. Terveen, and J. T. Riedl, "Evaluating collaborative filtering recommender systems", ACM Transactions on Information Systems (TOIS), Vol.22, No.1, 2004, pp. 5-53. https://doi.org/10.1145/963770.963772
  27. IDC, "New IDC Worldwide Big Data Technology and Services Forecast Shows Market Expected to Grow to $41.5 Billion in 2018", 2014, Available at http://www.idc.com/getdoc.jsp?containerId=prUS25132014.
  28. Im, I. and A. Hars, "Does a one-size recommendation system fit all? The effectiveness of collaborative filtering based recommendation systems across different domains and search modes", ACM Transactions on Information Systems (TOIS), Vol.26, No.1, 2007, p. 4, doi:10.1145/1292591.1292595.
  29. Kirk, D. S. and A. Sellen, "On human remains: Values and practice in the home archiving of cherished objects", ACM Transactions on Computer-Human Interaction (TOCHI), Vol.17, No.3, 2010, p. 10.
  30. Konstan, J. A. and J. Riedl, "Recommender systems: From algorithms to user experience", User Modeling and User-Adapted Interaction, Vol.22, No.1-2, 2012, pp. 101-123. https://doi.org/10.1007/s11257-011-9112-x
  31. Koren, Y., R. Bell, and C. Volinsky, "Matrix factorization techniques for recommender systems", Computer, Vol.42, No.8, pp. 30-37.
  32. Koufaris, M. and W. Hampton-Sosa, "The development of initial trust in an online company by new customers", Information & Management, Vol.41, No.3, 2004, pp. 377-397. https://doi.org/10.1016/j.im.2003.08.004
  33. Laurel, B., "Interface agents: Metaphors with character", in B. Friedman (ed.), Human Values and the Design of Computer Technology, Cambridge University Press, Stanford, CA, 1997, pp. 207-219.
  34. Linder, R., C. Snodgrass, and A. Kerne, "Everyday ideation: All of my ideas are on Pinterest", Paper Presented at the Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, 2014.
  35. Martin, F. J., J. Donaldson, A. Ashenfelter, M. Torrens, and R. Hangartner, "The big promise of recommender systems", AI Magazine, Vol.32, No.3, 2011, pp. 19-27. https://doi.org/10.1609/aimag.v32i3.2360
  36. McKnight, D. H., V. Choudhury, and C. Kacmar, "The impact of initial consumer trust on intentions to transact with a web site: A trust building model", The Journal of Strategic Information Systems, Vol.11, No.3, 2002, pp. 297-323. https://doi.org/10.1016/S0963-8687(02)00020-3
  37. Merriam, S. B., "Designing your study and selecting a sample", in Merriam, S. B. (eds.), Qualitative Research: A Guide to Design and Implementation, John Wiley & Sons, 2014, pp. 109-110.
  38. Mihailidis, P. and J. N. Cohen, "Exploring curation as a core competency in digital and media literacy education", Journal of Interactive Media in Education, Vol.2013, No.1, 2013, p. 2.
  39. Morey, T., T. T. Forbath, and A. Schoop, "Customer data: Designing for transparency and trust", Harvard Business Review, Vol.93, No.5, 2015, pp. 96-106.
  40. Nguyen, T. T., D. Kluver, T.-Y. Wang, P.-M. Hui, M. D. Ekstrand, M. C. Willemsen, and J. Riedl, "Rating support interfaces to improve user experience and recommender accuracy", Paper Presented at the Proceedings of the 7th ACM Conference on Recommender Systems, 2013.
  41. O'Donovan, J. and B. Smyth, "Trust in recommender systems", Paper Presented at the Proceedings of the 10th International Conference on Intelligent User Interfaces, 2005.
  42. Park, D. H., H. K. Kim, I. Y. Choi, and J. K. Kim, "A literature review and classification of recommender systems research", Expert Systems with Applications, Vol.39, No.11, 2012, pp. 10059-10072. https://doi.org/10.1016/j.eswa.2012.02.038
  43. Petrelli, D., S. Whittaker, and J. Brockmeier, "AutoTopography: What can physical mementos tell us about digital memories?", Paper Presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2008.
  44. Rosenbaum, S., "Curation-what is it?" in Rosenbaum, S. (eds.), Curation Nation: How to Win in a World Where Consumers are Creators, 2011. pp. 1-22.
  45. Ryan, M.-L., "Beyond myth and metaphor", Consultant 1983, 2001, p. 91.
  46. Shin, I.-H., J. Cha, G. W. Cheon, C. Lee, S. Y. Lee, H.-J. Yoon, and H. C. Kim, "Automatic stress-relieving music recommendation system based on photoplethysmography-derived heart rate variability analysis", Paper presented at the Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE, 2014.
  47. Sparling, E. I. and S. Sen, "Rating: How difficult is it?", Paper Presented at the Proceedings of the Fifth ACM Conference on Recommender Systems, 2011.
  48. Steedman, C., "The space of memory: In an archive", History of the Human Sciences, Vol.11, No.4, 1998, pp. 65-83. https://doi.org/10.1177/095269519801100405
  49. Wang, Y., X. Liao, H. Wu, and J. Wu, "Incremental Collaborative Filtering Considering Temporal Effects", arXiv preprint arXiv:1203.5415, 2012.
  50. Webb, A. M., R. Linder, A. Kerne, N. Lupfer, Y. Qu, B. Poffenberger, and C. Revia, "Promoting reflection and interpretation in education: Curating rich bookmarks as information composition", Paper Presented at the Proceedings of the 9th ACM Conference on Creativity & Cognition, 2013.
  51. Wingfield, N., "Unraveling the Mysteries Inside Web Shoppers 'Minds'", Wall Street Journal, 1998, available at http://www.wsj.com/articles/SB898117278721186500.
  52. Woodruff, A., "Necessary, unpleasant, and disempowering: Reputation management in the internet age", Paper Presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2014.
  53. Zarro, M. and C. Hall, "Exploring social curation", D-Lib Magazine, Vol.18, No.11, 2012, p. 6.