Journal of Service Research and Studies (서비스연구)

The Society of service Science (서비스사이언스학회)
권호 표지
DOI QR코드

DOI QR Code

An Exploratory Study of Generative AI Service Quality using LDA Topic Modeling and Comparison with Existing Dimensions

LDA토픽 모델링을 활용한 생성형 AI 챗봇의 탐색적 연구 : 기존 AI 챗봇 서비스 품질 요인과의 비교

  • YaeEun Ahn (Business School, Seoul National University) ;
  • Jungsuk Oh (Business School, Seoul National University)
  • 안예은 (서울대학교 경영대학 대학원 ) ;
  • 오정석 (서울대학교 경영대학 )
  • Received : 2023.12.04
  • Accepted : 2023.12.13
  • Published : 2023.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Artificial Intelligence (AI), especially in the domain of text-generative services, has witnessed a significant surge, with forecasts indicating the AI-as-a-Service (AIaaS) market reaching a valuation of $55.0 Billion by 2028. This research set out to explore the quality dimensions characterizing synthetic text media software, with a focus on four key players in the industry: ChatGPT, Writesonic, Jasper, and Anyword. Drawing from a comprehensive dataset of over 4,000 reviews sourced from a software evaluation platform, the study employed the Latent Dirichlet Allocation (LDA) topic modeling technique using the Gensim library. This process resulted the data into 11 distinct topics. Subsequent analysis involved comparing these topics against established AI service quality dimensions, specifically AICSQ and AISAQUAL. Notably, the reviews predominantly emphasized dimensions like availability and efficiency, while others, such as anthropomorphism, which have been underscored in prior literature, were absent. This observation is attributed to the inherent nature of the reviews of AI services examined, which lean more towards semantic understanding rather than direct user interaction. The study acknowledges inherent limitations, mainly potential biases stemming from the singular review source and the specific nature of the reviewer demographic. Possible future research includes gauging the real-world implications of these quality dimensions on user satisfaction and to discuss deeper into how individual dimensions might impact overall ratings.

인공 지능 (AI), 특히 텍스트 생성 서비스 분야에서의 발전은 두드러지게 나타나고 있으며, AI-as-a-Service (AIaaS) 시장은 2028년까지 550억 달러에 달할 것으로 예상된다. 본 연구는 합성 텍스트 미디어 소프트웨어의 품질 요소를 탐구하였으며, 이를 위해 ChatGPT, Writesonic, Jasper, 그리고 Anyword와 같은 산업의 주요 서비스에 주목하였다. 소프트웨어 평가 플랫폼에서 수집된 4,000개 이상의 리뷰를 바탕으로, Gensim 라이브러리를 활용한 잠재 디리클레 할당 (LDA) 주제 모델링 기법을 적용하였다. 이 분석을 통해 11개의 주제가 도출되었다. 이후 이 주제들을 AICSQ 및 AISAQUAL과 같은 기존 논문에서 다루었던 AI 서비스 품질 차원과 비교 분석하였다. 리뷰에서는 가용성 및 효율성과 같은 차원이 주로 강조되었으며, 이전 연구에서 중요하게 여겨졌던 사람다움과 같은 요소는 본 연구에서 강조되지 않았다. 이러한 결과는 AI 서비스의 본질적 특성, 즉 사용자와의 직접적인 상호작용보다 의미론적 이해에 더 중점을 둔다는 특성 때문으로 해석된다. 본 연구는 단일 리뷰 원천 및 평가자들의 인구 통계의 특정성과 같은 잠재적 편향을 인정하며, 향후 연구 방향으로는 이러한 품질 차원이 사용자 만족도에 어떻게 영향을 미치는지, 그리고 개별 차원이 전체 평점에 어떻게 영향을 미치는지에 대한 깊은 분석을 제안한다.

Keywords

Acknowledgement

This study was supported by the Institute of Management Research at Seoul National University.

References

  1. AMR(2020), Intelligent Virtual Assistant (IVA) Market to grow at 33% CAGR during forecast period (2020-2025) - Insights on Growth Drivers, Size and Share Analysis, Key Trends, Leading Players, and Business Opportunities: Adroit Market Research. www.globenewswire.com/news-release/2020/02/24/1988963/0/en/Intelligent-Virtual-Assistant-IVA-Market-to-grow-at-33-CAGR-during-forecast-period-2020-2025-Insights-on-Growth-Drivers-Size-and-Share-Analysis-Key-Trends-Leading-Players-andBusin.html (accessed 06 June 2023). 
  2. Barnes, S.J. and Vidgen, R.(2001). An evaluation of cyber-bookshops: the WebQual method. International Journal of Electronic Commerce, 11-30. 
  3. Barnes, S.J. and Vidgen, R.T.(2002). An integrative approach to the assessment of e-commerce quality. J. Electron. Commer. Res., 3(3), 114-127. 
  4. Chen, Q., Gong, Y., Lu, Y., and Tang, J. (2022). Classifying and measuring the service quality of AI chatbot in frontline service. Journal of Business Research, 145, 552-568.  https://doi.org/10.1016/j.jbusres.2022.02.088
  5. Davenport, T., Guha, A., Grewal, D. and Bressgott, T.(2020), How artificial intelligence will change the future of marketing, Journal of the Academy of Marketing Science, 48(1), 24-42.  https://doi.org/10.1007/s11747-019-00696-0
  6. Gartner(2019). How to manage customer service technology innovation. Gartner. https://www.gartner.com/smarterwithgartner/27297-2 
  7. Gartner(2021, October 18). Gartner identifies the top strategic technology trends for 2022. Gartner. https://www.gartner.com/en/newsroom/press-releases/2021-10-18-gartner-identifies-the-top-strategic-technology-trends-for-2022
  8. Hacker, P., Engel, A., and Mauer, M.(2023). Regulating ChatGPT and other large generative AI models. arXiv preprint arXiv:2302.02337. 
  9. IBM(2022, May). IBM Global AI Adoption Index 2022. IBM. https://www.ibm.com/downloads/cas/GVAGA3JP 
  10. Koury, L. and Murphy, T.(2023). The evolution of chatbots from ELIZA to Bard. TechTarget. TechTarget. Retrieved June 12, 2023, from https://www.techtarget.com/searchcustomerexperience/infographic/The-evolution-of-chatbots-and-generative-AI. 
  11. Lawton, G.(2023, April 18). Generative AI Ethics: 8 biggest concerns. Enterprise AI. https://www.techtarget.com/searchenterpriseai/tip/Generative-AI-ethics-8-biggest-concerns 
  12. Lee, S., Baker, J., Song, J., and Wetherbe, J.C.(2010). An Empirical Comparison of Four Text Mining Methods. Proceedings of the 43rd Hawaii International Conference on System Sciences, 1-10. 
  13. MarketsandMarkets. (2023, May 4). AI as a service market size & analysis, trends, growth, Revenue Forecast - 2028 & Opportunities: MarketsandMarketsTM. MarketsandMarkets. https://www.marketsandmarkets.com/Market-Reports/artificial-intelligence-ai-as-a-service-market-121842268.html 
  14. Noor, N., Rao Hill, S., and Troshani, I. (2022). Developing a service quality scale for artificial intelligence service agents. European Journal of Marketing, 56(5), 1301-1336.  https://doi.org/10.1108/EJM-09-2020-0672
  15. OpenAI. (2023). Introducing chatgpt. Introducing ChatGPT. https://openai.com/blog/chatgpt 
  16. Poole, D.L. and Mackworth, A.K.(2010), Artificial Intelligence: Foundations of Computational Agents, Cambridge University Press, Cambridge. 
  17. Sievert, C. and Shirley, K.E.(2014). LDAvis: A method for visualizing and interpreting topics. 
  18. Steyvers, M. and Griffiths, T.(2007). Probabilistic Topic Models. In T. Landauer, D. McNamara, S. Dennis, & W. Kintsch (Eds.), Latent Semantic Analysis: A Road to Meaning. Hillsdale, NJ: Laurence Erlbaum. 
  19. Thomas, R.(2020, January 8). Ai in 2020: From experimentation to adoption. THINK Blog. https://www.ibm.com/blogs/think/2020/01/ai-in-2020-from-experimentation-to-adoption/
  20. Thormudsson(2023), Rate of generative AI adoption in the workplace in the United States 2023, by industry, Statista, https://www.statista.com/statistics/1361251/generative-ai-adoption-rate-at-work-by-industry-us/ (accessed 06 June 2023). 
  21. Weizenbaum, J.(1966). Eliza-a computer program for the study of natural language communication between man and Machine. Communications of the ACM, 9(1), 36-45.  https://doi.org/10.1145/365153.365168
  22. Wolfinbarger, M. and Gilly, M.C.(2003). eTailQ: Dimensionalizing, measuring and predicting retail quality. Journal of Retailing, 79(3), 183-198.  https://doi.org/10.1016/S0022-4359(03)00034-4
  23. Yang, Z. and Jun, M.(2002). Consumer perception of e-service quality: from internet purchaser and non-purchaser perspectives. Journal of Business Strategies, 19(1), 19-42. https://doi.org/10.54155/jbs.19.1.19-42