Asia pacific journal of information systems

The Korea Society of Management Information Systems (한국경영정보학회)
권호 표지
DOI QR코드

DOI QR Code

An Experimental Comparison of the Usability of Rule-based and Natural Language Processing-based Chatbots

  • Yeji Lim (School of Business, Hanyang University) ;
  • Jeonghun Lim (School of Business, Hanyang University) ;
  • Namjae Cho (School of Business, Hanyang University)
  • Received : 2019.08.08
  • Accepted : 2020.11.04
  • Published : 2020.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Service organizations increasingly adopt data-based intelligent engines called chatbots in support of the interaction between customers and the companies. Two different types of chatbots have been suggested and introduced by companies leading the adoption of this emerging technology: rule-based chatbots and natural language processing-based chatbots. While the differences between these two types of technologies look relatively clear, the organizational and practical impacts of the differences have not been systematically explored. This study performed an experiment to compare the use of the two different types of chatbots used in practice by two comparable organizations. These two types of actual chatbots were used by Korean on-line shopping malls with similar business models (mobile shopping), length of history, size and reputation. The comparison was made based on such dimensions as usability, searchability, reliability and attractiveness. Contraty to conventional expectation that the superiority in technology will produce superior usability, the results show mixed superiority. The discussion on the reasons is presented.

Keywords

Acknowledgement

This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea(NRF-2017S1A5A2A03068426).

References

  1. Arhippainen, L., and Tahti, M. (2003). Empirical evaluation of user experience in two adaptive mobile application prototypes. In MUM 2003. Proceedings of the 2nd International Conference on Mobile and Ubiquitous Multimedia, 11, 27-34.
  2. Arsovski, S., Osipyan, H., Oladele, M. I., and Cheok, A. D. (2019). Automatic knowledge extraction of any Chatbot from conversation. Expert Systems with Applications, 137, 343-348. https://doi.org/10.1016/j.eswa.2019.07.014
  3. Bharti, U., Bajaj, D., Batra, H., Lalit, S., Lalit, S., and Gangwani, A. (2020). Medbot: Conversational artificial intelligence powered chatbot for delivering tele-health after COVID-19. In 2020 5th International Conference on Communication and Electronics Systems (ICCES), IEEE, 870-875.
  4. Cho, N., and Park, S. (2001). Development of electronic commerce user consumer satisfaction index(ECUSI) for internet shopping. Industrial Management & Data Systems, 101(8), 400-405. https://doi.org/10.1108/EUM0000000006170
  5. Hassenzahl, M., and Tractinsky, N. (2006). User experience-a research agenda. Behaviour & Information Technology, 25(2), 91-97. https://doi.org/10.1080/01449290500330331
  6. Hassenzahl, M. (2018). The thing and I: Understanding the relationship between user and product. In Funology 2. Springer, Cham, pp. 301-313.
  7. Han, S. R. (2017). Business trends and implications of Chatbot. Digieco Report.
  8. Keszocze, O., and Harris, I. G. (2019). Chatbot-based assertion generation from natural language specifications. In 2019 Forum for Specification and Design Languages(FDL), IEEE, 1-6.
  9. Kim, J. T., Jung, S. R., and Chung, H. (2017). Trends of messenger-based chatbot technology and military application. Defense & Technology, 459, 118-127.
  10. Kim, J. W. (2012). Human computer interaction. Ahn Graphics.
  11. Konstan, J. A., and Riedl, J. (2012). Recommender systems: From algorithms to user experience. User Modeling and User-Adapted Interaction, 22(1-2), 101-123. https://doi.org/10.1007/s11257-011-9112-x
  12. Mamani, J. R. C., Alamo, Y. J. R., Aguirre, J. A. A., and Toledo, E. E. G. (2019). Cognitive services to improve user experience in searching for academic information based on chatbot. In 2019 IEEE XXVI International Conference on Electronics, Electrical Engineering and Computing(INTERCON), IEEE, 1-4.
  13. Moon, J. H., Lim, S. T., Park, C. L., Lee, I. S., and Kim, J. W. (2008). Conceptual study on user experience in HCI: Definition of UX and introduction of a new concept of CX (Co-Experience). Journal of the HCI Society of Korea, 3(1), 9-17. https://doi.org/10.17210/jhsk.2008.05.3.1.9
  14. Megha, D. (2019). [Online] Available at https://medium.com/searce/paving-theway-to-a-chatbot-revolution-c7ea21fa2f2f
  15. Morville, P. (2006). Search 2.0: The evolution of discovery(Ambient findability). Seoul: Hanvit Media.
  16. Morville, P. (2004). User experience design. Ann Arbor: Semantic Studios LLC.
  17. Oh, S. W. (2016). (ro) Bot as media An experimental discussion on news chatbot. Korean Journal of Communication and Information, 79, 70-103.
  18. Park, D. A. (2017). A study on conversational public administration service of the Chatbot based on artificial intelligence. Journal of Korea Multimedia Society, 20(8), 1347-1356. https://doi.org/10.9717/KMMS.2017.20.8.1347
  19. Roto, V. (2006). Web browsing on mobile phones: Characteristics of user experience. Helsinki University of Technology, pp. 834-837.
  20. Shevat, A. (2017). Designing bots: Creating conversational experiences. O'Reilly Media, Inc., p. 120.
  21. Sidenko, I., Kondratenko, G., Kushneryk, P., and Kondratenko, Y. (2019). Peculiarities of human machine interaction for synthesis of the intelligent dialogue Chatbot. In 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications(IDAACS), IEEE, 1056-1061.
  22. Tanya, R., Tanlamai, U., and Cho, N. (2017). Design guidelines of a web portal's functions and layouts to fit individual user's personality traits. International Journal of Innovation and Learning, 22(3), 271-292. https://doi.org/10.1504/IJIL.2017.086729