Asia pacific journal of information systems

The Korea Society of Management Information Systems (한국경영정보학회)
권호 표지
DOI QR코드

DOI QR Code

An In-depth Investigation into the Influence of Chatbot Usability and Age on Continuous Intention to Use: A Comprehensive Study

  • Manigandan L. (VIT Business School, Vellore Institute of Technology) ;
  • Sivakumar Alur (VIT Business School, Vellore Institute of Technology)
  • Received : 2023.07.20
  • Accepted : 2024.01.12
  • Published : 2024.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

This study aims to assess the impact of chatbot usability and demographics on continuous intention to use across different sectors. The research employed Braun's Bot Usability Scale (BUS11) to measure chatbot usability, focusing on accessibility, quality, conversation quality, privacy risk, and response time. A total of 187 participants completed a survey as part of this study. Variance-based SEM was utilized to examine relationships and test hypotheses. This study contributes to the ongoing discourse on chatbot adoption and user behaviour. It enhances the understanding of chatbot usability, highlighting the role of age in continued intention to use chatbots. The findings suggest that different age groups may possess specific preferences and expectations regarding chatbot usability. These differing preferences can influence their intention to continue using this technology. The study reveals that chatbot usability significantly impacts continuous intention to use and that age moderates the relationship between perceived conversation quality, information, privacy, security, and continuous intention to use. Based on the study's results, it is recommended that chatbot designers enhance usability to promote long-term adoption and usage.

Keywords

References

  1. Ackerman, S., Anaby-Tavor, A., Farchi, E., Goldbraich, E., Kour, G., Rabinovich, E., ... and Zwerdling, N. (2022). High-quality Conversational Systems. Cornell University. https://doi.org/10.48550/ARXIV.2204.13043
  2. Adamopoulou, E., and Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006. https://doi.org/10.1016/j.mlwa.2020.100006
  3. Ait-Mlouk, A., Alawadi, S., Toor, S., and Hellander, A. (2023). FedBot: Enhancing Privacy in Chatbots with Federated Learning. Cornell University. https://doi.org/10.48550/ARXIV.2304.03228
  4. Akdogan, M., and Kuru, D. (2022). The Effects Of Sense And Information Quality In Virtual Travel Experience On Visit Intention. Kahramanmaras Sutcu Imam universitesi Sosyal Bilimler Dergisi. https://doi.org/10.33437/ksusbd.1133724
  5. Al-Emran, M., Arpaci, I., and Salloum, S. A. (2020). An empirical examination of continuous intention to use m-learning: An integrated mode. Education and Information Technologies, 25(4), 2899-2918. https://doi.org/10.1007/s10639-019-10094-2
  6. Antle, A. N., and Wise, A. F. (2013). Getting down to details: Using theories of cognition and learning to inform tangible user interface design. Interacting with Computers, 25(1), 1-20. https://doi.org/10.1093/iwc/iws007
  7. Azzahra, T. R., and Kusumawati, N. (2023). Impact of Mobile Service Quality, Perceived Value, Perceived Usefulness, Perceived Ease of Use, Customer Satisfaction Towards Continuance Intention to Use MyTelkomsel App. Journal of Consumer Studies and Applied Marketing, 1(1), 46-60. https://doi.org/10.58229/jcsam.v1i1.74
  8. Belanche, D., Casalo, L. V., and Flavian, C. (2019). Artificial Intelligence in FinTech: understanding robo-advisors adoption among customers. Industrial Management and Data Systems, 119(7), 1411-1430. https://doi.org/10.1108/IMDS-08-2018-0368
  9. Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F., Tariverdiyeva, G., Balaji, D., and Chamberlain, A. (2022). The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents. Personal and Ubiquitous Computing, 26, 95-119. https://doi.org/10.1007/s00779-021-01582-9
  10. Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., and van der Velde, F. (2022). A confirmatory factorial analysis of the Chatbot usability scale: A multilanguage validation. Pers Ubiquit Comput 27, 317-330 https://doi.org/10.1007/s00779-022-01690-0
  11. Braun, M. (2023a). Evaluating the Chatbot Usability Scale: A Psychometric and Designometric Perspective. University of Twente Student Theses
  12. Chagas, B. A., Pagano, A. S., Prates, R. O., Praes, E. C., Ferreguetti, K., Vaz, H., Nogueira Reis, Z. S., Ribeiro, L. B., Ribeiro, A. L. P., Pedroso, T. M., Beleigoli, A., Oliveira, C. R. A., and Marcolino, M. S. (2022). Evaluating user experience with a Chatbot designed as a public health response to the COVID-19 pandemic in Brazil: Mixed methods study. JMIR Human Factors, 10, e43135-e43135. https://doi.org/10.2196/43135
  13. Chen, H. L., Vicki Widarso, G., and Sutrisno, H. (2020). A ChatBot for learning Chinese: Learning achievement and technology acceptance. Journal of Educational Computing Research, 58(6), 1161-1189. https://doi.org/10.1177/0735633120929622
  14. Chung, K., Cho, H. Y. and Park, J. Y. (2021). A Chatbot for Perinatal Women's and Partners' Obstetric and Mental Health Care: Development and Usability Evaluation Study. JMIR Medical Informatics, 9(3), https://doi.org/10.2196/18607
  15. Chung, M., Ko, E., Joung, H., and Kim, S. J. (2020). Chatbot e-service and customer satisfaction regarding luxury brands. Journal of Business Research, 117, 587-595, https://doi.org/10.1016/J.JBUSRES.2018.10.004
  16. Colledge, A. (2018). Bridging the generational gap: Designing internet services for technologically-naive older people using familiar interfaces. Detta ar en Magister-uppsats fran Umea universitet/ Institutionen for informatik.
  17. Davenport, T., Guha, A., Grewal, D., and Bressgott, T. (2020). How artificial intelligence will change the future of marketing. Journal of the Academy of Marketing Science, 48(1), 24-42. https://doi.org/10.1007/s11747-019-00696-0
  18. De Cicco, R., e Silva, S. C., and Alparone, F. R. (2020). Millennials' attitude toward chatbots: An experimental study in a social relationship perspective. International Journal of Retail and Distribution Management, 48(11), 1213-1233. https://doi.org/10.1108/IJRDM-12-2019-0406
  19. de Cosmo, L. M., Piper, L., and Di Vittorio, A. (2021). The role of attitude toward chatbots and privacy concern on the relationship between attitude toward mobile advertising and behavioral intent to use chatbots. Italian Journal of Marketing, 2021(1-2), 83-102. https://doi.org/10.1007/s43039-021-00020-1
  20. Folstad, A., and Brandtzaeg, P. B. (2017). Chatbots and the New World of HCI. Interactions, 24(4), 38-42. https://doi.org/10.1145/3085558
  21. Fu, S. (2012). User Interface Design By Applying Theories Of Aesthetics.
  22. Gnewuch, U., Morana, S., and Maedche, A. (2017). Towards designing cooperative and social conversational agents for customer service. In ICIS 2017 Proceedings.
  23. Go, E., and Sundar, S. S. (2019). Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions Computers in Human Behavior, 97, 304-316. https://doi.org/10.1016/j.chb.2019.01.020.
  24. Gomez Reynoso, J. and Olfman, L. (2012). The impact of combining gestalt theories with interface design guidelines in designing user interfaces. In AMCIS 2012 Proceedings.
  25. Gray, H. M., Gray, K. and Wegner, D. M. (2007). Dimensions of mind perception. Science, 315(5812), 619. https://doi.org/10.1126/science.1134475
  26. Gupta, V., Joshi, V., Jain, A., and Garg, I. (2023). Chatbot for Mental health support using NLP. In 2023 4th International Conference for Emerging Technology, INCET 2023 (pp. 1-6). https://doi.org/10.1109/INCET57972.2023.10170573
  27. Hair, J. F., Ringle, C. M., and Sarstedt, M. (2011). PLS-SEM: Indeed a silver bullet. Journal of Marketing Theory and Practice, 19(2), 139-152. https://doi.org/10.2753/MTP1069-6679190202
  28. Hair, J.F., Sarstedt, M., Matthews, L. M., and Ringle, C. M. (2016). Identifying and treating unobserved heterogeneity with FIMIX-PLS: part I - method. European Business Review, 28(1), 63-76, https://doi.org/10.1108/EBR-09-2015-0094/FULL/XML.
  29. Han, R., Todd, A., Wardak, S., Partridge, S.R. and Raeside, R. (2022). Feasibility and acceptability of chatbots for nutrition and physical activity health promotion among adolescents: Systematic scoping review with adolescent consultation. JMIR Human Factors, 10, e43227-e43227. https://doi.org/10.2196/43227
  30. He, S. Y., Sun, K. K., and Luo, S. (2022). Factors affecting electric vehicle adoption intention: The impact of objective, perceived, and prospective charger accessibility. Journal of Transport and Land Use, University of Minnesota, 15(1), 779-801. https://doi.org/10.5198/jtlu.2022.2113
  31. Hendriks, F., Ou, C. X. J., Amiri, A. K. and Bockting, S. (2020). The power of computer- mediated communication theories in explaining the effect of chatbot introduction on user experience. In Proceedings of the Annual Hawaii International Conference on System Sciences (pp. 271-278). https://doi.org/10.24251/hicss.2020.034
  32. Hollebeek, L. D., Sprott, D. E., and Brady, M. K. (2021). Rise of the machines? Customer engagement in automated service interactions. Journal of Service Research, 24(1), https://doi.org/10.1177/1094670520975110.
  33. Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., and McTear, M. (2019, September). Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?. In Proceedings of the 31st European Conference on Cognitive Ergonomics (pp. 207-214). https://doi.org/10.1145/3335082.3335094
  34. Huang, M. H., and Rust, R. T. (2021). Engaged to a robot? The role of AI in service. Journal of Service Research, 24(1), 30-41. https://doi.org/10.1177/1094670520902266
  35. Iancu, I., and Iancu, B. (2023). Interacting with chatbots later in life: A technology acceptance perspective in COVID-19 pandemic situation. Frontiers in Psychology, 13, https://doi.org/10.3389/FPSYG.2022.1111003
  36. Islam, M. T., Porter, D. E., and Billah, S. M. (2023). A probabilistic model and metrics for estimating perceived accessibility of desktop applications in keystroke-based non- visual interactions. In Conference on Human Factors in Computing Systems - Proceedings, 43, (pp. 1-20). https://doi.org/10.1145/3544548.3581400
  37. Jenneboer, L., Herrando, C., and Constantinides, E. (2022). The impact of chatbots on customer loyalty: A systematic literature review. Journal of Theoretical and Applied Electronic Commerce Research, 17(1), 212-229. https://doi.org/10.3390/jtaer17010011.
  38. Joyce, M., and Kirakowski, J. (2015). Measuring attitudes towards the internet: The general internet attitude scale. International Journal of Human-Computer Interaction, 31(8), 506-517. https://doi.org/10.1080/10447318.2015.1064657
  39. Kim, S. H., Bae, J. H., and Jeon, H. M. (2019). Continuous intention on accommodation apps: Integrated value-based adoption and expectation-confirmation model analysis. Sustainability, 11(6), 1578-1595. https://doi.org/10.3390/su11061578
  40. Larbi, D., Denecke, K., and Gabarron, E. (2022), Usability testing of a social media chatbot for increasing physical activity behavior. Journal of Personalized Medicine, 12(5), 828. https://doi.org/10.3390/JPM12050828
  41. Lesselroth, B., Monkman, H., Adams, K., Wood, S., Corbett, A., Homco, J., Borycki, E. M., Spier, R., and Kushniruk, A. W. (2020). User experience theories, models, and frameworks: A focused review of the healthcare literature. Studies in Health Technology and Informatics, 270, 1076-1080. https://doi.org/10.3233/SHTI200327
  42. Li, Y. S., Lam, C. S. N., and See, C. (2021). Using a machine learning architecture to create an AI-powered chatbot for anatomy education. Medical Science Educator, 31(6), 1729-1730. https://doi.org/10.1007/s40670-021-01405-9
  43. Luo, X., Tong, S., Fang, Z., and Qu, Z. (2019). Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases. Marketing Science, 38(6), 937-947. https://doi.org/10.1287/MKSC.2019.1192
  44. Maria, I., Wijaya, V., and Keni, K. (2021). Pengaruh information quality dan service quality terhadap perceived value dan konsekuensinya terhadap customer engagement behavior intention (Studi Pada Social Commerce Instagram). Jurnal Muara Ilmu Ekonomi Dan Bisnis, 5(2), 321. https://doi.org/10.24912/jmieb.v5i2.12276
  45. Michael, B., Goodman-Deane, J., Waller, S. Tenneti., R., Langdon, P., and Clarkson, P. J. (2013). Age, technology prior experience and ease of use: Who's doing what?. In Contemporary Ergonomics and Human Factors (pp. 363-369).
  46. Ng, M., Coopamootoo, K. P. L., Toreini, E., Aitken, M., Elliot, K., and Van Moorsel, A. (2020). Simulating the effects of social presence on trust, privacy concerns & usage intentions in automated bots for finance. In Proceedings - 5th IEEE European Symposium on Security and Privacy Workshops, Euro S and PW 2020 (pp. 190-199). https://doi.org/10.1109/EUROSPW51379.2020.00034
  47. Oh, Y., Zhuang, Q., Welp, L. R., Liu, L., Lan, X., Basu, S., ... and Chanton, J. P. (2022). Improved global wetland carbon isotopic signatures support post-2006 microbial methane emission increase. Communications Earth & Environment, 3(1), 159. https://doi.org/10.1038/s43247-022-00488-5
  48. Petre, M., Minocha, S., and Roberts, D. (2011). Usability beyond the website: An empirically-grounded e-commerce evaluation instrument for the total customer experience. Behaviour & Information Technology, 25(2), 189-203. https://doi.org/10.1080/01449290500331198
  49. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., and Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. https://doi.org/10.1037/0021-9010.88.5.879
  50. Pot, F. J., Koster, S., and Tillema, T. (2023a). Perceived accessibility and residential self-selection in the Netherlands. Journal of Transport Geography, 108, 103555-103555. https://doi.org/10.1016/J.JTRANGEO.2023.103555
  51. Pot, F. J., Koster, S., and Tillema, T. (2023b). Perceived accessibility in Dutch rural areas. Transport Policy, 138, 170-184. https://doi.org/10.1016/J.TRANPOL.2023.04.014
  52. Prentice, C., Han, X. Y., Hua, L. L., and Hu, L. (2019). The influence of identity-driven customer engagement on purchase intention. Journal of Retailing and Consumer Services, 47, 339-347. https://doi.org/10.1016/J.JRETCONSER.2018.12.014
  53. Rajesh, V., Perumal, B., Ganesh, U. S., Rajkumar, V., Kumar, D. M., and Kumar, M. (2023). Building customer support chatbots with intent recognition. In ViTECoN 2023 - 2nd IEEE International Conference on Vision Towards Emerging Trends in Communication and Networking Technologies, Proceedings. https://doi.org/10.1109/VITECON58111.2023.10157329
  54. Ren, R., Castro, J. W., Acuna, S. T., and De Lara, J. (2019). Usability of chatbots: A systematic mapping study. In Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE (pp. 479-484). https://doi.org/10.18293/SEKE2019-029
  55. Roy, S. K., Shekhar, V., Lassar, W. M., and Chen, T. (2018). Customer engagement behaviors: The role of service convenience, fairness and quality. Journal of Retailing and Consumer Services, 44, 293-304. https://doi.org/10.1016/J.JRETCONSER.2018.07.018
  56. Sebastian, G. (2023). Privacy and data protection in ChatGPT and Other AI chatbots. In International Journal of Security and Privacy in Pervasive Computing, 15(1), 1-14. https://doi.org/10.4018/IJSPPC.325475
  57. Sidaoui, K., Jaakkola, M., and Burton, J. (2020). AI feel you: customer experience assessment via chatbot interviews. Journal of Service Management, 31(4), 745-766. https://doi.org/10.1108/JOSM-11-2019-0341/FULL/XML
  58. Sousa, M. J., and Rocha, A. (2019). Skills for disruptive digital business. Journal of Business Research, 94, 257-263. https://doi.org/10.1016/j.jbusres.2017.12.051
  59. Suryawirawan, O. A., Suhermin, S., and Shabrie, W. S. (2022). Service quality, satisfaction, continuous usage intention, and purchase intention toward freemium applications: The moderating effect of perceived value. Jurnal Ekonomi Bisnis Dan Kewirausahaan, 11(3), 383. https://doi.org/10.26418/jebik.v11i3.57483
  60. Tarmidi, D., Santoso, A. B., Marinda, V. S., and Amalia, S. (2022). Perceived Value and Perceived Quality on Repurchase Intention: The Case Study Of Spotify in Bandung. JIIP-Jurnal Ilmiah Ilmu Pendidikan, 5(8), 3212-3216. https://doi.org/10.54371/jiip.v5i8.826
  61. Terblanche, N. (2020). A design framework to create artificial intelligence coaches. International Journal of Evidence Based Coaching and Mentoring, 18(2), 152-165. https://doi.org/10.24384/b7gs-3h05
  62. Terblanche, N., and Kidd, M. (2022). Adoption factors and moderating effects of age and gender that influence the intention to use a non-directive reflective coaching chatbot. SAGE Open, 12(2), https://doi.org/10.1177/21582440221096136
  63. To, Q. G., Green, C., and Vandelanotte, C. (2021). Feasibility, usability, and effectiveness of a machine learning-based physical activity chatbot: Quasi-experimental study. JMIR MHealth and UHealth, 9(11), https://doi.org/10.2196/28577
  64. Verhagen, T., van Nes, J., Feldberg, F., and van Dolen, W. (2014). Virtual customer service agents: Using social presence and personalization to shape online service encounters. Journal of Computer-Mediated Communication, 19(3), 529-545. https://doi.org/10.1111/jcc4.12066
  65. Waheed, N., Ikram, M., Hashmi, S. S., He, X. and Nanda, P. (2022). An empirical assessment of security and privacy risks of web-based Chatbots. In R. Chbeir, H. Huang, F. Silvestri, Y. Manolopoulos, and Y. Zhang (Eds.), Web Information Systems Engineering - WISE 2022. WISE 2022. Lecture Notes in Computer Science (Vol. 13724). Springer, Cham. https://doi.org/10.1007/978-3-031-20891-1_23
  66. Wu, R., Liu, M., and Kardes, F. (2021). Aging and the preference for the human touch. Journal of Services Marketing, 35(1), 29-40. https://doi.org/10.1108/JSM-09-2019-0366
  67. Yang, J., Chen, Y. L., Por, L. Y., and Ku, C. S. (2023). A systematic literature review of information security in chatbots. Applied Sciences, 13(11), 6355. https://doi.org/10.3390/app13116355
  68. Zarouali, B., Van Den Broeck, E., Walrave, M., and Poels, K. (2018). Predicting consumer responses to a chatbot on Facebook. Cyberpsychology, Behavior, and Social Networking, 21(8), 491-497. https://doi.org/10.1089/cyber.2017.0518