Journal of the korean academy of Pediatric Dentistry (대한소아치과학회지)

Korean Academy of Pediatric Dentistry (대한소아치과학회)
권호 표지
DOI QR코드

DOI QR Code

Evaluating the Accuracy of Artificial Intelligence-Based Chatbots on Pediatric Dentistry Questions in the Korean National Dental Board Exam

  • Yun Sun Jung (Department of Pediatric Dentistry, Kyung Hee University College of Dentistry, Kyung Hee University Medical Center) ;
  • Yong Kwon Chae (Department of Pediatric Dentistry, Kyung Hee University College of Dentistry, Kyung Hee University Medical Center) ;
  • Mi Sun Kim (Department of Pediatric Dentistry, Kyung Hee University Dental Hospital at Gangdong) ;
  • Hyo-Seol Lee (Department of Pediatric Dentistry, Kyung Hee University College of Dentistry, Kyung Hee University Medical Center) ;
  • Sung Chul Choi (Department of Pediatric Dentistry, Kyung Hee University College of Dentistry, Kyung Hee University Medical Center) ;
  • Ok Hyung Nam (Department of Pediatric Dentistry, Kyung Hee University College of Dentistry, Kyung Hee University Medical Center)
  • Received : 2024.07.10
  • Accepted : 2024.08.12
  • Published : 2024.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

This study aimed to assess the competency of artificial intelligence (AI) in pediatric dentistry and compare it with that of dentists. We used open-source data obtained from the Korea Health Personnel Licensing Examination Institute. A total of 32 item multiple-choice pediatric dentistry exam questions were included. Two AI-based chatbots (ChatGPT 3.5 and Gemini) were evaluated. Each chatbot received the same questions seven times in separate chat sessions initiated on April 25, 2024. The accuracy was assessed by measuring the percentage of correct answers, and consistency was evaluated using Cronbach's alpha coefficient. Both ChatGPT 3.5 and Gemini demonstrated similar accuracy, with no significant differences observed between them. However, neither chatbot achieved the minimum passing score set by the Pediatric Dentistry National Examination. However, both chatbots exhibited acceptable consistency in their responses. Within the limits of this study, both AI-based chatbots did not sufficiently answer the pediatric dentistry exam questions. This finding suggests that pediatric dentists should be aware of the advantages and limitations of this new tool and effectively utilize it to promote patient health.

Keywords

References

  1. Arif TB, Munaf U, Ul-Haque I : The future of medical education and research: Is ChatGPT a blessing or blight in disguise? Med Educ Online, 28:2181052, 2023. 
  2. Manickam P, Mariappan SA, Murugesan SM, Hansda S, Kaushik A, Shinde R, Thipperudraswamy SP : Artificial Intelligence (AI) and Internet of Medical Things (IoMT) Assisted Biomedical Systems for Intelligent Healthcare. Biosensors (Basel), 12:562, 2022. 
  3. Lee JW, Yoo IS, Kim JH, Kim WT, Jeon HJ, Yoo HS, Shin JG, Kim GH, Hwang S, Park S, Kim YJ : Development of AI-generated medical responses using the ChatGPT for cancer patients. Comput Methods Programs Biomed, 254:108302, 2024. 
  4. Schwendicke F, Singh T, Lee JH, Gaudin R, Chaurasia A, Wiegand T, Uribe SE, Krois J : Artificial intelligence in dental research: Checklist for authors, reviewers, readers. J Dent, 107:103610, 2021. 
  5. McDermott MBA, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M : Reproducibility in machine learning for health research: Still a ways to go. Sci Transl Med, 13:eabb1655, 2021. 
  6. Vaishya R, Misra A, Vaish A : ChatGPT: Is this version good for healthcare and research? Diabetes Metab Syndr, 17:102744, 2023. 
  7. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D : How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ, 9:E45312, 2023. 
  8. Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD : Evaluating ChatGPT as an adjunct for radiologic decision-making. MedRxiv, 2023 Feb 7:2023.02.02. 23285399, 2023. 
  9. Potapenko I, Boberg-Ans LC, Stormly Hansen M, Klefter ON, van Dijk EH, Subhi Y : Artificial intelligence-based chatbot patient information on common retinal diseases using ChatGPT. Acta Ophthalmol, 101:829-831, 2023.  https://doi.org/10.1111/aos.15661
  10. Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, Ayoub W, Yang JD, Liran O, Spiegel B, Kuo A : Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol, 29:721-732, 2023.  https://doi.org/10.3350/cmh.2023.0089
  11. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V : Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health, 2:E0000198, 2023. 
  12. Frosolini A, Franz L, Benedetti S, Vaira LA, de Filippis C, Gennaro P, Marioni G, Gabriele G : Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines. Eur Arch Otorhinolaryngol, 280:5129-5133, 2023.  https://doi.org/10.1007/s00405-023-08205-4
  13. Lum ZC : Can Artificial Intelligence Pass the American Board of Orthopaedic Surgery Examination? Orthopaedic Residents Versus ChatGPT. Clin Orthop Relat Res, 481:1623-1630, 2023.  https://doi.org/10.1097/CORR.0000000000002704
  14. Schwendicke F, Samek W, Krois J : Artificial Intelligence in Dentistry: Chances and Challenges. J Dent Res, 99:769-774, 2020.  https://doi.org/10.1177/0022034520915714
  15. Taber KS : The use of Cronbach's alpha when developing and reporting research instruments in science education. Res Sci Educ, 48:1273-1296, 2018.  https://doi.org/10.1007/s11165-016-9602-2
  16. Korea Health Personnel Licensing Examination Institute : Results of the analysis of the 75th national examination of dentists in 2023. Available from URL: https://www.kuksiwon.or.kr/analysis/brd/m_91/view.do?seq=330&srchFr=&srchTo=&srchWord=&srchTp=&itm_seq_1=0&itm_seq_2=0&multi_itm_seq=0&-company_cd=&company_nm= (Accessed on August 14, 2024). 
  17. Strong E, DiGiammarino A, Weng Y, Kumar A, Hosamani P, Hom J, Chen JH : Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations. JAMA Intern Med, 183:1028-1030, 2023.  https://doi.org/10.1001/jamainternmed.2023.2909
  18. Hoch CC, Wollenberg B, Luers JC, Knoedler S, Knoedler L, Frank K, Cotofana S, Alfertshofer M : ChatGPT's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol, 280:4271-4278, 2023.  https://doi.org/10.1007/s00405-023-08051-4
  19. Alhaidry HM, Fatani B, Alrayes JO, Almana AM, Alfhaed NK : ChatGPT in Dentistry: A Comprehensive Review. Cureus, 15:E38317, 2023. 
  20. Liu J, Wang C, Liu S : Utility of ChatGPT in Clinical Practice. J Med Internet Res, 25:E48568, 2023. 
  21. Sallam M : ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare (Basel), 11:887, 2023. 
  22. Haupt CE, Marks M : AI-generated medical advice - GPT and beyond. JAMA, 329:1349-1350, 2023.  https://doi.org/10.1001/jama.2023.5321
  23. Athaluri SA, Manthena SV, Kesapragada V, Yarlagadda V, Dave T, Duddumpudi RTS : Exploring the Boundaries of Reality: Investigating the Phenomenon of Artificial Intelligence Hallucination in Scientific Writing Through ChatGPT References. Cureus, 15:E37432, 2023. 
  24. Ramgopal S, Sanchez-Pinto LN, Horvat CM, Carroll MS, Luo Y, Florin TA : Artificial intelligence-based clinical decision support in pediatrics. Pediatr Res, 93:334-341, 2023.  https://doi.org/10.1038/s41390-022-02226-1
  25. Liu Z, Zhang L, Wu Z, Yu X, Cao C, Dai H, Liu N, Liu J, Liu W, Li Q, Shen D, Li X, Zhu D, Liu T : Surviving ChatGPT in healthcare. Front Radiol, 3:1224682, 2024. 
  26. Dave T, Athaluri SA, Singh S : ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell, 6:1169595, 2023. 
  27. Stokel-Walker C : ChatGPT listed as author on research papers: many scientists disapprove. Nature, 613:620-621, 2023.  https://doi.org/10.1038/d41586-023-00107-z
  28. Chatterjee J, Dethlefs N : This new conversational AI model can be your friend, philosopher, and guide ... and even your worst enemy. Patterns (N Y), 4:100676, 2023. 
  29. Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, Moy L : ChatGPT and Other Large Language Models Are Double-edged Swords. Radiology, 307:E230163, 2023. 
  30. Biswas S : ChatGPT and the Future of Medical Writing. Radiology, 307:E223312, 2023. 
  31. WHO : WHO guideline Recommendations on Digital Interventions for Health System Strengthening. World Health Organization, Geneva, 2019. 
  32. Balel Y : Can ChatGPT be used in oral and maxillofacial surgery? J Stomatol Oral Maxillofac Surg, 124:101471, 2023. 
  33. Guerra GA, Hofmann H, Sobhani S, Hofmann G, Gomez D, Soroudi D, Hopkins BS, Dallas J, Pangal DJ, Cheok S, Nguyen VN, Mack WJ, Zada G : GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like Questions. World Neurosurg, 179:E160-E165, 2023.