Annals of Laboratory Medicine

Korean Society for Laboratory Medicine (대한진단검사의학회)
권호 표지
DOI QR코드

DOI QR Code

Evaluation of Six Large Language Models for Clinical Decision Support: Application in Transfusion Decision-making for RhD Blood-type Patients

  • Jong Kwon Lee (Department of Laboratory Medicine and Genetics, Samsung Medical Center, Sungkyunkwan University School of Medicine) ;
  • Sooin Choi (Department of Laboratory Medicine and Genetics, Soonchunhyang University Bucheon Hospital, Soonchunhyang University College of Medicine) ;
  • Sholhui Park (Department of Laboratory Medicine, Ewha Womans University College of Medicine) ;
  • Sang-Hyun Hwang (Department of Laboratory Medicine, Asan Medical Center, University of Ulsan College of Medicine) ;
  • Duck Cho (Department of Laboratory Medicine and Genetics, Samsung Medical Center, Sungkyunkwan University School of Medicine)
  • Received : 2024.10.29
  • Accepted : 2025.03.05
  • Published : 2025.09.01
Download PDF
⟨ Previous Next ⟩

Abstract

Background: Large language models (LLMs) have the potential for clinical decision support; however, their use in specific tasks, such as determining the RhD blood type for transfusion, remains underexplored. Therefore, we evaluated the accuracy of six LLMs in addressing RhD blood type-related issues in Korean healthcare. Methods: Fifteen multiple-choice and true/false questions, based on real-world transfusion scenarios and reviewed by specialists, were developed. The questions were administered twice to six LLMs (Clova X, Gemini 1.0, Gemini 1.5, ChatGPT-3.5, GPT-4.0, and GPT-4o) in both Korean and English. Results were compared against the performance of 22 transfusion medicine experts. For particularly challenging questions, prompt engineering was applied, and the questions were reevaluated. Results: GPT-4o demonstrated the highest accuracy rate in Korean (0.6), with significant differences compared with those of Clova X and Gemini (P <0.05). In English, the results were similar across all models. The transfusion experts achieved a higher accuracy rate (0.8). Among the five questions subjected to prompt engineering, only GPT-4o correctly responded to one, whereas the other models failed. All LLM models changed their responses or did not respond when the same question was repeated. Conclusions: GPT-4o showed the best overall performance among the models tested and may be beneficial in RhD blood product transfusion decision-making. However, its performance suggests that it may serve best in a supportive role rather than as a primary decision-making tool.

Keywords

Acknowledgement

We sincerely thank the transfusion medicine experts for their participation in the survey.

References

  1. Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst 2023;47:33. https://doi.org/10.1007/s10916-023-01925-4
  2. Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, et al. A guide to deep learning in healthcare. Nat Med 2019;25:24–9. https://doi.org/10.1038/s41591-018-0316-z
  3. Hurley NC, Schroeder KM, Hess AS. Would doctors dream of electric blood bankers? Large language model-based artificial intelligence performs well in many aspects of transfusion medicine. Transfusion 2023;63:1833–40.
  4. Alberts IL, Mercolli L, Pyka T, Prenosil G, Shi K, Rominger A, et al. Large language models (LLM) and ChatGPT: what will the impact on nuclear medicine be? Eur J Nucl Med Mol Imaging 2023;50:1549–52. https://doi.org/10.1007/s00259-023-06172-w
  5. Barile J, Margolis A, Cason G, Kim R, Kalash S, Tchaconas A, et al. Diagnostic accuracy of a large language model in pediatric case studies. JAMA Pediatr 2024;178:313–5. https://doi.org/10.1001/jamapediatrics.2023.5750
  6. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature 2023;620:172–80.
  7. Isiksacan Z, D'Alessandro A, Wolf SM, McKenna DH, Tessier SN, Kucukal E, et al. Assessment of stored red blood cells through lab-on-a-chip technologies for precision transfusion medicine. Proc Natl Acad Sci U S A 2023;120:e2115616120. https://doi.org/10.1073/pnas.2115616120
  8. Smit Sibinga CT. Transfusion medicine: from AB0 to AI (artificial intelligence). In: Linwood SL, ed. Digital health. Brisbane (AU). Exon Publications, 2022:107-20.
  9. Lopes MGM, Recktenwald SM, Simionato G, Eichler H, Wagner C, Quint S, et al. Big data in transfusion medicine and artificial intelligence analysis for red blood cell QC. Transfus Med Hemother 2023;50:163–73. https://doi.org/10.1159/000530458
  10. Levi R, Carli F, Arévalo AR, Altinel Y, Stein DJ, Naldini MM, et al. Artificial intelligence-based prediction of transfusion in the intensive care unit in patients with gastrointestinal bleeding. BMJ Health Care Inform 2021;28:e100245. https://doi.org/10.1136/bmjhci-2020-100245
  11. Jeong IH, Yu S, Kim TY, Oh SY, Cho D. Guide to Rho(D) immune globulin in women with molecularly defined Asian-type DEL (c.1227G>A). Ann Lab Med 2024;44:307–13. https://doi.org/10.3343/alm.2023.0356
  12. Ji Y, Luo Y, Wen J, Sun Y, Jia S, Ou C, et al. Patients with Asian-type DEL can safely be transfused with RhD-positive blood. Blood 2023;141:2141–50. https://doi.org/10.1182/blood.2022018152
  13. Wang L, Chen X, Deng X, Wen H, You M, Liu W, et al. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. NPJ Digit Med 2024;7:41. https://doi.org/10.1038/s41746-024-01029-4
  14. Lubušký M, Procházka M, Simetka O, Holusková I. [Guideline for prevention of RhD alloimmunization in RhD negative women]. Cesk Gynekol 2013;78:132–3.
  15. Ouchari M, Srivastava K, Romdhane H, Jemni Yacoub S, Flegel WA. Transfusion strategy for weak D type 4.0 based on RHD alleles and RH haplotypes in Tunisia. Transfusion 2018;58:306–12. https://doi.org/10.1111/trf.2018.58.issue-2
  16. Lim ZW, Pushpanathan K, Yew SME, Lai Y, Sun CH, Lam JSH, et al. Benchmarking large language models' performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine 2023;95:104770. https://doi.org/10.1016/j.ebiom.2023.104770
  17. Lecler A, Duron L, Soyer P. Revolutionizing radiology with GPT-based models: current applications, future possibilities and limitations of ChatGPT. Diagn Interv Imaging 2023;104:269–74. https://doi.org/10.1016/j.diii.2023.02.003
  18. Benary M, Wang XD, Schmidt M, Soll D, Hilfenhaus G, Nassir M, et al. Leveraging large language models for decision support in personalized oncology. JAMA Netw Open 2023;6:e2343689.
  19. Arslan S. Exploring the potential of Chat GPT in personalized obesity treatment. Ann Biomed Eng 2023;51:1887–8. https://doi.org/10.1007/s10439-023-03227-9
  20. Huang H, Zheng O, Wang D, Yin J, Wang Z, Ding S, et al. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. Int J Oral Sci 2023;15:29. https://doi.org/10.1038/s41368-023-00239-y
  21. Seghier ML. ChatGPT: not all languages are equal. Nature 2023;615:216.
  22. Madrid-García A, Rosales-Rosado Z, Freites-Nuñez D, Pérez-Sancristóbal I, Pato-Cour E, Plasencia-Rodríguez C, et al. Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training. Sci Rep 2023;13:22129. https://doi.org/10.1038/s41598-023-49483-6
  23. Wang H, Wu W, Dou Z, He L, Yang L. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inform 2023;177:105173. https://doi.org/10.1016/j.ijmedinf.2023.105173
  24. Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study. JMIR Med Educ 2023;9:e48002. https://doi.org/10.2196/48002
  25. Grimm DR, Lee YJ, Hu K, Liu L, Garcia O, Balakrishnan K, et al. The utility of ChatGPT as a generative medical translator. Eur Arch Otorhinolaryngol 2024;281:6161–5. https://doi.org/10.1007/s00405-024-08708-8
  26. Abdurahman S, Atari M, Karimi-Malekabadi F, Xue MJ, Trager J, Park PS, et al. Perils and opportunities in using large language models in psychological research. PNAS Nexus 2024;3:pgae245.
  27. Giannakopoulos K, Kavadella A, Aaqel Salim A, Stamatopoulos V, Kaklamanos EG. Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: comparative mixed methods study. J Med Internet Res 2023;25:e51580.
  28. Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res 2023;25:e50638. https://doi.org/10.2196/50638
  29. Fink MA, Bischoff A, Fink CA, Moll M, Kroschke J, Dulz L, et al. Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer. Radiology 2023;308:e231362.
  30. Shin E and Ramanathan M. Evaluation of prompt engineering strategies for pharmacokinetic data analysis with the ChatGPT large language model. J Pharmacokinet Pharmacodyn 2024;51:101–8. https://doi.org/10.1007/s10928-023-09892-6
  31. Bateman RM, Sharpe MD, Jagger JE, Ellis CG, Sole-Violan J, Lopez-Rodriguez M, et al. 36th International Symposium on Intensive Care and Emergency Medicine: Brussels, Belgium. 15-18 March 2016. Crit Care 2016;20(S2):94.
  32. Yazer M, Triulzi D, Sperry J, Corcos A, Seheult J. Rate of RhD-alloimmunization after the transfusion of RhD-positive red blood cell containing products among injured patients of childbearing age: single center experience and narrative literature review. Hematology 2021;26:321–7. https://doi.org/10.1080/16078454.2021.1905395
  33. Werch J and Todd C. Resolution by erythrocytapheresis of the exposure of an Rh-negative person to Rh-positive cells: an alternative treatment. Transfusion 1993;33:530–2. https://doi.org/10.1046/j.1537-2995.1993.33693296819.x
  34. Practice Bulletin No. 181: Prevention of Rh D alloimmunization. Obstet Gynecol 2017;130:e57–70. https://doi.org/10.1097/AOG.0000000000002232
  35. Subramaniyan R and Pereira K. Is antenatal RhIg completely safe? Rev Bras Hematol Hemoter 2016;38:161–2. https://doi.org/10.1016/j.bjhh.2016.01.006
  36. Longwell JB, Hirsch I, Binder F, Gonzalez Conchas GA, Mau D, Jang R, et al. Performance of large language models on medical oncology examination questions. JAMA Netw Open 2024;7:e2417641.
  37. Wagner FF, Frohmajer A, Flegel WA. RHD positive haplotypes in D negative Europeans. BMC Genet 2001;2:10. https://doi.org/10.1186/1471-2156-2-10
  38. Ma SL, Vega-Warner V, Gillies C, Sampson MG, Kher V, Sethi SK, et al. Whole exome sequencing reveals novel PHEX splice site mutations in patients with hypophosphatemic rickets. PLoS One 2015;10:e0130729. https://doi.org/10.1371/journal.pone.0130729
  39. Ohto H, Ito S, Srivastava K, Ogiyama Y, Uchikawa M, Nollet KE, et al. Asian-type DEL (RHD*DEL1) with an allo-anti-D: a paradoxical observation in a healthy multiparous woman. Transfusion 2023;63:1601–11. https://doi.org/10.1111/trf.v63.8
  40. Kang T, Choi R, Kim D, Cho D, Kim DW. A Case of Bidirectional ABO- and RhD-Incompatible Liver Transplantation in a Mongolian Patient With Asian-Type DEL. Ann Lab Med 2025;45:109–11. https://doi.org/10.3343/alm.2024.0399