Journal of Korean Medical Science

Korean Acad Medical Sciences (대한의학회)
권호 표지
DOI QR코드

DOI QR Code

A Novel Fundus Image Reading Tool for Efficient Generation of a Multi-dimensional Categorical Image Database for Machine Learning Algorithm Training

  • Park, Sang Jun (Department of Ophthalmology, Seoul National University Bundang Hospital, Seoul National University College of Medicine) ;
  • Shin, Joo Young (Department of Ophthalmology, Dongguk University Ilsan Hospital) ;
  • Kim, Sangkeun (VUNO Inc.) ;
  • Son, Jaemin (VUNO Inc.) ;
  • Jung, Kyu-Hwan (VUNO Inc.) ;
  • Park, Kyu Hyung (Department of Ophthalmology, Seoul National University Bundang Hospital, Seoul National University College of Medicine)
  • Received : 2018.05.04
  • Accepted : 2018.07.10
  • Published : 2018.10.22
⟨ Previous Next ⟩

Abstract

Background: We described a novel multi-step retinal fundus image reading system for providing high-quality large data for machine learning algorithms, and assessed the grader variability in the large-scale dataset generated with this system. Methods: A 5-step retinal fundus image reading tool was developed that rates image quality, presence of abnormality, findings with location information, diagnoses, and clinical significance. Each image was evaluated by 3 different graders. Agreements among graders for each decision were evaluated. Results: The 234,242 readings of 79,458 images were collected from 55 licensed ophthalmologists during 6 months. The 34,364 images were graded as abnormal by at-least one rater. Of these, all three raters agreed in 46.6% in abnormality, while 69.9% of the images were rated as abnormal by two or more raters. Agreement rate of at-least two raters on a certain finding was 26.7%-65.2%, and complete agreement rate of all-three raters was 5.7%-43.3%. As for diagnoses, agreement of at-least two raters was 35.6%-65.6%, and complete agreement rate was 11.0%-40.0%. Agreement of findings and diagnoses were higher when restricted to images with prior complete agreement on abnormality. Retinal/glaucoma specialists showed higher agreements on findings and diagnoses of their corresponding subspecialties. Conclusion: This novel reading tool for retinal fundus images generated a large-scale dataset with high level of information, which can be utilized in future development of machine learning-based algorithms for automated identification of abnormal conditions and clinical decision supporting system. These results emphasize the importance of addressing grader variability in algorithm developments.

Keywords

References

  1. Park SH, Kressel HY. Connecting technological innovation in artificial intelligence to real-world medical practice through rigorous clinical validation: what peer-reviewed medical journals could do. J Korean Med Sci 2018;33(22):e152. https://doi.org/10.3346/jkms.2018.33.e152
  2. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016;316(22):2402-10. https://doi.org/10.1001/jama.2016.17216
  3. Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology 2017;124(7):962-9. https://doi.org/10.1016/j.ophtha.2017.02.008
  4. Abramoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci 2016;57(13):5200-6. https://doi.org/10.1167/iovs.16-19964
  5. Ting DS, Cheung CY, Lim G, Tan GS, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 2017;318(22):2211-23. https://doi.org/10.1001/jama.2017.18152
  6. Takahashi H, Tampo H, Arai Y, Inoue Y, Kawashima H. Applying artificial intelligence to disease staging: deep learning for improved staging of diabetic retinopathy. PLoS One 2017;12(6):e0179790. https://doi.org/10.1371/journal.pone.0179790
  7. Choi JY, Yoo TK, Seo JG, Kwak J, Um TT, Rim TH. Multi-categorical deep learning neural network to classify retinal images: a pilot study employing small database. PLoS One 2017;12(11):e0187336. https://doi.org/10.1371/journal.pone.0187336
  8. Quellec G, Charriere K, Boudi Y, Cochener B, Lamard M. Deep image mining for diabetic retinopathy screening. Med Image Anal 2017;39:178-93. https://doi.org/10.1016/j.media.2017.04.012
  9. Wong TY, Bressler NM. Artificial intelligence with deep learning technology looks into diabetic retinopathy screening. JAMA 2016;316(22):2366-7. https://doi.org/10.1001/jama.2016.17563
  10. Arunkumar R, Karthigaikumar P. Multi-retinal disease classification by reduced deep learning features. Neural Comput Appl 2017;28(2):329-34. https://doi.org/10.1007/s00521-015-2059-9
  11. Quellec G, Lamard M, Josselin PM, Cazuguel G, Cochener B, Roux C. Optimal wavelet transform for the detection of microaneurysms in retina photographs. IEEE Trans Med Imaging 2008;27(9):1230-41. https://doi.org/10.1109/TMI.2008.920619
  12. Decenciere E, Zhang X, Cazuguel G, Lay B, Cochener B, Trone C, et al. Feedback on a publicly distributed image database: the Messidor database. Image Anal Stereol 2014;33(3):231-4. https://doi.org/10.5566/ias.1155
  13. Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 2018;125(8):1264-72. https://doi.org/10.1016/j.ophtha.2018.01.034

Cited by

  1. Artificial Intelligence in Ophthalmology: Evolutions in Asia vol.9, pp.2, 2018, https://doi.org/10.1097/01.apo.0000656980.41190.bf
  2. Predicting High Coronary Artery Calcium Score From Retinal Fundus Images With Deep Learning Algorithms vol.9, pp.2, 2020, https://doi.org/10.1167/tvst.9.2.28
  3. Evaluating the Viability of a Smartphone-Based Annotation Tool for Faster and Accurate Image Labelling for Artificial Intelligence in Diabetic Retinopathy vol.15, pp.None, 2018, https://doi.org/10.2147/opth.s289425
  4. Leveraging the Generalization Ability of Deep Convolutional Neural Networks for Improving Classifiers for Color Fundus Photographs vol.11, pp.2, 2018, https://doi.org/10.3390/app11020591
  5. Development of Decision Support Software for Deep Learning-Based Automated Retinal Disease Screening Using Relatively Limited Fundus Photograph Data vol.10, pp.2, 2018, https://doi.org/10.3390/electronics10020163
  6. Fully Leveraging Deep Learning Methods for Constructing Retinal Fundus Photomontages vol.11, pp.4, 2018, https://doi.org/10.3390/app11041754