Korean Journal of Radiology

The Korean Society of Radiology (대한영상의학회)
권호 표지
DOI QR코드

DOI QR Code

Reproducibility and Generalizability in Radiomics Modeling: Possible Strategies in Radiologic and Statistical Perspectives

  • Park, Ji Eun (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Park, Seo Young (Department of Clinical Epidemiology and Biostatistics, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Kim, Hwa Jung (Department of Clinical Epidemiology and Biostatistics, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Kim, Ho Sung (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center)
  • Received : 2019.01.25
  • Accepted : 2019.04.07
  • Published : 2019.07.01
⟨ Previous Next ⟩

Abstract

Radiomics, which involves the use of high-dimensional quantitative imaging features for predictive purposes, is a powerful tool for developing and testing medical hypotheses. Radiologic and statistical challenges in radiomics include those related to the reproducibility of imaging data, control of overfitting due to high dimensionality, and the generalizability of modeling. The aims of this review article are to clarify the distinctions between radiomics features and other omics and imaging data, to describe the challenges and potential strategies in reproducibility and feature selection, and to reveal the epidemiological background of modeling, thereby facilitating and promoting more reproducible and generalizable radiomics research.

Keywords

Acknowledgement

This study was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (grant number: NRF-2017R1C1B2007258 and NRF2017R1A2A2A05001217).

References

  1. Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology 2016;278:563-577 https://doi.org/10.1148/radiol.2015151169
  2. Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RG, Granton P, et al. Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 2012;48:441-446 https://doi.org/10.1016/j.ejca.2011.11.036
  3. Micheel CM, Nass SJ, Omenn GS; Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials; Board on Health Care Services; Board on Health Sciences Policy; Institute of Medicine. Evolution of translational omics: lessons learned and the path forward. Washington, DC: The National Academies Press, 2012
  4. Diehn M, Nardini C, Wang DS, McGovern S, Jayaraman M, Liang Y, et al. Identification of noninvasive imaging surrogates for brain tumor gene-expression modules. Proc Natl Acad Sci U S A 2008;105:5213-5218 https://doi.org/10.1073/pnas.0801279105
  5. Segal E, Sirlin CB, Ooi C, Adler AS, Gollub J, Chen X, et al. Decoding global gene expression programs in liver cancer by noninvasive imaging. Nat Biotechnol 2007;25:675-680 https://doi.org/10.1038/nbt1306
  6. Yip SS, Aerts HJ. Applications and limitations of radiomics. Phys Med Biol 2016;61:R150-R166 https://doi.org/10.1088/0031-9155/61/13/R150
  7. Cook GJR, Azad G, Owczarczyk K, Siddique M, Goh V. Challenges and promises of PET radiomics. Int J Radiat Oncol Biol Phys 2018;102:1083-1089 https://doi.org/10.1016/j.ijrobp.2017.12.268
  8. Limkin EJ, Sun R, Dercle L, Zacharaki EI, Robert C, Reuze S, et al. Promises and challenges for the implementation of computational medical imaging (radiomics) in oncology. Ann Oncol 2017;28:1191-1206 https://doi.org/10.1093/annonc/mdx034
  9. Kumar V, Gu Y, Basu S, Berglund A, Eschrich SA, Schabath MB, et al. Radiomics: the process and the challenges. Magn Reson Imaging 2012;30:1234-1248 https://doi.org/10.1016/j.mri.2012.06.010
  10. Park JE, Kim HS. Radiomics as a quantitative imaging biomarker: practical considerations and the current standpoint in neuro-oncologic studies. Nucl Med Mol Imaging 2018;52:99-108 https://doi.org/10.1007/s13139-017-0512-7
  11. Jain R, Lui YW. How far are we from using radiomics assessment of gliomas in clinical practice? Radiology 2018;289:807-808 https://doi.org/10.1148/radiol.2018182033
  12. Lee G, Lee HY, Park H, Schiebler ML, van Beek EJR, Ohno Y, et al. Radiomics and its emerging role in lung cancer research, imaging biomarkers and clinical management: State of the art. Eur J Radiol 2017;86:297-307 https://doi.org/10.1016/j.ejrad.2016.09.005
  13. Kristensen VN, Lingjaerde OC, Russnes HG, Vollan HK, Frigessi A, Borresen-Dale AL. Principles and methods of integrative genomic analyses in cancer. Nat Rev Cancer 2014;14:299-313 https://doi.org/10.1038/nrc3721
  14. Clarke R, Ressom HW, Wang A, Xuan J, Liu MC, Gehan EA, et al. The properties of high-dimensional data spaces: implications for exploring gene and protein expression data. Nat Rev Cancer 2008;8:37-49 https://doi.org/10.1038/nrc2294
  15. Jain AK, Duin RP, Mao J. Statistical pattern recognition: a review. IEEE Trans Pattern Anal Mach Intell 2000;22:4-37 https://doi.org/10.1109/34.824819
  16. Bellman RE. Adaptive control processes: a guided tour. Princeton, NJ: Princeton university press, 2015
  17. Ferte C, Trister AD, Huang E, Bot BM, Guinney J, Commo F, et al. Impact of bioinformatic procedures in the development and translation of high-throughput molecular classifiers in oncology. Clin Cancer Res 2013;19:4315-4325 https://doi.org/10.1158/1078-0432.CCR-12-3937
  18. Genders TS, Spronk S, Stijnen T, Steyerberg EW, Lesaffre E, Hunink MG. Methods for calculating sensitivity and specificity of clustered data: a tutorial. Radiology 2012;265:910-916 https://doi.org/10.1148/radiol.12120509
  19. Lin YC, Lin G, Hong JH, Lin YP, Chen FH, Ng SH, et al. Diffusion radiomics analysis of intratumoral heterogeneity in a murine prostate cancer model following radiotherapy: pixelwise correlation with histology. J Magn Reson Imaging 2017;46:483-489 https://doi.org/10.1002/jmri.25583
  20. O'Connor JP, Aboagye EO, Adams JE, Aerts HJ, Barrington SF, Beer AJ, et al. Imaging biomarker roadmap for cancer studies. Nat Rev Clin Oncol 2017;14:169-186 https://doi.org/10.1038/nrclinonc.2016.162
  21. Zinn PO, Singh SK, Kotrotsou A, Hassan I, Thomas G, Luedi MM, et al. A coclinical radiogenomic validation study: conserved magnetic resonance radiomic appearance of periostin-expressing glioblastoma in patients and xenograft models. Clin Cancer Res 2018;24:6288-6299 https://doi.org/10.1158/1078-0432.CCR-17-3420
  22. Cook TD, Campbell DT. Quasi-experimentation: design & analysis issues for field settings. Chicago, IL: Rand McNally, 1979
  23. Ferguson L. External validity, generalizability, and knowledge utilization. J Nurs Scholarsh 2004;36:16-22 https://doi.org/10.1111/j.1547-5069.2004.04006.x
  24. Murad MH, Katabi A, Benkhadra R, Montori VM. External validity, generalisability, applicability and directness: a brief primer. BMJ Evid Based Med 2018;23:17-19 https://doi.org/10.1136/ebmed-2017-110800
  25. Raunig DL, McShane LM, Pennello G, Gatsonis C, Carson PL, Voyvodic JT, et al.; QIBA Technical Performance Working Group. Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment. Stat Methods Med Res 2015;24:27-67 https://doi.org/10.1177/0962280214537344
  26. Kessler LG, Barnhart HX, Buckler AJ, Choudhury KR, Kondratovich MV, Toledano A, et al.; QIBA Terminology Working Group. The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions. Stat Methods Med Res 2015;24:9-26 https://doi.org/10.1177/0962280214537333
  27. Stodden V, Guo P, Ma Z. Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS One 2013;8:e67111 https://doi.org/10.1371/journal.pone.0067111
  28. Stodden V, Leisch F, Peng RD. Implementing reproducible research, 1st ed. Boca Raton, FL: CRC Press, 2014
  29. Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Carvalho S, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun 2014;5:4006 https://doi.org/10.1038/ncomms5006
  30. Antunes J, Viswanath S, Rusu M, Valls L, Hoimes C, Avril N, et al. Radiomics analysis on FLT-PET/MRI for characterization of early treatment response in renal cell carcinoma: a proof-of-concept study. Transl Oncol 2016;9:155-162 https://doi.org/10.1016/j.tranon.2016.01.008
  31. Balagurunathan Y, Kumar V, Gu Y, Kim J, Wang H, Liu Y, et al. Test-retest reproducibility analysis of lung CT image features. J Digit Imaging 2014;27:805-823 https://doi.org/10.1007/s10278-014-9716-x
  32. Gevaert O, Mitchell LA, Achrol AS, Xu J, Echegaray S, Steinberg GK, et al. Glioblastoma multiforme: exploratory radiogenomic analysis by using quantitative image features. Radiology 2014;273:168-174 https://doi.org/10.1148/radiol.14131731
  33. Hunter LA, Krafft S, Stingo F, Choi H, Martel MK, Kry SF, et al. High quality machine-robust image features: identification in nonsmall cell lung cancer computed tomography images. Med Phys 2013;40:121916 https://doi.org/10.1118/1.4829514
  34. Leijenaar RT, Carvalho S, Velazquez ER, van Elmpt WJ, Parmar C, Hoekstra OS, et al. Stability of FDG-PET Radiomics features: an integrated analysis of test-retest and inter-observer variability. Acta Oncol 2013;52:1391-1397 https://doi.org/10.3109/0284186X.2013.812798
  35. Tixier F, Hatt M, Le Rest CC, Le Pogam A, Corcos L, Visvikis D. Reproducibility of tumor uptake heterogeneity characterization through textural feature analysis in 18F-FDG PET. J Nucl Med 2012;53:693-700 https://doi.org/10.2967/jnumed.111.099127
  36. van Velden FH, Kramer GM, Frings V, Nissen IA, Mulder ER, de Langen AJ, et al. Repeatability of radiomic features in non-small-cell lung cancer [(18)F]FDG-PET/CT studies: impact of reconstruction and delineation. Mol Imaging Biol 2016;18:788-795 https://doi.org/10.1007/s11307-016-0940-2
  37. van Velden FH, Nissen IA, Jongsma F, Velasquez LM, Hayes W, Lammertsma AA, et al. Test-retest variability of various quantitative measures to characterize tracer uptake and/or tracer uptake heterogeneity in metastasized liver for patients with colorectal carcinoma. Mol Imaging Biol 2014;16:13-18 https://doi.org/10.1007/s11307-013-0660-9
  38. Zhao B, James LP, Moskowitz CS, Guo P, Ginsberg MS, Lefkowitz RA, et al. Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology 2009;252:263-272 https://doi.org/10.1148/radiol.2522081593
  39. Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. Measuring computed tomography scanner variability of radiomics features. Invest Radiol 2015;50:757-765 https://doi.org/10.1097/RLI.0000000000000180
  40. van Timmeren JE, Leijenaar RTH, van Elmpt W, Wang J, Zhang Z, Dekker A, et al. Test-retest data for radiomics feature stability analysis: generalizable or study-specific? Tomography 2016;2:361-365 https://doi.org/10.18383/j.tom.2016.00208
  41. Berenguer R, Pastor-Juan MDR, Canales-Vazquez J, Castro-Garcia M, Villas MV, Mansilla Legorburo F, et al. Radiomics of CT features may be nonreproducible and redundant: influence of CT acquisition parameters. Radiology 2018;288:407-415 https://doi.org/10.1148/radiol.2018172361
  42. Galavis PE, Hollensen C, Jallow N, Paliwal B, Jeraj R. Variability of textural features in FDG PET images due to different acquisition modes and reconstruction parameters. Acta Oncol 2010;49:1012-1016 https://doi.org/10.3109/0284186X.2010.498437
  43. He L, Huang Y, Ma Z, Liang C, Liang C, Liu Z. Effects of contrast-enhancement, reconstruction slice thickness and convolution kernel on the diagnostic performance of radiomics signature in solitary pulmonary nodule. Sci Rep 2016;6:34921 https://doi.org/10.1038/srep34921
  44. Lu L, Ehmke RC, Schwartz LH, Zhao B. Assessing agreement between radiomic features computed for multiple CT imaging settings. PLoS One 2016;11:e0166550 https://doi.org/10.1371/journal.pone.0166550
  45. Kickingereder P, Neuberger U, Bonekamp D, Piechotta PL, Gotz M, Wick A, et al. Radiomic subtyping improves disease stratification beyond key molecular, clinical, and standard imaging characteristics in patients with glioblastoma. Neuro Oncol 2018;20:848-857 https://doi.org/10.1093/neuonc/nox188
  46. Parmar C, Rios Velazquez E, Leijenaar R, Jermoumi M, Carvalho S, Mak RH, et al. Robust radiomics feature quantification using semiautomatic volumetric segmentation. PLoS One 2014;9:e102107 https://doi.org/10.1371/journal.pone.0102107
  47. Pavic M, Bogowicz M, Wurms X, Glatz S, Finazzi T, Riesterer O, et al. Influence of inter-observer delineation variability on radiomics stability in different tumor sites. Acta Oncol 2018;57:1070-1074 https://doi.org/10.1080/0284186X.2018.1445283
  48. Bogowicz M, Riesterer O, Bundschuh RA, Veit-Haibach P, Hullner M, Studer G, et al. Stability of radiomic features in CT perfusion maps. Phys Med Biol 2016;61:8736-8749 https://doi.org/10.1088/1361-6560/61/24/8736
  49. Li Q, Bai H, Chen Y, Sun Q, Liu L, Zhou S, et al. A fully-automatic multiparametric radiomics model: towards reproducible and prognostic imaging signature for prediction of overall survival in glioblastoma multiforme. Sci Rep 2017;7:14331 https://doi.org/10.1038/s41598-017-14753-7
  50. Lu L, Lv W, Jiang J, Ma J, Feng Q, Rahmim A, et al. Robustness of radiomic features in [11C]choline and [18F] FDG PET/CT imaging of nasopharyngeal carcinoma: impact of segmentation and discretization. Mol Imaging Biol 2016;18:935-945 https://doi.org/10.1007/s11307-016-0973-6
  51. Traverso A, Wee L, Dekker A, Gillies R. EP-2132: repeatability and reproducibility of radiomic features: results of a systematic review. Radiother Oncol 2018;127:S1174-S1175 https://doi.org/10.1016/S0167-8140(18)32441-1
  52. Hatt M, Majdoub M, Vallieres M, Tixier F, Le Rest CC, Groheux D, et al. 18F-FDG PET uptake characterization through texture analysis: investigating the complementary nature of heterogeneity and functional tumor volume in a multi-cancer site patient cohort. J Nucl Med 2015;56:38-44 https://doi.org/10.2967/jnumed.114.144055
  53. Kang D, Park JE, Kim YH, Kim JH, Oh JY, Kim J, et al. Diffusion radiomics as a diagnostic model for atypical manifestation of primary central nervous system lymphoma: development and multicenter external validation. Neuro Oncol 2018;20:1251-1261 https://doi.org/10.1093/neuonc/noy021
  54. Kim JY, Park JE, Jo Y, Shim WH, Nam SJ, Kim JH, et al. Incorporating diffusion- and perfusion-weighted MRI into a radiomics model improves diagnostic performance for pseudoprogression in glioblastoma patients. Neuro Oncol 2019;21:404-414 https://doi.org/10.1093/neuonc/noy133
  55. Beukinga RJ, Hulshoff JB, Mul VEM, Noordzij W, Kats-Ugurlu G, Slart RHJA, et al. Prediction of response to neoadjuvant chemotherapy and radiation therapy with baseline and restaging 18F-FDG PET imaging biomarkers in patients with esophageal cancer. Radiology 2018;287:983-992 https://doi.org/10.1148/radiol.2018172229
  56. Boldrini L, Cusumano D, Chiloiro G, Casa C, Masciocchi C, Lenkowicz J, et al. Delta radiomics for rectal cancer response prediction with hybrid 0.35 T magnetic resonance-guided radiotherapy (MRgRT): a hypothesis-generating study for an innovative personalized medicine approach. Radiol Med 2019;124:145-153 https://doi.org/10.1007/s11547-018-0951-y
  57. Crombe A, Perier C, Kind M, De Senneville BD, Le Loarer F, Italiano A, et al. T2-based MRI delta-radiomics improve response prediction in soft-tissue sarcomas treated by neoadjuvant chemotherapy. J Magn Reson Imaging 2018 Dec 19 [Epub ahead of print]. https://doi.org/10.1002/jmri.26589
  58. Fave X, Zhang L, Yang J, Mackin D, Balter P, Gomez D, et al. Delta-radiomics features for the prediction of patient outcomes in non-small cell lung cancer. Sci Rep 2017;7:588 https://doi.org/10.1038/s41598-017-00665-z
  59. Mazzei MA, Nardone V, Di Giacomo L, Bagnacci G, Gentili F, Tini P, et al. The role of delta radiomics in gastric cancer. Quant Imaging Med Surg 2018;8:719-721 https://doi.org/10.21037/qims.2018.07.08
  60. Lambin P, Leijenaar RTH, Deist TM, Peerlings J, de Jong EEC, van Timmeren J, et al. Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol 2017;14:749-762 https://doi.org/10.1038/nrclinonc.2017.141
  61. Doumou G, Siddique M, Tsoumpas C, Goh V, Cook GJ. The precision of textural analysis in (18)F-FDG-PET scans of oesophageal cancer. Eur Radiol 2015;25:2805-2812 https://doi.org/10.1007/s00330-015-3681-8
  62. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, et al. Brain tumor segmentation with Deep Neural Networks. Med Image Anal 2017;35:18-31 https://doi.org/10.1016/j.media.2016.05.004
  63. Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Trans Med Imaging 2015;34:1993-2024 https://doi.org/10.1109/TMI.2014.2377694
  64. van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res 2017;77:e104-e107 https://doi.org/10.1158/0008-5472.CAN-17-0339
  65. Johnson WE, Li C, Rabinovic A. Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics 2007;8:118-127 https://doi.org/10.1093/biostatistics/kxj037
  66. Orlhac F, Boughdad S, Philippe C, Stalla-Bourdillon H, Nioche C, Champion L, et al. A postreconstruction harmonization method for multicenter radiomic studies in PET. J Nucl Med 2018;59:1321-1328 https://doi.org/10.2967/jnumed.117.199935
  67. Orlhac F, Frouin F, Nioche C, Ayache N, Buvat I. Validation of a method to compensate multicenter effects affecting CT radiomics. Radiology 2019;291:53-59 https://doi.org/10.1148/radiol.2019182023
  68. Saeys Y, Inza I, Larranaga P. A review of feature selection techniques in bioinformatics. Bioinformatics 2007;23:2507-2517 https://doi.org/10.1093/bioinformatics/btm344
  69. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Series B Stat Methodol 1995;57:289-300
  70. Tusher VG, Tibshirani R, Chu G. Significance analysis of microarrays applied to the ionizing radiation response. Proc Natl Acad Sci U S A 2001;98:5116-5121 https://doi.org/10.1073/pnas.091062498
  71. Reiner A, Yekutieli D, Benjamini Y. Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 2003;19:368-375 https://doi.org/10.1093/bioinformatics/btf877
  72. Storey JD. The positive false discovery rate: a Bayesian interpretation and the q-value. Ann Stat 2003;31:2013-2035 https://doi.org/10.1214/aos/1074290335
  73. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. Berlin/Heidelberg: Springer Science & Business Media, 2008
  74. Shmueli G. To explain or to predict? Statistical Science 2010;25:289-310 https://doi.org/10.1214/10-STS330
  75. Gordis L. Assessing the validity and reliability of diagnostic and screening tests. In: Gordis L, ed. Epidemiology, 5th ed. London: Elsevier Health Sciences, 2013
  76. Park SH. Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance. Radiology 2019;290:272-273 https://doi.org/10.1148/radiol.2018182294
  77. Park SH, Han K. Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 2018;286:800-809 https://doi.org/10.1148/radiol.2017171920

Cited by

  1. Prediagnostic Image Data, Artificial Intelligence, and Pancreatic Cancer : A Tell-Tale Sign to Early Detection vol.49, pp.7, 2020, https://doi.org/10.1097/mpa.0000000000001603
  2. Comparison of Diagnostic Performance of Two-Dimensional and Three-Dimensional Fractal Dimension and Lacunarity Analyses for Predicting the Meningioma Grade vol.8, pp.1, 2019, https://doi.org/10.14791/btrt.2020.8.e3
  3. Radiomics and Deep Learning: Hepatic Applications vol.21, pp.4, 2019, https://doi.org/10.3348/kjr.2019.0752
  4. Radiomics in Breast Imaging from Techniques to Clinical Applications: A Review vol.21, pp.7, 2019, https://doi.org/10.3348/kjr.2019.0855
  5. Performance of Prediction Models for Diagnosing Severe Aortic Stenosis Based on Aortic Valve Calcium on Cardiac Computed Tomography: Incorporation of Radiomics and Machine Learning vol.21, 2019, https://doi.org/10.3348/kjr.2020.0099
  6. Quality Reporting of Radiomics Analysis in Mild Cognitive Impairment and Alzheimer's Disease: A Roadmap for Moving Forward vol.21, pp.12, 2019, https://doi.org/10.3348/kjr.2020.0715
  7. Characteristics of Recent Articles Published in the Korean Journal of Radiology Based on the Citation Frequency vol.21, pp.12, 2020, https://doi.org/10.3348/kjr.2020.1322
  8. Radiomics Signatures of Cardiovascular Risk Factors in Cardiac MRI: Results From the UK Biobank vol.7, 2019, https://doi.org/10.3389/fcvm.2020.591368
  9. Prediction of Core Signaling Pathway by Using Diffusion- and Perfusion-based MRI Radiomics and Next-generation Sequencing in Isocitrate Dehydrogenase Wild-type Glioblastoma vol.294, pp.2, 2020, https://doi.org/10.1148/radiol.2019190913
  10. Multiparametric radiomics methods for breast cancer tissue characterization using radiological imaging vol.180, pp.2, 2020, https://doi.org/10.1007/s10549-020-05533-5
  11. Current Applications and Future Perspectives of Brain Tumor Imaging vol.81, pp.3, 2020, https://doi.org/10.3348/jksr.2020.81.3.467
  12. CT-Based Radiomics Analysis to Predict Malignancy in Patients with Intraductal Papillary Mucinous Neoplasm (IPMN) of the Pancreas vol.12, pp.11, 2020, https://doi.org/10.3390/cancers12113089
  13. FAIR‐compliant clinical, radiomics and DICOM metadata of RIDER, interobserver, Lung1 and head‐Neck1 TCIA collections vol.47, pp.11, 2019, https://doi.org/10.1002/mp.14322
  14. Comparison of machine learning algorithms to predict clinically significant prostate cancer of the peripheral zone with multiparametric MRI using clinical assessment categories and radiomic features vol.30, pp.12, 2020, https://doi.org/10.1007/s00330-020-07064-5
  15. Reliability of CT radiomic features reflecting tumour heterogeneity according to image quality and image processing parameters vol.10, 2019, https://doi.org/10.1038/s41598-020-60868-9
  16. 3D-MCN: A 3D Multi-scale Capsule Network for Lung Nodule Malignancy Prediction vol.10, 2020, https://doi.org/10.1038/s41598-020-64824-5
  17. Deep-learning-based image quality enhancement of compressed sensing magnetic resonance imaging of vessel wall: comparison of self-supervised and unsupervised approaches vol.10, pp.1, 2020, https://doi.org/10.1038/s41598-020-69932-w
  18. Germline BRCA 1-2 status prediction through ovarian ultrasound images radiogenomics: a hypothesis generating study (PROBE study) vol.10, pp.1, 2019, https://doi.org/10.1038/s41598-020-73505-2
  19. Radiomics features of hippocampal regions in magnetic resonance imaging can differentiate medial temporal lobe epilepsy patients from healthy controls vol.10, pp.1, 2019, https://doi.org/10.1038/s41598-020-76283-z
  20. How to develop a meaningful radiomic signature for clinical use in oncologic patients vol.20, pp.1, 2019, https://doi.org/10.1186/s40644-020-00311-4
  21. Future artificial intelligence tools and perspectives in medicine vol.31, pp.4, 2019, https://doi.org/10.1097/mou.0000000000000884
  22. Volumetric CT Texture Analysis of Intrahepatic Mass-Forming Cholangiocarcinoma for the Prediction of Postoperative Outcomes: Fully Automatic Tumor Segmentation Versus Semi-Automatic Segmentation vol.22, 2021, https://doi.org/10.3348/kjr.2021.0055
  23. Spatiotemporal Heterogeneity in Multiparametric Physiologic MRI Is Associated with Patient Outcomes in IDH-Wildtype Glioblastoma vol.27, pp.1, 2019, https://doi.org/10.1158/1078-0432.ccr-20-2156
  24. Predictive performance of ultrasonography-based radiomics for axillary lymph node metastasis in the preoperative evaluation of breast cancer vol.40, pp.1, 2021, https://doi.org/10.14366/usg.20026
  25. Identification of robust and reproducible CT‐texture metrics using a customized 3D‐printed texture phantom vol.22, pp.2, 2019, https://doi.org/10.1002/acm2.13162
  26. Applications of artificial intelligence in pancreatic and biliary diseases vol.36, pp.2, 2019, https://doi.org/10.1111/jgh.15380
  27. Magnetic Resonance Imaging Based Radiomic Models of Prostate Cancer: A Narrative Review vol.13, pp.3, 2019, https://doi.org/10.3390/cancers13030552
  28. Understanding Sources of Variation to Improve the Reproducibility of Radiomics vol.11, 2019, https://doi.org/10.3389/fonc.2021.633176
  29. Diagnostic Performance of 2D and 3D T2WI-Based Radiomics Features With Machine Learning Algorithms to Distinguish Solid Solitary Pulmonary Lesion vol.11, 2019, https://doi.org/10.3389/fonc.2021.683587
  30. Radiomics for Identification and Prediction in Metastatic Prostate Cancer: A Review of Studies vol.11, 2019, https://doi.org/10.3389/fonc.2021.771787
  31. MR-Guided Hypofractionated Radiotherapy: Current Emerging Data and Promising Perspectives for Localized Prostate Cancer vol.13, pp.8, 2021, https://doi.org/10.3390/cancers13081791
  32. Radiomic biomarkers of tumor immune biology and immunotherapy response vol.28, 2019, https://doi.org/10.1016/j.ctro.2021.03.006
  33. HCC advances in diagnosis and prognosis: Digital and Imaging vol.41, pp.suppl, 2019, https://doi.org/10.1111/liv.14865
  34. Impact of Preprocessing and Harmonization Methods on the Removal of Scanner Effects in Brain MRI Radiomic Features vol.13, pp.12, 2021, https://doi.org/10.3390/cancers13123000
  35. Artificial Intelligence Based Algorithms for Prostate Cancer Classification and Detection on Magnetic Resonance Imaging: A Narrative Review vol.11, pp.6, 2019, https://doi.org/10.3390/diagnostics11060959
  36. Radiomics-based model for predicting pathological complete response to neoadjuvant chemotherapy in muscle-invasive bladder cancer vol.76, pp.8, 2019, https://doi.org/10.1016/j.crad.2021.03.001
  37. Deep Learning Segmentation of Triple-Negative Breast Cancer (TNBC) Patient Derived Tumor Xenograft (PDX) and Sensitivity of Radiomic Pipeline to Tumor Probability Boundary vol.13, pp.15, 2019, https://doi.org/10.3390/cancers13153795
  38. Multivariate Machine Learning Analyses in Identification of Major Depressive Disorder Using Resting-State Functional Connectivity: A Multicentral Study vol.12, pp.15, 2019, https://doi.org/10.1021/acschemneuro.1c00256
  39. Bone Radiomics Score Derived From DXA Hip Images Enhances Hip Fracture Prediction in Older Women vol.36, pp.9, 2019, https://doi.org/10.1002/jbmr.4342
  40. Radiomics of MRI for the Prediction of the Pathological Response to Neoadjuvant Chemotherapy in Breast Cancer Patients: A Single Referral Centre Analysis vol.13, pp.17, 2019, https://doi.org/10.3390/cancers13174271
  41. A systematic review and quality of reporting checklist for repeatability and reproducibility of radiomic features vol.20, 2019, https://doi.org/10.1016/j.phro.2021.10.007
  42. Radiomics, mirnomics, and radiomirRNomics in glioblastoma: defining tumor biology from shadow to light vol.21, pp.11, 2019, https://doi.org/10.1080/14737140.2021.1971518
  43. Radiomics side experiments and DAFIT approach in identifying pulmonary hypertension using Cardiac MRI derived radiomics based machine learning models vol.11, pp.1, 2019, https://doi.org/10.1038/s41598-021-92155-6
  44. Impact of rescanning and repositioning on radiomic features employing a multi-object phantom in magnetic resonance imaging vol.11, pp.1, 2019, https://doi.org/10.1038/s41598-021-93756-x
  45. Stability of MRI radiomic features according to various imaging parameters in fast scanned T2-FLAIR for acute ischemic stroke patients vol.11, pp.1, 2019, https://doi.org/10.1038/s41598-021-96621-z
  46. Delta radiomics analysis of Magnetic Resonance guided radiotherapy imaging data can enable treatment response prediction in pancreatic cancer vol.16, pp.1, 2019, https://doi.org/10.1186/s13014-021-01957-5
  47. Radiomics in hepatic metastasis by colorectal cancer vol.16, pp.1, 2021, https://doi.org/10.1186/s13027-021-00379-y
  48. Advances in liver US, CT, and MRI: moving toward the future vol.5, pp.1, 2021, https://doi.org/10.1186/s41747-021-00250-0
  49. A review in radiomics: Making personalized medicine a reality via routine imaging vol.42, pp.1, 2019, https://doi.org/10.1002/med.21846