Korean Journal of Radiology

The Korean Society of Radiology (대한영상의학회)
권호 표지
DOI QR코드

DOI QR Code

Selection and Reporting of Statistical Methods to Assess Reliability of a Diagnostic Test: Conformity to Recommended Methods in a Peer-Reviewed Journal

  • Park, Ji Eun (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Han, Kyunghwa (Department of Radiology, Research Institute of Radiological Science, Yonsei University College of Medicine) ;
  • Sung, Yu Sub (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Chung, Mi Sun (Department of Radiology, Chung-Ang University Hospital, Chung-Ang University College of Medicine) ;
  • Koo, Hyun Jung (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Yoon, Hee Mang (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Choi, Young Jun (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Lee, Seung Soo (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Kim, Kyung Won (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Shin, Youngbin (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • An, Suah (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center) ;
  • Cho, Hyo-Min (Korea Research Institute of Standards and Science) ;
  • Park, Seong Ho (Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center)
  • Received : 2017.07.28
  • Accepted : 2017.07.28
  • Published : 2017.12.01
⟨ Previous Next ⟩

Abstract

Objective: To evaluate the frequency and adequacy of statistical analyses in a general radiology journal when reporting a reliability analysis for a diagnostic test. Materials and Methods: Sixty-three studies of diagnostic test accuracy (DTA) and 36 studies reporting reliability analyses published in the Korean Journal of Radiology between 2012 and 2016 were analyzed. Studies were judged using the methodological guidelines of the Radiological Society of North America-Quantitative Imaging Biomarkers Alliance (RSNA-QIBA), and COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative. DTA studies were evaluated by nine editorial board members of the journal. Reliability studies were evaluated by study reviewers experienced with reliability analysis. Results: Thirty-one (49.2%) of the 63 DTA studies did not include a reliability analysis when deemed necessary. Among the 36 reliability studies, proper statistical methods were used in all (5/5) studies dealing with dichotomous/nominal data, 46.7% (7/15) of studies dealing with ordinal data, and 95.2% (20/21) of studies dealing with continuous data. Statistical methods were described in sufficient detail regarding weighted kappa in 28.6% (2/7) of studies and regarding the model and assumptions of intraclass correlation coefficient in 35.3% (6/17) and 29.4% (5/17) of studies, respectively. Reliability parameters were used as if they were agreement parameters in 23.1% (3/13) of studies. Reproducibility and repeatability were used incorrectly in 20% (3/15) of studies. Conclusion: Greater attention to the importance of reporting reliability, thorough description of the related statistical methods, efforts not to neglect agreement parameters, and better use of relevant terminology is necessary.

Keywords

Acknowledgement

Supported by : Ministry of Health & Welfare

References

  1. Bankier AA, Levine D, Halpern EF, Kressel HY. Consensus interpretation in imaging research: is there a better way? Radiology 2010;257:14-17 https://doi.org/10.1148/radiol.10100252
  2. Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research. Getting better but still not good. JAMA 1995;274:645-651 https://doi.org/10.1001/jama.1995.03530080061042
  3. Levine D, Bankier AA, Halpern EF. Submissions to radiology: our top 10 list of statistical errors. Radiology 2009;253:288-290 https://doi.org/10.1148/radiol.2532090759
  4. Hernaez R. Reliability and agreement studies: a guide for clinical investigators. Gut 2015;64:1018-1027 https://doi.org/10.1136/gutjnl-2014-308619
  5. Kessler LG, Barnhart HX, Buckler AJ, Choudhury KR, Kondratovich MV, Toledano A, et al. The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions. Stat Methods Med Res 2015;24:9-26 https://doi.org/10.1177/0962280214537333
  6. Raunig DL, McShane LM, Pennello G, Gatsonis C, Carson PL, Voyvodic JT, et al. Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment. Stat Methods Med Res 2015;24:27-67 https://doi.org/10.1177/0962280214537344
  7. Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, et al. Guidelines for reporting reliability and agreement Studies (GRRAS) were proposed. J Clin Epidemiol 2011;64:96-106 https://doi.org/10.1016/j.jclinepi.2010.03.002
  8. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 2015;277:826-832 https://doi.org/10.1148/radiol.2015151516
  9. Choi YJ, Chung MS, Koo HJ, Park JE, Yoon HM, Park SH. Does the reporting quality of diagnostic test accuracy studies, as defined by STARD 2015, affect citation? Korean J Radiol 2016;17:706-714 https://doi.org/10.3348/kjr.2016.17.5.706
  10. Gallo L, Hua N, Mercuri M, Silveira A, Worster A; Best Evidence in Emergency Medicine (BEEM; beem.ca). Adherence to standards for reporting diagnostic accuracy in emergency medicine research. Acad Emerg Med 2017 Jun 16 [Epub]. https://doi.org/10.1111/acem.13233
  11. Grob AT, van der Vaart LR, Withagen MI, van der Vaart CH. The quality of reporting of diagnostic accuracy studies in pelvic floor transperineal three-dimensional ultrasound: a systematic review. Ultrasound Obstet Gynecol 2016 Dec 21 [Epub]. https://doi.org/10.1002/uog.17390
  12. Hong PJ, Korevaar DA, McGrath TA, Ziai H, Frank R, Alabousi M, et al. Reporting of imaging diagnostic accuracy studies with focus on MRI subgroup: Adherence to STARD 2015. J Magn Reson Imaging 2017 Jun 22 [Epub]. https://doi.org/10.1002/jmri.25797
  13. Korevaar DA, Cohen JF, Hooft L, Bossuyt PM. Literature survey of high-impact journals revealed reporting weaknesses in abstracts of diagnostic accuracy studies. J Clin Epidemiol 2015;68:708-715 https://doi.org/10.1016/j.jclinepi.2015.01.014
  14. Korevaar DA, Wang J, van Enst WA, Leeflang MM, Hooft L, Smidt N, et al. Reporting diagnostic accuracy studies: some improvements after 10 years of STARD. Radiology 2015;274:781-789 https://doi.org/10.1148/radiol.14141160
  15. Xu XQ, Hu H, Su GY, Liu H, Shi HB, Wu FY. Diffusion weighted imaging for differentiating benign from malignant orbital tumors: diagnostic performance of the apparent diffusion coefficient based on region of interest selection method. Korean J Radiol 2016;17:650-656 https://doi.org/10.3348/kjr.2016.17.5.650
  16. Park HJ, Lee SY, Rho MH, Chung EC, Kim MS, Kwon HJ, et al. Single-shot echo-planar diffusion-weighted MR imaging at 3T and 1.5T for differentiation of benign vertebral fracture edema and tumor infiltration. Korean J Radiol 2016;17:590-597 https://doi.org/10.3348/kjr.2016.17.5.590
  17. Liu J, Ji Y, Ai H, Ning B, Zhao J, Zhang Y, et al. Liver shearwave velocity and serum fibrosis markers to diagnose hepatic fibrosis in patients with chronic viral hepatitis B. Korean J Radiol 2016;17:396-404 https://doi.org/10.3348/kjr.2016.17.3.396
  18. Lim HJ, Chung MJ, Shin KE, Hwang HS, Lee KS. The impact of iterative reconstruction in low-dose computed tomography on the evaluation of diffuse interstitial lung disease. Korean J Radiol 2016;17:950-960 https://doi.org/10.3348/kjr.2016.17.6.950
  19. Lee JE, Cho JS, Shin KS, Kim SS, You SK, Park JW, et al. Diffuse infiltrative splenic lymphoma: diagnostic efficacy of arterial-phase CT. Korean J Radiol 2016;17:734-741 https://doi.org/10.3348/kjr.2016.17.5.734
  20. Lee H, Lee WW, Park SY, Kim SE. F-18 sodium fluoride positron emission tomography/computed tomography for detection of thyroid cancer bone metastasis compared with bone scintigraphy. Korean J Radiol 2016;17:281-288 https://doi.org/10.3348/kjr.2016.17.2.281
  21. Lee EH, Kim KW, Kim YJ, Shin DR, Park YM, Lim HS, et al. Performance of screening mammography: a report of the alliance for breast cancer screening in Korea. Korean J Radiol 2016;17:489-496 https://doi.org/10.3348/kjr.2016.17.4.489
  22. Joo SM, Kim YP, Yum TJ, Eun NL, Lee D, Lee KH. Optimized performance of flightplan during chemoembolization for hepatocellular carcinoma: importance of the proportion of segmented tumor area. Korean J Radiol 2016;17:771-778 https://doi.org/10.3348/kjr.2016.17.5.771
  23. Hong HS, Cho HS, Woo JY, Lee Y, Yang I, Hwang JY, et al. Intra-appendiceal air at CT: is it a useful or a confusing sign for the diagnosis of acute appendicitis? Korean J Radiol 2016;17:39-46 https://doi.org/10.3348/kjr.2016.17.1.39
  24. Ha EJ, Moon WJ, Na DG, Lee YH, Choi N, Kim SJ, et al. A multicenter prospective validation study for the Korean thyroid imaging reporting and data system in patients with thyroid nodules. Korean J Radiol 2016;17:811-821 https://doi.org/10.3348/kjr.2016.17.5.811
  25. Ding Y, Zeng M, Rao S, Chen C, Fu C, Zhou J. Comparison of biexponential and monoexponential model of diffusionweighted imaging for distinguishing between common renal cell carcinoma and fat poor angiomyolipoma. Korean J Radiol 2016;17:853-863 https://doi.org/10.3348/kjr.2016.17.6.853
  26. Choi TW, Lee JM, Kim JH, Yu MH, Han JK, Choi BI. Comparison of multidetector CT and gadobutrol-enhanced MR imaging for evaluation of small, solid pancreatic lesions. Korean J Radiol 2016;17:509-521 https://doi.org/10.3348/kjr.2016.17.4.509
  27. Alan B, Goya C, Tunc S, Teke M, Hattapoglu S. Assessment of placental stiffness using acoustic radiation force impulse elastography in pregnant women with fetal anomalies. Korean J Radiol 2016;17:218-223 https://doi.org/10.3348/kjr.2016.17.2.218
  28. Ahn JH, Yu JS, Cho ES, Chung JJ, Kim JH, Kim KW. Diffusionweighted MRI of malignant versus benign portal vein thrombosis. Korean J Radiol 2016;17:533-540 https://doi.org/10.3348/kjr.2016.17.4.533
  29. Ryoo I, Suh S, Lee YH, Seo HS, Seol HY. Comparison of ultrasonographic findings of biopsy-proven tuberculous lymphadenitis and kikuchi disease. Korean J Radiol 2015;16:767-775 https://doi.org/10.3348/kjr.2015.16.4.767
  30. Park B, Kim HK, Choi YS, Kim J, Zo JI, Choi JY, et al. Prediction of pathologic grade and prognosis in mucoepidermoid carcinoma of the lung using 18F-FDG PET/CT. Korean J Radiol 2015;16:929-935 https://doi.org/10.3348/kjr.2015.16.4.929
  31. Niu XK, Bhetuwal A, Yang HF. CT-guided core needle biopsy of pleural lesions: evaluating diagnostic yield and associated complications. Korean J Radiol 2015;16:206-212 https://doi.org/10.3348/kjr.2015.16.1.206
  32. Lu Q, Huang BJ, Wang WP, Li CX, Xue LY. Qualitative and quantitative analysis with contrast-enhanced ultrasonography: diagnosis value in hypoechoic renal angiomyolipoma. Korean J Radiol 2015;16:334-341 https://doi.org/10.3348/kjr.2015.16.2.334
  33. Lee S, Lee YH, Chung TS, Jeong EK, Kim S, Yoo YH, et al. Accuracy of diffusion tensor imaging for diagnosing cervical spondylotic myelopathy in patients showing spinal cord compression. Korean J Radiol 2015;16:1303-1312 https://doi.org/10.3348/kjr.2015.16.6.1303
  34. Lee EK, Choi SH, Yun TJ, Kang KM, Kim TM, Lee SH, et al. Prediction of response to concurrent chemoradiotherapy with temozolomide in glioblastoma: application of immediate post-operative dynamic susceptibility contrast and diffusionweighted MR imaging. Korean J Radiol 2015;16:1341-1348 https://doi.org/10.3348/kjr.2015.16.6.1341
  35. Kim SA, Chang JM, Cho N, Yi A, Moon WK. Characterization of breast lesions: comparison of digital breast tomosynthesis and ultrasonography. Korean J Radiol 2015;16:229-238 https://doi.org/10.3348/kjr.2015.16.2.229
  36. Kim J, Kim YH, Lee KH, Lee YJ, Park JH. Diagnostic performance of CT angiography in patients visiting emergency department with overt gastrointestinal bleeding. Korean J Radiol 2015;16:541-549 https://doi.org/10.3348/kjr.2015.16.3.541
  37. Kim DW, Jung SL, Kim J, Ryu JH, Sung JY, Lim HK. Comparison between ultrasonography and computed tomography for detecting the pyramidal lobe of the thyroid gland: a prospective multicenter study. Korean J Radiol 2015;16:402-409 https://doi.org/10.3348/kjr.2015.16.2.402
  38. Kang KA, Kim YK, Kim E, Jeong WK, Choi D, Lee WJ, et al. T2-weighted liver MRI using the multivane technique at 3T: comparison with conventional T2-weighted MRI. Korean J Radiol 2015;16:1038-1046 https://doi.org/10.3348/kjr.2015.16.5.1038
  39. Jung SI, Park HS, Yim Y, Jeon HJ, Yu MH, Kim YJ, et al. Added value of using a CT coronal reformation to diagnose adnexal torsion. Korean J Radiol 2015;16:835-845 https://doi.org/10.3348/kjr.2015.16.4.835
  40. Chun KY, Choi YS, Lee SH, Kim JS, Young KW, Jeong MS, et al. Deltoid ligament and tibiofibular syndesmosis injury in chronic lateral ankle instability: magnetic resonance iImaging evaluation at 3T and comparison with arthroscopy. Korean J Radiol 2015;16:1096-1103 https://doi.org/10.3348/kjr.2015.16.5.1096
  41. Yu H, Cui JL, Cui SJ, Sun YC, Cui FZ. Differentiating benign from malignant bone tumors using fluid-fluid level features on magnetic resonance imaging. Korean J Radiol 2014;15:757-763 https://doi.org/10.3348/kjr.2014.15.6.757
  42. Yoon SH, Goo JM, Jung J, Hong H, Park EA, Lee CH, et al. Computer-aided classification of visual ventilation patterns in patients with chronic obstructive pulmonary disease at two-phase xenon-enhanced CT. Korean J Radiol 2014;15:386-396 https://doi.org/10.3348/kjr.2014.15.3.386
  43. Yi J, Lee EH, Kwak JJ, Cha JG, Jung SH. Retrieval rate and accuracy of ultrasound-guided 14-G semi-automated core needle biopsy of breast microcalcifications. Korean J Radiol 2014;15:12-19 https://doi.org/10.3348/kjr.2014.15.1.12
  44. Woo S, Kim SY, Cho JY, Kim SH. Shear wave elastography for detection of prostate cancer: a preliminary study. Korean J Radiol 2014;15:346-355 https://doi.org/10.3348/kjr.2014.15.3.346
  45. Ucar M, Guryildirim M, Tokgoz N, Kilic K, Borcek A, Oner Y, et al. Evaluation of aqueductal patency in patients with hydrocephalus: three-dimensional high-sampling-efficiency technique (SPACE) versus two-dimensional turbo spin echo at 3 Tesla. Korean J Radiol 2014;15:827-835 https://doi.org/10.3348/kjr.2014.15.6.827
  46. Luczyn' ska E, Heinze-Paluchowska S, Dyczek S, Blecharz P, Rys J, Reinfuss M. Contrast-enhanced spectral mammography: comparison with conventional mammography and histopathology in 152 women. Korean J Radiol 2014;15:689-696 https://doi.org/10.3348/kjr.2014.15.6.689
  47. Lee JH, Yoon YC, Jee S, Kwon JW, Cha JG, Yoo JC. Comparison of three-dimensional isotropic and twodimensional conventional indirect MR arthrography for the diagnosis of rotator cuff tears. Korean J Radiol 2014;15:771-780 https://doi.org/10.3348/kjr.2014.15.6.771
  48. Lee JH, Byun JH, Kim JH, Lee SS, Kim HJ, Lee MG. Solid pancreatic tumors with unilocular cyst-like appearance on CT: differentiation from unilocular cystic tumors using CT. Korean J Radiol 2014;15:704-711 https://doi.org/10.3348/kjr.2014.15.6.704
  49. Lee JE, Lee JM, Lee KB, Yoon JH, Shin CI, Han JK, et al. Noninvasive assessment of hepatic fibrosis in patients with chronic hepatitis B viral infection using magnetic resonance elastography. Korean J Radiol 2014;15:210-217 https://doi.org/10.3348/kjr.2014.15.2.210
  50. Kim YP, Kannengiesser S, Paek MY, Kim S, Chung TS, Yoo YH, et al. Differentiation between focal malignant marrowreplacing lesions and benign red marrow deposition of the spine with T2*-corrected fat-signal fraction map using a three-echo volume interpolated breath-hold gradient echo Dixon sequence. Korean J Radiol 2014;15:781-791 https://doi.org/10.3348/kjr.2014.15.6.781
  51. Jung SI, Park HS, Kim YJ, Jeon HJ. Multidetector computed tomography for the assessment of adnexal mass: is unenhanced CT scan necessary? Korean J Radiol 2014;15:72-79 https://doi.org/10.3348/kjr.2014.15.1.72
  52. Iannicelli E, Di Renzo S, Ferri M, Pilozzi E, Di Girolamo M, Sapori A, et al. Accuracy of high-resolution MRI with lumen distention in rectal cancer staging and circumferential margin involvement prediction. Korean J Radiol 2014;15:37-44 https://doi.org/10.3348/kjr.2014.15.1.37
  53. Bang SH, Lee JY, Woo H, Joo I, Lee ES, Han JK, et al. Differentiating between adenomyomatosis and gallbladder cancer: revisiting a comparative study of high-resolution ultrasound, multidetector CT, and MR imaging. Korean J Radiol 2014;15:226-234 https://doi.org/10.3348/kjr.2014.15.2.226
  54. Ahn HS, Kim SM, Jang M, Yun BL, Kim B, Ko ES, et al. A new full-field digital mammography system with and without the use of an advanced post-processing algorithm: comparison of image quality and diagnostic performance. Korean J Radiol 2014;15:305-312 https://doi.org/10.3348/kjr.2014.15.3.305
  55. Yoon JH, Lee JM, Woo HS, Yu MH, Joo I, Lee ES, et al. Staging of hepatic fibrosis: comparison of magnetic resonance elastography and shear wave elastography in the same individuals. Korean J Radiol 2013;14:202-212 https://doi.org/10.3348/kjr.2013.14.2.202
  56. Wu EH, Chen YL, Wu YM, Huang YT, Wong HF, Ng SH. CTguided core needle biopsy of deep suprahyoid head and neck lesions. Korean J Radiol 2013;14:299-306 https://doi.org/10.3348/kjr.2013.14.2.299
  57. Song YS, Choi SH, Park CK, Yi KS, Lee WJ, Yun TJ, et al. True progression versus pseudoprogression in the treatment of glioblastomas: a comparison study of normalized cerebral blood volume and apparent diffusion coefficient by histogram analysis. Korean J Radiol 2013;14:662-672 https://doi.org/10.3348/kjr.2013.14.4.662
  58. Rief M, Stenzel F, Kranz A, Schlattmann P, Dewey M. Time efficiency and diagnostic accuracy of new automated myocardial perfusion analysis software in 320-row CT cardiac imaging. Korean J Radiol 2013;14:21-29 https://doi.org/10.3348/kjr.2013.14.1.21
  59. Liu X, Peng W, Zhou L, Wang H. Biexponential apparent diffusion coefficients values in the prostate: comparison among normal tissue, prostate cancer, benign prostatic hyperplasia and prostatitis. Korean J Radiol 2013;14:222-232 https://doi.org/10.3348/kjr.2013.14.2.222
  60. Lim HJ, Chung MJ, Lee G, Yie M, Shin KE, Moon JW, et al. Interpretation of digital chest radiographs: comparison of light emitting diode versus cold cathode fluorescent lamp backlit monitors. Korean J Radiol 2013;14:968-976 https://doi.org/10.3348/kjr.2013.14.6.968
  61. Lee SJ, Lee WW, Kim SE. Bone positron emission tomography with or without CT is more accurate than bone scan for detection of bone metastasis. Korean J Radiol 2013;14:510-519 https://doi.org/10.3348/kjr.2013.14.3.510
  62. Lee KH, Lee JM, Park JH, Kim JH, Park HS, Yu MH, et al. MR imaging in patients with suspected liver metastases: value of liver-specific contrast agent gadoxetic acid. Korean J Radiol 2013;14:894-904 https://doi.org/10.3348/kjr.2013.14.6.894
  63. Lee DH, Lee JM, Klotz E, Kim SJ, Kim KW, Han JK, et al. Detection of recurrent hepatocellular carcinoma in cirrhotic liver after transcatheter arterial chemoembolization: value of quantitative color mapping of the arterial enhancement fraction of the liver. Korean J Radiol 2013;14:51-60 https://doi.org/10.3348/kjr.2013.14.1.51
  64. Koo JH, Kim CK, Choi D, Park BK, Kwon GY, Kim B. Diffusionweighted magnetic resonance imaging for the evaluation of prostate cancer: optimal B value at 3T. Korean J Radiol 2013;14:61-69 https://doi.org/10.3348/kjr.2013.14.1.61
  65. Ko ES, Han BK, Kim SM, Ko EY, Jang M, Lyou CY, et al. Comparison of new and established full-field digital mammography systems in diagnostic performance. Korean J Radiol 2013;14:164-170 https://doi.org/10.3348/kjr.2013.14.2.164
  66. Kim SH, Kang BJ, Choi BG, Choi JJ, Lee JH, Song BJ, et al. Radiologists' performance for detecting lesions and the interobserver variability of automated whole breast ultrasound. Korean J Radiol 2013;14:154-163 https://doi.org/10.3348/kjr.2013.14.2.154
  67. Kim MY, Cho N, Yi A, Koo HR, Yun BL, Moon WK. Sonoelastography in distinguishing benign from malignant complex breast mass and making the decision to biopsy. Korean J Radiol 2013;14:559-567 https://doi.org/10.3348/kjr.2013.14.4.559
  68. Kim JI, Kim YH, Lee KH, Kim SY, Lee YJ, Park YS, et al. Type-specific diagnosis and evaluation of longitudinal tumor extent of borrmann type IV gastric cancer: CT versus gastroscopy. Korean J Radiol 2013;14:597-606 https://doi.org/10.3348/kjr.2013.14.4.597
  69. Kim JE, Lee JY, Bae KS, Han JK, Choi BI. Acoustic radiation force impulse elastography for focal hepatic tumors: usefulness for differentiating hemangiomas from malignant tumors. Korean J Radiol 2013;14:743-753 https://doi.org/10.3348/kjr.2013.14.5.743
  70. Jeh SK, Kim SH, Kang BJ. Comparison of the diagnostic performance of response evaluation criteria in solid tumor 1.0 with response evaluation criteria in solid tumor 1.1 on MRI in advanced breast cancer response evaluation to neoadjuvant chemotherapy. Korean J Radiol 2013;14:13-20 https://doi.org/10.3348/kjr.2013.14.1.13
  71. Choi HS, Kim AH, Ahn SS, Shin NY, Kim J, Lee SK. Glioma grading capability: comparisons among parameters from dynamic contrast-enhanced MRI and ADC value on DWI. Korean J Radiol 2013;14:487-492 https://doi.org/10.3348/kjr.2013.14.3.487
  72. Yoo JY, Chung MJ, Choi B, Jung HN, Koo JH, Bae YA, et al. Digital tomosynthesis for PNS evaluation: comparisons of patient exposure and image quality with plain radiography. Korean J Radiol 2012;13:136-143 https://doi.org/10.3348/kjr.2012.13.2.136
  73. Wu CH, Huang CC, Wang LJ, Wong YC, Wang CJ, Lo WC, et al. Value of CT in the discrimination of fatal from non-fatal stercoral colitis. Korean J Radiol 2012;13:283-289 https://doi.org/10.3348/kjr.2012.13.3.283
  74. Sohn CH, Lee HP, Park JB, Chang HW, Kim E, Kim E, et al. Imaging findings of brain death on 3-Tesla MRI. Korean J Radiol 2012;13:541-549 https://doi.org/10.3348/kjr.2012.13.5.541
  75. Lee KH, Goo JM, Park CM, Lee HJ, Jin KN. Computer-aided detection of malignant lung nodules on chest radiographs: effect on observers' performance. Korean J Radiol 2012;13:564-571 https://doi.org/10.3348/kjr.2012.13.5.564
  76. Kang KM, Choi SI, Chun EJ, Kim JA, Youn TJ, Choi DJ. Coronary vasospastic angina: assessment by multidetector CT coronary angiography. Korean J Radiol 2012;13:27-33 https://doi.org/10.3348/kjr.2012.13.1.27
  77. Chung SY, Park SH, Lee SS, Lee JH, Kim AY, Park SK, et al. Comparison between CT colonography and double-contrast barium enema for colonic evaluation in patients with renal insufficiency. Korean J Radiol 2012;13:290-299 https://doi.org/10.3348/kjr.2012.13.3.290
  78. Ko ES, Han H, Han BK, Kim SM, Kim RB, Lee GW, et al. Prognostic significance of a complete response on breast MRI in patients who received neoadjuvant chemotherapy according to the molecular subtype. Korean J Radiol 2015;16:986-995 https://doi.org/10.3348/kjr.2015.16.5.986
  79. Shin CI, Kim HC, Song YS, Cho HR, Lee KB, Lee W, et al. Rat model of hindlimb ischemia induced via embolization with polyvinyl alcohol and N-butyl cyanoacrylate. Korean J Radiol 2013;14:923-930 https://doi.org/10.3348/kjr.2013.14.6.923
  80. Suh YJ, Kim YJ, Hong YJ, Lee HJ, Hur J, Im DJ, et al. Measurement of opening and closing angles of aortic valve prostheses in vivo using Dual-Source Computed Tomography: comparison with those of manufacturers' in 10 different types. Korean J Radiol 2015;16:1012-1023 https://doi.org/10.3348/kjr.2015.16.5.1012
  81. Yokoo T, Shiehmorteza M, Hamilton G, Wolfson T, Schroeder ME, Middleton MS, et al. Estimation of hepatic protondensity fat fraction by using MR imaging at 3.0 T. Radiology 2011;258:749-759 https://doi.org/10.1148/radiol.10100659
  82. Yoo YH, Yoon CS, Eun NL, Hwang MJ, Yoo H, Peters RD, et al. Interobserver and test-retest reproducibility of $T1\rho$ and T2 measurements of lumbar intervertebral discs by 3T magnetic resonance imaging. Korean J Radiol 2016;17:903-911 https://doi.org/10.3348/kjr.2016.17.6.903
  83. Yoo H, Lee JM, Yoon JH, Lee DH, Chang W, Han JK. Prospective comparison of liver stiffness measurements between two point shear wave elastography methods: virtual touch quantification and elastography point quantification. Korean J Radiol 2016;17:750-757 https://doi.org/10.3348/kjr.2016.17.5.750
  84. Song JS, Hwang SB, Chung GH, Jin GY. Intra-individual, inter-vendor comparison of diffusion-weighted MR imaging of upper abdominal organs at 3.0 Tesla with an emphasis on the value of normalization with the spleen. Korean J Radiol 2016;17:209-217 https://doi.org/10.3348/kjr.2016.17.2.209
  85. Yoon SJ, Yoon YC, Bae SY, Wang JH. Bone tunnel diameter measured with CT after anterior cruciate ligament reconstruction using double-bundle auto-hamstring tendons: clinical implications. Korean J Radiol 2015;16:1313-1318 https://doi.org/10.3348/kjr.2015.16.6.1313
  86. Yoo YH, Kim HS, Lee YH, Yoon CS, Paek MY, Yoo H, et al. Comparison of multi-echo Dixon methods with volume interpolated breath-hold gradient echo magnetic resonance imaging in fat-signal fraction quantification of paravertebral muscle. Korean J Radiol 2015;16:1086-1095 https://doi.org/10.3348/kjr.2015.16.5.1086
  87. Yi JS, Cha JG, Han JK, Kim HJ. Imaging of herniated discs of the cervical spine: inter-modality differences between 64-slice multidetector CT and 1.5-T MRI. Korean J Radiol 2015;16:881-888 https://doi.org/10.3348/kjr.2015.16.4.881
  88. Park HS, Han JK, Lee JM, Kim YI, Woo S, Yoon JH, et al. Dynamic contrast-enhanced MRI using a macromolecular MR contrast agent (P792): evaluation of antivascular drug effect in a rabbit VX2 liver tumor model. Korean J Radiol 2015;16:1029-1037 https://doi.org/10.3348/kjr.2015.16.5.1029
  89. Park EA, Lee W, Kim HK, Chung JW. Effect of papillary muscles and trabeculae on left ventricular measurement using cardiovascular magnetic resonance imaging in patients with hypertrophic cardiomyopathy. Korean J Radiol 2015;16:4-12 https://doi.org/10.3348/kjr.2015.16.1.4
  90. Lee GY, Lee JW, Choi SW, Lim HJ, Sun HY, Kang Y, et al. MRI inter-reader and intra-reader reliabilities for assessing injury morphology and posterior ligamentous complex integrity of the spine according to the thoracolumbar injury classification system and severity score. Korean J Radiol 2015;16:889-898 https://doi.org/10.3348/kjr.2015.16.4.889
  91. Lee E, Choi JA. Associations between alpha angle and herniation pit on MRI revisited in 185 asymptomatic hip joints. Korean J Radiol 2015;16:1319-1325 https://doi.org/10.3348/kjr.2015.16.6.1319
  92. Kim S, Lee JW, Chai JW, Yoo HJ, Kang Y, Seo J, et al. A new MRI grading system for cervical foraminal stenosis based on axial T2-weighted images. Korean J Radiol 2015;16:1294-1302 https://doi.org/10.3348/kjr.2015.16.6.1294
  93. Kim JR, Lee YS, Yu J. Assessment of bone age in prepubertal healthy Korean children: comparison among the Korean standard bone age chart, Greulich-Pyle method, and Tanner-Whitehouse method. Korean J Radiol 2015;16:201-205 https://doi.org/10.3348/kjr.2015.16.1.201
  94. Ding J, Xing W, Wu D, Chen J, Pan L, Sun J, et al. Evaluation of renal oxygenation level changes after water loading using susceptibility-weighted imaging and T2* mapping. Korean J Radiol 2015;16:827-834 https://doi.org/10.3348/kjr.2015.16.4.827
  95. Choi YJ, Baek JH, Hong MJ, Lee JH. Inter-observer variation in ultrasound measurement of the volume and diameter of thyroid nodules. Korean J Radiol 2015;16:560-565 https://doi.org/10.3348/kjr.2015.16.3.560
  96. Yun BL, Cho N, Li M, Jang MH, Park SY, Kang HC, et al. Intratumoral heterogeneity of breast cancer xenograft models: texture analysis of diffusion-weighted MR imaging. Korean J Radiol 2014;15:591-604 https://doi.org/10.3348/kjr.2014.15.5.591
  97. Lim HK, Hong SH, Yoo HJ, Choi JY, Kim SH, Choi JA, et al. Visual MRI grading system to evaluate atrophy of the supraspinatus muscle. Korean J Radiol 2014;15:501-507 https://doi.org/10.3348/kjr.2014.15.4.501
  98. Lee CB, Choi SJ, Ahn JH, Ryu DS, Park MS, Jung SM, et al. Ectopic insertion of the pectoralis minor tendon: inter-reader agreement and findings in the rotator interval on MRI. Korean J Radiol 2014;15:764-770 https://doi.org/10.3348/kjr.2014.15.6.764
  99. Ko SY, Kim EK, Kim MJ, Moon HJ. Mammographic density estimation with automated volumetric breast density measurement. Korean J Radiol 2014;15:313-321 https://doi.org/10.3348/kjr.2014.15.3.313
  100. Cho YD, Kim KM, Lee WJ, Sohn CH, Kang HS, Kim JE, et al. Time-of-flight magnetic resonance angiography for followup of coil embolization with enterprise stent for intracranial aneurysm: usefulness of source images. Korean J Radiol 2014;15:161-168 https://doi.org/10.3348/kjr.2014.15.1.161
  101. Yoo SY, Kim Y, Cho HH, Choi MJ, Shim SS, Lee JK, et al. Dual-energy CT in the assessment of mediastinal lymph nodes: comparative study of virtual non-contrast and true non-contrast images. Korean J Radiol 2013;14:532-539 https://doi.org/10.3348/kjr.2013.14.3.532
  102. Shin SM, Kim WS, Cheon JE, Kim HS, Lee W, Jung AY, et al. Bronchopulmonary dysplasia: new high resolution computed tomography scoring system and correlation between the high resolution computed tomography score and clinical severity. Korean J Radiol 2013;14:350-360 https://doi.org/10.3348/kjr.2013.14.2.350
  103. Kang Y, Choi JA, Chung JH, Hong SH, Kang HS. Accuracy of preoperative MRI with microscopy coil in evaluation of primary tumor thickness of malignant melanoma of the skin with histopathologic correlation. Korean J Radiol 2013;14:287-293 https://doi.org/10.3348/kjr.2013.14.2.287
  104. Zhao F, Deng M, Yuan J, Teng GJ, Ahuja AT, Wang YX. Experimental evaluation of accelerated T1rho relaxation quantification in human liver using limited spin-lock times. Korean J Radiol 2012;13:736-742 https://doi.org/10.3348/kjr.2012.13.6.736
  105. Song SE, Seo BK, Yie A, Ku BK, Kim HY, Cho KR, et al. Which phantom is better for assessing the image quality in fullfield digital mammography?: American College of Radiology Accreditation phantom versus digital mammography accreditation phantom. Korean J Radiol 2012;13:776-783 https://doi.org/10.3348/kjr.2012.13.6.776
  106. Seok JH, Choi HS, Jung SL, Ahn KJ, Kim MJ, Shin YS, et al. Artificial luminal narrowing on contrast-enhanced magnetic resonance angiograms on an occasion of stent-assisted coiling of intracranial aneurysm: in vitro comparison using two different stents with variable imaging parameters. Korean J Radiol 2012;13:550-556 https://doi.org/10.3348/kjr.2012.13.5.550
  107. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res 2010;19:539-549 https://doi.org/10.1007/s11136-010-9606-8
  108. de Vet HC, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol 2006;59:1033-1039 https://doi.org/10.1016/j.jclinepi.2005.10.015
  109. Kim JS, Jang HY, Park SH, Kim KJ, Han K, Yang SK, et al. MR enterography assessment of bowel inflammation severity in crohn disease using the MR index of activity score: modifying roles of DWI and effects of contrast phases. AJR Am J Roentgenol 2017;208:1022-1029 https://doi.org/10.2214/AJR.16.17324
  110. Sullivan DC, Obuchowski NA, Kessler LG, Raunig DL, Gatsonis C, Huang EP, et al. Metrology standards for quantitative imaging biomarkers. Radiology 2015;277:813-825 https://doi.org/10.1148/radiol.2015142202
  111. Barnhart HX, Barboriak DP. Applications of the repeatability of quantitative imaging biomarkers: a review of statistical analysis of repeat data sets. Transl Oncol 2009;2:231-235 https://doi.org/10.1593/tlo.09268
  112. Seo N, Park SH, Kim KJ, Kang BK, Lee Y, Yang SK, et al. MR enterography for the evaluation of small-bowel inflammation in crohn disease by using diffusion-weighted imaging without intravenous contrast material: a prospective noninferiority study. Radiology 2016;278:762-772 https://doi.org/10.1148/radiol.2015150809

Cited by

  1. Optimal Phase of Dynamic Computed Tomography for Reliable Size Measurement of Metastatic Neuroendocrine Tumors of the Liver: Comparison between Pre- and Post-Contrast Phases vol.19, pp.6, 2017, https://doi.org/10.3348/kjr.2018.19.6.1066
  2. Age of Data in Contemporary Research Articles Published in Representative General Radiology Journals vol.19, pp.6, 2018, https://doi.org/10.3348/kjr.2018.19.6.1172
  3. Magnetic resonance tumour regression grade and pathological correlates in patients with rectal cancer vol.105, pp.12, 2018, https://doi.org/10.1002/bjs.10898
  4. A Glimpse on Trends and Characteristics of Recent Articles Published in the Korean Journal of Radiology vol.20, pp.12, 2019, https://doi.org/10.3348/kjr.2019.0928
  5. High-Resolution Magnetic Resonance Imaging Using Compressed Sensing for Intracranial and Extracranial Arteries: Comparison with Conventional Parallel Imaging vol.20, pp.3, 2017, https://doi.org/10.3348/kjr.2018.0424
  6. Prognostic Value of Radiologic Extranodal Extension in Human Papillomavirus-Related Oropharyngeal Squamous Cell Carcinoma vol.20, pp.8, 2019, https://doi.org/10.3348/kjr.2018.0742
  7. Comparison of monoexponential, intravoxel incoherent motion diffusion-weighted imaging and diffusion kurtosis imaging for assessment of hepatic fibrosis vol.60, pp.12, 2019, https://doi.org/10.1177/0284185119840219
  8. Characteristics of Recent Articles Published in the Korean Journal of Radiology Based on the Citation Frequency vol.21, pp.12, 2020, https://doi.org/10.3348/kjr.2020.1322
  9. Repeatability of amide proton transfer-weighted signals in the brain according to clinical condition and anatomical location vol.30, pp.1, 2017, https://doi.org/10.1007/s00330-019-06285-7
  10. Magnetic resonance imaging-based 3-dimensional fractal dimension and lacunarity analyses may predict the meningioma grade vol.30, pp.8, 2020, https://doi.org/10.1007/s00330-020-06788-8
  11. Reliability of 3D image analysis and influence of contrast medium administration on measurement of Hounsfield unit values of the proximal femur vol.15, pp.10, 2017, https://doi.org/10.1371/journal.pone.0241012
  12. Test-retest repeatability of ultrasonographic shear wave elastography in a rat liver fibrosis model: toward a quantitative biomarker for preclinical trials vol.40, pp.1, 2017, https://doi.org/10.14366/usg.19088
  13. Interobserver Reproducibility in Sonographic Measurement of Diameter and Volume of Papillary Thyroid Microcarcinoma vol.31, pp.3, 2017, https://doi.org/10.1089/thy.2020.0317