• Journal of Digestive Cancer Research
• /
• v.11 no.3
• /
• pp.157-164
• /
• 2023

Prediction is a significant topic in clinical research. The development and validation of a prediction model has been increasingly published in clinical research. In this review, we investigated analytical methods and validation schemes for a clinical prediction model used in digestive cancer research. Deep learning and logistic regression, with split-sample validation as an internal or external validation, were the most commonly used methods. Furthermore, we briefly introduced and summarized the advantages and disadvantages of each method. Finally, we discussed several points to consider when conducting prediction model studies.

• Toxicological Research
• /
• v.25 no.1
• /
• pp.1-8
• /
• 2009

Validation specifies and coordinates all relevant activities to ensure compliance with good laboratory practices (GLP) according to suitable international standards. This includes validation activities of past, present and future for the best possible actions to ensure the integrity of non-clinical laboratory data. Recently, validation has become increasingly important, not only in good manufacturing practice (GMP) institutions but also in GLP facilities. In accordance with the guideline for GLP regulations, all equipments used to generate, measure, or assess data should undergo validation to ensure that this equipment is of appropriate design and capacity and that it will consistently function as intended. Therefore, the implantation of validation processes is considered to be an essential step in a global institution. This review describes the procedures and documentations required for validation of GLP. It introduces basic elements such as the validation master plan, risk assessment, gap analysis, design qualification, installation qualification, operational qualification, performance qualification, calibration, traceability, and revalidation.

• Korean Journal of Radiology
• /
• v.24 no.7
• /
• pp.626-639
• /
• 2023

Objective: To investigate the association of clinical, pathologic, and magnetic resonance imaging (MRI) variables with progressive disease (PD) during neoadjuvant chemotherapy (NAC) and distant metastasis-free survival (DMFS) in patients with triple-negative breast cancer (TNBC). Materials and Methods: This single-center retrospective study included 252 women with TNBC who underwent NAC between 2010 and 2019. Clinical, pathologic, and treatment data were collected. Two radiologists analyzed the pre-NAC MRI. After random allocation to the development and validation sets in a 2:1 ratio, we developed models to predict PD and DMFS using logistic regression and Cox proportional hazard regression, respectively, and validated them. Results: Among the 252 patients (age, 48.3 ± 10.7 years; 168 in the development set; 84 in the validation set), PD was occurred in 17 patients and 9 patients in the development and validation sets, respectively. In the clinical-pathologic-MRI model, the metaplastic histology (odds ratio [OR], 8.0; P = 0.032), Ki-67 index (OR, 1.02; P = 0.044), and subcutaneous edema (OR, 30.6; P = 0.004) were independently associated with PD in the development set. The clinical-pathologic-MRI model showed a higher area under the receiver-operating characteristic curve (AUC) than the clinical-pathologic model (AUC: 0.69 vs. 0.54; P = 0.017) for predicting PD in the validation set. Distant metastases occurred in 49 patients and 18 patients in the development and validation sets, respectively. Residual disease in both the breast and lymph nodes (hazard ratio [HR], 6.0; P = 0.005) and the presence of lymphovascular invasion (HR, 3.3; P < 0.001) were independently associated with DMFS. The model consisting of these pathologic variables showed a Harrell's C-index of 0.86 in the validation set. Conclusion: The clinical-pathologic-MRI model, which considered subcutaneous edema observed using MRI, performed better than the clinical-pathologic model for predicting PD. However, MRI did not independently contribute to the prediction of DMFS.

• Korean Journal of Radiology
• /
• v.22 no.3
• /
• pp.442-453
• /
• 2021

Artificial intelligence (AI) will likely affect various fields of medicine. This article aims to explain the fundamental principles of clinical validation, device approval, and insurance coverage decisions of AI algorithms for medical diagnosis and prediction. Discrimination accuracy of AI algorithms is often evaluated with the Dice similarity coefficient, sensitivity, specificity, and traditional or free-response receiver operating characteristic curves. Calibration accuracy should also be assessed, especially for algorithms that provide probabilities to users. As current AI algorithms have limited generalizability to real-world practice, clinical validation of AI should put it to proper external testing and assisting roles. External testing could adopt diagnostic case-control or diagnostic cohort designs. A diagnostic case-control study evaluates the technical validity/accuracy of AI while the latter tests the clinical validity/accuracy of AI in samples representing target patients in real-world clinical scenarios. Ultimate clinical validation of AI requires evaluations of its impact on patient outcomes, referred to as clinical utility, and for which randomized clinical trials are ideal. Device approval of AI is typically granted with proof of technical validity/accuracy and thus does not intend to directly indicate if AI is beneficial for patient care or if it improves patient outcomes. Neither can it categorically address the issue of limited generalizability of AI. After achieving device approval, it is up to medical professionals to determine if the approved AI algorithms are beneficial for real-world patient care. Insurance coverage decisions generally require a demonstration of clinical utility that the use of AI has improved patient outcomes.

• Korean Journal of Radiology
• /
• v.24 no.5
• /
• pp.395-405
• /
• 2023

Objective: This study aimed to develop and validate models using radiomics features on a native T1 map from cardiac magnetic resonance (CMR) to predict left ventricular reverse remodeling (LVRR) in patients with nonischemic dilated cardiomyopathy (NIDCM). Materials and Methods: Data from 274 patients with NIDCM who underwent CMR imaging with T1 mapping at Severance Hospital between April 2012 and December 2018 were retrospectively reviewed. Radiomic features were extracted from the native T1 maps. LVRR was determined using echocardiography performed ≥ 180 days after the CMR. The radiomics score was generated using the least absolute shrinkage and selection operator logistic regression models. Clinical, clinical + late gadolinium enhancement (LGE), clinical + radiomics, and clinical + LGE + radiomics models were built using a logistic regression method to predict LVRR. For internal validation of the result, bootstrap validation with 1000 resampling iterations was performed, and the optimism-corrected area under the receiver operating characteristic curve (AUC) with 95% confidence interval (CI) was computed. Model performance was compared using AUC with the DeLong test and bootstrap. Results: Among 274 patients, 123 (44.9%) were classified as LVRR-positive and 151 (55.1%) as LVRR-negative. The optimism-corrected AUC of the radiomics model in internal validation with bootstrapping was 0.753 (95% CI, 0.698-0.813). The clinical + radiomics model revealed a higher optimism-corrected AUC than that of the clinical + LGE model (0.794 vs. 0.716; difference, 0.078 [99% CI, 0.003-0.151]). The clinical + LGE + radiomics model significantly improved the prediction of LVRR compared with the clinical + LGE model (optimism-corrected AUC of 0.811 vs. 0.716; difference, 0.095 [99% CI, 0.022-0.139]). Conclusion: The radiomic characteristics extracted from a non-enhanced T1 map may improve the prediction of LVRR and offer added value over traditional LGE in patients with NIDCM. Additional external validation research is required.

• Journal of the Korean Medical Association
• /
• v.61 no.12
• /
• pp.765-775
• /
• 2018

With growing interest in novel digital healthcare devices, such as artificial intelligence (AI) software for medical diagnosis and prediction, and their potential impacts on healthcare, discussions have taken place regarding the regulatory approval, coverage, and clinical implementation of these devices. Despite their potential, 'digital exceptionalism' (i.e., skipping the rigorous clinical validation of such digital tools) is creating significant concerns for patients and healthcare stakeholders. This white paper presents the positions of the Korean Society of Radiology, a leader in medical imaging and digital medicine, on the clinical validation, regulatory approval, coverage decisions, and clinical implementation of novel digital healthcare devices, especially AI software for medical diagnosis and prediction, and explains the scientific principles underlying those positions. Mere regulatory approval by the Food and Drug Administration of Korea, the United States, or other countries should be distinguished from coverage decisions and widespread clinical implementation, as regulatory approval only indicates that a digital tool is allowed for use in patients, not that the device is beneficial or recommended for patient care. Coverage or widespread clinical adoption of AI software tools should require a thorough clinical validation of safety, high accuracy proven by robust external validation, documented benefits for patient outcomes, and cost-effectiveness. The Korean Society of Radiology puts patients first when considering novel digital healthcare tools, and as an impartial professional organization that follows scientific principles and evidence, strives to provide correct information to the public, make reasonable policy suggestions, and build collaborative partnerships with industry and government for the good of our patients.

• Korean Journal of Radiology
• /
• v.21 no.8
• /
• pp.1007-1017
• /
• 2020

Objective: The purpose of our study was to investigate the predictive abilities of clinical and computed tomography (CT) features for outcome prediction in patients with coronavirus disease (COVID-19). Materials and Methods: The clinical and CT data of 238 patients with laboratory-confirmed COVID-19 in our two hospitals were retrospectively analyzed. One hundred sixty-six patients (103 males; age 43.8 ± 12.3 years) were allocated in the training cohort and 72 patients (38 males; age 45.1 ± 15.8 years) from another independent hospital were assigned in the validation cohort. The primary composite endpoint was admission to an intensive care unit, use of mechanical ventilation, or death. Univariate and multivariate Cox proportional hazard analyses were performed to identify independent predictors. A nomogram was constructed based on the combination of clinical and CT features, and its prognostic performance was externally tested in the validation group. The predictive value of the combined model was compared with models built on the clinical and radiological attributes alone. Results: Overall, 35 infected patients (21.1%) in the training cohort and 10 patients (13.9%) in the validation cohort experienced adverse outcomes. Underlying comorbidity (hazard ratio [HR], 3.35; 95% confidence interval [CI], 1.67-6.71; p < 0.001), lymphocyte count (HR, 0.12; 95% CI, 0.04-0.38; p < 0.001) and crazy-paving sign (HR, 2.15; 95% CI, 1.03-4.48; p = 0.042) were the independent factors. The nomogram displayed a concordance index (C-index) of 0.82 (95% CI, 0.76-0.88), and its prognostic value was confirmed in the validation cohort with a C-index of 0.89 (95% CI, 0.82-0.96). The combined model provided the best performance over the clinical or radiological model (p < 0.050). Conclusion: Underlying comorbidity, lymphocyte count and crazy-paving sign were independent predictors of adverse outcomes. The prognostic nomogram based on the combination of clinical and CT features could be a useful tool for predicting adverse outcomes of patients with COVID-19.

• Korean Journal of Radiology
• /
• v.23 no.11
• /
• pp.1078-1088
• /
• 2022

Objective: To develop and validate a model using radiomics features from apparent diffusion coefficient (ADC) map to diagnose local tumor recurrence in head and neck squamous cell carcinoma (HNSCC). Materials and Methods: This retrospective study included 285 patients (mean age ± standard deviation, 62 ± 12 years; 220 male, 77.2%), including 215 for training (n = 161) and internal validation (n = 54) and 70 others for external validation, with newly developed contrast-enhancing lesions at the primary cancer site on the surveillance MRI following definitive treatment of HNSCC between January 2014 and October 2019. Of the 215 and 70 patients, 127 and 34, respectively, had local tumor recurrence. Radiomics models using radiomics scores were created separately for T2-weighted imaging (T2WI), contrast-enhanced T1-weighted imaging (CE-T1WI), and ADC maps using non-zero coefficients from the least absolute shrinkage and selection operator in the training set. Receiver operating characteristic (ROC) analysis was used to evaluate the diagnostic performance of each radiomics score and known clinical parameter (age, sex, and clinical stage) in the internal and external validation sets. Results: Five radiomics features from T2WI, six from CE-T1WI, and nine from ADC maps were selected and used to develop the respective radiomics models. The area under ROC curve (AUROC) of ADC radiomics score was 0.76 (95% confidence interval [CI], 0.62-0.89) and 0.77 (95% CI, 0.65-0.88) in the internal and external validation sets, respectively. These were significantly higher than the AUROC values of T2WI (0.53 [95% CI, 0.40-0.67], p = 0.006), CE-T1WI (0.53 [95% CI, 0.40-0.67], p = 0.012), and clinical parameters (0.53 [95% CI, 0.39-0.67], p = 0.021) in the external validation set. Conclusion: The radiomics model using ADC maps exhibited higher diagnostic performance than those of the radiomics models using T2WI or CE-T1WI and clinical parameters in the diagnosis of local tumor recurrence in HNSCC following definitive treatment.

• Journal of Biomedical Engineering Research
• /
• v.38 no.3
• /
• pp.111-115
• /
• 2017

Blood pressure is one of the important vital signs for monitoring the medical condition of a patient. Automated NIBP(non-invasive blood pressure) monitoring devices calculate systolic and diastolic blood pressures from the oscillation in cuff pressure caused by a pulsation of an artery. To validate the NIBP devices, we developed a simulator to supply the oscillometric waveforms obtained from human subjects. The simulator provided pressure pulses to device-under-test and device readings were compared to the auscultatory references. Fully automated simulation system including OCR(optical character recognition) were developed and used for NIBP monitoring devices. The validation results using the simulator agreed well with previous clinical validation. More validation studies using the standardized oscillometric waveforms would be required for the replacement of clinical trials to validate a new automated NIBP monitoring device.

• Radiation Oncology Journal
• /
• v.35 no.2
• /
• pp.101-111
• /
• 2017

The number of imaging data sets has significantly increased during radiation treatment after introducing a diverse range of advanced techniques into the field of radiation oncology. As a consequence, there have been many studies proposing meaningful applications of imaging data set use. These applications commonly require a method to align the data sets at a reference. Deformable image registration (DIR) is a process which satisfies this requirement by locally registering image data sets into a reference image set. DIR identifies the spatial correspondence in order to minimize the differences between two or among multiple sets of images. This article describes clinical applications, validation, and algorithms of DIR techniques. Applications of DIR in radiation treatment include dose accumulation, mathematical modeling, automatic segmentation, and functional imaging. Validation methods discussed are based on anatomical landmarks, physical phantoms, digital phantoms, and per application purpose. DIR algorithms are also briefly reviewed with respect to two algorithmic components: similarity index and deformation models.