• International Journal of Computer Science & Network Security
• /
• 제21권12spc호
• /
• pp.549-555
• /
• 2021

Machine Learning(ML) splits data into 3 parts, which are usually 60% for training, 20% for validation, and 20% for testing. It just splits quantitatively instead of selecting each set of data by a criterion, which is very important concept for the adequacy of test data. ML measures a model's accuracy by applying a set of validation data, and revises the model until the validation accuracy reaches on a certain level. After the validation process, the complete model is tested with the set of test data, which are not seen by the model yet. If the set of test data covers the model's attributes well, the test accuracy will be close to the validation accuracy of the model. To make sure that ML's set of test data works adequately, we design an experiment and see if the test accuracy of model is always close to its validation adequacy as expected. The experiment builds 100 different SVM models for each of six data sets published in UCI ML repository. From the test accuracy and its validation accuracy of 600 cases, we find some unexpected cases, where the test accuracy is very different from its validation accuracy. Consequently, it is not always true that ML's set of test data is adequate to assure a model's quality.

• 정보처리학회논문지D
• /
• 제8D권2호
• /
• pp.175-180
• /
• 2001

본 논문은 미래의 소프크웨어 공장 수나 고장시간 예측 정확성을 얻기 위해, 뉴로-피지 시스템을 이용할 경우 최적의 검증 데이터 할당 비율에 대한 연구이다. 훈련 데이터가 주어졌을 때, 과소 적합과 과잉 적합을 회피하면서 최적의 일반화 능력을 얻기 취해 Early Stopping 방법이 일반적으로 사용되고 있다. 그러나 훈련과 검증 데이터로 얼마나 많은 데이터를 할당갈 것인가는 시행착오법을 이용해 경험적으로 해를 구해야만 하며, 과다한 시간이 소요된다. 최적의 검증 데이터 양을 구하기 위해 규칙 수를 증가시키면서 다양한 검증 데이터 양을 할당하였다. 실험결과 최소의 검증 데이터로도 좋은 예측 능력을 보였다. 이 결과는 뉴로-퍼지 시스템을 소프트웨어 신뢰성 분야에 적용시 실질직언 지침을 제공할 수 있는 것이다.

• 디지털콘텐츠학회 논문지
• /
• 제18권3호
• /
• pp.525-534
• /
• 2017

열린 집합 인식 방법론은 테스트 데이터의 클래스를 학습 시에 모두 파악할 수 없는 경우에 대한 인식 방법론이다. 따라서 열린 집합 인식 방법론은 분류와 유효성 검증의 절차를 필요로 한다. 이러한 연구는 얼굴 인식 모듈의 상용화를 위해 필수적이지만 지금까지 국내에서 연구 결과들이 거의 발표되지 않았다. 우리는 두 개의 검증 단계를 가지는 열린 집합 얼굴 인식 방법론을 제안한다. 첫 번째 단계에서는 학습 클래스 외에 더미 클래스들을 설정하고 희소표현 기반 분류를 수행한다. 이 때 테스트 데이터가 더미 클래스로 분류되면 무효 데이터로 판별하고, 유효한 클래스로 분류되면 다음 유효성 검증 단계로 넘어간다. 두 번째 단계에서 제안하는 네 가지 특징을 추출하고, 확률분포에 기반을 둔 판별함수를 통해 유효성 검증을 수행한다. 우리는 실험을 통해 열린 집합 인식 방법론의 시뮬레이션 방법을 제안하였고 제안하는 방법론의 성능을 제시하고, 희소기반 분류 방식에서 널리 사용되는 SCI 지표를 이용한 유효성 테스트보다 높은 성능을 보임을 입증할 수 있었다.

• Korean Journal of Radiology
• /
• 제24권7호
• /
• pp.626-639
• /
• 2023

Objective: To investigate the association of clinical, pathologic, and magnetic resonance imaging (MRI) variables with progressive disease (PD) during neoadjuvant chemotherapy (NAC) and distant metastasis-free survival (DMFS) in patients with triple-negative breast cancer (TNBC). Materials and Methods: This single-center retrospective study included 252 women with TNBC who underwent NAC between 2010 and 2019. Clinical, pathologic, and treatment data were collected. Two radiologists analyzed the pre-NAC MRI. After random allocation to the development and validation sets in a 2:1 ratio, we developed models to predict PD and DMFS using logistic regression and Cox proportional hazard regression, respectively, and validated them. Results: Among the 252 patients (age, 48.3 ± 10.7 years; 168 in the development set; 84 in the validation set), PD was occurred in 17 patients and 9 patients in the development and validation sets, respectively. In the clinical-pathologic-MRI model, the metaplastic histology (odds ratio [OR], 8.0; P = 0.032), Ki-67 index (OR, 1.02; P = 0.044), and subcutaneous edema (OR, 30.6; P = 0.004) were independently associated with PD in the development set. The clinical-pathologic-MRI model showed a higher area under the receiver-operating characteristic curve (AUC) than the clinical-pathologic model (AUC: 0.69 vs. 0.54; P = 0.017) for predicting PD in the validation set. Distant metastases occurred in 49 patients and 18 patients in the development and validation sets, respectively. Residual disease in both the breast and lymph nodes (hazard ratio [HR], 6.0; P = 0.005) and the presence of lymphovascular invasion (HR, 3.3; P < 0.001) were independently associated with DMFS. The model consisting of these pathologic variables showed a Harrell's C-index of 0.86 in the validation set. Conclusion: The clinical-pathologic-MRI model, which considered subcutaneous edema observed using MRI, performed better than the clinical-pathologic model for predicting PD. However, MRI did not independently contribute to the prediction of DMFS.

• 한국정보처리학회논문지
• /
• 제7권4호
• /
• pp.1112-1121
• /
• 2000

Design metrics that quantify the design phase play an important role in reducing system development cost because the problems in early phases of software development seriously affected the quality of the late products. Real-time systems such as telecommunication systems are so large that design quantification is more important in real-time system design. Although many metrics have been proposed, few of them are correctly validated. This paper revises the SDL metrics set proposed in earlier study [9] and perform an analytical validation o the metrics set. Axiomatic approach and dimensional analysis are used for metrics validation and the old metrics are revised ot satisfy the validation criteria.

• 약학회지
• /
• 제48권1호
• /
• pp.60-64
• /
• 2004

Near-infrared (NIR) spectroscopy was used to determine rapidly and nondestructively the content of ambroxol in intact ambroxol tablets containing 30 mg (12.5％ m/m nominal concentration) by collecting NIR spectra in range 1100-1750 nm. The laboratory-made samples had 10.3∼15.9％ m/m nominal ambroxol concentration. The measurements were made by reflection using a fiber-optic probe and calibration was carried out by partial least square regression (PLSR) with autoscaling. Model validation was performed by randomly splitting the data set into calibration and validation data set (7 samples as a calibration data set and 5 samples as a validation data set). The developed NIR method gave results comparable to the known values of tablets in a laboratorial manufacturing Process, standard error of calibration (SEC) and standard error of prediction (SEP) being 0.49％ and 0.49％ m/m respectively. The method showed good accuracy and repeatability NIR spectroscopic determination in intact tablets allowed the potential use of real time monitoring for a running production process.

• 한국군사과학기술학회지
• /
• 제17권1호
• /
• pp.108-114
• /
• 2014

This paper describes a method to set an intensity-based initial validation gate for tracking filter while preserves the ability of tracking a target with maximum speed. First, we collected real data set of signal versus distance of an airplane target. And at each data point, we computed maximum distance the target can move. And a function is modeled to expect the maximum moving pixels on the lateral direction based on the intensity of the detected target in IR image sequence. The initial prediction error covariance can be computed using this function to decide the size of the initial validation gate. The simulation results show the proposed method can set the appropriate initial validation gates to track the targets with the maximum speed.

• 지능정보연구
• /
• 제5권1호
• /
• pp.95-101
• /
• 1999

In this paper, a comparison between different methods to combine predictions from neural networks will be given. These methods are bagging, bumping, and balancing. Those are based on the analysis of the ensemble generalization error into an ambiguity term and a term incorporating generalization performances of individual networks. Neural Networks and AI machine learning models are prone to overfitting. A strategy to prevent a neural network from overfitting, is to stop training in early stage of the learning process. The complete data set is spilt up into a training set and a validation set. Training is stopped when the error on the validation set starts increasing. The stability of the networks is highly dependent on the division in training and validation set, and also on the random initial weights and the chosen minimization procedure. This causes early stopped networks to be rather unstable: a small change in the data or different initial conditions can produce large changes in the prediction. Therefore, it is advisable to apply the same procedure several times starting from different initial weights. This technique is often referred to as training ensembles of neural networks. In this paper, we presented a comparison of three statistical methods to prevent overfitting of neural network.

• 한국통계학회:학술대회논문집
• /
• 한국통계학회 2004년도 학술발표논문집
• /
• pp.195-198
• /
• 2004

One Important problem in a cancer microarray study is to identify a set of genes from which a molecular prognostic indicator can be developed. In parallel with this problem is to validate the chosen set of genes. We develop in this note a K-fold cross validation procedure by combining a 'pre-validation' technique and a bootstrap resampling procedure in the Cox regression . The pre-validation technique predicts the microarray predictor of a case without having seen the true class level of the case. It was suggested by Tibshirani and Efron (2002) to avoid the possible over-fitting in the regression in which a microarray based predictor is employed. The bootstrap resampling procedure for the Cox regression was proposed by Sauerbrei and Schumacher (1992) as a means of overcoming the instability of a stepwise selection procedure. We apply this K-fold cross validation to the microarray data of 92 gastric cancers of which the experiment was conducted at Cancer Metastasis Research Center, Yonsei University. We also share some of our experience on the 'false positive' result due to the information leak.

• Korean Journal of Radiology
• /
• 제21권6호
• /
• pp.670-683
• /
• 2020

Objective: The presence of coagulative necrosis (CN) in clear cell renal cell carcinoma (ccRCC) indicates a poor prognosis, while the absence of CN indicates a good prognosis. The purpose of this study was to build and validate a radiomics signature based on preoperative CT imaging data to estimate CN status in ccRCC. Materials and Methods: Altogether, 105 patients with pathologically confirmed ccRCC were retrospectively enrolled in this study and then divided into training (n = 72) and validation (n = 33) sets. Thereafter, 385 radiomics features were extracted from the three-dimensional volumes of interest of each tumor, and 10 traditional features were assessed by two experienced radiologists using triple-phase CT-enhanced images. A multivariate logistic regression algorithm was used to build the radiomics score and traditional predictors in the training set, and their performance was assessed and then tested in the validation set. The radiomics signature to distinguish CN status was then developed by incorporating the radiomics score and the selected traditional predictors. The receiver operating characteristic (ROC) curve was plotted to evaluate the predictive performance. Results: The area under the ROC curve (AUC) of the radiomics score, which consisted of 7 radiomics features, was 0.855 in the training set and 0.885 in the validation set. The AUC of the traditional predictor, which consisted of 2 traditional features, was 0.843 in the training set and 0.858 in the validation set. The radiomics signature showed the best performance with an AUC of 0.942 in the training set, which was then confirmed with an AUC of 0.969 in the validation set. Conclusion: The CT-based radiomics signature that incorporated radiomics and traditional features has the potential to be used as a non-invasive tool for preoperative prediction of CN in ccRCC.