• Journal of the Korean Institute of Intelligent Systems
• /
• v.14 no.6
• /
• pp.789-792
• /
• 2004

In various fields as web mining, bioinformatics, statistical data analysis, and so forth, very diversely missing values are found. These values make training data to be sparse. Largely, the missing values are replaced by predicted values using mean and mode. We can used the advanced missing value imputation methods as conditional mean, tree method, and Markov Chain Monte Carlo algorithm. But general imputation models have the property that their predictive accuracy is decreased according to increase the ratio of missing in training data. Moreover the number of available imputations is limited by increasing missing ratio. To settle this problem, we proposed statistical learning theory to preprocess for missing values. Our statistical learning theory is the support vector regression by Vapnik. The proposed method can be applied to sparsely training data. We verified the performance of our model using the data sets from UCI machine learning repository.

• Survey Research
• /
• v.9 no.2
• /
• pp.1-27
• /
• 2008

Problems of incomplete data are pervasive in statistical analysis. In particular, incomplete data have been an important challenge in repeated measures studies. The objective of this study is to give a brief introduction to missing data mechanisms and conventional/recent missing data methods and to assess the performance of various missing data methods under ignorable and non-ignorable missingness mechanisms. Given the inadequate attention to longitudinal studies with missing data, this study applied recent advances in missing data methods to repeated measures models and investigated the performance of various missing data methods, such as FIML (Full Information Maximum Likelihood Estimation) and MICE(Multivariate Imputation by Chained Equations), under MCAR, MAR, and MNAR mechanisms. Overall, the results showed that listwise deletion and mean imputation performed poorly compared to other recommended missing data procedures. The better performance of EM, FIML, and MICE was more noticeable under MAR compared to MCAR. With the non-ignorable missing data, this study showed that missing data methods did not perform well. In particular, this problem was noticeable in slope-related estimates. Therefore, this study suggests that if missing data are suspected to be non-ignorable, developmental research may underestimate true rates of change over the life course. This study also suggests that bias from non-ignorable missing data can be substantially reduced by considering rich information from variables related to missingness.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2018.10a
• /
• pp.342-344
• /
• 2018

This paper shows the possibility to substitute statistical methods such as mean imputation, correlation coefficient analysis, graph correlation analysis for the proposed algorithm, and replace statistician for processing various abnormal data measured in the water treatment process with it. In addition, this study aims to model a data-filtering system based on a recent fractile pattern and a deep learning-based LSTM algorithm in order to improve the reliability and validation of the algorithm, using the open-sourced libraries such as KERAS, THEANO, TENSORFLOW, etc.

• Asian Pacific Journal of Cancer Prevention
• /
• v.16 no.15
• /
• pp.6773-6777
• /
• 2015

We used to LASSO-Cox method for determining prognostic factors of male breast cancer survival and showed the superiority of this method compared to Cox proportional hazard model in low sample size setting. In order to identify and estimate exactly the relative hazard of the most important factors effective for the survival duration of male breast cancer, the LASSO-Cox method has been used. Our data includes the information of male breast cancer patients in Fars province, south of Iran, from 1989 to 2008. Cox proportional hazard and LASSO-Cox models were fitted for 20 classified variables. To reduce the impact of missing data, the multiple imputation method was used 20 times through the Markov chain Mont Carlo method and the results were combined with Rubin's rules. In 50 patients, the age at diagnosis was 59.6 (SD=12.8) years with a minimum of 34 and maximum of 84 years and the mean of survival time was 62 months. Three, 5 and 10 year survival were 92%, 77% and 26%, respectively. Using the LASSO-Cox method led to eliminating 8 low effect variables and also decreased the standard error by 2.5 to 7 times. The relative efficiency of LASSO-Cox method compared with the Cox proportional hazard method was calculated as 22.39. The19 years follow of male breast cancer patients show that the age, having a history of alcohol use, nipple discharge, laterality, histological grade and duration of symptoms were the most important variables that have played an effective role in the patient's survival. In such situations, estimating the coefficients by LASSO-Cox method will be more efficient than the Cox's proportional hazard method.

• Journal of the Korea Convergence Society
• /
• v.12 no.8
• /
• pp.39-45
• /
• 2021

This paper discusses how to handle incomplete data including missing values. Optimally processing the missing value means obtaining an estimate that is the closest to the original value from the information contained in the training data, and replacing the missing value with this value. The way to achieve this is to use a decision tree that is completed in the process of classifying information by the classifier. In other words, this decision tree is obtained in the process of learning by inputting only complete information that does not include loss values among all training data into the C4.5 classifier. The nodes of this decision tree have classification variable information, and the higher node closer to the root contains more information, and the leaf node forms a classification region through a path from the root. In addition, the average of classified data events is recorded in each region. Events including the missing value are input to this decision tree, and the region closest to the event is searched through a traversal process according to the information of each node. The average value recorded in this area is regarded as an estimate of the missing value, and the compensation process is completed.

• Journal of Nutrition and Health
• /
• v.51 no.6
• /
• pp.567-579
• /
• 2018

Purpose: This study examined the mediation effects of dietary habits, nutrient intake, daily life in the relationship between the working hours of Korean shift workers and metabolic syndrome. Methods: Data were collected from the sixth (2013-2015) Korea National Health and Nutrition Examination Survey (KNHANES). The stochastic regression imputation was used to fill missing data. Statistical analysis was performed in Korean shift workers with metabolic syndrome using the SPSS 24 program for Windows and a structural equation model (SEM) using an analysis of moment structure (AMOS) 21.0 package. Results: The model fitted the data well in terms of the goodness of fit index (GFI) = 0.939, root mean square error of approximation (RMSEA) = 0.025, normed fit index (NFI) = 0.917, Tucker-Lewis index (TLI) = 0.984, comparative fit index (CFI) = 0.987, and adjusted goodness of fit index (AGFI) = 0.915. Specific mediation effect of dietary habits (p = 0.023) was statistically significant in the impact of the working hours of shift workers on nutrient intake, and specific mediation effect of daily life (p = 0.019) was statistically significant in the impact of the working hours of shift workers on metabolic syndrome. On the other hand, the dietary habits, nutrient intake and daily life had no significant multiple mediator effects on the working hours of shift workers with metabolic syndrome. Conclusion: The appropriate model suggests that working hours have direct effect on the daily life, which has the mediation effect on the risk of metabolic syndrome in shift workers.