• Journal of Information Processing Systems
• /
• v.18 no.1
• /
• pp.115-129
• /
• 2022

Adopting Internet of Things (IoT)-based technologies in smart homes helps users analyze home appliances electricity consumption for better overall cost monitoring. The IoT application like smart home system (SHS) could suffer from large missing values gaps due to several factors such as security attacks, sensor faults, or connection errors. In this paper, a novel framework has been proposed to predict large gaps of missing values from the SHS home appliances electricity consumption time-series datasets. The framework follows a series of steps to detect, predict and reconstruct the input time-series datasets of missing values. A hybrid convolutional neural network-long short term memory (CNN-LSTM) neural network used to forecast large missing values gaps. A comparative experiment has been conducted to evaluate the performance of hybrid CNN-LSTM with its single variant CNN and LSTM in forecasting missing values. The experimental results indicate a performance superiority of the CNN-LSTM model over the single CNN and LSTM neural networks.

• Journal of the Korean Data and Information Science Society
• /
• v.24 no.1
• /
• pp.189-200
• /
• 2013

In this paper, it will be assumed that there are two distinct populations which are multivariate normal with equal covariance matrix. We also assume that the two populations are equally likely and the costs of misclassification are equal. The classification rule depends on the situation whether the training samples include missing values or not. We consider the conditional bootstrap confidence intervals for classification error rate when a block of observation is missing.

• The Journal of the Korea Contents Association
• /
• v.21 no.11
• /
• pp.135-144
• /
• 2021

Recently, data-driven decision-making technology has become a key technology leading the data industry, and machine learning technology for this requires high-quality training datasets. However, real-world data contains missing values for various reasons, which degrades the performance of prediction models learned from the poor training data. Therefore, in order to build a high-performance model from real-world datasets, many studies on automatically imputing missing values in initial training data have been actively conducted. Many of conventional machine learning-based imputation techniques for handling missing data involve very time-consuming and cumbersome work because they are applied only to numeric type of columns or create individual predictive models for each columns. Therefore, this paper proposes a new data imputation technique called 'Denoising Self-Attention Network (DSAN)', which can be applied to mixed-type dataset containing both numerical and categorical columns. DSAN can learn robust feature expression vectors by combining self-attention and denoising techniques, and can automatically interpolate multiple missing variables in parallel through multi-task learning. To verify the validity of the proposed technique, data imputation experiments has been performed after arbitrarily generating missing values for several mixed-type training data. Then we show the validity of the proposed technique by comparing the performance of the binary classification models trained on imputed data together with the errors between the original and imputed values.

• The Journal of The Korea Institute of Intelligent Transport Systems
• /
• v.10 no.6
• /
• pp.63-73
• /
• 2011

Travel speed is an important parameter for measuring road traffic. UTIS(Urban Traffic Information System) was developed as a mobile detector for measuring link travel speeds in South Korea. After investigation, we founded that UTIS includes some missing data caused by the lack of probe vehicles on road segments, system failures and etc. Imputation is the practice of filling in missing data with estimated values. In this paper, we suggests a new model for imputing missing data to provide accurate link travel speeds to the public. In the field test, new model showed the travel speed measuring accuracy of 93.6%. Therefore, it can be concluded that the proposed model significantly improves travel speed measuring accuracy.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.7 no.4
• /
• pp.684-689
• /
• 2006

Rainfall is the most basic input data to analyze the hydrologic system. When we measure the rainfall data, the rainfall data can be missing due to various reasons. Therefore, various interpolation methods are available for compensating the missing data. However, the interpolation methods were used without considering their applicability and accuracy. This study compares the interpolation methods such as the arithmetic mean method, normal ratio method, modified normal ratio method, inverse distance method, linear programming, Kriging method to estimate the existing rainfall correction method.

• Journal of the Korean Statistical Society
• /
• v.26 no.4
• /
• pp.493-505
• /
• 1997

The analysis of binary data appears to many areas such as statistics, biometrics and econometrics. In many cases, data are often collected in which some observations are incomplete. Assume that the missing covariates are missing at random and the responses are completely observed. A method to Bayesian analysis of the binary regression model with incomplete data is presented. In particular, the desired marginal posterior moments of regression parameter are obtained using Meterpolis algorithm (Metropolis et al. 1953) within Gibbs sampler (Gelfand and Smith, 1990). Also, we compare logit model with probit model using Bayes factor which is approximated by importance sampling method. One example is presented.

• Journal of the Korean Data and Information Science Society
• /
• v.19 no.4
• /
• pp.1211-1218
• /
• 2008

Maximum likelihood estimator (MLE) and restricted maximum likelihood estimator (REMLE) approaches are available in analyzing the linear mixed model (LMM) like bioequivalence trials. US FDA (2001) guides that REMLE may be useful to assess bioequivalence (BE) test. This paper studies the statistical behaviors of the methods in assessing BE tests when some of observations are missing at random. The simulation results show that the REMLE maintains the given nominal level well and the MLE gives a bit higher power. Considering the levels and the powers, the REMLE approach is recommended when the sample sizes are small to moderate and the MLE approach should be used when the sample size is greater than 30.

• Journal of the Korean Data and Information Science Society
• /
• v.16 no.1
• /
• pp.95-106
• /
• 2005

Complete-case analysis is easy to carry out and it may be fine with small amount of missing data. However, this method is not recommended in general because the estimates are usually biased and not efficient. There are numerous alternatives to complete-case analysis. A natural alternative procedure is available-case analysis. Available-case analysis uses all cases that contain the variables required for a specific task. The EM algorithm is a general approach for computing maximum likelihood estimates of parameters from incomplete data. These methods and multiple imputation(MI) are reviewed and the performances are compared by simulation studies in monotone missing pattern.

• Communications for Statistical Applications and Methods
• /
• v.30 no.4
• /
• pp.355-368
• /
• 2023

This article introduces the R package ELCIC (https://cran.r-project.org/web/packages/ELCIC/index.html), which provides an empirical likelihood-based information criterion (ELCIC) for model selection that includes, but is not limited to, variable selection. The empirical likelihood is a semi-parametric approach to draw statistical inference that does not require distribution assumptions for data generation. Therefore, ELCIC is more robust and versatile in the context of model selection compared to the currently existing information criteria. This paper illustrates several applications of ELCIC, including its use in generalized linear models, generalized estimating equations (GEE) for longitudinal data, and weighted GEE (WGEE) for missing longitudinal data under the mechanisms of missing at random and dropout.

• Geophysics and Geophysical Exploration
• /
• v.23 no.3
• /
• pp.192-207
• /
• 2020

We acquire seismic data with regularly or irregularly missing traces, due to economic, environmental, and mechanical problems. Since these missing data adversely affect the results of seismic data processing and analysis, we need to reconstruct the missing data before subsequent processing. However, there are economic and temporal burdens to conducting further exploration and reconstructing missing parts. Many researchers have been studying interpolation methods to accurately reconstruct missing data. Recently, various machine learning technologies such as support vector regression, autoencoder, U-Net, ResNet, and generative adversarial network (GAN) have been applied in seismic data interpolation. In this study, by reviewing these studies, we found that not only neural network models, but also support vector regression models that have relatively simple structures can interpolate missing parts of seismic data effectively. We expect that future research can improve the interpolation performance of these machine learning models by using open-source field data, data augmentation, transfer learning, and regularization based on conventional interpolation technologies.