• The Korean Journal of Applied Statistics
• /
• v.34 no.3
• /
• pp.507-521
• /
• 2021

Detecting outliers among high-dimensional data encounters a challenging problem of screening the variables since relevant information is often contained in only a few of the variables. Otherwise, when a number of irrelevant variables are included in the data, the distances between all observations tend to become similar which leads to making the degree of outlierness of all observations alike. The subspace outlier detection method overcomes the problem by measuring the degree of outlierness of the observation based on the relevant subsets of the entire variables. In this paper, we survey recent subspace outlier detection techniques, classifying them into three major types according to the subspace selection method. And we summarize the techniques of each type based on how to select the relevant subspaces and how to measure the degree of outlierness. In addition, we introduce some computing tools for implementing the subspace outlier detection techniques and present results from the simulation study and real data analysis.

• Communications for Statistical Applications and Methods
• /
• v.7 no.1
• /
• pp.141-150
• /
• 2000

In this paper we propose a method of computing Bayes factor to detect an outlier in a random effects model. When no information is available and hence improper noninformative priors should be used Bayes factor includes the unspecified constants and has complicated computational burden. To solve this problem we use the fractional Bayes factor (FBF) of O-Hagan(1995) and the generalized Savage0-Dickey density ratio of Verdinelli and Wasserman (1995) The proposed method is applied to outlier deterction problem We perform a simulation of the proposed approach with a simulated data set including an outlier and also analyze a real data set.

• The Korean Journal of Applied Statistics
• /
• v.27 no.7
• /
• pp.1229-1241
• /
• 2014

Most of sampling surveys have outliers and non-response missing values simultaneously. In that case, due to the effect of outliers, the result of imputation is not good enough to meet a given precision. To overcome this situation, outlier treatment should be conducted before imputation. In this paper in order for reducing the effect of outlier, we study outlier imputation methods and outlier weight adjustment methods. For the outlier detection, the method suggested by She and Owen (2011) is used. A small simulation study is conducted and for real data analysis, Monthly Labor Statistic and Briquette Consumption Survey Data are used.

• The Korean Journal of Applied Statistics
• /
• v.35 no.3
• /
• pp.395-406
• /
• 2022

This paper studies the outlier detection method for multivariate long memory time series. The existing outlier detection methods are based on a short memory VARMA model, so they are not suitable for multivariate long memory time series. It is because higher order of autoregressive model is necessary to account for long memory, however, it can also induce estimation instability as the number of parameter increases. To resolve this issue, we propose outlier detection methods based on the VHAR structure. We also adapt the robust estimation method to estimate VHAR coefficients more efficiently. Our simulation results show that our proposed method performs well in detecting outliers in multivariate long memory time series. Empirical analysis with stock index shows RVHAR model finds additional outliers that existing model does not detect.

• Journal of Internet Computing and Services
• /
• v.18 no.1
• /
• pp.27-35
• /
• 2017

In near future, IoT data is expected to be a major portion of Big Data. Moreover, sensor data is expected to be major portion of IoT data, and its' research is actively carried out currently. However, processed results may not be trusted and used if outlier data is included in the processing of sensor data. Therefore, method for detection and deletion of those outlier data before processing is studied in this paper. Moreover, we used Spark which is memory based distributed processing environment for fast processing of big sensor data. The detection and deletion of outlier data consist of four stages, and each stage is implemented with Mapper and Reducer operation. The proposed method is compared in three different processing environments, and it is expected that the outlier detection and deletion performance is best in the distributed Spark environment as data volume is increasing.

• Communications for Statistical Applications and Methods
• /
• v.26 no.6
• /
• pp.575-582
• /
• 2019

The problem of selecting variables in the presence of outliers is considered. Variable selection and outlier detection are not separable problems because each observation affects the fitted regression equation differently and has a different influence on each variable. We suggest a simultaneous method for variable selection and outlier detection in a linear regression model. The suggested procedure uses a sequential method to detect outliers and uses all possible subset regressions for model selections. A simplified version of the procedure is also proposed to reduce the computational burden. The procedures are compared to other variable selection methods using real data sets known to contain outliers. Examples show that the proposed procedures are effective and superior to robust algorithms in selecting the best model.

• The Korean Journal of Applied Statistics
• /
• v.26 no.4
• /
• pp.635-647
• /
• 2013

Imputation is a commonly used method to handle missing survey data. The performance of the imputation method is influenced by various factors, especially an outlier. The removal of the outlier in a data set is a simple and effective approach to reduce the effect of an outlier. In this paper in order to improve the precision of multiple imputation, we study a imputation method which reduces the effect of outlier using various weight adjustment methods that include the removal of an outlier method. The regression method in PROC/MI in SAS is used for multiple imputation and the obtained final adjusted weight is used as a weight variable to obtain the imputed values. Simulation studies compared the performance of various weight adjustment methods and Monthly Labor Statistic data is used for real data analysis.

• Journal of the Korean Society for Marine Environment & Energy
• /
• v.15 no.2
• /
• pp.156-162
• /
• 2012

Outlier detection and treatment process is highly required as the first step for the statistical analysis of the monitoring data having many outliers frequently occurred in the coastal environmental monitoring projects. In this study, the outlier detection method using the approximate and detail (or residual) components of the (raw) data is suggested. The approximate and detail components of the data can be separated by the diverse filtering and smoothing methods. The decomposition of the data is carried out by the harmonic analysis and local regression curve, respectively. Then, the Grubbs' test and modified z-score method widely used to detect outliers in the data are applied to the detail components of the water temperature data. The new data set is reconstructed after removed the outliers detected by these methods. It can be shown that the suggested process is successfully applied to the outlier detection of the coastal water temperature monitoring data provided by the Real-time Information System for Aquaculture Environment, National Fisheries Research and Development Institute (NFRDI).

• Journal of Positioning, Navigation, and Timing
• /
• v.9 no.4
• /
• pp.397-403
• /
• 2020

In order to synchronize a remote system time to the reference time like Coordinated Universal Time (UTC), it is required to compare the time difference between the two clocks. The time comparison data may have some outliers and the time synchronization performance can be significantly degraded if the outliers are not removed. Therefore, it is required to employ an effective outlier detection algorithm for keeping high accurate system time. In this paper, an outlier detection method is presented for the time difference data of GNSS time transfer receivers. The time difference data between the system time and the GNSS usually have slopes because the remote system clock is under free running until synchronized to the reference clock time. For investigating the outlier detection performance of the proposed algorithm, simulations are performed by using the time difference data of a GNSS time transfer receiver corrected to a free running Cesium clock with intentionally inserted outliers. From the simulation, it is investigated that the proposed algorithm can effectively detect the inserted outliers while conventional methods such as modified Z-score and adjusted boxplot cannot. Furthermore, it is also observed that the synchronization performance can be degraded to more than 15% with 20 outliers compared to that of original data without outliers.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.10 no.12
• /
• pp.5643-5656
• /
• 2016

The detection accuracy of steganalysis depends on many factors, including the embedding algorithm, the payload size, the steganalysis feature space and the properties of the cover source. In practice, the cover source mismatch (CSM) problem has been recognized as the single most important factor negatively affecting the performance. To address this problem, we propose a new framework for blind, universal steganalysis which uses traditional steganalyst features. Firstly, cover images with the same statistical properties are searched from a reference image database as aided samples. The test image and its aided samples form a whole test set. Then, by assuming that most of the aided samples are innocent, we conduct outlier detection on the test set to judge the test image as cover or stego. In this way, the framework has removed the need for training. Hence, it does not suffer from cover source mismatch. Because it performs anomaly detection rather than classification, this method is totally unsupervised. The results in our study show that this framework works superior than one-class support vector machine and the outlier detector without considering the image retrieval process.