• Journal of the Korean Data and Information Science Society
• /
• v.23 no.4
• /
• pp.657-663
• /
• 2012

There are very small values and/or very big values which get out of the normal range for survey data in various fields. The reasons of occurrence for outlier are two. One of them is the error in process of data input and the other is the strange response of the respondent. If the data has outliers, then the summary statistics such as the mean and the variance produce misleading information. Therefore, researcher should be careful in detecting the outlier in data. In particular, it is very important problem for clinical fields because the cost of experiment is very high. This article introduce the Grubb test and Cochran test to detect outliers in the data and we apply this method for clinical data.

• Ocean and Polar Research
• /
• v.38 no.2
• /
• pp.149-159
• /
• 2016

The statistical measures of the coastal environmental data are used in a variety of statistical inferences, hypothesis tests, and data-driven modeling. If the measures are biased, then the statistical estimations and models may also be biased and this potential for bias is great when data contain some outliers defined as extraordinary large or small data values. This study aims to suggest more robust statistical measures as alternatives to more commonly used measures and to assess the performance these robust measures through a quantitative evaluation of more typical measures, such as in terms of locations, spreads, and shapes, with regard to environmental monitoring data in the Busan coastal sea. The detection of outliers within the data was carried out on the basis of Rosner's test. About 5-10% of the nutrient data were found to contain outliers based on Rosner's test. After removal (zero-weighting) of the outliers in the data sets, the relative change ratios of the mean and standard deviation between before and after outlier-removal conditions revealed the figures 13 and 33%, respectively. The variation magnitudes of skewness and kurtosis are 1.36 and 8.11 in a decreasing trend, respectively. On the other hand, the change ratios for more robust measures regarding the mean and standard deviation are 3.7-10.5%, and the variation magnitudes of robust skewness and kurtosis are about only 2-4% of the magnitude of the non-robust measures. The robust measures can be regarded as outlier-resistant statistical measures based on the relatively small changes in the scenarios before and after outlier removal conditions.

• Journal of Aerospace System Engineering
• /
• v.8 no.4
• /
• pp.39-43
• /
• 2014

This paper examines the statistical process that should be performed with caution in the composite material qualification and equivalency process, and describes statistically significant considerations on outlier finding and handling process, data pooling through normalization process, review for data distributions and design allowables determination process for structural analysis. Based on these considerations, the need for guidance on statistical process for aircraft manufacturers who use the composite material properties database are proposed.

• Journal of Korean Society on Water Environment
• /
• v.26 no.4
• /
• pp.700-706
• /
• 2010

Along the 4 major rivers in korea, there are automatic water quality monitoring (AWQM) stations to immediately respond to any pollution incident. Real-time data (temperature, DO, pH, EC and TOC) collected at each station were statistically treated to exclude outliers and keep valid data using Dixon's test and Discordance test. These applied methods were compared in terms of the number of the outliers sorted out. There was no significant difference between these methods. On the other hand, more outliers were sorted out from EC and TOC data, comparing with other water quality items. EC data did not show partly any variation for a long time at H station. If measured signal does not exceed ${\pm}0.001mS/cm$ from the sectional mean, the signal should be treated as normal data. Therefore, another routine was added to the data screening system, some data which were removed as outlier were restored.

• The Korean Journal of Applied Statistics
• /
• v.28 no.5
• /
• pp.847-858
• /
• 2015

Deep-sea fishery in the Antarctic Ocean has been actively progressed by the developed countries including Korea. In order to prevent the environmental destruction of the Antarctic Ocean, related countries have established the Commission for the Conservation of Antarctic Marine Living Resources (CCAMLR) and have monitored any illegal unreported or unregulated fishing. Fishing of tooth fish, an expensive fish, in the Antarctic Ocean has increased recently and high catches per unit effort (CPUE) of fishing boats, which is suspicious for an illegal activity, have been frequently reported. The data of CPUEs in a fishing area of the Antarctic Ocean often show an extreme Distribution or a mixture of two extreme distributions. This paper proposes an algorithm to detect an outlier of CPUEs by using the mixture of two extreme distributions. The parameters of the mixture distribution are estimated by the EM algorithm. Log likelihood value and posterior probabilities are used to detect an outlier. Experiments show that the proposed algorithm to detect outlier of the data can be adopted instead of simple criteria such as a CPUE is greater than 1.

• Smart Media Journal
• /
• v.13 no.4
• /
• pp.48-56
• /
• 2024

In the livestock industry, detecting environmental outliers and predicting data are crucial tasks. Outliers in livestock environment data, typically gathered through time-series methods, can signal rapid changes in the environment and potential unexpected epidemics. Prompt detection and response to these outliers are essential to minimize stress in livestock and reduce economic losses for farmers by early detection of epidemic conditions. This study employs two methods to experiment and compare performances in setting thresholds that define outliers in livestock environment data outlier detection. The first method is an outlier detection using Mean Squared Error (MSE), and the second is an outlier detection using a Dynamic Threshold, which analyzes variability against the average value of previous data to identify outliers. The MSE-based method demonstrated a 94.98% accuracy rate, while the Dynamic Threshold method, which uses standard deviation, showed superior performance with 99.66% accuracy.

• Journal of the Korean Statistical Society
• /
• v.36 no.4
• /
• pp.457-469
• /
• 2007

Many procedures are available to identify a single outlier or an isolated influential point in linear regression and logistic regression. But the detection of influential points or multiple outliers is more difficult, owing to masking and swamping problems. The multiple outlier detection methods for logistic regression have not been studied from the points of direct procedure yet. In this paper we consider the direct methods for logistic regression by extending the $Pe\tilde{n}a$ and Yohai (1995) influence matrix algorithm. We define the influence matrix in logistic regression by using Cook's distance in logistic regression, and test multiple outliers by using the mean shift model. To show accuracy of the proposed multiple outlier detection algorithm, we simulate artificial data including multiple outliers with masking and swamping.

• Journal of Korean Society of Coastal and Ocean Engineers
• /
• v.33 no.6
• /
• pp.265-274
• /
• 2021

Outlier detection research in ocean data has traditionally been performed using statistical and distance-based machine learning algorithms. Recently, AI-based methods have received a lot of attention and so-called supervised learning methods that require classification information for data are mainly used. This supervised learning method requires a lot of time and costs because classification information (label) must be manually designated for all data required for learning. In this study, an autoencoder based on unsupervised learning was applied as an outlier detection to overcome this problem. For the experiment, two experiments were designed: one is univariate learning, in which only SST data was used among the observation data of Deokjeok Island and the other is multivariate learning, in which SST, air temperature, wind direction, wind speed, air pressure, and humidity were used. Period of data is 25 years from 1996 to 2020, and a pre-processing considering the characteristics of ocean data was applied to the data. An outlier detection of actual SST data was tried with a learned univariate and multivariate autoencoder. We tried to detect outliers in real SST data using trained univariate and multivariate autoencoders. To compare model performance, various outlier detection methods were applied to synthetic data with artificially inserted errors. As a result of quantitatively evaluating the performance of these methods, the multivariate/univariate accuracy was about 96%/91%, respectively, indicating that the multivariate autoencoder had better outlier detection performance. Outlier detection using an unsupervised learning-based autoencoder is expected to be used in various ways in that it can reduce subjective classification errors and cost and time required for data labeling.

• Journal of the Korean Data and Information Science Society
• /
• v.22 no.3
• /
• pp.449-458
• /
• 2011

Illegal waste dumping is one of the major problems that the government agency monitoring water quality has to face. Recently government agency installed COD (chemical oxygen demand) auto-monitering machines in river. In this article we provide an outlier detection algorithm using R based on the time series intervention model that detects some outlier values among those COD time series values generated from an auto-monitering machine. Through this algorithm using R, we can achieve an automatic algorithm that does not need manual intervention in each step, and that can further be used in simulation study.

• Annual Conference of KIPS
• /
• 2019.10a
• /
• pp.760-761
• /
• 2019

Smart manufacturing is a process of integrating computer-related technologies in production and by doing so, achieving more efficient production management. The recent development of supercomputers has led to the broad utilization of artificial intelligence (AI) and machine learning techniques useful in predicting specific patterns. Despite the usefulness of AI and machine learning techniques in smart manufacturing processes, there are many fundamental issues with the direct deployment of these technologies related to data management. In this paper, we focus on solving the outlier detection issue in smart manufacturing applications. More specifically, we apply a state-of-the-art outlier detection technique, called Elliptic Envelope, to detect anomalies in simulation-based collected data.