• Title/Summary/Keyword: Outlier Detection Method

Search Result 128, Processing Time 0.023 seconds

Anomaly Detection using Geometric Transformation of Normal Sample Images (정상 샘플 이미지의 기하학적 변환을 사용한 이상 징후 검출)

  • Kwon, Yong-Wan;Kang, Dong-Joong
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.22 no.4
    • /
    • pp.157-163
    • /
    • 2022
  • Recently, with the development of automation in the industrial field, research on anomaly detection is being actively conducted. An application for anomaly detection used in factory automation is camera-based defect inspection. Vision camera inspection shows high performance and efficiency in factory automation, but it is difficult to overcome the instability of lighting and environmental conditions. Although camera inspection using deep learning can solve the problem of vision camera inspection with much higher performance, it is difficult to apply to actual industrial fields because it requires a huge amount of normal and abnormal data for learning. Therefore, in this study, we propose a network that overcomes the problem of collecting abnormal data with 72 geometric transformation deep learning methods using only normal data and adds an outlier exposure method for performance improvement. By applying and verifying this to the MVTec data set, which is a database for auto-mobile parts data and outlier detection, it is shown that it can be applied in actual industrial sites.

Correction of Erroneous Individual Vehicle Speed Data Using Locally Weighted Regression (LWR) (국소가중다항회귀분석을 이용한 이상치제거 및 자료보정기법 개발 (GPS를 이용한 개별차량 주행속도를 중심으로))

  • Im, Hui-Seop;O, Cheol;Park, Jun-Hyeong;Lee, Geon-U
    • Journal of Korean Society of Transportation
    • /
    • v.27 no.2
    • /
    • pp.47-56
    • /
    • 2009
  • Effective detection and correction of outliers of raw traffic data collected from the field is of keen interest because reliable traffic information is highly dependent on the quality of raw data. Global positioning system (GPS) based traffic surveillance systems are capable of producing individual vehicle speeds that are invaluable for various traffic management and information strategies. This study proposed a locally weighted regression (LWR) based filtering method for individual vehicle speed data. An important feature of this study was to propose a technique to generate synthetic outliers for more systematic evaluation of the proposed method. It was identified by performance evaluations that the proposed LWR-based method outperformed an exponential smoothing. The proposed method is expected to be effectively utilized for filtering out raw individual vehicle speed data.

Outlier detection using Grubb test and Cochran test in clinical data (그럽 및 코크란 검정을 이용한 임상자료의 이상치 판단)

  • Sohn, Ki-Cheul;Shin, Im-Hee
    • Journal of the Korean Data and Information Science Society
    • /
    • v.23 no.4
    • /
    • pp.657-663
    • /
    • 2012
  • There are very small values and/or very big values which get out of the normal range for survey data in various fields. The reasons of occurrence for outlier are two. One of them is the error in process of data input and the other is the strange response of the respondent. If the data has outliers, then the summary statistics such as the mean and the variance produce misleading information. Therefore, researcher should be careful in detecting the outlier in data. In particular, it is very important problem for clinical fields because the cost of experiment is very high. This article introduce the Grubb test and Cochran test to detect outliers in the data and we apply this method for clinical data.

Wheel tread defect detection for high-speed trains using FBG-based online monitoring techniques

  • Liu, Xiao-Zhou;Ni, Yi-Qing
    • Smart Structures and Systems
    • /
    • v.21 no.5
    • /
    • pp.687-694
    • /
    • 2018
  • The problem of wheel tread defects has become a major challenge for the health management of high-speed rail as a wheel defect with small radius deviation may suffice to give rise to severe damage on both the train bogie components and the track structure when a train runs at high speeds. It is thus highly desirable to detect the defects soon after their occurrences and then conduct wheel turning for the defective wheelsets. Online wheel condition monitoring using wheel impact load detector (WILD) can be an effective solution, since it can assess the wheel condition and detect potential defects during train passage. This study aims to develop an FBG-based track-side wheel condition monitoring method for the detection of wheel tread defects. The track-side sensing system uses two FBG strain gauge arrays mounted on the rail foot, measuring the dynamic strains of the paired rails excited by passing wheelsets. Each FBG array has a length of about 3 m, slightly longer than the wheel circumference to ensure a full coverage for the detection of any potential defect on the tread. A defect detection algorithm is developed for using the online-monitored rail responses to identify the potential wheel tread defects. This algorithm consists of three steps: 1) strain data pre-processing by using a data smoothing technique to remove the trends; 2) diagnosis of novel responses by outlier analysis for the normalized data; and 3) local defect identification by a refined analysis on the novel responses extracted in Step 2. To verify the proposed method, a field test was conducted using a test train incorporating defective wheels. The train ran at different speeds on an instrumented track with the purpose of wheel condition monitoring. By using the proposed method to process the monitoring data, all the defects were identified and the results agreed well with those from the static inspection of the wheelsets in the depot. A comparison is also drawn for the detection accuracy under different running speeds of the test train, and the results show that the proposed method can achieve a satisfactory accuracy in wheel defect detection when the train runs at a speed higher than 30 kph. Some minor defects with a depth of 0.05 mm~0.06 mm are also successfully detected.

Malicious Users Detection and Nullifying their Effects on Cooperative Spectrum Sensing

  • Prasain, Prakash;Choi, Dong-You
    • Journal of Information Technology Services
    • /
    • v.15 no.1
    • /
    • pp.167-178
    • /
    • 2016
  • Spectrum sensing in cognitive radio (CR) has a great role in order to utilize idle spectrum opportunistically, since it is responsible for making available dynamic spectrum access efficiently. In this research area, collaboration among multiple cognitive radio users has been proposed for the betterment of detection reliability. Even though cooperation among them improves the spectrum sensing performance, some falsely reporting malicious users may degrade the performance rigorously. In this article, we have studied the detection and nullifying the harmful effects of such malicious users by applying some well known outlier detection methods based on Grubb's test, Boxplot method and Dixon's test in cooperative spectrum sensing. Initially, the performance of each technique is compared and found that Boxplot method outperforms both Grubb's and Dixon's test for the case where multiple malicious users are present. Secondly, a new algorithm based on reputation and weight is developed to identify malicious users and cancel out their negative impact in final decision making. Simulation results demonstrate that the proposed scheme effectively identifies the malicious users and suppress their harmful effects at the fusion center to decide whether the spectrum is idle.

On study for change point regression problems using a difference-based regression model

  • Park, Jong Suk;Park, Chun Gun;Lee, Kyeong Eun
    • Communications for Statistical Applications and Methods
    • /
    • v.26 no.6
    • /
    • pp.539-556
    • /
    • 2019
  • This paper derive a method to solve change point regression problems via a process for obtaining consequential results using properties of a difference-based intercept estimator first introduced by Park and Kim (Communications in Statistics - Theory Methods, 2019) for outlier detection in multiple linear regression models. We describe the statistical properties of the difference-based regression model in a piecewise simple linear regression model and then propose an efficient algorithm for change point detection. We illustrate the merits of our proposed method in the light of comparison with several existing methods under simulation studies and real data analysis. This methodology is quite valuable, "no matter what regression lines" and "no matter what the number of change points".

Multivariate Outlier Removing for the Risk Prediction of Gas Leakage based Methane Gas (메탄 가스 기반 가스 누출 위험 예측을 위한 다변량 특이치 제거)

  • Dashdondov, Khongorzul;Kim, Mi-Hye
    • Journal of the Korea Convergence Society
    • /
    • v.11 no.12
    • /
    • pp.23-30
    • /
    • 2020
  • In this study, the relationship between natural gas (NG) data and gas-related environmental elements was performed using machine learning algorithms to predict the level of gas leakage risk without directly measuring gas leakage data. The study was based on open data provided by the server using the IoT-based remote control Picarro gas sensor specification. The naturel gas leaks into the air, it is a big problem for air pollution, environment and the health. The proposed method is multivariate outlier removing method based Random Forest (RF) classification for predicting risk of NG leak. After, unsupervised k-means clustering, the experimental dataset has done imbalanced data. Therefore, we focusing our proposed models can predict medium and high risk so best. In this case, we compared the receiver operating characteristic (ROC) curve, accuracy, area under the ROC curve (AUC), and mean standard error (MSE) for each classification model. As a result of our experiments, the evaluation measurements include accuracy, area under the ROC curve (AUC), and MSE; 99.71%, 99.57%, and 0.0016 for MOL_RF respectively.

Online condition assessment of high-speed trains based on Bayesian forecasting approach and time series analysis

  • Zhang, Lin-Hao;Wang, You-Wu;Ni, Yi-Qing;Lai, Siu-Kai
    • Smart Structures and Systems
    • /
    • v.21 no.5
    • /
    • pp.705-713
    • /
    • 2018
  • High-speed rail (HSR) has been in operation and development in many countries worldwide. The explosive growth of HSR has posed great challenges for operation safety and ride comfort. Among various technological demands on high-speed trains, vibration is an inevitable problem caused by rail/wheel imperfections, vehicle dynamics, and aerodynamic instability. Ride comfort is a key factor in evaluating the operational performance of high-speed trains. In this study, online monitoring data have been acquired from an in-service high-speed train for condition assessment. The measured dynamic response signals at the floor level of a train cabin are processed by the Sperling operator, in which the ride comfort index sequence is used to identify the train's operation condition. In addition, a novel technique that incorporates salient features of Bayesian inference and time series analysis is proposed for outlier detection and change detection. The Bayesian forecasting approach enables the prediction of conditional probabilities. By integrating the Bayesian forecasting approach with time series analysis, one-step forecasting probability density functions (PDFs) can be obtained before proceeding to the next observation. The change detection is conducted by comparing the current model and the alternative model (whose mean value is shifted by a prescribed offset) to determine which one can well fit the actual observation. When the comparison results indicate that the alternative model performs better, then a potential change is detected. If the current observation is a potential outlier or change, Bayes factor and cumulative Bayes factor are derived for further identification. A significant change, if identified, implies that there is a great alteration in the train operation performance due to defects. In this study, two illustrative cases are provided to demonstrate the performance of the proposed method for condition assessment of high-speed trains.

Outlier Detection Using Dynamic Plots (동적 그림을 이용한 이상치 검색)

  • Ahn, Byung-Jin;Seo, Han-Son
    • The Korean Journal of Applied Statistics
    • /
    • v.24 no.5
    • /
    • pp.979-986
    • /
    • 2011
  • A linear regression method is commonly used to analyze data because of its simplicity and applicability; however, it is well known that data may contain some outliers and influential cases that may have a harmful effect on a statistical analysis. Thus detection and examination of outliers or influential cases are important parts of data analysis. In detecting multiple outliers, masking effects usually occur and make it difficult to identify the true outliers. We propose to use dynamic plots as a method resistant to masking effect. The procedure using dynamic plots is useful to find appropriate basic sets with which a dependent outliers detection method start and detect a true outliers set. Examples are given to demonstrate the effectiveness of the suggested idea.

Variogram Analysis for Spatial Similarity Measures : A Case Study using Geochemical Data Sets in the Taebaek Area (공간적 상관도 측정을 위한 변이도 분석 : 태백지역의 지화학자료를 이용한 사례 연구)

  • Lee, Kiwon;Kwon, Byung-Doo
    • Economic and Environmental Geology
    • /
    • v.28 no.3
    • /
    • pp.271-277
    • /
    • 1995
  • The geological information analysis based on spatial statistical techniques have been studied in relation to mineral exploration. The applicability of outlier detection using moving-window statistics and directional cross-variography analysis have been verified by using geochemical data sets surveyed in the Taebaek area for mineral exploration. The directional variogram analysis has been basically known as a geostatistical method for spatial continuity measures. In this study, the application of this proposed method was extended to measure spatial correlation or similarity problems between two geochemical elements. For the appraisal of the usefulness of this scheme, five kinds of variogram functions were computed for original data and revised data, obtained by removing outliers detected by moving-window statistics and the results were compared. It is concluded that these advanced spatial statistical methods at the interpretation stage of spatial similarity provide us with valuable quantitative results as decision-supporting information for regional mineral exploration task.

  • PDF