• Title/Summary/Keyword: outlier detection method

Search Result 126, Processing Time 0.021 seconds

Automatic Identification of the Lumen Border in Intravascular Ultrasound Images (혈관 내 초음파 영상에서 내강 경계면 자동 분할)

  • Park, Jun-Oh;Ko, Byoung-Chul;Park, Hee-Jun;Nam, Jae-Yeal
    • The KIPS Transactions:PartB
    • /
    • v.19B no.3
    • /
    • pp.201-208
    • /
    • 2012
  • Accurately segmenting lumen border in intravascular ultrasound images (IVUS) is very important to study vascular wall architecture for diagnosis of the cardiovascular diseases. After each of IVUS image is transformed to a polar coordinated image, initial points are detected using wavelet transform. Then, lumen border is initialized as the set of important points using non parametric probability density function and smoothing function by removing outlier initial points occurred by noises and artifacts. Finally, polynomial curve fitting is applied to obtain real lumen border using filtered important points. The evaluation of proposed method was performed with related method and the proposed method produced accurate lumen contour detection when compared to another method in most types of IVUS images.

A Study of Anomaly Detection for ICT Infrastructure using Conditional Multimodal Autoencoder (ICT 인프라 이상탐지를 위한 조건부 멀티모달 오토인코더에 관한 연구)

  • Shin, Byungjin;Lee, Jonghoon;Han, Sangjin;Park, Choong-Shik
    • Journal of Intelligence and Information Systems
    • /
    • v.27 no.3
    • /
    • pp.57-73
    • /
    • 2021
  • Maintenance and prevention of failure through anomaly detection of ICT infrastructure is becoming important. System monitoring data is multidimensional time series data. When we deal with multidimensional time series data, we have difficulty in considering both characteristics of multidimensional data and characteristics of time series data. When dealing with multidimensional data, correlation between variables should be considered. Existing methods such as probability and linear base, distance base, etc. are degraded due to limitations called the curse of dimensions. In addition, time series data is preprocessed by applying sliding window technique and time series decomposition for self-correlation analysis. These techniques are the cause of increasing the dimension of data, so it is necessary to supplement them. The anomaly detection field is an old research field, and statistical methods and regression analysis were used in the early days. Currently, there are active studies to apply machine learning and artificial neural network technology to this field. Statistically based methods are difficult to apply when data is non-homogeneous, and do not detect local outliers well. The regression analysis method compares the predictive value and the actual value after learning the regression formula based on the parametric statistics and it detects abnormality. Anomaly detection using regression analysis has the disadvantage that the performance is lowered when the model is not solid and the noise or outliers of the data are included. There is a restriction that learning data with noise or outliers should be used. The autoencoder using artificial neural networks is learned to output as similar as possible to input data. It has many advantages compared to existing probability and linear model, cluster analysis, and map learning. It can be applied to data that does not satisfy probability distribution or linear assumption. In addition, it is possible to learn non-mapping without label data for teaching. However, there is a limitation of local outlier identification of multidimensional data in anomaly detection, and there is a problem that the dimension of data is greatly increased due to the characteristics of time series data. In this study, we propose a CMAE (Conditional Multimodal Autoencoder) that enhances the performance of anomaly detection by considering local outliers and time series characteristics. First, we applied Multimodal Autoencoder (MAE) to improve the limitations of local outlier identification of multidimensional data. Multimodals are commonly used to learn different types of inputs, such as voice and image. The different modal shares the bottleneck effect of Autoencoder and it learns correlation. In addition, CAE (Conditional Autoencoder) was used to learn the characteristics of time series data effectively without increasing the dimension of data. In general, conditional input mainly uses category variables, but in this study, time was used as a condition to learn periodicity. The CMAE model proposed in this paper was verified by comparing with the Unimodal Autoencoder (UAE) and Multi-modal Autoencoder (MAE). The restoration performance of Autoencoder for 41 variables was confirmed in the proposed model and the comparison model. The restoration performance is different by variables, and the restoration is normally well operated because the loss value is small for Memory, Disk, and Network modals in all three Autoencoder models. The process modal did not show a significant difference in all three models, and the CPU modal showed excellent performance in CMAE. ROC curve was prepared for the evaluation of anomaly detection performance in the proposed model and the comparison model, and AUC, accuracy, precision, recall, and F1-score were compared. In all indicators, the performance was shown in the order of CMAE, MAE, and AE. Especially, the reproduction rate was 0.9828 for CMAE, which can be confirmed to detect almost most of the abnormalities. The accuracy of the model was also improved and 87.12%, and the F1-score was 0.8883, which is considered to be suitable for anomaly detection. In practical aspect, the proposed model has an additional advantage in addition to performance improvement. The use of techniques such as time series decomposition and sliding windows has the disadvantage of managing unnecessary procedures; and their dimensional increase can cause a decrease in the computational speed in inference.The proposed model has characteristics that are easy to apply to practical tasks such as inference speed and model management.

Prediction of Chemical Composition in Distillers Dried Grain with Solubles and Corn Using Real-Time Near-Infrared Reflectance Spectroscopy

  • Choi, Sung Won;Park, Chang Hee;Lee, Chang Sug;Kim, Dong Hee;Park, Sung Kwon;Kim, Beob Gyun;Moon, Sang Ho
    • Journal of The Korean Society of Grassland and Forage Science
    • /
    • v.33 no.3
    • /
    • pp.177-184
    • /
    • 2013
  • This work was conducted to assess the use of Near-infrared reflectance spectroscopy (NIRS) as a technique to analyze nutritional constituents of Distillers dried grain with solubles (DDGS) and corn quickly and accurately, and to apply an NIRS-based indium gallium arsenide array detector, rather than a NIRS-based scanning system, to collect spectra and induce and analyze calibration equations using equipment which is better suited to field application. As a technique to induce calibration equations, Partial Least Squares (PLS) was used, and for better accuracy, various mathematical transformations were applied. A multivariate outlier detection method was applied to induce calibration equations, and, as a result, the way of structuring a calibration set significantly affected prediction accuracy. The prediction of nutritional constituents of distillers dried grains with solubles resulted in the following: moisture ($R^2$=0.80), crude protein ($R^2$=0.71), crude fat ($R^2$=0.80), crude fiber ($R^2$=0.32), and crude ash ($R^2$=0.72). All constituents except crude fiber showed good results. The prediction of nutritional constituents of corn resulted in the following: moisture ($R^2$=0.79), crude protein ($R^2$=0.61), crude fat ($R^2$=0.79), crude fiber ($R^2$=0.63), and crude ash ($R^2$=0.75). Therefore, all constituents except for crude fat and crude fiber were predicted for their chemical composition of DDGS and corn through Near-infrared reflectance spectroscopy.

Iterative Precision Geometric Correction for High-Resolution Satellite Images (고해상도 위성영상의 반복 정밀 기하보정)

  • Son, Jong-Hwan;Yoon, Wansang;Kim, Taejung;Rhee, Sooahm
    • Korean Journal of Remote Sensing
    • /
    • v.37 no.3
    • /
    • pp.431-447
    • /
    • 2021
  • Recently, the use of high-resolution satellites is increasing in many areas. In order to supply useful satellite images stably, it is necessary to establish automatic precision geometric correction technic. Geometric correction is the process that corrected geometric errors of satellite imagery based on the GCP (Ground Control Point), which is correspondence point between accurate ground coordinates and image coordinates. Therefore, in the automatic geometric correction process, it is the key to acquire high-quality GCPs automatically. In this paper, we proposed iterative precision geometry correction method. we constructed an image pyramid and repeatedly performed GCP chip matching, outlier detection, and precision sensor modeling in each layer of the image pyramid. Through this method, we were able to acquire high-quality GCPs automatically. we then improved the performance of geometric correction of high-resolution satellite images. To analyze the performance of the proposed method, we used KOMPSAT-3 and 3A Level 1R 8 scenes. As a result of the experiment, the proposed method showed the geometric correction accuracy of 1.5 pixels on average and a maximum of 2 pixels.

Pupil Data Measurement and Social Emotion Inference Technology by using Smart Glasses (스마트 글래스를 활용한 동공 데이터 수집과 사회 감성 추정 기술)

  • Lee, Dong Won;Mun, Sungchul;Park, Sangin;Kim, Hwan-jin;Whang, Mincheol
    • Journal of Broadcast Engineering
    • /
    • v.25 no.6
    • /
    • pp.973-979
    • /
    • 2020
  • This study aims to objectively and quantitatively determine the social emotion of empathy by collecting pupillary response. 52 subjects (26 men and 26 women) voluntarily participated in the experiment. After the measurement of the reference of 30 seconds, the experiment was divided into the task of imitation and spontaneously self-expression. The two subjects were interacted through facial expressions, and the pupil images were recorded. The pupil data was processed through binarization and circular edge detection algorithm, and outlier detection and removal technique was used to reject eye-blinking. The pupil size according to the empathy was confirmed for statistical significance with test of normality and independent sample t-test. Statistical analysis results, the pupil size was significantly different between empathy (M ± SD = 0.050 ± 1.817)) and non-empathy (M ± SD = 1.659 ± 1.514) condition (t(92) = -4.629, p = 0.000). The rule of empathy according to the pupil size was defined through discriminant analysis, and the rule was verified (Estimation accuracy: 75%) new 12 subjects (6 men and 6 women, mean age ± SD = 22.84 ± 1.57 years). The method proposed in this study is non-contact camera technology and is expected to be utilized in various virtual reality with smart glasses.

Analysis of the Optimal Window Size of Hampel Filter for Calibration of Real-time Water Level in Agricultural Reservoirs (농업용저수지의 실시간 수위 보정을 위한 Hampel Filter의 최적 Window Size 분석)

  • Joo, Dong-Hyuk;Na, Ra;Kim, Ha-Young;Choi, Gyu-Hoon;Kwon, Jae-Hwan;Yoo, Seung-Hwan
    • Journal of The Korean Society of Agricultural Engineers
    • /
    • v.64 no.3
    • /
    • pp.9-24
    • /
    • 2022
  • Currently, a vast amount of hydrologic data is accumulated in real-time through automatic water level measuring instruments in agricultural reservoirs. At the same time, false and missing data points are also increasing. The applicability and reliability of quality control of hydrological data must be secured for efficient agricultural water management through calculation of water supply and disaster management. Considering the characteristics of irregularities in hydrological data caused by irrigation water usage and rainfall pattern, the Korea Rural Community Corporation is currently applying the Hampel filter as a water level data quality management method. This method uses window size as a key parameter, and if window size is large, distortion of data may occur and if window size is small, many outliers are not removed which reduces the reliability of the corrected data. Thus, selection of the optimal window size for individual reservoir is required. To ensure reliability, we compared and analyzed the RMSE (Root Mean Square Error) and NSE (Nash-Sutcliffe model efficiency coefficient) of the corrected data and the daily water level of the RIMS (Rural Infrastructure Management System) data, and the automatic outlier detection standards used by the Ministry of Environment. To select the optimal window size, we used the classification performance evaluation index of the error matrix and the rainfall data of the irrigation period, showing the optimal values at 3 h. The efficient reservoir automatic calibration technique can reduce manpower and time required for manual calibration, and is expected to improve the reliability of water level data and the value of water resources.