• 제목/요약/키워드: data anomaly detection

검색결과 380건 처리시간 0.026초

패턴사전과 비정형성을 통한 이상치 탐지방법 적용 (Anomaly Detection via Pattern Dictionary Method and Atypicality in Application)

  • 오세홍;박종성;윤영삼
    • 센서학회지
    • /
    • 제32권6호
    • /
    • pp.481-486
    • /
    • 2023
  • Anomaly detection holds paramount significance across diverse fields, encompassing fraud detection, risk mitigation, and sensor evaluation tests. Its pertinence extends notably to the military, particularly within the Warrior Platform, a comprehensive combat equipment system with wearable sensors. Hence, we propose a data-compression-based anomaly detection approach tailored to unlabeled time series and sequence data. This method entailed the construction of two distinctive features, typicality and atypicality, to discern anomalies effectively. The typicality of a test sequence was determined by evaluating the compression efficacy achieved through the pattern dictionary. This dictionary was established based on the frequency of all patterns identified in a training sequence generated for each sensor within Warrior Platform. The resulting typicality served as an anomaly score, facilitating the identification of anomalous data using a predetermined threshold. To improve the performance of the pattern dictionary method, we leveraged atypicality to discern sequences that could undergo compression independently without relying on the pattern dictionary. Consequently, our refined approach integrated both typicality and atypicality, augmenting the effectiveness of the pattern dictionary method. Our proposed method exhibited heightened capability in detecting a spectrum of unpredictable anomalies, fortifying the stability of wearable sensors prevalent in military equipment, including the Army TIGER 4.0 system.

딥러닝 기반의 투명 렌즈 이상 탐지 알고리즘 성능 비교 및 적용 (Comparison and Application of Deep Learning-Based Anomaly Detection Algorithms for Transparent Lens Defects)

  • 김한비;서대호
    • 산업경영시스템학회지
    • /
    • 제47권1호
    • /
    • pp.9-19
    • /
    • 2024
  • Deep learning-based computer vision anomaly detection algorithms are widely utilized in various fields. Especially in the manufacturing industry, the difficulty in collecting abnormal data compared to normal data, and the challenge of defining all potential abnormalities in advance, have led to an increasing demand for unsupervised learning methods that rely on normal data. In this study, we conducted a comparative analysis of deep learning-based unsupervised learning algorithms that define and detect abnormalities that can occur when transparent contact lenses are immersed in liquid solution. We validated and applied the unsupervised learning algorithms used in this study to the existing anomaly detection benchmark dataset, MvTecAD. The existing anomaly detection benchmark dataset primarily consists of solid objects, whereas in our study, we compared unsupervised learning-based algorithms in experiments judging the shape and presence of lenses submerged in liquid. Among the algorithms analyzed, EfficientAD showed an AUROC and F1-score of 0.97 in image-level tests. However, the F1-score decreased to 0.18 in pixel-level tests, making it challenging to determine the locations where abnormalities occurred. Despite this, EfficientAD demonstrated excellent performance in image-level tests classifying normal and abnormal instances, suggesting that with the collection and training of large-scale data in real industrial settings, it is expected to exhibit even better performance.

An Effective Anomaly Detection Approach based on Hybrid Unsupervised Learning Technologies in NIDS

  • Kangseok Kim
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제18권2호
    • /
    • pp.494-510
    • /
    • 2024
  • Internet users are exposed to sophisticated cyberattacks that intrusion detection systems have difficulty detecting. Therefore, research is increasing on intrusion detection methods that use artificial intelligence technology for detecting novel cyberattacks. Unsupervised learning-based methods are being researched that learn only from normal data and detect abnormal behaviors by finding patterns. This study developed an anomaly-detection method based on unsupervised machines and deep learning for a network intrusion detection system (NIDS). We present a hybrid anomaly detection approach based on unsupervised learning techniques using the autoencoder (AE), Isolation Forest (IF), and Local Outlier Factor (LOF) algorithms. An oversampling approach that increased the detection rate was also examined. A hybrid approach that combined deep learning algorithms and traditional machine learning algorithms was highly effective in setting the thresholds for anomalies without subjective human judgment. It achieved precision and recall rates respectively of 88.2% and 92.8% when combining two AEs, IF, and LOF while using an oversampling approach to learn more unknown normal data improved the detection accuracy. This approach achieved precision and recall rates respectively of 88.2% and 94.6%, further improving the detection accuracy compared with the hybrid method. Therefore, in NIDS the proposed approach provides high reliability for detecting cyberattacks.

Design and Evaluation of a Dynamic Anomaly Detection Scheme Considering the Age of User Profiles

  • Lee, Hwa-Ju;Bae, Ihn-Han
    • Journal of the Korean Data and Information Science Society
    • /
    • 제18권2호
    • /
    • pp.315-326
    • /
    • 2007
  • The rapid proliferation of wireless networks and mobile computing applications has changed the landscape of network security. Anomaly detection is a pattern recognition task whose goal is to report the occurrence of abnormal or unknown behavior in a given system being monitored. This paper presents a dynamic anomaly detection scheme that can effectively identify a group of especially harmful internal masqueraders in cellular mobile networks. Our scheme uses the trace data of wireless application layer by a user as feature value. Based on the feature values, the use pattern of a mobile's user can be captured by rough sets, and the abnormal behavior of the mobile can be also detected effectively by applying a roughness membership function with both the age of the user profile and weighted feature values. The performance of our scheme is evaluated by a simulation. Simulation results demonstrate that the anomalies are well detected by the proposed dynamic scheme that considers the age of user profiles.

  • PDF

Anomaly Detection in Smart Homes Using Bayesian Networks

  • Saqaeeyan, Sasan;javadi, Hamid Haj Seyyed;Amirkhani, Hossein
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제14권4호
    • /
    • pp.1796-1816
    • /
    • 2020
  • The health and safety of elderly and disabled patients who cannot live alone is an important issue. Timely detection of sudden events is necessary to protect these people, and anomaly detection in smart homes is an efficient approach to extracting such information. In the real world, there is a causal relationship between an occupant's behaviour and the order in which appliances are used in the home. Bayesian networks are appropriate tools for assessing the probability of an effect due to the occurrence of its causes, and vice versa. This paper defines different subsets of random variables on the basis of sensory data from a smart home, and it presents an anomaly detection system based on various models of Bayesian networks and drawing upon these variables. We examine different models to obtain the best network, one that has higher assessment scores and a smaller size. Experimental evaluations of real datasets show the effectiveness of the proposed method.

산업제어시스템의 이상 탐지 성능 개선을 위한 데이터 보정 방안 연구 (Research on Data Tuning Methods to Improve the Anomaly Detection Performance of Industrial Control Systems)

  • 전상수;이경호
    • 정보보호학회논문지
    • /
    • 제32권4호
    • /
    • pp.691-708
    • /
    • 2022
  • 머신러닝과 딥러닝의 기술이 보편화되면서 산업제어시스템의 이상(비정상) 탐지 연구에도 적용이 되기 시작하였다. 국내에서는 산업제어시스템의 이상 탐지를 위한 인공지능 연구를 활성화시키기 위하여 HAI 데이터셋을 개발하여 공개하였고, 산업제어시스템 보안위협 탐지 AI 경진대회를 시행하고 있다. 이상 탐지 연구들은 대개 기존의 딥러닝 학습 알고리즘을 변형하거나 다른 알고리즘과 함께 적용하는 앙상블 학습 모델의 방법을 통해 향상된 성능의 학습 모델을 만드는 연구가 대부분 이었다. 본 연구에서는 학습 모델과 데이터 전처리(pre-processing)의 개선을 통한 방법이 아니라, 비정상 데이터를 탐지하여 라벨링 한 결과를 보정하는 후처리(post-processing) 방법으로 이상 탐지의 성능을 개선시키는 연구를 진행하였고, 그 결과 기존 모델의 이상 탐지 성능 대비 약 10%이상의 향상된 결과를 확인하였다.

Development of deep autoencoder-based anomaly detection system for HANARO

  • Seunghyoung Ryu;Byoungil Jeon ;Hogeon Seo ;Minwoo Lee;Jin-Won Shin;Yonggyun Yu
    • Nuclear Engineering and Technology
    • /
    • 제55권2호
    • /
    • pp.475-483
    • /
    • 2023
  • The high-flux advanced neutron application reactor (HANARO) is a multi-purpose research reactor at the Korea Atomic Energy Research Institute (KAERI). HANARO has been used in scientific and industrial research and developments. Therefore, stable operation is necessary for national science and industrial prospects. This study proposed an anomaly detection system based on deep learning, that supports the stable operation of HANARO. The proposed system collects multiple sensor data, displays system information, analyzes status, and performs anomaly detection using deep autoencoder. The system comprises communication, visualization, and anomaly-detection modules, and the prototype system is implemented on site in 2021. Finally, an analysis of the historical data and synthetic anomalies was conducted to verify the overall system; simulation results based on the historical data show that 12 cases out of 19 abnormal events can be detected in advance or on time by the deep learning AD model.

하둡 에코시스템을 활용한 로그 데이터의 이상 탐지 기법 (Anomaly Detection Technique of Log Data Using Hadoop Ecosystem)

  • 손시운;길명선;문양세
    • 정보과학회 컴퓨팅의 실제 논문지
    • /
    • 제23권2호
    • /
    • pp.128-133
    • /
    • 2017
  • 최근 대용량 데이터 분석을 위해 다수의 서버를 사용하는 시스템이 증가하고 있다. 대표적인 빅데이터 기술인 하둡은 대용량 데이터를 다수의 서버로 구성된 분산 환경에 저장하여 처리한다. 이러한 분산 시스템에서는 각 서버의 시스템 자원 관리가 매우 중요하다. 본 논문은 다수의 서버에서 수집된 로그 데이터를 토대로 간단하면서 효율적인 이상 탐지 기법을 사용하여 로그 데이터의 변화가 급증하는 이상치를 탐지하고자 한다. 이를 위해, 각 서버로부터 로그 데이터를 수집하여 하둡 에코시스템에 저장할 수 있도록 Apache Hive의 저장 구조를 설계하고, 이동 평균 및 3-시그마를 사용한 세 가지 이상 탐지 기법을 설계한다. 마지막으로 실험을 통해 세 가지 기법이 모두 올바로 이상 구간을 탐지하며, 또한 가중치가 적용된 이상 탐지 기법이 중복을 제거한 더 정확한 탐지 기법임을 확인한다. 본 논문은 하둡 에코시스템을 사용하여 간단한 방법으로 로그 데이터의 이상을 탐지하는 우수한 결과라 사료된다.

Robust transformer-based anomaly detection for nuclear power data using maximum correntropy criterion

  • Shuang Yi;Sheng Zheng;Senquan Yang;Guangrong Zhou;Junjie He
    • Nuclear Engineering and Technology
    • /
    • 제56권4호
    • /
    • pp.1284-1295
    • /
    • 2024
  • Due to increasing operational security demands, digital and intelligent condition monitoring of nuclear power plants is becoming more significant. However, establishing an accurate and effective anomaly detection model is still challenging. This is mainly because of data characteristics of nuclear power data, including the lack of clear class labels combined with frequent interference from outliers and anomalies. In this paper, we introduce a Transformer-based unsupervised model for anomaly detection of nuclear power data, a modified loss function based on the maximum correntropy criterion (MCC) is applied in the model training to improve the robustness. Experimental results on simulation datasets demonstrate that the proposed Trans-MCC model achieves equivalent or superior detection performance to the baseline models, and the use of the MCC loss function is proven can obviously alleviate the negative effect of outliers and anomalies in the training procedure, the F1 score is improved by up to 0.31 compared to Trans-MSE on a specific dataset. Further studies on genuine nuclear power data have verified the model's capability to detect anomalies at an earlier stage, which is significant to condition monitoring.

멀티모달 딥 러닝 기반 이상 상황 탐지 방법론 (Anomaly Detection Methodology Based on Multimodal Deep Learning)

  • 이동훈;김남규
    • 지능정보연구
    • /
    • 제28권2호
    • /
    • pp.101-125
    • /
    • 2022
  • 최근 컴퓨팅 기술의 발전과 클라우드 환경의 개선에 따라 딥 러닝 기술이 발전하게 되었으며, 다양한 분야에 딥 러닝을 적용하려는 시도가 많아지고 있다. 대표적인 예로 정상적인 데이터에서 벗어나는 값이나 패턴을 식별하는 기법인 이상 탐지가 있으며, 이상 탐지의 대표적 유형인 점 이상, 집단적 이상, 맥락적 이중 특히 전반적인 상황을 파악해야 하는 맥락적 이상을 탐지하는 것은 매우 어려운 것으로 알려져 있다. 일반적으로 이미지 데이터의 이상 상황 탐지는 대용량 데이터로 학습된 사전학습 모델을 사용하여 이루어진다. 하지만 이러한 사전학습 모델은 이미지의 객체 클래스 분류에 초점을 두어 생성되었기 때문에, 다양한 객체들이 만들어내는 복잡한 상황을 탐지해야 하는 이상 상황 탐지에 그대로 적용되기에는 한계가 있다. 이에 본 연구에서는 객체 클래스 분류를 학습한 사전학습 모델을 기반으로 이미지 캡셔닝 학습을 추가적으로 수행하여, 객체 파악뿐만 아니라 객체들이 만들어내는 상황까지 이해해야 하는 이상 상황 탐지에 적절한 2 단계 사전학습 모델 구축 방법론을 제안한다. 구체적으로 제안 방법론은 ImageNet 데이터로 클래스 분류를 학습한 사전학습 모델을 이미지 캡셔닝 모델에 전이하고, 이미지가 나타내는 상황을 설명한 캡션을 입력 데이터로 사용하여 학습을 진행한다. 이후 이미지와 캡션을 통해 상황 특질을 학습한 가중치를 추출하고 이에 대한 미세 조정을 수행하여 이상 상황 탐지 모델을 생성한다. 제안 방법론의 성능을 평가하기 위해 직접 구축한 데이터 셋인 상황 이미지 400장에 대해 이상 탐지 실험을 수행하였으며, 실험 결과 제안 방법론이 기존의 단순 사전학습 모델에 비해 이상 상황 탐지 정확도와 F1-score 측면에서 우수한 성능을 나타냄을 확인하였다.