• Title/Summary/Keyword: anomaly detection

Search Result 668, Processing Time 0.025 seconds

Synthetic Data Generation and Performance Analysis for Anomaly Detection (이상 탐지를 위한 합성 데이터 생성 및 성능 분석)

  • Hwang, Ju-hyo;Jin, Kyo-hong
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2022.10a
    • /
    • pp.19-21
    • /
    • 2022
  • Anomaly detection using self-supervised learning typically generates synthetic data to learn to classify normal and abnormal, and uses real abnormal data as test data to measure anomaly detection performance. In a study using this method to generate synthetic data similar to normal data, anomaly detection was carried out by generating synthetic data by cutting and pasting a specific patch from the original image. In this way, the degree of similarity to normal data depends on the number and size of patches, which affects anomaly detection performance. In this paper, synthetic data were generated by varying patch sizes and numbers, and then similarity and analysis with normal data were conducted using a pre-trained model, and anomaly detection performance was measured by learning the model.

  • PDF

Performance Comparison of Anomaly Detection Algorithms: in terms of Anomaly Type and Data Properties (이상탐지 알고리즘 성능 비교: 이상치 유형과 데이터 속성 관점에서)

  • Jaeung Kim;Seung Ryul Jeong;Namgyu Kim
    • Journal of Intelligence and Information Systems
    • /
    • v.29 no.3
    • /
    • pp.229-247
    • /
    • 2023
  • With the increasing emphasis on anomaly detection across various fields, diverse anomaly detection algorithms have been developed for various data types and anomaly patterns. However, the performance of anomaly detection algorithms is generally evaluated on publicly available datasets, and the specific performance of each algorithm on anomalies of particular types remains unexplored. Consequently, selecting an appropriate anomaly detection algorithm for specific analytical contexts poses challenges. Therefore, in this paper, we aim to investigate the types of anomalies and various attributes of data. Subsequently, we intend to propose approaches that can assist in the selection of appropriate anomaly detection algorithms based on this understanding. Specifically, this study compares the performance of anomaly detection algorithms for four types of anomalies: local, global, contextual, and clustered anomalies. Through further analysis, the impact of label availability, data quantity, and dimensionality on algorithm performance is examined. Experimental results demonstrate that the most effective algorithm varies depending on the type of anomaly, and certain algorithms exhibit stable performance even in the absence of anomaly-specific information. Furthermore, in some types of anomalies, the performance of unsupervised anomaly detection algorithms was observed to be lower than that of supervised and semi-supervised learning algorithms. Lastly, we found that the performance of most algorithms is more strongly influenced by the type of anomalies when the data quantity is relatively scarce or abundant. Additionally, in cases of higher dimensionality, it was noted that excellent performance was exhibited in detecting local and global anomalies, while lower performance was observed for clustered anomaly types.

Anomaly Intrusion Detection Based on Hyper-ellipsoid in the Kernel Feature Space

  • Lee, Hansung;Moon, Daesung;Kim, Ikkyun;Jung, Hoseok;Park, Daihee
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.9 no.3
    • /
    • pp.1173-1192
    • /
    • 2015
  • The Support Vector Data Description (SVDD) has achieved great success in anomaly detection, directly finding the optimal ball with a minimal radius and center, which contains most of the target data. The SVDD has some limited classification capability, because the hyper-sphere, even in feature space, can express only a limited region of the target class. This paper presents an anomaly detection algorithm for mitigating the limitations of the conventional SVDD by finding the minimum volume enclosing ellipsoid in the feature space. To evaluate the performance of the proposed approach, we tested it with intrusion detection applications. Experimental results show the prominence of the proposed approach for anomaly detection compared with the standard SVDD.

Tropospheric Anomaly Detection in Multi-reference Stations Environment during Localized Atmosphere Conditions-(1) : Basic Concept of Anomaly Detection Algorithm

  • Yoo, Yun-Ja
    • Journal of Navigation and Port Research
    • /
    • v.40 no.5
    • /
    • pp.265-270
    • /
    • 2016
  • Extreme tropospheric anomalies such as typhoons or regional torrential rain can degrade positioning accuracy of the GPS signal. It becomes one of the main error terms affecting high-precision positioning solutions in network RTK. This paper proposed a detection algorithm to be used during atmospheric anomalies in order to detect the tropospheric irregularities that can degrade the quality of correction data due to network errors caused by inhomogeneous atmospheric conditions between multi-reference stations. It uses an atmospheric grid that consists of four meteorological stations and estimates the troposphere zenith total delay difference at a low performance point in an atmospheric grid. AWS (automatic weather station) meteorological data can be applied to the proposed tropospheric anomaly detection algorithm when there are different atmospheric conditions between the stations. The concept of probability density distribution of the delta troposphere slant delay was proposed for the threshold determination.

An Anomaly Detection Algorithm for Cathode Voltage of Aluminum Electrolytic Cell

  • Cao, Danyang;Ma, Yanhong;Duan, Lina
    • Journal of Information Processing Systems
    • /
    • v.15 no.6
    • /
    • pp.1392-1405
    • /
    • 2019
  • The cathode voltage of aluminum electrolytic cell is relatively stable under normal conditions and fluctuates greatly when it has an anomaly. In order to detect the abnormal range of cathode voltage, an anomaly detection algorithm based on sliding window was proposed. The algorithm combines the time series segmentation linear representation method and the k-nearest neighbor local anomaly detection algorithm, which is more efficient than the direct detection of the original sequence. The algorithm first segments the cathode voltage time series, then calculates the length, the slope, and the mean of each line segment pattern, and maps them into a set of spatial objects. And then the local anomaly detection algorithm is used to detect abnormal patterns according to the local anomaly factor and the pattern length. The experimental results showed that the algorithm can effectively detect the abnormal range of cathode voltage.

Structural health monitoring data anomaly detection by transformer enhanced densely connected neural networks

  • Jun, Li;Wupeng, Chen;Gao, Fan
    • Smart Structures and Systems
    • /
    • v.30 no.6
    • /
    • pp.613-626
    • /
    • 2022
  • Guaranteeing the quality and integrity of structural health monitoring (SHM) data is very important for an effective assessment of structural condition. However, sensory system may malfunction due to sensor fault or harsh operational environment, resulting in multiple types of data anomaly existing in the measured data. Efficiently and automatically identifying anomalies from the vast amounts of measured data is significant for assessing the structural conditions and early warning for structural failure in SHM. The major challenges of current automated data anomaly detection methods are the imbalance of dataset categories. In terms of the feature of actual anomalous data, this paper proposes a data anomaly detection method based on data-level and deep learning technique for SHM of civil engineering structures. The proposed method consists of a data balancing phase to prepare a comprehensive training dataset based on data-level technique, and an anomaly detection phase based on a sophisticatedly designed network. The advanced densely connected convolutional network (DenseNet) and Transformer encoder are embedded in the specific network to facilitate extraction of both detail and global features of response data, and to establish the mapping between the highest level of abstractive features and data anomaly class. Numerical studies on a steel frame model are conducted to evaluate the performance and noise immunity of using the proposed network for data anomaly detection. The applicability of the proposed method for data anomaly classification is validated with the measured data of a practical supertall structure. The proposed method presents a remarkable performance on data anomaly detection, which reaches a 95.7% overall accuracy with practical engineering structural monitoring data, which demonstrates the effectiveness of data balancing and the robust classification capability of the proposed network.

A Moving Window Principal Components Analysis Based Anomaly Detection and Mitigation Approach in SDN Network

  • Wang, Mingxin;Zhou, Huachun;Chen, Jia
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.12 no.8
    • /
    • pp.3946-3965
    • /
    • 2018
  • Network anomaly detection in Software Defined Networking, especially the detection of DDoS attack, has been given great attention in recent years. It is convenient to build the Traffic Matrix from a global view in SDN. However, the monitoring and management of high-volume feature-rich traffic in large networks brings significant challenges. In this paper, we propose a moving window Principal Components Analysis based anomaly detection and mitigation approach to map data onto a low-dimensional subspace and keep monitoring the network state in real-time. Once the anomaly is detected, the controller will install the defense flow table rules onto the corresponding data plane switches to mitigate the attack. Furthermore, we evaluate our approach with experiments. The Receiver Operating Characteristic curves show that our approach performs well in both detection probability and false alarm probability compared with the entropy-based approach. In addition, the mitigation effect is impressive that our approach can prevent most of the attacking traffic. At last, we evaluate the overhead of the system, including the detection delay and utilization of CPU, which is not excessive. Our anomaly detection approach is lightweight and effective.

A Pre-processing Process Using TadGAN-based Time-series Anomaly Detection (TadGAN 기반 시계열 이상 탐지를 활용한 전처리 프로세스 연구)

  • Lee, Seung Hoon;Kim, Yong Soo
    • Journal of Korean Society for Quality Management
    • /
    • v.50 no.3
    • /
    • pp.459-471
    • /
    • 2022
  • Purpose: The purpose of this study was to increase prediction accuracy for an anomaly interval identified using an artificial intelligence-based time series anomaly detection technique by establishing a pre-processing process. Methods: Significant variables were extracted by applying feature selection techniques, and anomalies were derived using the TadGAN time series anomaly detection algorithm. After applying machine learning and deep learning methodologies using normal section data (excluding anomaly sections), the explanatory power of the anomaly sections was demonstrated through performance comparison. Results: The results of the machine learning methodology, the performance was the best when SHAP and TadGAN were applied, and the results in the deep learning, the performance was excellent when Chi-square Test and TadGAN were applied. Comparing each performance with the papers applied with a Conventional methodology using the same data, it can be seen that the performance of the MLR was significantly improved to 15%, Random Forest to 24%, XGBoost to 30%, Lasso Regression to 73%, LSTM to 17% and GRU to 19%. Conclusion: Based on the proposed process, when detecting unsupervised learning anomalies of data that are not actually labeled in various fields such as cyber security, financial sector, behavior pattern field, SNS. It is expected to prove the accuracy and explanation of the anomaly detection section and improve the performance of the model.

Effective Dimensionality Reduction of Payload-Based Anomaly Detection in TMAD Model for HTTP Payload

  • Kakavand, Mohsen;Mustapha, Norwati;Mustapha, Aida;Abdullah, Mohd Taufik
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.8
    • /
    • pp.3884-3910
    • /
    • 2016
  • Intrusion Detection System (IDS) in general considers a big amount of data that are highly redundant and irrelevant. This trait causes slow instruction, assessment procedures, high resource consumption and poor detection rate. Due to their expensive computational requirements during both training and detection, IDSs are mostly ineffective for real-time anomaly detection. This paper proposes a dimensionality reduction technique that is able to enhance the performance of IDSs up to constant time O(1) based on the Principle Component Analysis (PCA). Furthermore, the present study offers a feature selection approach for identifying major components in real time. The PCA algorithm transforms high-dimensional feature vectors into a low-dimensional feature space, which is used to determine the optimum volume of factors. The proposed approach was assessed using HTTP packet payload of ISCX 2012 IDS and DARPA 1999 dataset. The experimental outcome demonstrated that our proposed anomaly detection achieved promising results with 97% detection rate with 1.2% false positive rate for ISCX 2012 dataset and 100% detection rate with 0.06% false positive rate for DARPA 1999 dataset. Our proposed anomaly detection also achieved comparable performance in terms of computational complexity when compared to three state-of-the-art anomaly detection systems.

Techniques for Improving Host-based Anomaly Detection Performance using Attack Event Types and Occurrence Frequencies

  • Juyeon Lee;Daeseon Choi;Seung-Hyun Kim
    • Journal of the Korea Society of Computer and Information
    • /
    • v.28 no.11
    • /
    • pp.89-101
    • /
    • 2023
  • In order to prevent damages caused by cyber-attacks on nations, businesses, and other entities, anomaly detection techniques for early detection of attackers have been consistently researched. Real-time reduction and false positive reduction are essential to promptly prevent external or internal intrusion attacks. In this study, we hypothesized that the type and frequency of attack events would influence the improvement of anomaly detection true positive rates and reduction of false positive rates. To validate this hypothesis, we utilized the 2015 login log dataset from the Los Alamos National Laboratory. Applying the preprocessed data to representative anomaly detection algorithms, we confirmed that using characteristics that simultaneously consider the type and frequency of attack events is highly effective in reducing false positives and execution time for anomaly detection.