• KSII Transactions on Internet and Information Systems (TIIS)
• /
• 제18권2호
• /
• pp.494-510
• /
• 2024

Internet users are exposed to sophisticated cyberattacks that intrusion detection systems have difficulty detecting. Therefore, research is increasing on intrusion detection methods that use artificial intelligence technology for detecting novel cyberattacks. Unsupervised learning-based methods are being researched that learn only from normal data and detect abnormal behaviors by finding patterns. This study developed an anomaly-detection method based on unsupervised machines and deep learning for a network intrusion detection system (NIDS). We present a hybrid anomaly detection approach based on unsupervised learning techniques using the autoencoder (AE), Isolation Forest (IF), and Local Outlier Factor (LOF) algorithms. An oversampling approach that increased the detection rate was also examined. A hybrid approach that combined deep learning algorithms and traditional machine learning algorithms was highly effective in setting the thresholds for anomalies without subjective human judgment. It achieved precision and recall rates respectively of 88.2% and 92.8% when combining two AEs, IF, and LOF while using an oversampling approach to learn more unknown normal data improved the detection accuracy. This approach achieved precision and recall rates respectively of 88.2% and 94.6%, further improving the detection accuracy compared with the hybrid method. Therefore, in NIDS the proposed approach provides high reliability for detecting cyberattacks.

• 한국컴퓨터정보학회논문지
• /
• 제24권9호
• /
• pp.21-27
• /
• 2019

In this paper, we propose a deep auto-encoder-based pipe leak detection (PLD) technique from time-series acoustic data collected by microphone sensor nodes. The key idea of the proposed technique is to learn representative features of the leak-free state using leak-free time-series acoustic data and the deep auto-encoder. The proposed technique can be used to create a PLD model that detects leaks in the pipeline in an unsupervised learning manner. This means that we only use leak-free data without labeling while training the deep auto-encoder. In addition, when compared to the previous supervised learning-based PLD method that uses image features, this technique does not require complex preprocessing of time-series acoustic data owing to the unsupervised feature extraction scheme. The experimental results show that the proposed PLD method using the deep auto-encoder can provide reliable PLD accuracy even considering unsupervised learning-based feature extraction.

• Asia-Pacific Journal of Atmospheric Sciences
• /
• 제54권4호
• /
• pp.527-544
• /
• 2018

This paper presents a nighttime sea fog detection algorithm incorporating unsupervised learning technique. The algorithm is based on data sets that combine brightness temperatures from the $3.7{\mu}m$ and $10.8{\mu}m$ channels of the meteorological imager (MI) onboard the Communication, Ocean and Meteorological Satellite (COMS), with sea surface temperature from the Operational Sea Surface Temperature and Sea Ice Analysis (OSTIA). Previous algorithms generally employed threshold values including the brightness temperature difference between the near infrared and infrared. The threshold values were previously determined from climatological analysis or model simulation. Although this method using predetermined thresholds is very simple and effective in detecting low cloud, it has difficulty in distinguishing fog from stratus because they share similar characteristics of particle size and altitude. In order to improve this, the unsupervised learning approach, which allows a more effective interpretation from the insufficient information, has been utilized. The unsupervised learning method employed in this paper is the expectation-maximization (EM) algorithm that is widely used in incomplete data problems. It identifies distinguishing features of the data by organizing and optimizing the data. This allows for the application of optimal threshold values for fog detection by considering the characteristics of a specific domain. The algorithm has been evaluated using the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) vertical profile products, which showed promising results within a local domain with probability of detection (POD) of 0.753 and critical success index (CSI) of 0.477, respectively.

• 산업경영시스템학회지
• /
• 제47권1호
• /
• pp.9-19
• /
• 2024

Deep learning-based computer vision anomaly detection algorithms are widely utilized in various fields. Especially in the manufacturing industry, the difficulty in collecting abnormal data compared to normal data, and the challenge of defining all potential abnormalities in advance, have led to an increasing demand for unsupervised learning methods that rely on normal data. In this study, we conducted a comparative analysis of deep learning-based unsupervised learning algorithms that define and detect abnormalities that can occur when transparent contact lenses are immersed in liquid solution. We validated and applied the unsupervised learning algorithms used in this study to the existing anomaly detection benchmark dataset, MvTecAD. The existing anomaly detection benchmark dataset primarily consists of solid objects, whereas in our study, we compared unsupervised learning-based algorithms in experiments judging the shape and presence of lenses submerged in liquid. Among the algorithms analyzed, EfficientAD showed an AUROC and F1-score of 0.97 in image-level tests. However, the F1-score decreased to 0.18 in pixel-level tests, making it challenging to determine the locations where abnormalities occurred. Despite this, EfficientAD demonstrated excellent performance in image-level tests classifying normal and abnormal instances, suggesting that with the collection and training of large-scale data in real industrial settings, it is expected to exhibit even better performance.

• 전기전자학회논문지
• /
• 제24권4호
• /
• pp.1011-1016
• /
• 2020

We propose a framework called Stacked Gated Recurrent Unit - Infrequent Residual Analysis (SG-IRA) that detects anomalies in time-series data that can be trained on streams of raw sensor data without any pre-labeled dataset. To enable such unsupervised learning, SG-IRA includes an estimation model that uses a stacked Gated Recurrent Unit (GRU) structure and an analysis method that detects anomalies based on the difference between the estimated value and the actual measurement (residual). SG-IRA's residual analysis method dynamically adapts the detection threshold from the population using frequency analysis, unlike the baseline model that relies on a constant threshold. In this paper, SG-IRA is evaluated using the industrial control systems (ICS) datasets. SG-IRA improves the detection performance (F1 score) by 5.9% compared to the baseline model.

• 한국측량학회:학술대회논문집
• /
• 한국측량학회 2006년도 춘계학술발표회 논문집
• /
• pp.243-248
• /
• 2006

In this paper, we propose the unsupervised change detection algorithm that apply the similarity measure techniques to the hyperspectral image. The general similarity measures including euclidean distance and spectral angle were compared. The spectral similarity scale algorithm for reducing the problems of those techniques was studied and tested with Hyperion data. The thresholds for detecting the change area were estimated through EM(Expectation-Maximization) algorithm. The experimental result shows that the similarity measure techniques and EM algorithm can be applied effectively for the unsupervised change detection of the hyperspectral data.

• 정보보호학회논문지
• /
• 제33권1호
• /
• pp.1-12
• /
• 2023

다변량 시계열 이상 탐지 과업에서 정답 값이 존재하는 데이터를 얻는 것은 매우 시간 집약적인 일이다. 따라서 최근 정답 값이 필요 없는 비지도 학습법(unsupervised learning)에 관한 많은 연구가 진행되었다. 하지만 다변량 시계열 이상 탐지 과업에 특화된 주요 구조와 세부적인 특성에 대한 심화 있는 논의는 이루어지지 않았다. 본 논문에서는 비지도 학습 기반의 다변량 시계열 이상 탐지 모델과 특장점을 포괄적으로 분석하여 분류하였다. 전력 계통(power grid) 또는 Cyber Physical System(CPS)과 같은 현실 세계 데이터 집합에서 현실적인 이상 상황을 고려하여 학습을 진행하였고, 실험 결과를 바탕으로 각 모델의 정량적 성능을 비교 분석하였다. 성능 지표로는 정밀도(precision), 재현율(recall)과 F1 점수를 사용하여 성능을 측정하였다.

• 정보처리학회논문지:소프트웨어 및 데이터공학
• /
• 제4권3호
• /
• pp.129-134
• /
• 2015

역 사회공학 기반 스팸공격은 공격자가 직접적인 공격을 수행하는 것이 아니라 피해자가 문제 있는 사이트 주소, 문자, 이메일 수신 및 친구 수락 등을 통해 유도하기 때문에 온라인 소셜 네트워크에서 활성화되기 쉽다. 스팸 탐지 관련 기존 연구들은 소셜 네트워크 특성을 반영하지 않은 채, 관리자의 수동적인 판단 및 라벨링을 바탕으로 스팸을 정상 데이터와 구분하는 단계에 머물러있다. 본 논문에서는 소셜 네트워크 데이터 중 하나인 Twitter spam데이터 셋을 실제로 분석하고 소셜 네트워크에서 다양한 속성들을 반영하여 정상 (ham)과 비정상 (spam)을 구분할 수 있는 탐지 메트릭을 제안한다. 또한, 관리자의 관여 없이도 실시간 및 점진적으로 스팸의 특성을 학습하여 새로운 스팸에 대해서도 탐지할 수 있는 비지도 학습 기법(unsupervised scheme)을 제안한다. 실험 결과, 제안하는 기법은 90% 이상의 정확도로 정상과 스팸을 구별했고 실시간 및 점진적 학습 결과도 정확함을 보였다.

• ETRI Journal
• /
• 제41권5호
• /
• pp.684-695
• /
• 2019

In a cloud environment, performance degradation, or even downtime, of virtual machines (VMs) usually appears gradually along with anomalous states of VMs. To better characterize the state of a VM, all possible performance metrics are collected. For such high-dimensional datasets, this article proposes a feature extraction algorithm based on unsupervised fuzzy linear discriminant analysis with kernel (UFKLDA). By introducing the kernel method, UFKLDA can not only effectively deal with non-Gaussian datasets but also implement nonlinear feature extraction. Two sets of experiments were undertaken. In discriminability experiments, this article introduces quantitative criteria to measure discriminability among all classes of samples. The results show that UFKLDA improves discriminability compared with other popular feature extraction algorithms. In detection accuracy experiments, this article computes accuracy measures of an anomaly detection algorithm (i.e., C-SVM) on the original performance metrics and extracted features. The results show that anomaly detection with features extracted by UFKLDA improves the accuracy of detection in terms of sensitivity and specificity.

• 대한원격탐사학회:학술대회논문집
• /
• 대한원격탐사학회 2005년도 Proceedings of ISRS 2005
• /
• pp.233-235
• /
• 2005

This paper presents an unsupervised change detection methodology designed for the detection of landslide areas. The proposed methodology consists of two analytical steps: one for multi-temporal segmentation and the other for automatic selection of thresholding values. By considering the conditions of landslide occurrences and the spectral behavior of multi-temporal remote sensing images, some specific procedures are included in the analytical steps mentioned above. The effectiveness and applicability of the methodology proposed here were illustrated by a case study of the Gangneung area, Korea. The case study demonstrated that the proposed methodology could detect about $83\%$ of landslide occurrences.