• Title/Summary/Keyword: data anomaly detection

Search Result 383, Processing Time 0.025 seconds

Design and Evaluation of a Rough Set Based Anomaly Detection Scheme Considering the Age of User Profiles

  • Bae, Ihn-Han
    • Journal of Korea Multimedia Society
    • /
    • v.10 no.12
    • /
    • pp.1726-1732
    • /
    • 2007
  • The rapid proliferation of wireless networks and mobile computing applications has changed the landscape of network security. Anomaly detection is a pattern recognition task whose goal is to report the occurrence of abnormal or unknown behavior in a given system being monitored. This paper presents an efficient rough set based anomaly detection method that can effectively identify a group of especially harmful internal attackers - masqueraders in cellular mobile networks. Our scheme uses the trace data of wireless application layer by a user as feature value. Based on this, the used pattern of a mobile's user can be captured by rough sets, and the abnormal behavior of the mobile can be also detected effectively by applying a roughness membership function with the age of the user profile. The performance of the proposed scheme is evaluated by using a simulation. Simulation results demonstrate that the anomalies are well detected by the proposed scheme that considers the age of user profiles.

  • PDF

Mutual Information Applied to Anomaly Detection

  • Kopylova, Yuliya;Buell, Duncan A.;Huang, Chin-Tser;Janies, Jeff
    • Journal of Communications and Networks
    • /
    • v.10 no.1
    • /
    • pp.89-97
    • /
    • 2008
  • Anomaly detection systems playa significant role in protection mechanism against attacks launched on a network. The greatest challenge in designing systems detecting anomalous exploits is defining what to measure. Effective yet simple, Shannon entropy metrics have been successfully used to detect specific types of malicious traffic in a number of commercially available IDS's. We believe that Renyi entropy measures can also adequately describe the characteristics of a network as a whole as well as detect abnormal traces in the observed traffic. In addition, Renyi entropy metrics might boost sensitivity of the methods when disambiguating certain anomalous patterns. In this paper we describe our efforts to understand how Renyi mutual information can be applied to anomaly detection as an offline computation. An initial analysis has been performed to determine how well fast spreading worms (Slammer, Code Red, and Welchia) can be detected using our technique. We use both synthetic and real data audits to illustrate the potentials of our method and provide a tentative explanation of the results.

Data abnormal detection using bidirectional long-short neural network combined with artificial experience

  • Yang, Kang;Jiang, Huachen;Ding, Youliang;Wang, Manya;Wan, Chunfeng
    • Smart Structures and Systems
    • /
    • v.29 no.1
    • /
    • pp.117-127
    • /
    • 2022
  • Data anomalies seriously threaten the reliability of the bridge structural health monitoring system and may trigger system misjudgment. To overcome the above problem, an efficient and accurate data anomaly detection method is desiderated. Traditional anomaly detection methods extract various abnormal features as the key indicators to identify data anomalies. Then set thresholds artificially for various features to identify specific anomalies, which is the artificial experience method. However, limited by the poor generalization ability among sensors, this method often leads to high labor costs. Another approach to anomaly detection is a data-driven approach based on machine learning methods. Among these, the bidirectional long-short memory neural network (BiLSTM), as an effective classification method, excels at finding complex relationships in multivariate time series data. However, training unprocessed original signals often leads to low computation efficiency and poor convergence, for lacking appropriate feature selection. Therefore, this article combines the advantages of the two methods by proposing a deep learning method with manual experience statistical features fed into it. Experimental comparative studies illustrate that the BiLSTM model with appropriate feature input has an accuracy rate of over 87-94%. Meanwhile, this paper provides basic principles of data cleaning and discusses the typical features of various anomalies. Furthermore, the optimization strategies of the feature space selection based on artificial experience are also highlighted.

Power Quality Early Warning Based on Anomaly Detection

  • Gu, Wei;Bai, Jingjing;Yuan, Xiaodong;Zhang, Shuai;Wang, Yuankai
    • Journal of Electrical Engineering and Technology
    • /
    • v.9 no.4
    • /
    • pp.1171-1181
    • /
    • 2014
  • Different power quality (PQ) disturbance sources can have major impacts on the power supply grid. This study proposes, for the first time, an early warning approach to identifying PQ problems and providing early warning prompts based on the monitored data of PQ disturbance sources. To establish a steady-state power quality early warning index system, the characteristics of PQ disturbance sources are analyzed and summed up. The higher order statistics anomaly detection (HOSAD) algorithm, based on skewness and kurtosis, and hierarchical power quality early warning flow, were then used to mine limit-exceeding and abnormal data and analyze their severity. Cases studies show that the proposed approach is effective and feasible, and that it is possible to provide timely power quality early warnings for limit-exceeding and abnormal data.

Irregularly-Sampled Time Series Correction Method for Anomaly Detection in Manufacturing Facility (생산 설비의 이상탐지를 위한 불규칙 샘플링 시계열 데이터 보정 기법)

  • Shin, Kang-hyeon;Jin, Kyo-hong
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2021.10a
    • /
    • pp.85-88
    • /
    • 2021
  • There are many irregularly-sampled time series in the manufacturing data which are collected from manufacturing facilities by short intervals. Those time series often have large variance. In this paper, we propose irregularly-sampled time series correction method based on simple moving average. This method corrects time intervals between neighboring values in time series regularly and reduces the variance of the values at the same time. We examine that this method improves performance of anomaly detection in manufacturing facility.

  • PDF

FADA: A fuzzy anomaly detection algorithm for MANETs (모바일 애드-혹 망을 위한 퍼지 비정상 행위 탐지 알고리즘)

  • Bae, Ihn-Han
    • Journal of the Korean Data and Information Science Society
    • /
    • v.21 no.6
    • /
    • pp.1125-1136
    • /
    • 2010
  • Lately there exist increasing demands for online abnormality monitoring over trajectory stream, which are obtained from moving object tracking devices. This problem is challenging due to the requirement of high speed data processing within limited space cost. In this paper, we present a FADA (Fuzzy Anomaly Detection Algorithm) which constructs normal profile by computing mobility feature information from the GPS (Global Positioning System) logs of mobile devices in MANETs (Mobile Ad-hoc Networks), computes a fuzzy dissimilarity between the current mobility feature information of the mobile device and the mobility feature information in the normal profile, and detects effectively the anomaly behaviors of mobile devices on the basis of the computed fuzzy dissimilarity. The performance of proposed FADA is evaluated through simulation.

Host Anomaly Detection of Neural Networks and Neural-fuzzy Techniques with Soundex Algorithm (사운덱스 알고리즘을 적용한 신경망라 뉴로-처지 기법의 호스트 이상 탐지)

  • Cha, Byung-Rae;Kim, Hyung-Jong;Park, Bong-Gu;Cho, Hyug-Hyun
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.15 no.2
    • /
    • pp.13-22
    • /
    • 2005
  • To improve the anomaly IDS using system calls, this study focuses on Neural Networks Learning using the Soundex algorithm which is designed to change feature selection and variable length data into a fixed length learning pattern. That is, by changing variable length sequential system call data into a fixed length behavior pattern using the Soundex algorithm, this study conducted neural networks learning by using a backpropagation algorithm with fuzzy membership function. The back-propagation neural networks and Neuro-Fuzzy technique are applied for anomaly intrusion detection of system calls using Sendmail Data of UNM to demonstrate its aspect of he complexity of time, space and MDL performance.

Detection of Defect Patterns on Wafer Bin Map Using Fully Convolutional Data Description (FCDD) (FCDD 기반 웨이퍼 빈 맵 상의 결함패턴 탐지)

  • Seung-Jun Jang;Suk Joo Bae
    • Journal of Korean Society of Industrial and Systems Engineering
    • /
    • v.46 no.2
    • /
    • pp.1-12
    • /
    • 2023
  • To make semiconductor chips, a number of complex semiconductor manufacturing processes are required. Semiconductor chips that have undergone complex processes are subjected to EDS(Electrical Die Sorting) tests to check product quality, and a wafer bin map reflecting the information about the normal and defective chips is created. Defective chips found in the wafer bin map form various patterns, which are called defective patterns, and the defective patterns are a very important clue in determining the cause of defects in the process and design of semiconductors. Therefore, it is desired to automatically and quickly detect defective patterns in the field, and various methods have been proposed to detect defective patterns. Existing methods have considered simple, complex, and new defect patterns, but they had the disadvantage of being unable to provide field engineers the evidence of classification results through deep learning. It is necessary to supplement this and provide detailed information on the size, location, and patterns of the defects. In this paper, we propose an anomaly detection framework that can be explained through FCDD(Fully Convolutional Data Description) trained only with normal data to provide field engineers with details such as detection results of abnormal defect patterns, defect size, and location of defect patterns on wafer bin map. The results are analyzed using open dataset, providing prominent results of the proposed anomaly detection framework.

Anomaly Intrusion Detection using Fuzzy Membership Function and Neural Networks (퍼지 멤버쉽 함수와 신경망을 이용한 이상 침입 탐지)

  • Cha, Byung-Rae
    • The KIPS Transactions:PartC
    • /
    • v.11C no.5
    • /
    • pp.595-604
    • /
    • 2004
  • By the help of expansion of computer network and rapid growth of Internet, the information infrastructure is now able to provide a wide range of services. Especially open architecture - the inherent nature of Internet - has not only got in the way of offering QoS service, managing networks, but also made the users vulnerable to both the threat of backing and the issue of information leak. Thus, people recognized the importance of both taking active, prompt and real-time action against intrusion threat, and at the same time, analyzing the similar patterns of in-trusion already known. There are now many researches underway on Intrusion Detection System(IDS). The paper carries research on the in-trusion detection system which hired supervised learning algorithm and Fuzzy membership function especially with Neuro-Fuzzy model in order to improve its performance. It modifies tansigmoid transfer function of Neural Networks into fuzzy membership function, so that it can reduce the uncertainty of anomaly intrusion detection. Finally, the fuzzy logic suggested here has been applied to a network-based anomaly intrusion detection system, tested against intrusion data offered by DARPA 2000 Intrusion Data Sets, and proven that it overcomes the shortcomings that Anomaly Intrusion Detection usually has.

Data Fusion Algorithm based on Inference for Anomaly Detection in the Next-Generation Intrusion Detection (차세대 침입탐지에서 이상탐지를 위한 추론 기반 데이터 융합 알고리즘)

  • Kim, Dong-Wook;Han, Myung-Mook
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.26 no.3
    • /
    • pp.233-238
    • /
    • 2016
  • In this paper, we propose the algorithms of processing the uncertainty data using data fusion for the next generation intrusion detection. In the next generation intrusion detection, a lot of data are collected by many of network sensors to discover knowledge from generating information in cyber space. It is necessary the data fusion process to extract knowledge from collected sensors data. In this paper, we have proposed method to represent the uncertainty data, by classifying where is a confidence interval in interval of uncertainty data through feature analysis of different data using inference method with Dempster-Shafer Evidence Theory. In this paper, we have implemented a detection experiment that is classified by the confidence interval using IRIS plant Data Set for anomaly detection of uncertainty data. As a result, we found that it is possible to classify data by confidence interval.