• Title/Summary/Keyword: Anomaly data detection

Search Result 402, Processing Time 0.022 seconds

A semi-supervised interpretable machine learning framework for sensor fault detection

  • Martakis, Panagiotis;Movsessian, Artur;Reuland, Yves;Pai, Sai G.S.;Quqa, Said;Cava, David Garcia;Tcherniak, Dmitri;Chatzi, Eleni
    • Smart Structures and Systems
    • /
    • v.29 no.1
    • /
    • pp.251-266
    • /
    • 2022
  • Structural Health Monitoring (SHM) of critical infrastructure comprises a major pillar of maintenance management, shielding public safety and economic sustainability. Although SHM is usually associated with data-driven metrics and thresholds, expert judgement is essential, especially in cases where erroneous predictions can bear casualties or substantial economic loss. Considering that visual inspections are time consuming and potentially subjective, artificial-intelligence tools may be leveraged in order to minimize the inspection effort and provide objective outcomes. In this context, timely detection of sensor malfunctioning is crucial in preventing inaccurate assessment and false alarms. The present work introduces a sensor-fault detection and interpretation framework, based on the well-established support-vector machine scheme for anomaly detection, combined with a coalitional game-theory approach. The proposed framework is implemented in two datasets, provided along the 1st International Project Competition for Structural Health Monitoring (IPC-SHM 2020), comprising acceleration and cable-load measurements from two real cable-stayed bridges. The results demonstrate good predictive performance and highlight the potential for seamless adaption of the algorithm to intrinsically different data domains. For the first time, the term "decision trajectories", originating from the field of cognitive sciences, is introduced and applied in the context of SHM. This provides an intuitive and comprehensive illustration of the impact of individual features, along with an elaboration on feature dependencies that drive individual model predictions. Overall, the proposed framework provides an easy-to-train, application-agnostic and interpretable anomaly detector, which can be integrated into the preprocessing part of various SHM and condition-monitoring applications, offering a first screening of the sensor health prior to further analysis.

An Abnormal Worker Movement Detection System Based on Data Stream Processing and Hierarchical Clustering

  • Duong, Dat Van Anh;Lan, Doi Thi;Yoon, Seokhoon
    • International Journal of Internet, Broadcasting and Communication
    • /
    • v.14 no.4
    • /
    • pp.88-95
    • /
    • 2022
  • Detecting anomalies in human movement is an important task in industrial applications, such as monitoring industrial disasters or accidents and recognizing unauthorized factory intruders. In this paper, we propose an abnormal worker movement detection system based on data stream processing and hierarchical clustering. In the proposed system, Apache Spark is used for streaming the location data of people. A hierarchical clustering-based anomalous trajectory detection algorithm is designed for detecting anomalies in human movement. The algorithm is integrated into Apache Spark for detecting anomalies from location data. Specifically, the location information is streamed to Apache Spark using the message queuing telemetry transport protocol. Then, Apache Spark processes and stores location data in a data frame. When there is a request from a client, the processed data in the data frame is taken and put into the proposed algorithm for detecting anomalies. A real mobility trace of people is used to evaluate the proposed system. The obtained results show that the system has high performance and can be used for a wide range of industrial applications.

Implementation of Realtime Face Recognition System using Haar-Like Features and PCA in Mobile Environment (모바일 환경에서 Haar-Like Features와 PCA를 이용한 실시간 얼굴 인증 시스템)

  • Kim, Jung Chul;Heo, Bum Geun;Shin, Na Ra;Hong, Ki Cheon
    • Journal of Korea Society of Digital Industry and Information Management
    • /
    • v.6 no.2
    • /
    • pp.199-207
    • /
    • 2010
  • Recently, large amount of information in IDS(Intrusion Detection System) can be un manageable and also be mixed with false prediction error. In this paper, we propose a data mining methodology for IDS, which contains uncertainty based on training process and post-processing analysis additionally. Our system is trained to classify the existing attack for misuse detection, to detect the new attack pattern for anomaly detection, and to define border patter between attack and normal pattern. In experimental results show that our approach improve the performance against existing attacks and new attacks, from 0.62 to 0.84 about 35%.

Seasonal Variations and Characteristics of the Stratification Depth and Strength in the Seas Near the Korea Peninsular using the Relative Potential Energy Anomaly (한반도 근해의 상대적 위치에너지 편차 변화를 이용한 성층화의 특성과 계절별 변화에 대한 연구)

  • Cho, Chang-Bong;Kim, Young-Gyu;Chang, Kyung-Il
    • Journal of the Korea Institute of Military Science and Technology
    • /
    • v.14 no.2
    • /
    • pp.205-212
    • /
    • 2011
  • In this paper, we have proposed a method for quantization of the stratification strength in the sea water and analysing the distributions of the maximum stratification depths calculated by the method at the seas near the Korean peninsular. For calculating the stratification strength, modified and applied the potential energy anomaly formular which was suggested by Simpson in 1977. The data had been collected by NFRDI from 1971 to 2008 were used to determine the maximum vertical density gradient depth and the relative potential energy anomaly at that depth. In the East Sea, the stratification depth has become deepened about 20m in February and April since 1971. In Yellow-South Sea, the maximum density gradient depth has been deepened about 10m only in December during the same period and the difference of the stratification depth between summer and winter has been enlarged. These trends of variation of stratification strength and depth near the Korean peninsular should be investigated more carefully and continuously. And the results of these studies could be adopted for the more efficient operation of underwater weapon and detection systems.

Structural novelty detection based on sparse autoencoders and control charts

  • Finotti, Rafaelle P.;Gentile, Carmelo;Barbosa, Flavio;Cury, Alexandre
    • Structural Engineering and Mechanics
    • /
    • v.81 no.5
    • /
    • pp.647-664
    • /
    • 2022
  • The powerful data mapping capability of computational deep learning methods has been recently explored in academic works to develop strategies for structural health monitoring through appropriate characterization of dynamic responses. In many cases, these studies concern laboratory prototypes and finite element models to validate the proposed methodologies. Therefore, the present work aims to investigate the capability of a deep learning algorithm called Sparse Autoencoder (SAE) specifically focused on detecting structural alterations in real-case studies. The idea is to characterize the dynamic responses via SAE models and, subsequently, to detect the onset of abnormal behavior through the Shewhart T control chart, calculated with SAE extracted features. The anomaly detection approach is exemplified using data from the Z24 bridge, a classical benchmark, and data from the continuous monitoring of the San Vittore bell-tower, Italy. In both cases, the influence of temperature is also evaluated. The proposed approach achieved good performance, detecting structural changes even under temperature variations.

Distributed and Scalable Intrusion Detection System Based on Agents and Intelligent Techniques

  • El-Semary, Aly M.;Mostafa, Mostafa Gadal-Haqq M.
    • Journal of Information Processing Systems
    • /
    • v.6 no.4
    • /
    • pp.481-500
    • /
    • 2010
  • The Internet explosion and the increase in crucial web applications such as ebanking and e-commerce, make essential the need for network security tools. One of such tools is an Intrusion detection system which can be classified based on detection approachs as being signature-based or anomaly-based. Even though intrusion detection systems are well defined, their cooperation with each other to detect attacks needs to be addressed. Consequently, a new architecture that allows them to cooperate in detecting attacks is proposed. The architecture uses Software Agents to provide scalability and distributability. It works in two modes: learning and detection. During learning mode, it generates a profile for each individual system using a fuzzy data mining algorithm. During detection mode, each system uses the FuzzyJess to match network traffic against its profile. The architecture was tested against a standard data set produced by MIT's Lincoln Laboratory and the primary results show its efficiency and capability to detect attacks. Finally, two new methods, the memory-window and memoryless-window, were developed for extracting useful parameters from raw packets. The parameters are used as detection metrics.

Minimization Method of Measurement Noise for Satellite Clock Anomaly Detection (위성시계 이상검출을 위한 측정잡음 최소화 기법)

  • Seo, Kiyeol;Park, Sanghyun;Jang, Wonseok;Kim, Youngki
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.23 no.6
    • /
    • pp.505-510
    • /
    • 2013
  • In order to detect and identify the GPS clock anomaly in the Differential GPS real environment, this paper addresses a method for minimizing the measurement noise of reference receivers. It estimates the real measurement noise that removed the uncommon error source from pseudorange measurement to minimize the measurement noise. Based on the output of two reference receivers, it first removes the uncommon errors, then optimizes the measurement noise by applying the correction data. Finally, it detects and identifies the satellite clock anomaly using the minimized measurement noise. The method will increase the availability of current DGPS reference system.

Density-based Outlier Detection for Very Large Data (대용량 자료 분석을 위한 밀도기반 이상치 탐지)

  • Kim, Seung;Cho, Nam-Wook;Kang, Suk-Ho
    • Journal of the Korean Operations Research and Management Science Society
    • /
    • v.35 no.2
    • /
    • pp.71-88
    • /
    • 2010
  • A density-based outlier detection such as an LOF (Local Outlier Factor) tries to find an outlying observation by using density of its surrounding space. In spite of several advantages of a density-based outlier detection method, the computational complexity of outlier detection has been one of major barriers in its application. In this paper, we present an LOF algorithm that can reduce computation time of a density based outlier detection algorithm. A kd-tree indexing and approximated k-nearest neighbor search algorithm (ANN) are adopted in the proposed method. A set of experiments was conducted to examine performance of the proposed algorithm. The results show that the proposed method can effectively detect local outliers in reduced computation time.

Monitoring of Climatological Variability Using EOS and OSMl Data

  • Lim, Hyo-Suk;Kim, Jeong-Yeon;Lee, Sang-Hee
    • Korean Journal of Remote Sensing
    • /
    • v.19 no.3
    • /
    • pp.209-216
    • /
    • 2003
  • Dramatic changes in the patterns of satellite-derived pigment concentrations, sea-level height anomaly, sea surface temperature anomaly, and zonal wind anomaly are observed during the 1997-1998 El Ni$\bar{n}$o. By some measures, the 1997-1998 El Ni$\bar{n}$o was the strongest one of the 20$^{th}$ century. A very strong El Ni$\bar{n}$o developed during 1997 and matured late in the year. A dramatic recovery occurred in mid-1998 and led to La Nina condition. The largest spatial extent of the phytoplankton bloom was fellowed recovery from El Ni$\bar{n}$o over the equatorial Pacific. The evolution towards a warm episode (El Ni$\bar{n}$o) started from spring of 2002 and continued during January 2003, while equatorial SSTA remained greater than +1$^{\circ}C$ in the central equatorial Pacific. The OSMI (Ocean Scanning Multispectral Imager) data are used for detection of dramatic changes in the patterns of pigment concentration during next El Ni$\bar{n}$o.

Anomaly detection on simulation conditions for ship-handling safety assessment (시뮬레이션 실험조건 이상 진단 연구)

  • Kwon, Se-Hyug
    • Journal of the Korean Data and Information Science Society
    • /
    • v.21 no.5
    • /
    • pp.853-861
    • /
    • 2010
  • Experimental conditions are set with environmental factors which can affect ship navigation. In FTS simulation, infinite simulation can be theoretically tested with no time constraint but the simulated result with the same experimental condition is repeated due to mathematical model. RTS simulation can give more resonable results but costs at lest 30 minutes for only experimental time. The mixture of two simulation methods using probability density function has been proposed: some of experimental conditions in which ship-handling is most difficult are selected with FTS and are tested in RTS. It has drawback that it does not consider the navigation route but aggregated track index. In this paper, anomaly detection approach is suggested to select some experimental conditions of FTS simulation which are most difficult in ship-handling during the navigation route. An empirical result has been shown.