• Title/Summary/Keyword: data anomaly detection

Search Result 400, Processing Time 0.03 seconds

A case study of red tide detection around Korean waters using satellite remote sensing

  • Suh, Y.S.;Lee, N.K.;Jang, L.H.;Kim, H.G.;Hwang, J.D.
    • Proceedings of the KSRS Conference
    • /
    • 2003.11a
    • /
    • pp.654-655
    • /
    • 2003
  • Korea has experienced 10 a Cochlodinium polykrikoides red tide outbreaks during the last 10 years (1993-2002). The monitoring activities at National Fisheries Research and Development Institute (NFRDI) in Korea have been extended to all the coastal waters after the worst of fish killing by C. polykrikoides blooms in 1995. NFRDI is looking forward to finding out the feasibility of red tide detection around Korean waters using satellite remote sensing of NOAA/AVHRR, Orbview-2/SeaWiFS, IRS-P4/OCM and Terra/MODIS on real time base. In this study, we used several alternative methods including climatological analysis, spectral and optical methods which may offer a potential detection of the major species of red tide in Korean waters. The relationship between the distribution of SST and C. polykrikoides bloom areas was studied. In climatological analysis, NOAA, SeaWiFS, OCM satellite data in 20th and 26th August 2001 were chosen using the known C. polykrikoides red tide bloom area mapped by helicopter reconnaissance and ground observation. The 26th August, 2001 SeaWiFS chlorophyll a anomaly imageries against the imageries of non-occurring red tide for August 20, 2001 showed the areas C. polykrikoides occurred. The anomalies of chlorophyll a concentration from satellite data between before and after red tide outbreaks showed the similar distribution of C. polykrikoides red tide in 26th August, 2001. The distribution of the difference in SST between daytime and nighttime also showed the possibility of red tide detection. We used corrected vegetation index (CVI) to detect floating vegetation and submerged vegetation containing algal blooms. The simple result of optical absorption from C. polykrikoides showed that if we use the optical characteristics of each red tide we will be able to get the feasibility of the red tide detection.

  • PDF

Damaged cable detection with statistical analysis, clustering, and deep learning models

  • Son, Hyesook;Yoon, Chanyoung;Kim, Yejin;Jang, Yun;Tran, Linh Viet;Kim, Seung-Eock;Kim, Dong Joo;Park, Jongwoong
    • Smart Structures and Systems
    • /
    • v.29 no.1
    • /
    • pp.17-28
    • /
    • 2022
  • The cable component of cable-stayed bridges is gradually impacted by weather conditions, vehicle loads, and material corrosion. The stayed cable is a critical load-carrying part that closely affects the operational stability of a cable-stayed bridge. Damaged cables might lead to the bridge collapse due to their tension capacity reduction. Thus, it is necessary to develop structural health monitoring (SHM) techniques that accurately identify damaged cables. In this work, a combinational identification method of three efficient techniques, including statistical analysis, clustering, and neural network models, is proposed to detect the damaged cable in a cable-stayed bridge. The measured dataset from the bridge was initially preprocessed to remove the outlier channels. Then, the theory and application of each technique for damage detection were introduced. In general, the statistical approach extracts the parameters representing the damage within time series, and the clustering approach identifies the outliers from the data signals as damaged members, while the deep learning approach uses the nonlinear data dependencies in SHM for the training model. The performance of these approaches in classifying the damaged cable was assessed, and the combinational identification method was obtained using the voting ensemble. Finally, the combination method was compared with an existing outlier detection algorithm, support vector machines (SVM). The results demonstrate that the proposed method is robust and provides higher accuracy for the damaged cable detection in the cable-stayed bridge.

Detecting Insider Threat Based on Machine Learning: Anomaly Detection Using RNN Autoencoder (기계학습 기반 내부자위협 탐지기술: RNN Autoencoder를 이용한 비정상행위 탐지)

  • Ha, Dong-wook;Kang, Ki-tae;Ryu, Yeonseung
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.27 no.4
    • /
    • pp.763-773
    • /
    • 2017
  • In recent years, personal information leakage and technology leakage accidents are frequently occurring. According to the survey, the most important part of this spill is the 'insider' within the organization, and the leakage of technology by insiders is considered to be an increasingly important issue because it causes huge damage to the organization. In this paper, we try to learn the normal behavior of employees using machine learning to prevent insider threats, and to investigate how to detect abnormal behavior. Experiments on the detection of abnormal behavior by implementing an Autoencoder composed of Recurrent Neural Network suitable for learning time series data among the neural network models were conducted and the validity of this method was verified.

A Study on the Detection of Small Cavity Located in the Hard Rock by Crosswell Seismic Survey (경암 내 소규모 공동 탐지를 위한 시추공간 탄성파탐사 기법의 적용성 연구)

  • Ko, Kwang-Beom;Lee, Doo-Sung
    • Geophysics and Geophysical Exploration
    • /
    • v.6 no.2
    • /
    • pp.57-63
    • /
    • 2003
  • For the dectection of small cavity in the hard rock, we investigated the feasibility of crosswell travel-time tomography and Kirchhoff migration technique. In travel-time tomography, first arrival anomaly caused by small cavity was investigated by numerical modeling based on the knowledge of actual field information. First arrival delay was very small (<0.125 msec) and detectable receiver offset range was limited to 4m with respect to $1\%$ normalized first arrival anomaly. As a consequence, it was turned out that carefully designed survey array with both sufficient narrow spatial spacing and temporal (<0.03125 msec) sampling were required for small cavity detection. Also, crosswell Kirchhoff migration technique was investigated with both numerical and real data. Stack section obtained by numerical data shows the good cavity image. In crosswell seismic data, various unwanted seismic events such as direct wave and various mode converted waves were alto recorded. To remove these noises und to enhance the diffraction signal, combination of median and bandpass filtering was applied and prestack and stacked migration images were created. From this, we viewed the crosswell migration technique as one of the adoptable method for small cavity detection.

Orbit Ephemeris Failure Detection in a GNSS Regional Application

  • Ahn, Jongsun;Lee, Young Jae;Won, Dae Hee;Jun, Hyang-Sig;Yeom, Chanhong;Sung, Sangkyung;Lee, Jeong-Oog
    • International Journal of Aeronautical and Space Sciences
    • /
    • v.16 no.1
    • /
    • pp.89-101
    • /
    • 2015
  • To satisfy civil aviation requirements using the Global Navigation Satellite System (GNSS), it is important to guarantee system integrity. In this work, we propose a fault detection algorithm for GNSS ephemeris anomalies. The basic principle concerns baseline length estimation with GNSS measurements (pseudorange, broadcasted ephemerides). The estimated baseline length is subtracted from the true baseline length, computed using the exact surveyed ground antenna positions. If this subtracted value differs by more than a given threshold, this indicates that an ephemeris anomaly has been detected. This algorithm is suitable for detecting Type A ephemeris failure, and more advantageous for use with multiple stations with various long baseline vectors. The principles of the algorithm, sensitivity analysis, minimum detectable error (MDE), and protection level derivation are described and we verify the sensitivity analysis and algorithm availability based on real GPS data in Korea. Consequently, this algorithm is appropriate for GNSS regional implementation.

ELECTRICAL IMPEDANCE IMAGING FOR SEARCHING ANOMALIES

  • Ohin Kwon;Seo, Jin-Keun;Woo, Eung-Je;Yoon, Jeong-Rock
    • Communications of the Korean Mathematical Society
    • /
    • v.16 no.3
    • /
    • pp.459-485
    • /
    • 2001
  • The aim of EIT (electrical impedance tomography) system is to image cross-section conductivity distribution of a human body by means of both generating and sensing electrodes attached on to the surface of the body, where currents are injected and voltages are measured. EIT has been suffered from the severe ill-posedness which is caused by the inherent low sensitivity of boundary measurements to any changes of internal tissue conductivity values. With a limited set of current-to-voltage data, figuring out full structure of the conductivity distribution could be extremely difficult at present time, so it could be worthwhile to extract some necessary partial information of the internal conductivity. We try to extract some key patterns of current-to-voltage data that furnish some core information on the conductivity distribution such s location and size. This overview provides our recent observation on the location search and the size estimation.

  • PDF

Effective Payload-based Anomaly Detection Method Using Pre-trained Model (사전학습 모델을 활용한 효과적인 Http Payload 이상 탐지 방법)

  • LEE, Unggi;KIM, Wonchul
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2022.11a
    • /
    • pp.228-230
    • /
    • 2022
  • 딥러닝 기반의 인공지능 기술이 발달함에 따라 이상 탐지 방법에도 딥러닝이 적용되었다. 네트워크 트래픽으로부터 요약 및 집계된 Feature 를 학습하는 방법과 Packet 자체를 학습하는 등의 방법이 있었다. 그러나 모두 정보의 제한적으로 사용한다는 단점이 있었다. 본 연구에서는 Http Request에 대한 사전학습 기반의 효과적인 이상 탐지 방법을 제안한다. 사전학습에 고려되는 토큰화 방법, Padding 방법, Feature 결합 방법, Feature 선택 방법과 전이학습 시 Numerical 정보를 추가하는 방법을 소개하고 각 실험을 통해 최적의 방법을 제안한다.

Hot Place Detection Based on ConvLSTM AutoEncoder Using Foot Traffic Data (유동인구를 활용한 ConvLSTM AutoEncoder 기반 핫플레이스 탐지)

  • Ju-Young Lee;Heon-Jin Park
    • The Journal of Bigdata
    • /
    • v.8 no.2
    • /
    • pp.97-107
    • /
    • 2023
  • Small business owners are relatively likely to be alienated from various benefits caused by the change to a big data/AI-based society. To support them, we would like to detect a hot place based on the floating population to support small business owners' decision-making in the start-up area. Through various studies, it is known that the population size of the region has an important effect on the sales of small business owners. In this study, inland regions were extracted from the Incheon floating population data from January 2019 to June 2022. the Data is consisted of a grid of 50m intervals, central coordinates and the population for each grid are presented, made image structure through imputation to maintain spatial information. Spatial outliers were removed and imputated using LOF and GAM, and temporal outliers were removed and imputated through LOESS. We used ConvLSTM which can take both temporal and spatial characteristics into account as a predictive model, and used AutoEncoder structure, which performs outliers detection based on reconstruction error to define an area with high MAPE as a hot place.

Reinforcement Mining Method for Anomaly Detection and Misuse Detection using Post-processing and Training Method (이상탐지(Anomaly Detection) 및 오용탐지(Misuse Detection) 분석의 정확도 향상을 위한 개선된 데이터마이닝 방법 연구)

  • Choi Yun-Jeong;Park Seung-Soo
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2006.06b
    • /
    • pp.238-240
    • /
    • 2006
  • 네트워크상에서 발생하는 다양한 형태의 대량의 데이터를 정확하고 효율적으로 분석하기 위해 설계되고 있는 마이닝 시스템들은 목표지향적으로 훈련데이터들을 어떻게 구축하여 다룰 것인지에 대한 문제보다는 대부분 얼마나 많은 데이터 마이닝 기법을 지원하고 이를 적용할 수 있는지 등의 기법에 초점을 두고 있다. 따라서, 점점 더 에이전트화, 분산화, 자동화 및 은닉화 되는 최근의 보안공격기법을 정확하게 탐지하기 위한 방법은 미흡한 실정이다. 본 연구에서는 유비쿼터스 환경 내에서 발생 가능한 문제 중 복잡하고 지능화된 침입패턴의 탐지를 위해 데이터 마이닝 기법과 결함허용방법을 이용하는 개선된 학습알고리즘과 후처리 방법에 의한 RTPID(Refinement Training and Post-processing for Intrusion Detection)시스템을 제안한다. 본 논문에서의 RTPID 시스템은 active learning과 post-processing을 이용하여, 네트워크 내에서 발생 가능한 침입형태들을 정확하고 효율적으로 다루어 분석하고 있다. 이는 기법에만 초점을 맞춘 기존의 데이터마이닝 분석을 개선하고 있으며, 특히 제안된 분석 프로세스를 진행하는 동안 능동학습방법의 장점을 수용하여 학습효과는 높이며 비용을 감소시킬 수 있는 자가학습방법(self learning)방법의 효과를 기대할 수 있다. 이는 관리자의 개입을 최소화하는 학습방법이면서 동시에 False Positive와 False Negative 의 오류를 매우 효율적으로 개선하는 방법으로 기대된다. 본 논문의 제안방법은 분석도구나 시스템에 의존하지 않기 때문에, 유사한 문제를 안고 있는 여러 분야의 네트웍 환경에 적용될 수 있다.더욱 높은성능을 가짐을 알 수 있다.의 각 노드의 전력이 위험할 때 에러 패킷을 발생하는 기법을 추가하였다. NS-2 시뮬레이터를 이용하여 실험을 한 결과, 제안한 기법이 AOMDV에 비해 경로 탐색 횟수가 최대 36.57% 까지 감소되었음을 알 수 있었다.의 작용보다 더 강력함을 시사하고 있다.TEX>로 최고값을 나타내었으며 그 후 감소하여 담금 10일에는 $1.61{\sim}2.34%$였다. 시험구간에는 KKR, SKR이 비교적 높은 값을 나타내었다. 무기질 함량은 발효기간이 경과할수록 증하였고 Ca는 $2.95{\sim}36.76$, Cu는 $0.01{\sim}0.14$, Fe는 $0.71{\sim}3.23$, K는 $110.89{\sim}517.33$, Mg는 $34.78{\sim}122.40$, Mn은 $0.56{\sim}5.98$, Na는 $0.19{\sim}14.36$, Zn은 $0.90{\sim}5.71ppm$을 나타내었으며, 시험구별로 보면 WNR, BNR구가 Na만 제외한 다른 무기성분 함량이 가장 높았다.O to reduce I/O cost by reusing data already present in the memory of other nodes. Finally, chunking and on-line compression mechanisms are included in both models. We demonstrate that we can obtain significantly high-performanc

  • PDF

Data Mining Approaches for DDoS Attack Detection (분산 서비스거부 공격 탐지를 위한 데이터 마이닝 기법)

  • Kim, Mi-Hui;Na, Hyun-Jung;Chae, Ki-Joon;Bang, Hyo-Chan;Na, Jung-Chan
    • Journal of KIISE:Information Networking
    • /
    • v.32 no.3
    • /
    • pp.279-290
    • /
    • 2005
  • Recently, as the serious damage caused by DDoS attacks increases, the rapid detection and the proper response mechanisms are urgent. However, existing security mechanisms do not effectively defend against these attacks, or the defense capability of some mechanisms is only limited to specific DDoS attacks. In this paper, we propose a detection architecture against DDoS attack using data mining technology that can classify the latest types of DDoS attack, and can detect the modification of existing attacks as well as the novel attacks. This architecture consists of a Misuse Detection Module modeling to classify the existing attacks, and an Anomaly Detection Module modeling to detect the novel attacks. And it utilizes the off-line generated models in order to detect the DDoS attack using the real-time traffic. We gathered the NetFlow data generated at an access router of our network in order to model the real network traffic and test it. The NetFlow provides the useful flow-based statistical information without tremendous preprocessing. Also, we mounted the well-known DDoS attack tools to gather the attack traffic. And then, our experimental results show that our approach can provide the outstanding performance against existing attacks, and provide the possibility of detection against the novel attack.