• Title/Summary/Keyword: 스트리밍데이터

Search Result 512, Processing Time 0.021 seconds

A Study of Anomaly Detection for ICT Infrastructure using Conditional Multimodal Autoencoder (ICT 인프라 이상탐지를 위한 조건부 멀티모달 오토인코더에 관한 연구)

  • Shin, Byungjin;Lee, Jonghoon;Han, Sangjin;Park, Choong-Shik
    • Journal of Intelligence and Information Systems
    • /
    • v.27 no.3
    • /
    • pp.57-73
    • /
    • 2021
  • Maintenance and prevention of failure through anomaly detection of ICT infrastructure is becoming important. System monitoring data is multidimensional time series data. When we deal with multidimensional time series data, we have difficulty in considering both characteristics of multidimensional data and characteristics of time series data. When dealing with multidimensional data, correlation between variables should be considered. Existing methods such as probability and linear base, distance base, etc. are degraded due to limitations called the curse of dimensions. In addition, time series data is preprocessed by applying sliding window technique and time series decomposition for self-correlation analysis. These techniques are the cause of increasing the dimension of data, so it is necessary to supplement them. The anomaly detection field is an old research field, and statistical methods and regression analysis were used in the early days. Currently, there are active studies to apply machine learning and artificial neural network technology to this field. Statistically based methods are difficult to apply when data is non-homogeneous, and do not detect local outliers well. The regression analysis method compares the predictive value and the actual value after learning the regression formula based on the parametric statistics and it detects abnormality. Anomaly detection using regression analysis has the disadvantage that the performance is lowered when the model is not solid and the noise or outliers of the data are included. There is a restriction that learning data with noise or outliers should be used. The autoencoder using artificial neural networks is learned to output as similar as possible to input data. It has many advantages compared to existing probability and linear model, cluster analysis, and map learning. It can be applied to data that does not satisfy probability distribution or linear assumption. In addition, it is possible to learn non-mapping without label data for teaching. However, there is a limitation of local outlier identification of multidimensional data in anomaly detection, and there is a problem that the dimension of data is greatly increased due to the characteristics of time series data. In this study, we propose a CMAE (Conditional Multimodal Autoencoder) that enhances the performance of anomaly detection by considering local outliers and time series characteristics. First, we applied Multimodal Autoencoder (MAE) to improve the limitations of local outlier identification of multidimensional data. Multimodals are commonly used to learn different types of inputs, such as voice and image. The different modal shares the bottleneck effect of Autoencoder and it learns correlation. In addition, CAE (Conditional Autoencoder) was used to learn the characteristics of time series data effectively without increasing the dimension of data. In general, conditional input mainly uses category variables, but in this study, time was used as a condition to learn periodicity. The CMAE model proposed in this paper was verified by comparing with the Unimodal Autoencoder (UAE) and Multi-modal Autoencoder (MAE). The restoration performance of Autoencoder for 41 variables was confirmed in the proposed model and the comparison model. The restoration performance is different by variables, and the restoration is normally well operated because the loss value is small for Memory, Disk, and Network modals in all three Autoencoder models. The process modal did not show a significant difference in all three models, and the CPU modal showed excellent performance in CMAE. ROC curve was prepared for the evaluation of anomaly detection performance in the proposed model and the comparison model, and AUC, accuracy, precision, recall, and F1-score were compared. In all indicators, the performance was shown in the order of CMAE, MAE, and AE. Especially, the reproduction rate was 0.9828 for CMAE, which can be confirmed to detect almost most of the abnormalities. The accuracy of the model was also improved and 87.12%, and the F1-score was 0.8883, which is considered to be suitable for anomaly detection. In practical aspect, the proposed model has an additional advantage in addition to performance improvement. The use of techniques such as time series decomposition and sliding windows has the disadvantage of managing unnecessary procedures; and their dimensional increase can cause a decrease in the computational speed in inference.The proposed model has characteristics that are easy to apply to practical tasks such as inference speed and model management.

Research on Archive Opening and Sharing Projects of Korean Terrestrial Broadcasters and External Users of Shared Archives : Focusing on the Case of the 5.18 Footage Video Sharing Project 〈May Story(Owol-Iyagi)〉 Contest Organized by KBS (국내 지상파 방송사의 아카이브 개방·공유 사업과 아카이브 이용자 연구 KBS 5.18 아카이브 시민공유 프로젝트 <5월이야기> 공모전 사례를 중심으로)

  • Choi, Hyojin
    • The Korean Journal of Archival Studies
    • /
    • no.78
    • /
    • pp.197-249
    • /
    • 2023
  • This paper focus on the demand for broadcast and video archive contents by users outside broadcasters as the archive openness and sharing projects of terrestrial broadcasters have become more active in recent years. In the process of creating works using broadcasters' released video footage, the study examined the criteria by which video footage is selected and the methods and processes utilized for editing. To this end, the study analyzed the the case of the 5.18 footage video sharing project 〈May Story(Owol-Iyagi)〉 contest organized by KBS in 2022, in which KBS released its footage about the May 18 Democratic Uprising and invited external users to create new content using them. Analyzing the works that were selected as the winners of the contest, the research conducts in-depth interviews with the creators of each work. As a result, the following points are identified. Among the submitted works, many works deal with the direct or indirect experience of the May 18 Democratic Uprising and focus on the impact of this historical event on individuals and our current society. The study also examined the ways in which broadcasters' footage is used in secondary works. We found ways to use video as a means to share historical events, or to present video as evidence or metaphor. It is found that the need for broadcasters to provide a wider range of public video materials such as the May 18 Democratic Uprising, describing more metadata including copyright information before releasing selected footage, ensuring high-definition and high-fidelity videos that can be used for editing, and strengthening streaming or downloading functions for user friendliness. Through this, the study explores the future direction of broadcasters' video data openness and sharing business, and confirms that broadcasters' archival projects can be an alternative to fulfill public responsibilities such as strengthening social integration between regions, generations, and classes through moving images.

  • PDF