Browse > Article
http://dx.doi.org/10.6109/jkiice.2018.22.2.382

A Security Log Analysis System using Logstash based on Apache Elasticsearch  

Lee, Bong-Hwan (Department of Electronics, Information and Communications Engineering, Daejeon University)
Yang, Dong-Min (Grade School of Archive and Records Management, Chonbuk National University)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.22, no.2, 2018 , pp. 382-389 More about this Journal
Abstract
Recently cyber attacks can cause serious damage on various information systems. Log data analysis would be able to resolve this problem. Security log analysis system allows to cope with security risk properly by collecting, storing, and analyzing log data information. In this paper, a security log analysis system is designed and implemented in order to analyze security log data using the Logstash in the Elasticsearch, a distributed search engine which enables to collect and process various types of log data. The Kibana, an open source data visualization plugin for Elasticsearch, is used to generate log statistics and search report, and visualize the results. The performance of Elasticsearch-based security log analysis system is compared to the existing log analysis system which uses the Flume log collector, Flume HDFS sink and HBase. The experimental results show that the proposed system tremendously reduces both database query processing time and log data analysis time compared to the existing Hadoop-based log analysis system.
Keywords
Security; Logstacy; Elasticsearch; Hadoop;
Citations & Related Records
Times Cited By KSCI : 4  (Citation Analysis)
연도 인용수 순위
1 J. H. Kim, "Big Data-based System Fault Detection through Log Data Analysis Techniques," Master's Thesis, Graduate School of Gunkook University, 2014.
2 V. Bhavana, "Data Security in Cloud environments", Asia-pacific Journal of Convergent Research Interchange, HSST, ISSN : 2508-9080, vol.1, no.4,pp.25-31, December 2015.), http://dx.doi.org/10.21742/APJCRI.2015.12.04.
3 N. Y. Kim, S. H. Kim, W. Y. Sohn, and J. G. Song, "Design and Implementation of Log Analysis for Intrusion Responsible System," KSII Transactions on Internet and Information Systems, vol. 2, no. 2, pp. 123-126, Nov. 2004.
4 S. R, Cho, "Big Data, Environmental Changes and Distributed Database System," Communications of KIISE, vol. 30, no. 5, pp. 21-28, May 2012.
5 D. H. Lee, J. C. Park, C. G. Yu, and H. S. Yun, "On the Design of a Big Data based Real-Time Network Traffic Analysis Platform," Journal of the Korea Institute of Information Security & Cryptology, vol. 23, no. 8, pp. 721-728, Aug. 2013.   DOI
6 B. M. Choi, J. H. Kong, S. S. Hong, and M. M. Han, "The Method of Analyzing Firewall Log Data using MapReduce based on NoSQL," Journal of the Korea Institute of Information Security & Cryptology, vol. 23, no. 4, pp. 667-677, Aug. 2013.   DOI
7 B. M. Choi, J. H. Kong, and M. M. Ha, "The Model of Network Packet Analysis based on Big Data," Journal of Korean Institute of Intelligent Systems, vol. 23, no. 5, pp. 392-39, Oct. 2013.   DOI
8 J. H. Lee and K. Raj, "Hybrid Data Management in Cloud Security", Asia-pacific Journal of Convergent Research Interchange, HSST, ISSN : 2508-9080, vol.1, no.4, pp.33-39, December, 2015. http://dx.doi.org/10.21742/APJCRI.2015.12.05
9 K. M. Ahn, J. Y. Lee, D. M. Yang, and B. H. Lee, "Design and Implementation of a Hadoop-based Efficient Security Log Analysis System," Journal of the Korea Institute of Information and Communication Engineering, vol. 19, no. 8, pp. 1797-1804. Aug. 2015.   DOI
10 M. J. Kim, S. H. Han, W. Choi, and H. G. Lee, "Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment," KSII Transactions on Internet and Information Systems, vol. 14, no. 6, pp.71-84, Dec. 2013.
11 NoSQL Databases: An Overview [Internet], Available: https://www.thoughtworks.com/insights/blog/nosql-databases-overview.
12 Elastic Search: Introduction, Basics, Architecture and Usage of Elastic Search [Internet], Available: https://hassantariqblog.wordpress.com/.
13 Apache Flume [Internet], Available: https://flume.apache.org.