• Title/Summary/Keyword: Database Log Analysis

Search Result 68, Processing Time 0.036 seconds

A Log Analyzer for Database Tuning (데이타베이스 튜닝을 위한 로그 분석 도구)

  • Lee, Sang-Hyup;Kim, Sung-Jin;Lee, Sang-Ho
    • The KIPS Transactions:PartD
    • /
    • v.11D no.5
    • /
    • pp.1041-1048
    • /
    • 2004
  • Database logs contain various information on database operations, but they are used to recover database systems from failures generally. This paper proposes a log analysis tool that provides useful information for database tuning. This tool provides users with information on work-load organization, database schemas, and resources usages of queries. This paper describes the tool in views of its architecture, functions, implementation, and verification. The tool is verified by running the TPC-W benchmark, and representative analysis results are also presented.

Improvement of RocksDB Performance via Large-Scale Parameter Analysis and Optimization

  • Jin, Huijun;Choi, Won Gi;Choi, Jonghwan;Sung, Hanseung;Park, Sanghyun
    • Journal of Information Processing Systems
    • /
    • v.18 no.3
    • /
    • pp.374-388
    • /
    • 2022
  • Database systems usually have many parameters that must be configured by database administrators and users. RocksDB achieves fast data writing performance using a log-structured merged tree. This database has many parameters associated with write and space amplifications. Write amplification degrades the database performance, and space amplification leads to an increased storage space owing to the storage of unwanted data. Previously, it was proven that significant performance improvements can be achieved by tuning the database parameters. However, tuning the multiple parameters of a database is a laborious task owing to the large number of potential configuration combinations. To address this problem, we selected the important parameters that affect the performance of RocksDB using random forest. We then analyzed the effects of the selected parameters on write and space amplifications using analysis of variance. We used a genetic algorithm to obtain optimized values of the major parameters. The experimental results indicate an insignificant reduction (-5.64%) in the execution time when using these optimized values; however, write amplification, space amplification, and data processing rates improved considerably by 20.65%, 54.50%, and 89.68%, respectively, as compared to the performance when using the default settings.

An Analysis and Visualization of Creative Tendency appeared in Query Log of a Story Database Service (스토리 검색 서비스의 사용자 기록에 나타난 창작 성향 분석 및 가시화)

  • Kim, Myoung-Jun
    • Journal of Korea Multimedia Society
    • /
    • v.19 no.8
    • /
    • pp.1609-1618
    • /
    • 2016
  • is a service providing the story synopses that match user's query. This paper analyzes the user log of which is the answers to the queries to find stories from database, and shows the tendency distribution of user creation. Specially, we analyze a joint distribution of the genres and actions of stories to get better understanding of the tendencies that cannot be found in the analysis of independent distribution. Furthermore, we define a correlation factor between genre and action, and investigate what combinations of the genres and actions are highly, less, and negatively correlated. Finally, we investigate how the tendencies of characters are related to genres and actions, and propose a visualization method to show the tendencies.

An Analysis of Search Log from a Story Database Service and a New Story Search Method based on Story Map (스토리 검색 서비스의 사용자 기록 분석 및 스토리맵에 의한 새로운 스토리 검색 방법)

  • Kim, Myoung-Jun
    • Journal of Digital Contents Society
    • /
    • v.16 no.5
    • /
    • pp.795-803
    • /
    • 2015
  • is a service providing story synopsis that matches user's query. This paper analyzes the user log of and shows the tendency distribution of user creation in comparison to stories in database. We also investigate the log to see possible improvements on search method. This paper proposes a concept of Story Map, in which the query-answer information is projected into spatial coordinates, and a new story search UI based on it. Using the Story Map, users are able to see entire spatial distribution of story database so that they can quickly and intuitively find a story on the map.

HBase based Business Process Event Log Schema Design of Hadoop Framework

  • Ham, Seonghun;Ahn, Hyun;Kim, Kwanghoon Pio
    • Journal of Internet Computing and Services
    • /
    • v.20 no.5
    • /
    • pp.49-55
    • /
    • 2019
  • Organizations design and operate business process models to achieve their goals efficiently and systematically. With the advancement of IT technology, the number of items that computer systems can participate in and the process becomes huge and complicated. This phenomenon created a more complex and subdivide flow of business process.The process instances that contain workcase and events are larger and have more data. This is an essential resource for process mining and is used directly in model discovery, analysis, and improvement of processes. This event log is getting bigger and broader, which leads to problems such as capacity management and I / O load in management of existing row level program or management through a relational database. In this paper, as the event log becomes big data, we have found the problem of management limit based on the existing original file or relational database. Design and apply schemes to archive and analyze large event logs through Hadoop, an open source distributed file system, and HBase, a NoSQL database system.

A Security Log Analysis System using Logstash based on Apache Elasticsearch (아파치 엘라스틱서치 기반 로그스태시를 이용한 보안로그 분석시스템)

  • Lee, Bong-Hwan;Yang, Dong-Min
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.22 no.2
    • /
    • pp.382-389
    • /
    • 2018
  • Recently cyber attacks can cause serious damage on various information systems. Log data analysis would be able to resolve this problem. Security log analysis system allows to cope with security risk properly by collecting, storing, and analyzing log data information. In this paper, a security log analysis system is designed and implemented in order to analyze security log data using the Logstash in the Elasticsearch, a distributed search engine which enables to collect and process various types of log data. The Kibana, an open source data visualization plugin for Elasticsearch, is used to generate log statistics and search report, and visualize the results. The performance of Elasticsearch-based security log analysis system is compared to the existing log analysis system which uses the Flume log collector, Flume HDFS sink and HBase. The experimental results show that the proposed system tremendously reduces both database query processing time and log data analysis time compared to the existing Hadoop-based log analysis system.

Design and Implementation of a Hadoop-based Efficient Security Log Analysis System (하둡 기반의 효율적인 보안로그 분석시스템 설계 및 구현)

  • Ahn, Kwang-Min;Lee, Jong-Yoon;Yang, Dong-Min;Lee, Bong-Hwan
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.19 no.8
    • /
    • pp.1797-1804
    • /
    • 2015
  • Integrated log management system can help to predict the risk of security and contributes to improve the security level of the organization, and leads to prepare an appropriate security policy. In this paper, we have designed and implemented a Hadoop-based log analysis system by using distributed database model which can store large amount of data and reduce analysis time by automating log collecting procedure. In the proposed system, we use the HBase in order to store a large amount of data efficiently in the scale-out fashion and propose an easy data storing scheme for analysing data using a Hadoop-based normal expression, which results in improving data processing speed compared to the existing system.

Design of Main-Memory Database Prototype System using Fuzzy Checkpoint Technique in Real-Time Environment (실시간 시스템에서 퍼지 검사점을 이용한 주기억 데이터베이스 프로토타입 시스템의설계)

  • Park, Yong-Mun;Lee, Chan-Seop;Choe, Ui-In
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.6
    • /
    • pp.1753-1765
    • /
    • 2000
  • As the areas of computer application are expanded, real-time application environments that must process as many transactions as possible within their deadlines, such as a stock transaction systems, ATM switching systems etc, have been increased recently. The reason why the conventional database systems can't process soft real-time applications is the lack of prediction and poor performance on processing transaction's deadline. If transactions want to access data stored at the secondary storage, they can not satisfy requirements of real-time applications because of the disk delay time. This paper designs a main-memory database prototype systems to be suitable to real-time applications and then this system can produce rapid results without disk i/o as all of the information are loaded in main memory database. In thesis proposed the improved techniques with respect to logging, checkpointing, and recovering in our environment. In order to improve the performance of the system, a) the frequency of log analysis and redo processing is reduced by the proposed redo technique at system failure, b) database consistency is maintained by improved fuzzy checkpointing. The performance model is proposed which consists of two parts. The first part evaluates log processing time for recovery and compares with other research activities. The second part examines checkpointing behavior.

  • PDF

A Study on the Analysis of Validity and Importance of Event Log for the Detection of Insider Threats to Control System (제어시스템의 내부자 위협 탐지를 위한 Event Log 타당성 및 중요도 분석에 관한 연구)

  • Kim, Jongmin;Kim, DongMin;Lee, DongHwi
    • Convergence Security Journal
    • /
    • v.18 no.3
    • /
    • pp.77-85
    • /
    • 2018
  • With the convergence of communications network between control system and public network, such threats like information leakage/falsification could be fully shown in control system through diverse routes. Due to the recent diversification of security issues and violation cases of new attack techniques, the security system based on the information database that simply blocks and identifies, is not good enough to cope with the new types of threat. The current control system operates its security system focusing on the outside threats to the inside, and it is insufficient to detect the security threats by insiders with the authority of security access. Thus, this study conducted the importance analysis based on the main event log list of "Spotting the Adversary with Windows Event Log Monitoring" announced by NSA. In the results, the matter of importance of event log for the detection of insider threats to control system was understood, and the results of this study could be contributing to researches in this area.

  • PDF

A Study on Event Log Correlation Analysis for Control System Threat Analysis (제어시스템 위협분석을 위한 Event Log 상관분석에 관한 연구)

  • Kim, Jongmin;Kim, Minsu;Lee, DongHwi
    • Convergence Security Journal
    • /
    • v.17 no.5
    • /
    • pp.35-40
    • /
    • 2017
  • The control system can have such threats as information leakage and falsification through various routes due to communications network fusion with public network. As the issues about security and the infringe cases by new attack methods are diversified recently, with the security system that makes information data database by simply blocking and checking it is difficult to cope with new types of threats. It is also difficult to respond security threats by insiders who have security access authority with the existing security equipment. To respond the threats by insiders, it is necessary to collect and analyze Event Log occurring in the internal system realtime. Therefore, this study could find out whether there is correlation of the elements among Event Logs through correlation analysis based on Event Logs that occur real time in the control system, and based on the analysis result, the study is expected to contribute to studies in this field.