• Title/Summary/Keyword: 데이터 삽입

Search Result 765, Processing Time 0.022 seconds

Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)

  • Kim, Myoungjin;Han, Seungho;Cui, Yun;Lee, Hanku
    • Journal of Internet Computing and Services
    • /
    • v.14 no.6
    • /
    • pp.71-84
    • /
    • 2013
  • Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.

An Image Concealment Algorithm Using Fuzzy Inference (퍼지 추론을 이용한 영상은닉 알고리즘)

  • Kim, Ha-Sik;Kim, Yoon-Ho
    • Journal of Advanced Navigation Technology
    • /
    • v.11 no.4
    • /
    • pp.485-492
    • /
    • 2007
  • In this paper, we propose the receiver block error detection of the video codec and the image concealment algorithm using fuzzy inference. The proposed error detection and concealment algorithm gets SSD(Summation of Squared Difference) and BMC(Boundary Matching Coefficient) using the temporal and spatial similarity between corresponded blocks in the two successive frames. Proportional constant, ${\alpha}$, for threshold value, TH1 and TH2, is decided after fuzzy data is generated by each parameter. To examine the propriety of the proposed algorithm, random errors are inserted into the QCIF Susie standard image, then the error detection and concealment performance is simulated. To evaluate the efficiency of the algorithm, image quality is evaluated by PSNR for the error detection and concealed image by the existing VLC table and by the proposed method. In the experimental results, the error detection algorithm could detect all of the inserted error, the image quality is improved over 15dB after the error concealment compare to existing error detection algorithm.

  • PDF

A Study on Digital Image Watermarking for Embedding Audio Logo (음성로고 삽입을 위한 디지털 영상 워터마킹에 관한 연구)

  • Cho, Gang-Seok;Koh, Sung-Shik
    • Journal of the Institute of Electronics Engineers of Korea TE
    • /
    • v.39 no.3
    • /
    • pp.21-27
    • /
    • 2002
  • The digital watermarking methods have been proposed as a solution for solving the illegal copying and proof of ownership problems in the context of multimedia data. But it is still difficult to have been overcame the problem of the protection of property to multimedia data, such as digital images, digital video, and digital audio. This paper describes a watermarking algorithm that embeds non-linearly audio logo watermark data which is converted from audio signal of the ownership in the components of pixel intensities in an original image and that insists of ownership by hearing the audio signal transformed from the extracted audio logo through the speaker. Experimental results show that our algorithm using audio logo proposed in this paper is robust against attacks such as particularly lossy JPEG image compression. 

Development of Bypass Unit for Ship Area Network Based on Legacy-line Communication (무배선 통신을 위한 선박 네트워크용 바이패스 장치 개발)

  • Jun, Ho-Ik;Kim, Hyun-Sik;Jung, Kyun Sik;Sohn, Kyung-Rak
    • Journal of Advanced Marine Engineering and Technology
    • /
    • v.39 no.3
    • /
    • pp.292-297
    • /
    • 2015
  • In this paper, we reported the bypass unit for ship area networks in order to detour the communication failure regions and poor communication links. The device was composed of three parts of circuits for power cut-off, protection, and coupling transformer. Since the coupling transformer exerts a dominant influence on the performance of the by-pass unit, we have tried to find the optimal magnetic core materials and its dimensions. The prototype was passed through the performance test of insertion loss, temperature, and vibration characteristics. The insertion loss was around -2 dB in the range of 90 kHz ~ 30 MHz and the average communication speed was 59.2 Mbps in the laboratory. A pilot communication test using the developed tool was conducted in the training ship of the Korea Maritime and Ocean University. As a results of experiment, we showed that the wired communication among the heterogeneous-links in the ship area networks are possible by the bypass unit and also a high speed communication services are available in ~ Mbps by using a power-line.

Determination of Optimal Checkpoint Intervals for Real-Time Tasks Using Distributed Fault Detection (분산 고장 탐지 방식을 이용한 실시간 태스크에서의 최적 체크포인터 구간 선정)

  • Kwak, Seong Woo;Yang, Jung-Min
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.26 no.3
    • /
    • pp.202-207
    • /
    • 2016
  • Checkpoint placement is an effective fault tolerance technique against transient faults in which the task is re-executed from the latest checkpoint when a fault is detected. In this paper, we propose a new checkpoint placement strategy separating data saving and fault detection processes that are performed together in conventional checkpoints. Several fault detection processes are performed in one checkpoint interval in order to decrease the latency between the occurrence and detection of faults. We address the placement method of fault detection processes to maximize the probability of successful execution of a task within the given deadline. We develop the Markov chain model for a real-time task having the proposed checkpoints, and derive the optimal fault detection and checkpoint interval.

A Robust DNA Watermarking in Lifting Based 1D DWT Domain (Lifting 기반 1D DWT 영역 상의 강인한 DNA 워터마킹)

  • Lee, Suk-Hwan;Kwon, Ki-Ryong;Kwon, Seong-Geun
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.49 no.10
    • /
    • pp.91-101
    • /
    • 2012
  • DNA watermarking have been interested for both the security of private genetic information or huge DNA storage information and the copyright protection of GMO. Multimedia watermarking has been mainly designed on the basis of frequency domain, such as DCT, DWT, FMT, and so on, for the robustness and invisibility. But a frequency domain watermarking for coding DNA sequence has a considerable constraint for embedding the watermark because transform and inverse transform must be performed without completely changing the amino acid sequence. This paper presents a coding sequence watermarking on lifting based DWT domain and brings up the availability of frequency domain watermarking for DNA sequence. From experimental results, we verified that the proposed scheme has the robustness to until a combination of 10% point mutations, 5% insertion and deletion mutations and also the amino preservation and the security.

A Wavelet Based Robust Logo Watermarking Algorithm for Digital Content Protection (디지털 콘텐트 보호를 위한 강인한 웨이블릿 기반 로고 워터마킹 알고리즘)

  • Kim, Tae-Jung;Hwang, Jae-Ho;Hong, Choong-Seon
    • Journal of Internet Computing and Services
    • /
    • v.9 no.1
    • /
    • pp.33-41
    • /
    • 2008
  • Due to the advantage of wavelet transform such as the compatibility with JPEG2000, multi-resolution decomposition, and application of HVS(Human Visual System), watermarking algorithm based on wavelet transform (DWT) is recently mast interesting research subject. However, mast of researches are focused on theoretical aspects for the robustness rather than practical usage, and. may be not suitable too much complicated to use in practice. In this paper, robust logo watermarking algorithm based on DWT is proposed for large and huge data processing. The proposed method embeds the logo watermark by mapping of $8{\times}8$ blocks in order of the number of '1' of the original image and the randomized watermark image with LFSR. The original image is transformed by 2 level wavelet. The experimental results shows that the watermark is embedded successfully, and the proposed algorithm has the valuable robustness from the image processing like JPEG compression, low pass filter, high pass filter and changes in brightness and contrast.

  • PDF

Attacks and Defenses for Vulnerability of Cross Site Scripting (크로스 사이트 스크립팅(XSS) 취약점에 대한 공격과 방어)

  • Choi, Eun-Jung;Jung, Whi-Chan;Kim, Seung-Yeop
    • Journal of Digital Convergence
    • /
    • v.13 no.2
    • /
    • pp.177-183
    • /
    • 2015
  • Cross Site Scripting enables hackers to steal other user's information (such as cookie, session etc.) or to do abnormal functions automatically using vulnerability of web application. This attack patterns of Cross Site Scripting(XSS) can be divided into two types. One is Reflect XSS which can be executed in one request for HTTP and its reply, and the other is Stored XSS which attacks those many victim users whoever access to the page which accepted the payload transmitted. To correspond to these XSS attacks, some measures have been suggested. They are data validation for user input, output validation during HTML encoding procedures, and removal of possible risk injection point to prevent from trying to insert malicious code into web application. In this paper, the methods and procedures for these two types are explained and a penetration testing is done. With these suggestions, the attack by XSS could be understood and prepared by its countermeasures.

Development of Audio Watermark Decoding Model Using Support Vector Machine (Support Vector Machine을 이용한 오디오 워터마크 디코딩 모델 개발)

  • Seo, Yejin;Cho, Sangjin
    • The Journal of the Acoustical Society of Korea
    • /
    • v.33 no.6
    • /
    • pp.400-406
    • /
    • 2014
  • This paper describes a robust watermark decoding model using a SVM(Support Vector Machine). First, the embedding process is performed inversely for a watermarked signal. And then the watermark is extracted using the proposed model. For SVM training of the proposed model, data are generated that are watermarks extracted from sounds containing watermarks by four different embedding schemes. BER(Bit Error Rate) values of the data are utilized to determine a threshold value employed to create training set. To evaluate the robustness, 14 attacks selected in StirMark, SMDI and STEP2000 benchmarking are applied. Consequently, the proposed model outperformed previous method in PSNR(Peak Signal to Noise Ratio) and BER. It is noticeable that the proposed method achieves BER 1% below in the case of PSNR greater than 10 dB.

A Study on Effective Digital Watermark Generation Method to Overcome Capacity Limit (저장 한계를 극복한 효율적인 디지털 워터마크 생성 방법 연구)

  • Kim Hee-Sun;Cho Dae-Jea
    • The Journal of the Korea Contents Association
    • /
    • v.5 no.6
    • /
    • pp.343-350
    • /
    • 2005
  • During the design of a successful digital watermarking systems, Pseudo-Noise(PN) sequences are widely used to modulate information bits into watermark signals. In this method, the number of bits that can be hidden within a small image by means of frequency domain watermarking is limited. In this paper, we show the possibility of introducing chaotic sequences into digital watermarking systems as potential substitutes to commonly used PN-sequences. And we propose a method that transforms the text to chaotic sequence. In our current implementation, we show how the sample text is expressed by an implied unit data(watermark) and the implied unit data is regenerated into the original left. Because we use this implied data as watermark for information hiding, we can insert much more watermark compared with previous method.

  • PDF