• Title/Summary/Keyword: Distributed File System

Search Result 251, Processing Time 0.021 seconds

MapReduce-Based Partitioner Big Data Analysis Scheme for Processing Rate of Log Analysis (로그 분석 처리율 향상을 위한 맵리듀스 기반 분할 빅데이터 분석 기법)

  • Lee, Hyeopgeon;Kim, Young-Woon;Park, Jiyong;Lee, Jin-Woo
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.11 no.5
    • /
    • pp.593-600
    • /
    • 2018
  • Owing to the advancement of Internet and smart devices, access to various media such as social media became easy; thus, a large amount of big data is being produced. Particularly, the companies that provide various Internet services are analyzing the big data by using the MapReduce-based big data analysis techniques to investigate the customer preferences and patterns and strengthen the security. However, with MapReduce, when the big data is analyzed by defining the number of reducer objects generated in the reduce stage as one, the processing rate of big data analysis decreases. Therefore, in this paper, a MapReduce-based split big data analysis method is proposed to improve the log analysis processing rate. The proposed method separates the reducer partitioning stage and the analysis result combining stage and improves the big data processing rate by decreasing the bottleneck phenomenon by generating the number of reducer objects dynamically.

Performance Evaluation of Web Image Server for sharing e-Commerce System's Image (전자 상거래의 이미지 공유를 위한 웹 이미지 서버의 성능 평가)

  • Kim, Myoung-Eun;Cho, Dong-Sub
    • The KIPS Transactions:PartB
    • /
    • v.9B no.5
    • /
    • pp.533-540
    • /
    • 2002
  • We can buy products everywhere from Web-based shopping mall using desktop, cellular phone or PDA. To guarantee the various services for different equipment, shopping mail systems should allow a lot of different size or quality of images to provide a good service to their customers. Despite of same product image, each shopping mall saves the image in its storage space individually. Furthermore, all the product images in each shopping mall are stored as images of different quality. It may waste resources of shopping mail server and bring us developmental overhead. It is difficult to update all the images for product that is used by distributed e-catalog in everywhere. In this paper, we extended the proposed Web Image Server (WIS) for sharing one image with all clients and processing Image dynamically, so that we strengthened the function of managing shopping mall as a client of WIS and added the function of recording clients'log file and image catalog for shopping mall. We measured the response time from WIS and conventional e-Commerce site using by WAS which is one of the stress test tools for Web application. Furthermore, we measured WIS responses image requests in reasonable time when the current user is increased.

Design and Implementation of an Efficient Web Services Data Processing Using Hadoop-Based Big Data Processing Technique (하둡 기반 빅 데이터 기법을 이용한 웹 서비스 데이터 처리 설계 및 구현)

  • Kim, Hyun-Joo
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.16 no.1
    • /
    • pp.726-734
    • /
    • 2015
  • Relational databases used by structuralizing data are the most widely used in data management at present. However, in relational databases, service becomes slower as the amount of data increases because of constraints in the reading and writing operations to save or query data. Furthermore, when a new task is added, the database grows and, consequently, requires additional infrastructure, such as parallel configuration of hardware, CPU, memory, and network, to support smooth operation. In this paper, in order to improve the web information services that are slowing down due to increase of data in the relational databases, we implemented a model to extract a large amount of data quickly and safely for users by processing Hadoop Distributed File System (HDFS) files after sending data to HDFSs and unifying and reconstructing the data. We implemented our model in a Web-based civil affairs system that stores image files, which is irregular data processing. Our proposed system's data processing was found to be 0.4 sec faster than that of a relational database system. Thus, we found that it is possible to support Web information services with a Hadoop-based big data processing technique in order to process a large amount of data, as in conventional relational databases. Furthermore, since Hadoop is open source, our model has the advantage of reducing software costs. The proposed system is expected to be used as a model for Web services that provide fast information processing for organizations that require efficient processing of big data because of the increase in the size of conventional relational databases.

A Study on Non-Fungible Token Platform for Usability and Privacy Improvement (사용성 및 프라이버시 개선을 위한 NFT 플랫폼 연구)

  • Kang, Myung Joe;Kim, Mi Hui
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.11 no.11
    • /
    • pp.403-410
    • /
    • 2022
  • Non-Fungible Tokens (NFTs) created on the basis of blockchain have their own unique value, so they cannot be forged or exchanged with other tokens or coins. Using these characteristics, NFTs can be issued to digital assets such as images, videos, artworks, game characters, and items to claim ownership of digital assets among many users and objects in cyberspace, as well as proving the original. However, interest in NFTs exploded from the beginning of 2020, causing a lot of load on the blockchain network, and as a result, users are experiencing problems such as delays in computational processing or very large fees in the mining process. Additionally, all actions of users are stored in the blockchain, and digital assets are stored in a blockchain-based distributed file storage system, which may unnecessarily expose the personal information of users who do not want to identify themselves on the Internet. In this paper, we propose an NFT platform using cloud computing, access gate, conversion table, and cloud ID to improve usability and privacy problems that occur in existing system. For performance comparison between local and cloud systems, we measured the gas used for smart contract deployment and NFT-issued transaction. As a result, even though the cloud system used the same experimental environment and parameters, it saved about 3.75% of gas for smart contract deployment and about 4.6% for NFT-generated transaction, confirming that the cloud system can handle computations more efficiently than the local system.

Development of cardiopulmonary resuscitation nursing education program of web-based instruction (웹 기반의 심폐소생술 간호교육 프로그램 개발)

  • Sin, Hae-Won;Hong, Hae-Sook
    • Journal of Korean Biological Nursing Science
    • /
    • v.4 no.1
    • /
    • pp.25-39
    • /
    • 2002
  • The purpose of this study is to develop and evaluate a web-based instruction Program(WBI) to help nurses improving their knowledge and skill of cardiopulmonary resuscitation. Using the model of web-based instruction(WBI) program designed by Rhu(1999), this study was carried out during February-April 2002 in five different steps; analysis, design, data collection and reconstruction, programming and publishing, and evaluation. The results of the study were as follows; 1) The goal of this program was focused on improving accuracy of knowledge and skills of cardiopulmonary resuscitation. The program texts consists of the concepts and importances of cardiopulmonary resuscitation(CPR), basic life support(BLS), advanced cardiac life support(ACLS), treatment of CPR, nursing care after CPR treatment. And in the file making step, photographs, drawings and image files were collected and edited by web-editor(Namo), scanner and Adobe photoshop program. Then, the files were modified and posted on the web by file transfer protocol(FTP). Finally, the program was demonstrated and once again revised by the result, and then completed. 2) For the evaluation of the program, 36 nurses who in K university hospital located in D city, and related questionnaire were distributed to them as well. Higher scores were given by the nurses in its learning contents with $4.2{\pm}.67$, and in its structuring and interaction of the program with $4.0{\pm}.79$, and also in its satisfactory of the program with $4.2{\pm}.58$ respectively. In conclusion, if the contents of this WBI educational program upgrade further based upon analysis and applying of the results the program evaluation, it is considered as an effective tool to implement for continuing education as life-long educational system for nurse.

  • PDF

Design and Implementation of Game Server using the Efficient Load Balancing Technology based on CPU Utilization (게임서버의 CPU 사용율 기반 효율적인 부하균등화 기술의 설계 및 구현)

  • Myung, Won-Shig;Han, Jun-Tak
    • Journal of Korea Game Society
    • /
    • v.4 no.4
    • /
    • pp.11-18
    • /
    • 2004
  • The on-line games in the past were played by only two persons exchanging data based on one-to-one connections, whereas recent ones (e.g. MMORPG: Massively Multi-player Online Role-playings Game) enable tens of thousands of people to be connected simultaneously. Specifically, Korea has established an excellent network infrastructure that can't be found anywhere in the world. Almost every household has a high-speed Internet access. What made this possible was, in part, high density of population that has accelerated the formation of good Internet infrastructure. However, this rapid increase in the use of on-line games may lead to surging traffics exceeding the limited Internet communication capacity so that the connection to the games is unstable or the server fails. expanding the servers though this measure is very costly could solve this problem. To deal with this problem, the present study proposes the load distribution technology that connects in the form of local clustering the game servers divided by their contents used in each on-line game reduces the loads of specific servers using the load balancer, and enhances performance of sewer for their efficient operation. In this paper, a cluster system is proposed where each Game server in the system has different contents service and loads are distributed efficiently using the game server resource information such as CPU utilization. Game sewers having different contents are mutually connected and managed with a network file system to maintain information consistency required to support resource information updates, deletions, and additions. Simulation studies show that our method performs better than other traditional methods. In terms of response time, our method shows shorter latency than RR (Round Robin) and LC (Least Connection) by about 12%, 10% respectively.

  • PDF

Program Development for Automatic Extraction and Transformation of Standard Metadata of Geo-spatial Data (공간정보 표준 메타데이터 추출 및 변환 프로그램 개발)

  • Han, Sun-Mook;Lee, Ki-Won
    • Korean Journal of Remote Sensing
    • /
    • v.26 no.5
    • /
    • pp.549-559
    • /
    • 2010
  • In geo-spatial information system building and operation, metadata is one of the crucial factors. Therefore, international and domestic organizations or associations for standardization have developed and distributed geo-based standard metadata to meet public demands. However, because metadata is composed of complicated elements and needs XML storage and management, individual organization which implement and operate practical application system is inclined to define and use its own metadata specifications. In this study, metadata extraction program, that metadata elements are directly extracted from geo-based file formats was developed to easily utilize standard metadata such as ISO/TC 19115, TTAS.KO-10.0139 and TTAS.IS-19115, and those elements are processed into XML. Furthermore, geo-based images sets are applied to another metadata of ISO/TC 19115-2. As well, metadata transformation is needed due to inconsistent or non-corresponding definition among standard metadata; in this program, transformation modules are also implemented to interoperable uses between standard metadata specifications. Widely used data formats are dealt with in this program, but extension for other formats and other metadata specifications is possible, and it is expected that availability of standard metadata is increased, through this kind of development.

An Analysis of Big Video Data with Cloud Computing in Ubiquitous City (클라우드 컴퓨팅을 이용한 유시티 비디오 빅데이터 분석)

  • Lee, Hak Geon;Yun, Chang Ho;Park, Jong Won;Lee, Yong Woo
    • Journal of Internet Computing and Services
    • /
    • v.15 no.3
    • /
    • pp.45-52
    • /
    • 2014
  • The Ubiquitous-City (U-City) is a smart or intelligent city to satisfy human beings' desire to enjoy IT services with any device, anytime, anywhere. It is a future city model based on Internet of everything or things (IoE or IoT). It includes a lot of video cameras which are networked together. The networked video cameras support a lot of U-City services as one of the main input data together with sensors. They generate huge amount of video information, real big data for the U-City all the time. It is usually required that the U-City manipulates the big data in real-time. And it is not easy at all. Also, many times, it is required that the accumulated video data are analyzed to detect an event or find a figure among them. It requires a lot of computational power and usually takes a lot of time. Currently we can find researches which try to reduce the processing time of the big video data. Cloud computing can be a good solution to address this matter. There are many cloud computing methodologies which can be used to address the matter. MapReduce is an interesting and attractive methodology for it. It has many advantages and is getting popularity in many areas. Video cameras evolve day by day so that the resolution improves sharply. It leads to the exponential growth of the produced data by the networked video cameras. We are coping with real big data when we have to deal with video image data which are produced by the good quality video cameras. A video surveillance system was not useful until we find the cloud computing. But it is now being widely spread in U-Cities since we find some useful methodologies. Video data are unstructured data thus it is not easy to find a good research result of analyzing the data with MapReduce. This paper presents an analyzing system for the video surveillance system, which is a cloud-computing based video data management system. It is easy to deploy, flexible and reliable. It consists of the video manager, the video monitors, the storage for the video images, the storage client and streaming IN component. The "video monitor" for the video images consists of "video translater" and "protocol manager". The "storage" contains MapReduce analyzer. All components were designed according to the functional requirement of video surveillance system. The "streaming IN" component receives the video data from the networked video cameras and delivers them to the "storage client". It also manages the bottleneck of the network to smooth the data stream. The "storage client" receives the video data from the "streaming IN" component and stores them to the storage. It also helps other components to access the storage. The "video monitor" component transfers the video data by smoothly streaming and manages the protocol. The "video translator" sub-component enables users to manage the resolution, the codec and the frame rate of the video image. The "protocol" sub-component manages the Real Time Streaming Protocol (RTSP) and Real Time Messaging Protocol (RTMP). We use Hadoop Distributed File System(HDFS) for the storage of cloud computing. Hadoop stores the data in HDFS and provides the platform that can process data with simple MapReduce programming model. We suggest our own methodology to analyze the video images using MapReduce in this paper. That is, the workflow of video analysis is presented and detailed explanation is given in this paper. The performance evaluation was experiment and we found that our proposed system worked well. The performance evaluation results are presented in this paper with analysis. With our cluster system, we used compressed $1920{\times}1080(FHD)$ resolution video data, H.264 codec and HDFS as video storage. We measured the processing time according to the number of frame per mapper. Tracing the optimal splitting size of input data and the processing time according to the number of node, we found the linearity of the system performance.

Development of Information Technology Infrastructures through Construction of Big Data Platform for Road Driving Environment Analysis (도로 주행환경 분석을 위한 빅데이터 플랫폼 구축 정보기술 인프라 개발)

  • Jung, In-taek;Chong, Kyu-soo
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.3
    • /
    • pp.669-678
    • /
    • 2018
  • This study developed information technology infrastructures for building a driving environment analysis platform using various big data, such as vehicle sensing data, public data, etc. First, a small platform server with a parallel structure for big data distribution processing was developed with H/W technology. Next, programs for big data collection/storage, processing/analysis, and information visualization were developed with S/W technology. The collection S/W was developed as a collection interface using Kafka, Flume, and Sqoop. The storage S/W was developed to be divided into a Hadoop distributed file system and Cassandra DB according to the utilization of data. Processing S/W was developed for spatial unit matching and time interval interpolation/aggregation of the collected data by applying the grid index method. An analysis S/W was developed as an analytical tool based on the Zeppelin notebook for the application and evaluation of a development algorithm. Finally, Information Visualization S/W was developed as a Web GIS engine program for providing various driving environment information and visualization. As a result of the performance evaluation, the number of executors, the optimal memory capacity, and number of cores for the development server were derived, and the computation performance was superior to that of the other cloud computing.

GWB: An integrated software system for Managing and Analyzing Genomic Sequences (GWB: 유전자 서열 데이터의 관리와 분석을 위한 통합 소프트웨어 시스템)

  • Kim In-Cheol;Jin Hoon
    • Journal of Internet Computing and Services
    • /
    • v.5 no.5
    • /
    • pp.1-15
    • /
    • 2004
  • In this paper, we explain the design and implementation of GWB(Gene WorkBench), which is a web-based, integrated system for efficiently managing and analyzing genomic sequences, Most existing software systems handling genomic sequences rarely provide both managing facilities and analyzing facilities. The analysis programs also tend to be unit programs that include just single or some part of the required functions. Moreover, these programs are widely distributed over Internet and require different execution environments. As lots of manual and conversion works are required for using these programs together, many life science researchers suffer great inconveniences. in order to overcome the problems of existing systems and provide a more convenient one for helping genomic researches in effective ways, this paper integrates both managing facilities and analyzing facilities into a single system called GWB. Most important issues regarding the design of GWB are how to integrate many different analysis programs into a single software system, and how to provide data or databases of different formats required to run these programs. In order to address these issues, GWB integrates different analysis programs byusing common input/output interfaces called wrappers, suggests a common format of genomic sequence data, organizes local databases consisting of a relational database and an indexed sequential file, and provides facilities for converting data among several well-known different formats and exporting local databases into XML files.

  • PDF