• Title/Summary/Keyword: I/O Processing

Search Result 568, Processing Time 0.028 seconds

An Analysis of Utilization on Virtualized Computing Resource for Hadoop and HBase based Big Data Processing Applications (Hadoop과 HBase 기반의 빅 데이터 처리 응용을 위한 가상 컴퓨팅 자원 이용률 분석)

  • Cho, Nayun;Ku, Mino;Kim, Baul;Xuhua, Rui;Min, Dugki
    • Journal of Information Technology and Architecture
    • /
    • v.11 no.4
    • /
    • pp.449-462
    • /
    • 2014
  • In big data era, there are a number of considerable parts in processing systems for capturing, storing, and analyzing stored or streaming data. Unlike traditional data handling systems, a big data processing system needs to concern the characteristics (format, velocity, and volume) of being handled data in the system. In this situation, virtualized computing platform is an emerging platform for handling big data effectively, since virtualization technology enables to manage computing resources dynamically and elastically with minimum efforts. In this paper, we analyze resource utilization of virtualized computing resources to discover suitable deployment models in Apache Hadoop and HBase-based big data processing environment. Consequently, Task Tracker service shows high CPU utilization and high Disk I/O overhead during MapReduce phases. Moreover, HRegion service indicates high network resource consumption for transfer the traffic data from DataNode to Task Tracker. DataNode shows high memory resource utilization and Disk I/O overhead for reading stored data.

Independent I/O Relay Class Design Using Modbus Protocol for Embedded Systems

  • Kim, Ki-Su;Lee, Jong-Chan
    • Journal of the Korea Society of Computer and Information
    • /
    • v.25 no.6
    • /
    • pp.1-8
    • /
    • 2020
  • Communication between system modules is applied using the Modbus protocol in industrial sites including smart factories, industrial drones, building energy management systems, PLCs, ships, trains, and airplanes. The existing Modbus was used for serial communication, but the recent Modbus protocol is used for TCP/IP communication.The Modbus protocol supports RTU, TCP and ASCII, and implements and uses protocols in embedded systems. However, the transmission I/O devices for RTU, TCP, and ASCII-based protocols may differ. For example, RTU and ASCII communications transmit on a serial-based communication protocol, but in some cases, Ethernet TCP/IP transmission is required. In particular, since the C language (object-oriented) is used in embedded systems, the complexity of source code related to I/O registers increases. In this study, we designed software that can logically separate I/O functions from embedded devices, and designed the execution logic of each instance requiring I/O processing through a delegate class instance with Modbus RTU, TCP, and ASCII protocol generation. We designed and experimented with software that can separate communication I/O processing and logical execution logic for each instance.

Sim-Hadoop : Leveraging Hadoop Distributed File System and Parallel I/O for Reliable and Efficient N-body Simulations (Sim-Hadoop : 신뢰성 있고 효율적인 N-body 시뮬레이션을 위한 Hadoop 분산 파일 시스템과 병렬 I / O)

  • Awan, Ammar Ahmad;Lee, Sungyoung;Chung, Tae Choong
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.05a
    • /
    • pp.476-477
    • /
    • 2013
  • Gadget-2 is a scientific simulation code has been used for many different types of simulations like, Colliding Galaxies, Cluster Formation and the popular Millennium Simulation. The code is parallelized with Message Passing Interface (MPI) and is written in C language. There is also a Java adaptation of the original code written using MPJ Express called Java Gadget. Java Gadget writes a lot of checkpoint data which may or may not use the HDF-5 file format. Since, HDF-5 is MPI-IO compliant, we can use our MPJ-IO library to perform parallel reading and writing of the checkpoint files and improve I/O performance. Additionally, to add reliability to the code execution, we propose the usage of Hadoop Distributed File System (HDFS) for writing the intermediate (checkpoint files) and final data (output files). The current code writes and reads the input, output and checkpoint files sequentially which can easily become bottleneck for large scale simulations. In this paper, we propose Sim-Hadoop, a framework to leverage HDFS and MPJ-IO for improving the I/O performance of Java Gadget code.

I/O Scheduler Scheme for User Responsiveness in Mobile Systems (모바일 시스템에서 사용자 반응성을 고려한 입출력 스케줄링 기법)

  • Park, Jong Woo;Yoon, Jun Young;Seo, Dae-Wha
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.5 no.11
    • /
    • pp.379-384
    • /
    • 2016
  • NAND flash storage is widely used for computer systems, because of it has faster response time, lower power consumption, and larger capacity per unit area than hard disk. However, currently used I/O scheduler in the operating system is optimized for characteristics of the hard disk. Therefore, the conventional I/O scheduler includes the unnecessary overhead in the case of the NAND flash storage to be applied. Particularly, when the write requests performed intensively, garbage collection is performed intensively. So, it occurs the problem that the processing of the I/O request delay. In this paper, we propose the new I/O scheduler to solve the problem of garbage collection performs intensively, and to optimize for NAND flash storage. In the result of performance evaluation, proposed scheme shows an improvement the user responsiveness by reducing 1% of the average read response time and 78% of the maximum response time.

Performance Evaluation of Disk Scheduling Schemes in a VOD System (VOD 시스템에서의 디스크 스케줄링 기법의 성능 평가)

  • Jeon, Yong-Hee;Lee, Sang-Hag
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.11
    • /
    • pp.3519-3533
    • /
    • 2000
  • In a VOD(Video-On-Demand) system, I/O performance and storage requirements are generally more important than computing capability, Due to the development of CPU tecnology, the gap between the performance of processors and desk speed are widening. Therefore, the disk system typically becomes the bohleneck ina VOD system. In order to allevte those effects from physical constramts, disk array system is used. The I/O performance of disk arrays can be improved by the capability of processors, I/O schefuling polcy, the number of member disks in the array, disk block size, block placemet method etc. In this paper, we considered the disk scheduling schemes including EDF( Earliest Deadine First), SCAN, SGAN-EDF, Round-robm. GSS(Grouped Sweeping scheme), and C-SCAN, LOOK, C-LOOK which are variations of SCAN Mamly focusing on those disk scheduling schemes, we performed the simulation in order to compare and eview the perionmance of disk arrays considening seveial related parameters under diverse environments and analyzed the effects of those parameters to the performance.

  • PDF

A Study on the Design and Development of Automatic Optical Fiber Aligner (자동 광섬유 정렬 장치의 설계 및 제작에 관한 연구)

  • Kim, Byung-Hee;Uhm, Chul;Choi, Young-Suk
    • Journal of Industrial Technology
    • /
    • v.22 no.B
    • /
    • pp.241-249
    • /
    • 2002
  • Optical fiber is indispensable for optical communication systems that transmit large volumes of data at high speed, but super precision technology in sub-micron units is required for optical axis adjustment. We developed the automatic optical fiber by image processing and automatic loading system. we have developed 6-axis micro stage system for I/O optical fiber arrays, the initial automatic aligning system software for a input optical array by the image processing technique, fast I/O-synchronous aligning strategy, the automatic loading/unloading system and the automatic UV bonding mechanism. In order to adjust the alignment it used on PC based motion controller, a $10{\mu}m$ repeat-detailed drawing of automatic loading system is developed by a primary line up for high detailed drawing. Also, at this researches used the image processing system and algorithm instead of the existing a primary hand-line up and fiber input array and waveguide chip formed in line by automatic.

  • PDF

On the Study Of AlSiCa($Al_2O_3-SiC-C$) refractories: (I) Synthesis of raw material using domestic chnmotte (AlSiCa($Al_2O_3-SiC-C$)계 내화물 재료에 관한 연구: (I) 국산 chamotte로부터 원료분말합성)

  • Shim, Kwang-Bo;Joo, Kyoung;Auh, Keun-Ho
    • Journal of the Korean Crystal Growth and Crystal Technology
    • /
    • v.7 no.4
    • /
    • pp.626-631
    • /
    • 1997
  • AlSiCa powders were prepared from the domestic Hadong Kaolin ($Al_2O_3{\cdot}2SiO_2{\cdot}2H_2O$). As a result of the reaction of Hadong Kaolin and carbon powder at reducing atmosphere, $Al_2O_3{\cdot}SiC$ composite started to form at $1300^{\circ}C$ and completed at $1400^{\circ}C$. The optimum amount of carbon was 1:4 in mole ratio. It was found that only bright-green $\beta-SiC$ phase forms when the mixture was packed without carbon powder in alumina crucible.

  • PDF

Performance Evaluation and Analysis on Single and Multi-Network Virtualization Systems with Virtio and SR-IOV (가상화 시스템에서 Virtio와 SR-IOV 적용에 대한 단일 및 다중 네트워크 성능 평가 및 분석)

  • Jaehak Lee;Jongbeom Lim;Heonchang Yu
    • The Transactions of the Korea Information Processing Society
    • /
    • v.13 no.2
    • /
    • pp.48-59
    • /
    • 2024
  • As functions that support virtualization on their own in hardware are developed, user applications having various workloads are operating efficiently in the virtualization system. SR-IOV is a virtualization support function that takes direct access to PCI devices, thus giving a high I/O performance by minimizing the need for hypervisor or operating system interventions. With SR-IOV, network I/O acceleration can be realized in virtualization systems that have relatively long I/O paths compared to bare-metal systems and frequent context switches between the user area and kernel area. To take performance advantages of SR-IOV, network resource management policies that can derive optimal network performance when SR-IOV is applied to an instance such as a virtual machine(VM) or container are being actively studied.This paper evaluates and analyzes the network performance of SR-IOV implementing I/O acceleration is compared with Virtio in terms of 1) network delay, 2) network throughput, 3) network fairness, 4) performance interference, and 5) multi-network. The contributions of this paper are as follows. First, the network I/O process of Virtio and SR-IOV was clearly explained in the virtualization system, and second, the evaluation results of the network performance of Virtio and SR-IOV were analyzed based on various performance metrics. Third, the system overhead and the possibility of optimization for the SR-IOV network in a virtualization system with high VM density were experimentally confirmed. The experimental results and analysis of the paper are expected to be referenced in the network resource management policy for virtualization systems that operate network-intensive services such as smart factories, connected cars, deep learning inference models, and crowdsourcing.

Performance Evaluation of iSER on Storage system using Infiniband fabric (인피니밴드기반 저장장치에서의 iSER(iSCSI Extension for RDMA) 성능평가)

  • Kim, Young Hwan;Son, Jae-Gi;Jung, Hye-Dong
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.05a
    • /
    • pp.148-150
    • /
    • 2013
  • 최근 TCP/IP에서 세션을 통하여 노드들 간의 통신을 연결하는 방식에서 현재는 하나의 채널을 통해 고속의 I/O가 가능하도록 하는 인피니밴드 같은 기술이 많이 연구되고 있다. 인피니밴드는 프로세싱 노드와 입출력 장치 사이의 통신, 프로세스간 통신에 대한 산업 표준이 되고 있고 프로세싱 노드와 입출력 장치를 연결하기 위해 스위치 기반의 상호 연결은 전통적인 버스 입출력을 대체하는 새로운 입출력 방식이다. 또한 인피니밴드에서는 현재 이슈가 되고 있는 RDMA 방식을 이용해 원격지 서버들간에 직접 메모리 접근 방식을 통해 CPU와 OS의 로드를 최소화하고 있다. 본 논문에서는 인피니밴드 네트워크를 이용하는 저장장치 접근 프로토콜인 iSER(iSCSI Extension RDMA Protocol)와 기존 이더넷망에서 사용되는 iSCSI(Internet SCSI) 프로토콜을 이용하여 서버와 저장장치 간의 IOPS 와 초당 데이터 전송량에 대한 성능을 평가한다. 우리는 성능평가를 위해 Intel에서 제공하는 저장장치 I/O 성능평가 도구인 IO meter를 이용했다.