• Title/Summary/Keyword: Distributed Computing

Search Result 1,279, Processing Time 0.029 seconds

Design of Distributed Processing Framework Based on H-RTGL One-class Classifier for Big Data (빅데이터를 위한 H-RTGL 기반 단일 분류기 분산 처리 프레임워크 설계)

  • Kim, Do Gyun;Choi, Jin Young
    • Journal of Korean Society for Quality Management
    • /
    • v.48 no.4
    • /
    • pp.553-566
    • /
    • 2020
  • Purpose: The purpose of this study was to design a framework for generating one-class classification algorithm based on Hyper-Rectangle(H-RTGL) in a distributed environment connected by network. Methods: At first, we devised one-class classifier based on H-RTGL which can be performed by distributed computing nodes considering model and data parallelism. Then, we also designed facilitating components for execution of distributed processing. In the end, we validate both effectiveness and efficiency of the classifier obtained from the proposed framework by a numerical experiment using data set obtained from UCI machine learning repository. Results: We designed distributed processing framework capable of one-class classification based on H-RTGL in distributed environment consisting of physically separated computing nodes. It includes components for implementation of model and data parallelism, which enables distributed generation of classifier. From a numerical experiment, we could observe that there was no significant change of classification performance assessed by statistical test and elapsed time was reduced due to application of distributed processing in dataset with considerable size. Conclusion: Based on such result, we can conclude that application of distributed processing for generating classifier can preserve classification performance and it can improve the efficiency of classification algorithms. In addition, we suggested an idea for future research directions of this paper as well as limitation of our work.

A Distributed Real-time Self-Diagnosis System for Processing Large Amounts of Log Data (대용량 로그 데이터 처리를 위한 분산 실시간 자가 진단 시스템)

  • Son, Siwoon;Kim, Dasol;Moon, Yang-Sae;Choi, Hyung-Jin
    • Database Research
    • /
    • v.34 no.3
    • /
    • pp.58-68
    • /
    • 2018
  • Distributed computing helps to efficiently store and process large data on a cluster of multiple machines. The performance of distributed computing is greatly influenced depending on the state of the servers constituting the distributed system. In this paper, we propose a self-diagnosis system that collects log data in a distributed system, detects anomalies and visualizes the results in real time. First, we divide the self-diagnosis process into five stages: collecting, delivering, analyzing, storing, and visualizing stages. Next, we design a real-time self-diagnosis system that meets the goals of real-time, scalability, and high availability. The proposed system is based on Apache Flume, Apache Kafka, and Apache Storm, which are representative real-time distributed techniques. In addition, we use simple but effective moving average and 3-sigma based anomaly detection technique to minimize the delay of log data processing during the self-diagnosis process. Through the results of this paper, we can construct a distributed real-time self-diagnosis solution that can diagnose server status in real time in a complicated distributed system.

Implementation of Distributed Computing Environment using Java Agent (자바 에이젼트를 이용한 분산컴퓨팅 환경 구현)

  • 서건원;이길흥
    • Journal of the Korea Computer Industry Society
    • /
    • v.3 no.2
    • /
    • pp.199-208
    • /
    • 2002
  • Because of the change of computing environment, an agent technology is spotlighted recently. By deploying agent distributedly in network and offering the necessary service quickly in place easing realization of service, the enlargement of the effectiveness of network is necessary more and more. In this paper, agents programmed in java are distributed in place easing realization of service in network. And a manager connects to agents and informs URL of service code of the agent. This paper implements distributed computing environment in which agent downloads service code from URL of service code, executes the code, and returns the result of execution to manager.

  • PDF

Distributed Optimal Path Generation Based on Delayed Routing in Smart Camera Networks

  • Zhang, Yaying;Lu, Wangyan;Sun, Yuanhui
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.7
    • /
    • pp.3100-3116
    • /
    • 2016
  • With the rapid development of urban traffic system and fast increasing of vehicle numbers, the traditional centralized ways to generate the source-destination shortest path in terms of travel time(the optimal path) encounter several problems, such as high server pressure, low query efficiency, roads state without in-time updating. With the widespread use of smart cameras in the urban traffic and surveillance system, this paper maps the optimal path finding problem in the dynamic road network to the shortest routing problem in the smart camera networks. The proposed distributed optimal path generation algorithm employs the delay routing and caching mechanism. Real-time route update is also presented to adapt to the dynamic road network. The test result shows that this algorithm has advantages in both query time and query packet numbers.

Infrastructure of Grid-based Distributed Remotely Sensed Images Processing Environment and its Parallel Intelligence Algorithms

  • ZHENG, Jiang;LUO, Jian-Cheng;Hu, Cheng;CHEN, Qiu-Xiao
    • Proceedings of the KSRS Conference
    • /
    • 2003.11a
    • /
    • pp.1284-1286
    • /
    • 2003
  • There is a growing demand on remotely sensed and GIS data services in modern society. However, conventional WEB applications based on client/server pattern can not meet the criteria in the future . Grid computing provides a promising resolution for establishing spatial information system toward future applications. Here, a new architecture of the distributed environment for remotely sensed data processing based on the middleware technology was proposed. In addition, in order to utilize the new environment, a problem had to be algorithmically expressed as comprising a set of concurrently executing sub-problems or tasks. Experiment of the algorithm was implemented, and the results show that the new environmental can achieve high speedups for applications compared with conventional implementation.

  • PDF

Design of the Power Management Middleware for Embedded System (소형 임베디드 시스템의 전력관리를 위한 미들웨어 설계)

  • Hwang, Sun-Yeop;Park, Choong-Bum;Choi, Hoon
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2006.10a
    • /
    • pp.326-330
    • /
    • 2006
  • 배터리를 사용하는 소형 임베디드 시스템은 효과적인 전력관리가 필요하다. 그러나 현재까지 개발된 전력관리 기법들은 대부분 운영체제와 하드웨어를 대상으로 하고 있어 개발 플랫폼에 종속적인 특징을 갖는다. 따라서 시스템을 개발할 때마다 지원하는 전력관리 기법을 하드웨어나 운영체제에 따라 여러 번 개발을 해야 하는 단점이 있다. 본 논문에서는 하위 계층에서 지원하는 전력관리 기법에 관계없이 동일한 기능을 제공하는 웨어러블 컴퓨터용 전력관리 미들웨어를 설계하였다.

  • PDF

Operational Modeling of Modern Disk Drives for Improving Disk I/O Performance (최근의 디스크 드라이브의 내부 동작 모델링을 통한 컴퓨터 I/O 시스템의 성능 향상 모색)

  • Shin, Dong-In;Yu, Young-Jin;Yeom, Heon-Young
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2007.11a
    • /
    • pp.531-534
    • /
    • 2007
  • 우리는 이 논문에서 현대 마그네틱 디스크 드라이브를 대상으로 하여 내부의 세부적인 동작 메커니즘을 상위 레벨의 검침 프로그램을 이용하여 정확하게 측정하고 이를 통해서 디스크 드라이브를 사용하는 다양한 응용 프로그램 및 운영 체제 시스템의 디스크 입/출력 성능을 개선하고자 한다.

An Empirical Inspection of Modern Disk Drive Internals (실험적인 방법으로 현대 디스크의 내부구조 추측)

  • Yu, Young-Jin;Shin, Dong-In;Yeom, Heon-Young
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2007.11a
    • /
    • pp.535-537
    • /
    • 2007
  • 디스크는 내부의 정보를 최대한 숨기고 추상화하여 운영체제에 읽기와 쓰기같은 최소한의 인터페이스만을 제공한다. 결과적으로 상위 레이어의 소프트웨어는 디바이스에 대해 최소한의 가정만을 가지고 결정에 임할 수밖에 없으며 이는 여러가지 최적화에 걸림돌이 될 수 밖에 없다. 본 논문에서는 디스크가 제공하는 최소한의 인터페이스만을 가지고 내부 구조를 정확히 추측해 내는 기법을 소개한다. 기존에 SCSI 디스크에 대해 매핑 정보를 추출해내는 연구[1,2] 이미 존재했으나, 널리 사용되고 있는 ATA 디스크의 경우 이를 밝혀낸 논문은 알려진 바 없다. 이 논문에서는 ATA 뿐만 아니라 SCSI 디스크에서도 적용할 수 있는 더 빠르고 정확한 알고리즘을 제안하고, 실제 실험 결과를 제시하였다. 이러한 결과는 차후에 입출력 시스템을 최적화하는데 큰 도움을 줄 수 있을 것이라 여겨진다.

Design and Implementation of a Distributed Data Mining Framework (분산된 데이터마이닝을 위한 프레임워크의 설계 및 구현)

  • Kadel, Prakash;Choi, Ho-Jin
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2007.06c
    • /
    • pp.336-340
    • /
    • 2007
  • We envisage that grid computing environments allow us to implement distributed data mining services, that is, those applications which analyze large sets of geographically distributed databases and information using the computational power and resources of a grid environment. This paper describes an experimental framework towards such a distributed data mining approach, including design considerations and a prototype implementation. Based on the "Knowledge Grid" architecture suggested by Cannataro et al., we identify four major components - user node, broker node, data node, and computation node - and define their individual roles. For implementing the prototype, we have investigated methods for utilizing distributed resources within a grid computing environment, e.g., communication and coordination among the various resources available.

  • PDF

Distributed Recommendation System Using Clustering-based Collaborative Filtering Algorithm (클러스터링 기반 협업 필터링 알고리즘을 사용한 분산 추천 시스템)

  • Jo, Hyun-Je;Rhee, Phill-Kyu
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.14 no.1
    • /
    • pp.101-107
    • /
    • 2014
  • This paper presents an efficient distributed recommendation system using clustering collaborative filtering algorithm in distributed computing environments. The system was built based on Hadoop distributed computing platform, where distributed Min-hash clustering algorithm is combined with user based collaborative filtering algorithm to optimize recommendation performance. Experiments using Movie Lens benchmark data show that the proposed system can reduce the execution time for recommendation compare to sequential system.