• Title/Summary/Keyword: Distributed Processing Platform

Search Result 130, Processing Time 0.027 seconds

Interaction Contortion a Distributed Multiagent System (분산 멀티에이전트 시스템의 상호협력 제어)

  • Baek, Sun-Cheol;Choe, Jung-Min;Im, Yeong-Hwan;Jang, Myeong-Uk;Park, Sang-Gyu;Lee, Gwang-Ro
    • The Transactions of the Korea Information Processing Society
    • /
    • v.3 no.7
    • /
    • pp.1803-1811
    • /
    • 1996
  • In a distributed multiagent framework, the capabilities of each agent are known to other agents. Namely, each agent in a multiagent society is aware of what agents are available in the whole society, which is able to solve a query, and how to contact them. This characteristic leads to the simplicity in controling both local and remote interactions among agents by using a fixed form for communication packes. This paper presents methods for controlling interactions among agents in this distributed multiagent frame-work. Agent interactions are described within the platform of MASCOT that is a tightly coupled multiagent system developed for the role of a computer secretary. A frame-like form of a commumication packet is defined, and protocols for message exchanges are presented. Also, ascenarios given to demonstrate how the communication mechanism controls agent intercations in MASCOT.

  • PDF

Analysis of the Influence Factors of Data Loading Performance Using Apache Sqoop (아파치 스쿱을 사용한 하둡의 데이터 적재 성능 영향 요인 분석)

  • Chen, Liu;Ko, Junghyun;Yeo, Jeongmo
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.4 no.2
    • /
    • pp.77-82
    • /
    • 2015
  • Big Data technology has been attracted much attention in aspect of fast data processing. Research of practicing Big Data technology is also ongoing to process large-scale structured data much faster in Relatioinal Database(RDB). Although there are lots of studies about measuring analyzing performance, studies about structured data loading performance, prior step of analyzing, is very rare. Thus, in this study, structured data in RDB is tested the performance that loads distributed processing platform Hadoop using Apache sqoop. Also in order to analyze the influence factors of data loading, it is tested repeatedly with different options of data loading and compared with data loading performance among RDB based servers. Although data loading performance of Apache Sqoop in test environment was low, but in large-scale Hadoop cluster environment we can expect much better performance because of getting more hardware resources. It is expected to be based on study improving data loading performance and whole steps of performance analyzing structured data in Hadoop Platform.

Energy Big Data Pre-processing System for Energy New Industries (에너지신산업을 위한 에너지 빅데이터 전처리 시스템)

  • Yang, Soo-Young;Kim, Yo-Han;Kim, Sang-Hyun;Kim, Won-Jung
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.16 no.5
    • /
    • pp.851-858
    • /
    • 2021
  • Due to the increase in renewable energy and distributed resources, not only traditional data but also various energy-related data are being generated in the new energy industry. In other words, there are various renewable energy facilities and power generation data, system operation data, metering and rate-related data, as well as weather and energy efficiency data necessary for new services and analysis. Energy big data processing technology can systematically analyze and diagnose data generated in the first half of the power production and consumption infrastructure, including distributed resources, systems, and AMI. Through this, it will be a technology that supports the creation of new businesses in convergence between the ICT industry and the energy industry. To this end, research on the data analysis system, such as itemized characteristic analysis of the collected data, correlation sampling, categorization of each feature, and element definition, is needed. In addition, research on data purification technology for data loss and abnormal state processing should be conducted. In addition, it is necessary to develop and structure NIFI, Spark, and HDFS systems so that energy data can be stored and managed in real time. In this study, the overall energy data processing technology and system for various power transactions as described above were proposed.

Development of Information Technology Infrastructures through Construction of Big Data Platform for Road Driving Environment Analysis (도로 주행환경 분석을 위한 빅데이터 플랫폼 구축 정보기술 인프라 개발)

  • Jung, In-taek;Chong, Kyu-soo
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.3
    • /
    • pp.669-678
    • /
    • 2018
  • This study developed information technology infrastructures for building a driving environment analysis platform using various big data, such as vehicle sensing data, public data, etc. First, a small platform server with a parallel structure for big data distribution processing was developed with H/W technology. Next, programs for big data collection/storage, processing/analysis, and information visualization were developed with S/W technology. The collection S/W was developed as a collection interface using Kafka, Flume, and Sqoop. The storage S/W was developed to be divided into a Hadoop distributed file system and Cassandra DB according to the utilization of data. Processing S/W was developed for spatial unit matching and time interval interpolation/aggregation of the collected data by applying the grid index method. An analysis S/W was developed as an analytical tool based on the Zeppelin notebook for the application and evaluation of a development algorithm. Finally, Information Visualization S/W was developed as a Web GIS engine program for providing various driving environment information and visualization. As a result of the performance evaluation, the number of executors, the optimal memory capacity, and number of cores for the development server were derived, and the computation performance was superior to that of the other cloud computing.

Design and Implementation of Real-Time Information Appliance Controlling Simulator Based on TMO Scheme (TMO 스킴 기반의 실시간 정보가전 제어 시뮬레이터의 설계 및 구현)

  • Shin Chang-Sun;Kim Woon-Mi;Ryu Eun-Soon;Joo Su-Chong
    • The KIPS Transactions:PartD
    • /
    • v.12D no.2 s.98
    • /
    • pp.319-326
    • /
    • 2005
  • Recently, the information appliance devices integrating information technology, appliance technology, and communication technology are appeared in home. And this environment requires that the monitoring technology should include the interaction and the real-time controlling among these devices using home network. In this paper, we implemented the active objects for information appliances by applying the Time-triggered Message-triggered Object(TMO) scheme supporting real-time service. Based on distributed real-time services supporting platform using the TMO Support Middleware(TMOSM), we also constructed the real-time information appliance controlling simulator supporting the remote controlling and monitoring service among active objects. The information appliance devices implemented by the TMO scheme can exchange the housing information by autonomous triggering. And we can conveniently reconfigure the executing environment for information appliances when new devices were inserted. For constructing the simulator on the distributed real-time service supporting platform, we described the functions of the active TMOs for information appliances that execute the temperature management, the illuminance management, and the time management controlling services, and designed the remote interaction among them. Finally, by implementing the simulating environment of information appliance devices on distributed platform, we showed whether the procedures of the real-time controlling and the monitoring service for the active objects of information appliances corresponding to physical devices are processed in given executing conditions correctly.

Research on a New Approach to Enhance IoT Security Using Blockchain Technology (블록체인 기술을 이용하여 IoT 보안 강화를 위한 새로운 접근방법 연구)

  • Hong, Sunghyuck
    • Journal of Digital Convergence
    • /
    • v.17 no.12
    • /
    • pp.235-241
    • /
    • 2019
  • The structure of the IoT can be divided into devices, gateways, and servers. First, the gateway collects data from the device, and the gateway sends data to the server through HTTP protocol, Websocket protocol, and MQTT protocol. The processing server then processes, analyzes, and transforms the data, and the database makes it easy to store and use this data. These IoT services are basically centralized structures with servers, so attacks on the entire platform are concentrated only on the central server, which makes hacking more successful than distributed structures. One way to solve this problem is to develop IoT that combines blockchain. Therefore, the proposed research suggests that the blockchain is a distributed structure, in which blocks containing small data are connected in a chain form, so that each node agrees and verifies the data with each other, thereby increasing reliability and lowering the probability of data forgery.

Design of Distributed Hadoop Full Stack Platform for Big Data Collection and Processing (빅데이터 수집 처리를 위한 분산 하둡 풀스택 플랫폼의 설계)

  • Lee, Myeong-Ho
    • Journal of the Korea Convergence Society
    • /
    • v.12 no.7
    • /
    • pp.45-51
    • /
    • 2021
  • In accordance with the rapid non-face-to-face environment and mobile first strategy, the explosive increase and creation of many structured/unstructured data every year demands new decision making and services using big data in all fields. However, there have been few reference cases of using the Hadoop Ecosystem, which uses the rapidly increasing big data every year to collect and load big data into a standard platform that can be applied in a practical environment, and then store and process well-established big data in a relational database. Therefore, in this study, after collecting unstructured data searched by keywords from social network services based on Hadoop 2.0 through three virtual machine servers in the Spring Framework environment, the collected unstructured data is loaded into Hadoop Distributed File System and HBase based on the loaded unstructured data, it was designed and implemented to store standardized big data in a relational database using a morpheme analyzer. In the future, research on clustering and classification and analysis using machine learning using Hive or Mahout for deep data analysis should be continued.

Design and Implementation of HDFS Data Encryption Scheme Using ARIA Algorithms on Hadoop (하둡 상에서 ARIA 알고리즘을 이용한 HDFS 데이터 암호화 기법의 설계 및 구현)

  • Song, Youngho;Shin, YoungSung;Chang, Jae-Woo
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.5 no.2
    • /
    • pp.33-40
    • /
    • 2016
  • Due to the growth of social network systems (SNS), big data are realized and Hadoop was developed as a distributed platform for analyzing big data. Enterprises analyze data containing users' sensitive information by using Hadoop and utilize them for marketing. Therefore, researches on data encryption have been done to protect the leakage of sensitive data stored in Hadoop. However, the existing researches support only the AES encryption algorithm, the international standard of data encryption. Meanwhile, Korean government choose ARIA algorithm as a standard data encryption one. In this paper, we propose a HDFS data encryption scheme using ARIA algorithms on Hadoop. First, the proposed scheme provide a HDFS block splitting component which performs ARIA encryption and decryption under the distributed computing environment of Hadoop. Second, the proposed scheme also provide a variable-length data processing component which performs encryption and decryption by adding dummy data, in case when the last block of data does not contains 128 bit data. Finally, we show from performance analysis that our proposed scheme can be effectively used for both text string processing applications and science data analysis applications.

A Remote Cache Coherence Protocol for Single Shared Memory in Multiprocessor System (단일 공유 메모리를 가지는 다중 프로세서 시스템의 원격 캐시 일관성 유지 프로토콜)

  • Kim, Seong-Woon;Kim, Bo-Gwan
    • Journal of the Institute of Electronics Engineers of Korea CI
    • /
    • v.42 no.6
    • /
    • pp.19-28
    • /
    • 2005
  • The multiprocessor architecture is a good method to improve the computer system performance. The CC-NUMA provides a single shared space with the physically distributed memories is used widely in the multiprocessor computer system. A CC-NUMA has the full-mapped directory for the shared memory md uses a remote cache memory for tile fast memory access. In this paper, we propose a processing node architecture for a CC-NUMA system and a cache coherency protocol on the physically distributed but logically shared system. We show an implementation result of the system which is adopted the cache coherency protocol.

Distributed Edge Computing for DNA-Based Intelligent Services and Applications: A Review (딥러닝을 사용하는 IoT빅데이터 인프라에 필요한 DNA 기술을 위한 분산 엣지 컴퓨팅기술 리뷰)

  • Alemayehu, Temesgen Seyoum;Cho, We-Duke
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.12
    • /
    • pp.291-306
    • /
    • 2020
  • Nowadays, Data-Network-AI (DNA)-based intelligent services and applications have become a reality to provide a new dimension of services that improve the quality of life and productivity of businesses. Artificial intelligence (AI) can enhance the value of IoT data (data collected by IoT devices). The internet of things (IoT) promotes the learning and intelligence capability of AI. To extract insights from massive volume IoT data in real-time using deep learning, processing capability needs to happen in the IoT end devices where data is generated. However, deep learning requires a significant number of computational resources that may not be available at the IoT end devices. Such problems have been addressed by transporting bulks of data from the IoT end devices to the cloud datacenters for processing. But transferring IoT big data to the cloud incurs prohibitively high transmission delay and privacy issues which are a major concern. Edge computing, where distributed computing nodes are placed close to the IoT end devices, is a viable solution to meet the high computation and low-latency requirements and to preserve the privacy of users. This paper provides a comprehensive review of the current state of leveraging deep learning within edge computing to unleash the potential of IoT big data generated from IoT end devices. We believe that the revision will have a contribution to the development of DNA-based intelligent services and applications. It describes the different distributed training and inference architectures of deep learning models across multiple nodes of the edge computing platform. It also provides the different privacy-preserving approaches of deep learning on the edge computing environment and the various application domains where deep learning on the network edge can be useful. Finally, it discusses open issues and challenges leveraging deep learning within edge computing.