• Title/Summary/Keyword: Distribute Processing

Search Result 152, Processing Time 0.03 seconds

Dynamic Load Management Method for Spatial Data Stream Processing on MapReduce Online Frameworks (맵리듀스 온라인 프레임워크에서 공간 데이터 스트림 처리를 위한 동적 부하 관리 기법)

  • Jeong, Weonil
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.8
    • /
    • pp.535-544
    • /
    • 2018
  • As the spread of mobile devices equipped with various sensors and high-quality wireless network communications functionsexpands, the amount of spatio-temporal data generated from mobile devices in various service fields is rapidly increasing. In conventional research into processing a large amount of real-time spatio-temporal streams, it is very difficult to apply a Hadoop-based spatial big data system, designed to be a batch processing platform, to a real-time service for spatio-temporal data streams. This paper extends the MapReduce online framework to support real-time query processing for continuous-input, spatio-temporal data streams, and proposes a load management method to distribute overloads for efficient query processing. The proposed scheme shows a dynamic load balancing method for the nodes based on the inflow rate and the load factor of the input data based on the space partition. Experiments show that it is possible to support efficient query processing by distributing the spatial data stream in the corresponding area to the shared resources when load management in a specific area is required.

Study on the distribution law and influencing factors of pressure field distribution before exploitation in heavy oilfield

  • Zhang, Xing;Jiang, Ting T.;Zhang, Jian H.;Li, Bo;Li, Yu B.;Zhang, Chun Y.;Xu, Bing B.;Qi, Peng
    • Geomechanics and Engineering
    • /
    • v.18 no.2
    • /
    • pp.205-213
    • /
    • 2019
  • A calculation model of reservoir pressure field distribution around multiple production wells in a heavy oil reservoir is established, which can overcome the unreasonable uniform-pressure value calculated by the traditional mathematical model in the multiwell mining areas. A calculating program is developed based on the deduced equations by using Visual Basic computer language. Based on the proposed mathematical model, the effects of drainage rate and formation permeability on the distribution of reservoir pressure are studied. Results show that the reservoir pressure drops most at the wellbore. The farther the distance away from the borehole, the sparser the isobaric lines distribute. Increasing drainage rate results in decreasing reservoir pressure and bottom-hole pressure, especially the latter. The permeability has a significant effect on bottom hole pressure. The study provides a reference basis for studying the dynamic pressure field distribution before thermal recovery technology in heavy oilfield and optimizing construction parameters.

Load Balancing for Distributed Processing of Real-time Spatial Big Data Stream (실시간 공간 빅데이터 스트림 분산 처리를 위한 부하 균형화 방법)

  • Yoon, Susik;Lee, Jae-Gil
    • Journal of KIISE
    • /
    • v.44 no.11
    • /
    • pp.1209-1218
    • /
    • 2017
  • A variety of sensors is widely used these days, and it has become much easier to acquire spatial big data streams from various sources. Since spatial data streams have inherently skewed and dynamically changing distributions, the system must effectively distribute the load among workers. Previous studies to solve this load imbalance problem are not directly applicable to processing spatial data. In this research, we propose Adaptive Spatial Key Grouping (ASKG). The main idea of ASKG is, by utilizing the previous distribution of the data streams, to adaptively suggest a new grouping scheme that evenly distributes the future load among workers. We evaluate the validity of the proposed algorithm in various environments, by conducting an experiment with real datasets while varying the number of workers, input rate, and processing overhead. Compared to two other alternative algorithms, ASKG improves the system performance in terms of load imbalance, throughput, and latency.

The File Splitting Distribution Scheme Using the P2P Networks with The Mesh topology (그물망 위상의 P2P 네트워크를 활용한 파일 분리 분산 방안)

  • Lee Myoung-Hoon;Park Jung-Su;Kim Jin-Hong;Jo In-June
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.9 no.8
    • /
    • pp.1669-1675
    • /
    • 2005
  • Recently, the small sized wireless terminals have problems of processing of large sized file because of the trends of a small sized terminals and a large sized files. Moreover, the web servers or the file servers have problems of the overload because of the concentration with many number of files to the them. Also, There is a security vulnerability of the data processing caused by the processing with a unit of the independent file. To resolve the problems, this paper proposes a new scheme of fat splining distribution using the P2P networks with the mesh topology. The proposed scheme is to distribute blocks of file into any peer of P2P networks. It can do that the small sized wireless terminals can process the large size file, the overload problems of a web or file servers can solve because of the decentralized files, and, the security vulnerability of the data processing is mitigated because of the distributed processing with a unit of the blocks to the peers.

A Performance Analysis of IP Multicasting using Resource Reservation over ATM Networks (ATM 망에서 자원 예약 방식을 이용한 IP 멀티캐스팅의 성능 분석)

  • 봉은철;김동일;최영진;양기원;오창석
    • The KIPS Transactions:PartC
    • /
    • v.8C no.1
    • /
    • pp.41-50
    • /
    • 2001
  • With growth of Internet, real time multimedia application services require guarantee of QoS (Quality of Service). However, the existing Internet based on best-effort service is not enough to support those services, so MARS model to support IP multicast over ATM networks and RSVP to provide heterogeneous QoS by resource reservation scheme are proposed. As a simulation result, it is proved that processing delay of MARS is increased and success of resource reservation is decreased as a number of QoS change request of a receiver is increased. Therefore, it is important to provide heterogeneous QoS that distribute a load of MARS caused by convergence of messages and efficiently manage the resource of network as to find out scale of receivers and frequency of QoS change request.

  • PDF

Sequential Pattern Mining with Optimization Calling MapReduce Function on MapReduce Framework (맵리듀스 프레임웍 상에서 맵리듀스 함수 호출을 최적화하는 순차 패턴 마이닝 기법)

  • Kim, Jin-Hyun;Shim, Kyu-Seok
    • The KIPS Transactions:PartD
    • /
    • v.18D no.2
    • /
    • pp.81-88
    • /
    • 2011
  • Sequential pattern mining that determines frequent patterns appearing in a given set of sequences is an important data mining problem with broad applications. For example, sequential pattern mining can find the web access patterns, customer's purchase patterns and DNA sequences related with specific disease. In this paper, we develop the sequential pattern mining algorithms using MapReduce framework. Our algorithms distribute input data to several machines and find frequent sequential patterns in parallel. With synthetic data sets, we did a comprehensive performance study with varying various parameters. Our experimental results show that linear speed up can be achieved through our algorithms with increasing the number of used machines.

Process Design of Isothermal Forging for Three-Dimensional Ti-6Al-4V Wing-Shape (Ti-6Al-4V 합금 3D 날개형상의 항온단조 공정설계)

  • Yeom J. T.;Park N. K.;Lee Y. H.;Shin T. J.;Hong S. S.;Shim I. O.;Hwang S. M.;Lee C. S.
    • Transactions of Materials Processing
    • /
    • v.14 no.2 s.74
    • /
    • pp.126-132
    • /
    • 2005
  • The isothermal forging design of a Ti-6Al-4V wing shape was performed by 3D FE simulation. The design focuses on near-net shape forming by the single stage. The process variables such as the die design, pre-form shape and size, ram speed and forging temperature were investigated. The main design priorities were to minimize forging loads and to distribute strain uniformly in a given forging condition. The FE simulation results for the final process design were compared with the isothermal forging tests. The instability of deformation was evaluated using a processing map based on the dynamic materials model(DMM), including flow stability criteria. Finally, a modified process design for producing a uniform Ti-6Al-4V wing product without forming defects was suggested.

New GPU computing algorithm for wind load uncertainty analysis on high-rise systems

  • Wei, Cui;Luca, Caracoglia
    • Wind and Structures
    • /
    • v.21 no.5
    • /
    • pp.461-487
    • /
    • 2015
  • In recent years, the Graphics Processing Unit (GPU) has become a competitive computing technology in comparison with the standard Central Processing Unit (CPU) technology due to reduced unit cost, energy and computing time. This paper describes the derivation and implementation of GPU-based algorithms for the analysis of wind loading uncertainty on high-rise systems, in line with the research field of probability-based wind engineering. The study begins by presenting an application of the GPU technology to basic linear algebra problems to demonstrate advantages and limitations. Subsequently, Monte-Carlo integration and synthetic generation of wind turbulence are examined. Finally, the GPU architecture is used for the dynamic analysis of three high-rise structural systems under uncertain wind loads. In the first example the fragility analysis of a single degree-of-freedom structure is illustrated. Since fragility analysis employs sampling-based Monte Carlo simulation, it is feasible to distribute the evaluation of different random parameters among different GPU threads and to compute the results in parallel. In the second case the fragility analysis is carried out on a continuum structure, i.e., a tall building, in which double integration is required to evaluate the generalized turbulent wind load and the dynamic response in the frequency domain. The third example examines the computation of the generalized coupled wind load and response on a tall building in both along-wind and cross-wind directions. It is concluded that the GPU can perform computational tasks on average 10 times faster than the CPU.

An Efficient Central Queue Management Algorithm for High-speed Parallel Packet Filtering (고속 병렬 패킷 여과를 위한 효율적인 단일버퍼 관리 방안)

  • 임강빈;박준구;최경희;정기현
    • Journal of the Institute of Electronics Engineers of Korea TC
    • /
    • v.41 no.7
    • /
    • pp.63-73
    • /
    • 2004
  • This paper proposes an efficient centralized sin91e buffer management algorithm to arbitrate access contention mon processors on the multi-processor system for high-speed Packet filtering and proves that the algorithm provides reasonable performance by implementing it and applying it to a real multi-processor system. The multi-processor system for parallel packet filtering is modeled based on a network processor to distribute the packet filtering rules throughout the processors to speed up the filtering. In this paper we changed the number of processors and the processing time of the filtering rules as variables and measured the packet transfer rates to investigate the performance of the proposed algorithm.

An Optimized Approach of Fault Distribution for Debugging in Parallel

  • Srivasatav, Maneesha;Singh, Yogesh;Chauhan, Durg Singh
    • Journal of Information Processing Systems
    • /
    • v.6 no.4
    • /
    • pp.537-552
    • /
    • 2010
  • Software Debugging is the most time consuming and costly process in the software development process. Many techniques have been proposed to isolate different faults in a program thereby creating separate sets of failing program statements. Debugging in parallel is a technique which proposes distribution of a single faulty program segment into many fault focused program slices to be debugged simultaneously by multiple debuggers. In this paper we propose a new technique called Faulty Slice Distribution (FSD) to make parallel debugging more efficient by measuring the time and labor associated with a slice. Using this measure we then distribute these faulty slices evenly among debuggers. For this we propose an algorithm that estimates an optimized group of faulty slices using as a parameter the priority assigned to each slice as computed by value of their complexity. This helps in the efficient merging of two or more slices for distribution among debuggers so that debugging can be performed in parallel. To validate the effectiveness of this proposed technique we explain the process using example.