• Title/Summary/Keyword: Data-intensive processing

Search Result 131, Processing Time 0.031 seconds

Scalable Data Provisioning Scheme on Large-Scale Distributed Computing Environment (대규모 분산 컴퓨팅 환경에서 확장성을 고려한 실시간 데이터 공급 기법)

  • Kim, Byungs-Sang;Youn, Chan-Hyun
    • The KIPS Transactions:PartA
    • /
    • v.18A no.4
    • /
    • pp.123-128
    • /
    • 2011
  • As the global grid has grown in size, large-scale distributed data analysis schemes have gained momentum. Over the last few years, a number of methods have been introduced for allocating data intensive tasks across distributed and heterogeneous computing platforms. However, these approaches have a limited potential for scaling up computing nodes so that they can serve more tasks simultaneously. This paper tackles the scalability and communication delay for computing nodes. We propose a distributed data node for storing and allocating the data. This paper also provides data provisioning method based on the steady states for minimizing the communication delay between the data source and the computing nodes. The experimental results show that scalability and communication delay can be achieved in our system.

The Performance Potential of Data Dependent Computation on Asynchronous Superscalar Processor

  • Kim, Suk-Jin;Park, Byung-Soo;Park, Chan-Ho;Lee, Dong-Ik
    • Proceedings of the IEEK Conference
    • /
    • 2000.07a
    • /
    • pp.414-416
    • /
    • 2000
  • We investigate potential advantages and problems when a superscalar processor is designed and implemented using asynchronous design methods. Conventional techniques of superscalar processing are applied and data dependent adder is considered as an asynchronous component. Intensive simulations on SPEC INT95 benchmark suites are made for the purpose of performance comparison between a synchronous and an asynchronous superscalar processor, respectively. The simulation results show about 5% speedup with asynchronous design methods in the sense of Issue Rate.

  • PDF

Causality join query processing for data stream by spatio-temporal sliding window (시공간 슬라이딩윈도우기법을 이용한 데이터스트림의 인과관계 결합질의처리방법)

  • Kwon, O-Je;Li, Ki-Joune
    • Spatial Information Research
    • /
    • v.16 no.2
    • /
    • pp.219-236
    • /
    • 2008
  • Data stream collected from sensors contain a large amount of useful information including causality relationships. The causality join query for data stream is to retrieve a set of pairs (cause, effect) from streams of data. A part of causality pairs may however be lost from the query result, due to the delay from sensors to a data stream management system, and the limited size of sliding windows. In this paper, we first investigate spatial, temporal, and spatio-temporal aspects of the causality join query for data stream. Second, we propose several strategies for sliding window management based on these observations. The accuracy of the proposed strategies is studied by intensive experiments, and the result shows that we improve the accuracy of causality join query in data stream from simple FIFO strategy.

  • PDF

Low-grade waste heat recovery and repurposing to reduce the load on cooling towers

  • McLean, Shannon H.;Chenier, Jeff;Muinonen, Sari;Laamanen, Corey A.;Scott, John A.
    • Advances in Energy Research
    • /
    • v.7 no.2
    • /
    • pp.147-166
    • /
    • 2020
  • Industrial cooling towers are often ageing infrastructure that is expensive to maintain and operate. A novel approach is introduced in which a heat pump circuit is incorporated to reduce the load upon the towers by extracting low-grade energy from the stream sent to the towers and repurposing in on-site processing operations. To demonstrate the concept, a model was constructed, which uses industrial data on cooling towers linked to a smelter's sulphuric acid plant, to allow direct economic and environmental impact comparison between different heat recovery and repurposing scenarios. The model's results showed that implementing a heat pump system would significantly decrease annual operating costs and achieve a payback period of 3 years. In addition, overall CO2 emissions could be reduced by 42% (430,000 kg/year) and a 5% heat load reduction on the cooling towers achieved. The concept is significant as the outcomes introduce a new way for energy intensive industrial sectors, such as mineral processing, to reduce energy consumption and improve long-term sustainable performance.

Initial Authentication Protocol of Hadoop Distribution System based on Elliptic Curve (타원곡선기반 하둡 분산 시스템의 초기 인증 프로토콜)

  • Jeong, Yoon-Su;Kim, Yong-Tae;Park, Gil-Cheol
    • Journal of Digital Convergence
    • /
    • v.12 no.10
    • /
    • pp.253-258
    • /
    • 2014
  • Recently, the development of cloud computing technology is developed as soon as smartphones is increases, and increased that users want to receive big data service. Hadoop framework of the big data service is provided to hadoop file system and hadoop mapreduce supported by data-intensive distributed applications. But, smpartphone service using hadoop system is a very vulnerable state to data authentication. In this paper, we propose a initial authentication protocol of hadoop system assisted by smartphone service. Proposed protocol is combine symmetric key cryptography techniques with ECC algorithm in order to support the secure multiple data processing systems. In particular, the proposed protocol to access the system by the user Hadoop when processing data, the initial authentication key and the symmetric key instead of the elliptic curve by using the public key-based security is improved.

A Fast SAD Algorithm for Area-based Stereo Matching Methods (영역기반 스테레오 영상 정합을 위한 고속 SAD 알고리즘)

  • Lee, Woo-Young;Kim, Cheong Ghil
    • Journal of Satellite, Information and Communications
    • /
    • v.7 no.2
    • /
    • pp.8-12
    • /
    • 2012
  • Area-based stereo matchng algorithms are widely used for image analysis for stereo vision. SAD (Sum of Absolute Difference) algorithm is one of well known area-based stereo matchng algorithms with the characteristics of data intensive computing application. Therefore, it requires very high computation capabilities and its processing speed becomes very slow with software realization. This paper proposes a fast SAD algorithm utilizing SSE (Streaming SIMD Extensions) instructions based on SIMD (Single Instruction Multiple Data) parallism. CPU supporing SSE instructions has 16 XMM registers with 128 bits. For the performance evaluation of the proposed scheme, we compare the processing speed between SAD with/without SSE instructions. The proposed scheme achieves four times performance improvement over the general SAD, which shows the possibility of the software realization of real time SAD algorithm.

An Efficient ROLAP Cube Generation Scheme (효율적인 ROLAP 큐브 생성 방법)

  • Kim, Myung;Song, Ji-Sook
    • Journal of KIISE:Databases
    • /
    • v.29 no.2
    • /
    • pp.99-109
    • /
    • 2002
  • ROLAP(Relational Online Analytical Processing) is a process and methodology for a multidimensional data analysis that is essential to extract desired data and to derive value-added information from an enterprise data warehouse. In order to speed up query processing, most ROLAP systems pre-compute summary tables. This process is called 'cube generation' and it mostly involves intensive table sorting stages. (1) showed that it is much faster to generate ROLAP summary tables indirectly using a MOLAP(multidimensional OLAP) cube generation algorithm. In this paper, we present such an indirect ROLAP cube generation algorithm that is fast and scalable. High memory utilization is achieved by slicing the input fact table along one or more dimensions before generating summary tables. High speed is achieved by producing summary tables from their smallest parents. We showed the efficiency of our algorithm through experiments.

Intelligent Character Recognition System for Account Payable by using SVM and RBF Kernel

  • Farooq, Muhammad Umer;Kazi, Abdul Karim;Latif, Mustafa;Alauddin, Shoaib;Kisa-e-Zehra, Kisa-e-Zehra;Baig, Mirza Adnan
    • International Journal of Computer Science & Network Security
    • /
    • v.22 no.11
    • /
    • pp.213-221
    • /
    • 2022
  • Intelligent Character Recognition System for Account Payable (ICRS AP) Automation represents the process of capturing text from scanned invoices and extracting the key fields from invoices and storing the captured fields into properly structured document format. ICRS plays a very critical role in invoice data streamlining, we are interested in data like Vendor Name, Purchase Order Number, Due Date, Total Amount, Payee Name, etc. As companies attempt to cut costs and upgrade their processes, accounts payable (A/P) is an example of a paper-intensive procedure. Invoice processing is a possible candidate for digitization. Most of the companies dealing with an enormous number of invoices, these manual invoice matching procedures start to show their limitations. Receiving a paper invoice and matching it to a purchase order (PO) and general ledger (GL) code can be difficult for businesses. Lack of automation leads to more serious company issues such as accruals for financial close, excessive labor costs, and a lack of insight into corporate expenditures. The proposed system offers tighter control on their invoice processing to make a better and more appropriate decision. AP automation solutions provide tighter controls, quicker clearances, smart payments, and real-time access to transactional data, allowing financial managers to make better and wiser decisions for the bottom line of their organizations. An Intelligent Character Recognition System for AP Automation is a process of extricating fields like Vendor Name, Purchase Order Number, Due Date, Total Amount, Payee Name, etc. based on their x-axis and y-axis position coordinates.

Conjugate Point Extraction for High-Resolution Stereo Satellite Images Orientation

  • Oh, Jae Hong;Lee, Chang No
    • Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
    • /
    • v.37 no.2
    • /
    • pp.55-62
    • /
    • 2019
  • The stereo geometry establishment based on the precise sensor modeling is prerequisite for accurate stereo data processing. Ground control points are generally required for the accurate sensor modeling though it is not possible over the area where the accessibility is limited or reference data is not available. For the areas, the relative orientation should be carried out to improve the geometric consistency between the stereo data though it does not improve the absolute positional accuracy. The relative orientation requires conjugate points that are well distributed over the entire image region. Therefore the automatic conjugate point extraction is required because the manual operation is labor-intensive. In this study, we applied the method consisting of the key point extraction, the search space minimization based on the epipolar line, and the rigorous outlier detection based on the RPCs (Rational Polynomial Coefficients) bias compensation modeling. We tested different parameters of window sizes for Kompsat-2 across track stereo data and analyzed the RPCs precision after the bias compensation for the cases whether the epipolar line information is used or not. The experimental results showed that matching outliers were inevitable for the different matching parameterization but they were successfully detected and removed with the rigorous method for sub-pixel level of stereo RPCs precision.

Developing a Dynamic Materialized View Index for Efficiently Discovering Usable Views for Progressive Queries

  • Zhu, Chao;Zhu, Qiang;Zuzarte, Calisto;Ma, Wenbin
    • Journal of Information Processing Systems
    • /
    • v.9 no.4
    • /
    • pp.511-537
    • /
    • 2013
  • Numerous data intensive applications demand the efficient processing of a new type of query, which is called a progressive query (PQ). A PQ consists of a set of unpredictable but inter-related step-queries (SQ) that are specified by its user in a sequence of steps. A conventional DBMS was not designed to efficiently process such PQs. In our earlier work, we introduced a materialized view based approach for efficiently processing PQs, where the focus was on selecting promising views for materialization. The problem of how to efficiently find usable views from the materialized set in order to answer the SQs for a PQ remains open. In this paper, we present a new index technique, called the Dynamic Materialized View Index (DMVI), to rapidly discover usable views for answering a given SQ. The structure of the proposed index is a special ordered tree where the SQ domain tables are used as search keys and some bitmaps are kept at the leaf nodes for refined filtering. A two-level priority rule is adopted to order domain tables in the tree, which facilitates the efficient maintenance of the tree by taking into account the dynamic characteristics of various types of materialized views for PQs. The bitmap encoding methods and the strategies/algorithms to construct, search, and maintain the DMVI are suggested. The extensive experimental results demonstrate that our index technique is quite promising in improving the performance of the materialized view based query processing approach for PQs.