• Title/Summary/Keyword: Data processing

Search Result 16,486, Processing Time 0.055 seconds

Extended Storage Management System for Spatial Data Processing (공간데이타 처리를 위한 확장된 저장관리시스템)

  • 김재홍;배해영
    • Spatial Information Research
    • /
    • v.1 no.1
    • /
    • pp.7-16
    • /
    • 1993
  • Current computer technologies developing. our requirements are changing from simple alpha-numeric processing to graphic and image. spatial data processing which are easy for user to understand and use. Geographic information system is a kind of spatial database system that can not only print out the data in the form of maps but also manipulate. store. retrieve. and analyze the geographic data. It is efficient system that can process the spatial data which has a geographical feature and its relative attribute data. Conventional relational database management systems are not suitable for spatial data processing, so we need to design the spatial database mana-gement system which is suitable for efficient spatial data processing. In this paper we design the extended storage management system that supports the spatial index technique that allows user to access fast and store and manage the enormous spatial data efficiently like geographic information system.

  • PDF

High-Volume Data Processing using Complex Event Processing Engine in the Web of Next Generation (차세대 웹 환경에서 Complex Event Processing 엔진을 이용한 대용량데이터 처리)

  • Kang, Man-Mo;Koo, Ra-Rok;Lee, Dong-Hyung
    • Journal of KIISE:Databases
    • /
    • v.37 no.6
    • /
    • pp.300-307
    • /
    • 2010
  • According to growth of web, data processing technology is developing. In the Web of next generation, high-speed or high-volume data processing technologies for various wire-wireless users, USN and RFID are developing too. In this paper, we propose a high-volume data processing technology using Complex Event Processing(CEP) engine. CEP is the technology to process complex events. CEP Engine is the following characteristics. First it collects a high-volume event(data). Secondly it analyses events. Finally it lets event connect to new actions. In other words, CEP engine collects, analyses, filters high-volume events. Also it extracts events using pattern-matching for registered events and new events. As the results extracted. We use it by an input event of other work, real-time response for demanded event and can trigger to database for only valid data.

Implement of MapReduce-based Big Data Processing Scheme for Reducing Big Data Processing Delay Time and Store Data (빅데이터 처리시간 감소와 저장 효율성이 향상을 위한 맵리듀스 기반 빅데이터 처리 기법 구현)

  • Lee, Hyeopgeon;Kim, Young-Woon;Kim, Ki-Young
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.10
    • /
    • pp.13-19
    • /
    • 2018
  • MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

Qualification Test of ROCSAT -2 Image Processing System

  • Liu, Cynthia;Lin, Po-Ting;Chen, Hong-Yu;Lee, Yong-Yao;Kao, Ricky;Wu, An-Ming
    • Proceedings of the KSRS Conference
    • /
    • 2003.11a
    • /
    • pp.1197-1199
    • /
    • 2003
  • ROCSAT-2 mission is to daily image over Taiwan and the surrounding area for disaster monitoring, land use, and ocean surveillance during the 5-year mission lifetime. The satellite will be launched in December 2003 into its mission orbit, which is selected as a 14 rev/day repetitive Sun-synchronous orbit descending over (120 deg E, 24 deg N) and 9:45 a.m. over the equator with the minimum eccentricity. National Space Program Office (NSPO) is developing a ROCSAT-2 Image Processing System (IPS), which aims to provide real-time high quality image data for ROCSAT-2 mission. A simulated ROCSAT-2 image, based on Level 1B QuickBird Data, is generated for IPS verification. The test image is comprised of one panchromatic data and four multispectral data. The qualification process consists of four procedures: (a) QuickBird image processing, (b) generation of simulated ROCSAT-2 image in Generic Raw Level Data (GERALD) format, (c) ROCSAT-2 image processing, and (d) geometric error analysis. QuickBird standard photogrammetric parameters of a camera that models the imaging and optical system is used to calculate the latitude and longitude of each line and sample. The backward (inverse model) approach is applied to find the relationship between geodetic coordinate system (latitude, longitude) and image coordinate system (line, sample). The bilinear resampling method is used to generate the test image. Ground control points are used to evaluate the error for data processing. The data processing contains various coordinate system transformations using attitude quaternion and orbit elements. Through the qualification test process, it is verified that the IPS is capable of handling high-resolution image data with the accuracy of Level 2 processing within 500 m.

  • PDF

Efficiency Low-Power Signal Processing for Multi-Channel LiDAR Sensor-Based Vehicle Detection Platform (멀티채널 LiDAR 센서 기반 차량 검출 플랫폼을 위한 효율적인 저전력 신호처리 기법)

  • Chong, Taewon;Park, Daejin
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.25 no.7
    • /
    • pp.977-985
    • /
    • 2021
  • The LiDAR sensor is attracting attention as a key sensor for autonomous driving vehicle. LiDAR sensor provides measured three-dimensional lengths within range using LASER. However, as much data is provided to the external system, it is difficult to process such data in an external system or processor of the vehicle. To resolve these issues, we develop integrated processing system for LiDAR sensor. The system is configured that client receives data from LiDAR sensor and processes data, server gathers data from clients and transmits integrated data in real-time. The test was carried out to ensure real-time processing of the system by changing the data acquisition, processing method and process driving method of process. As a result of the experiment, when receiving data from four LiDAR sensors, client and server process was operated using background or multi-core processing, the system response time of each client was about 13.2 ms and the server was about 12.6 ms.

Strategy for Establishing a Rights Processing Platform to Enhance the Utilization of Open Data (공공데이터 활용성 제고를 위한 권리처리 플랫폼 구축 전략)

  • Sim, Junbo;Kwon, Hun-yeong
    • Journal of Information Technology Services
    • /
    • v.21 no.3
    • /
    • pp.27-42
    • /
    • 2022
  • Open Data is an essential resource for the data industry. 'Act On Promotion Of The Provision And Use Of Public Data', enacted on July 30, 2013, mandates public institutions to manage the quality of Open Data and provide it to the public. Via such a legislation, the legal basis for the public to Open Data is prepared. Furthermore, public institutions are prohibited from developing and providing open data services that are duplicated or similar to those of the private sector, and private start-ups using open data are supported. However, as the demand for Open Data gradually increases, the cases of refusal to provide or interruption of Open Data held by public institutions are also increasing. Accordingly, the 'Open Data Mediation Committee' is established and operated so that the right to use data can be rescued through a simple dispute mediation procedure rather than complicated administrative litigation. The main issues dealt with in dispute settlement so far are usually the rights of third parties, such as open data including personal information, private information such as trade secrets, and copyrights. Plus, non-open data cannot be provided without the consent of the information subject. Rather than processing non-open data into open data through de-identification processing, positive results can be expected if consent is provided through active rights processing of the personal information subject. Not only can the Public Mydata Service be used by the information subject, but Open Data applicants will also be able to secure higher quality Open Data, which will have a positive impact on fostering the private data industry. This study derives a plan to establish a rights processing platform to enhance the usability of Open Data, including private information such as personal information, trade secrets, and copyright, which have become an issue when providing Open Data since 2014. With that, the proposals in this study are expected to serve as a stepping stone to revitalize private start-ups through the use of wide Open Data and improve public convenience through Public MyData services of information subjects.

A Study on the Transformation of CAD Data Using the Image Data Processing (화상처리를 이용한 CAD 데이터의 생성에 관한 연구)

  • Koo, Bon-Kwon;Roh, Woo-Joon
    • Journal of the Korean Society of Manufacturing Technology Engineers
    • /
    • v.7 no.6
    • /
    • pp.72-79
    • /
    • 1998
  • In this paper, image processing algorithm is studied to enhance the preciseness of the geometry while converting captured images to CAD data. A program is developed as a result. The code, in the image processing, utilizes outline trace, point data smoothing algorithm. It is capable of automatically generating design data by converting input image data to the CAD data. The output can be made in DXF, IGES formats. The current research can be utilized as a base data for the development of factory automation or flexible manufacturing system which adopt image processing based automatic inspection and measuring system.

  • PDF

Enhanced Locality Sensitive Clustering in High Dimensional Space

  • Chen, Gang;Gao, Hao-Lin;Li, Bi-Cheng;Hu, Guo-En
    • Transactions on Electrical and Electronic Materials
    • /
    • v.15 no.3
    • /
    • pp.125-129
    • /
    • 2014
  • A dataset can be clustered by merging the bucket indices that come from the random projection of locality sensitive hashing functions. It should be noted that for this to work the merging interval must be calculated first. To improve the feasibility of large scale data clustering in high dimensional space we propose an enhanced Locality Sensitive Hashing Clustering Method. Firstly, multiple hashing functions are generated. Secondly, data points are projected to bucket indices. Thirdly, bucket indices are clustered to get class labels. Experimental results showed that on synthetic datasets this method achieves high accuracy at much improved cluster speeds. These attributes make it well suited to clustering data in high dimensional space.

Designing Cost Effective Open Source System for Bigdata Analysis (빅데이터 분석을 위한 비용효과적 오픈 소스 시스템 설계)

  • Lee, Jong-Hwa;Lee, Hyun-Kyu
    • Knowledge Management Research
    • /
    • v.19 no.1
    • /
    • pp.119-132
    • /
    • 2018
  • Many advanced products and services are emerging in the market thanks to data-based technologies such as Internet (IoT), Big Data, and AI. The construction of a system for data processing under the IoT network environment is not simple in configuration, and has a lot of restrictions due to a high cost for constructing a high performance server environment. Therefore, in this paper, we will design a development environment for large data analysis computing platform using open source with low cost and practicality. Therefore, this study intends to implement a big data processing system using Raspberry Pi, an ultra-small PC environment, and open source API. This big data processing system includes building a portable server system, building a web server for web mining, developing Python IDE classes for crawling, and developing R Libraries for NLP and visualization. Through this research, we will develop a web environment that can control real-time data collection and analysis of web media in a mobile environment and present it as a curriculum for non-IT specialists.

Query Optimization on Large Scale Nested Data with Service Tree and Frequent Trajectory

  • Wang, Li;Wang, Guodong
    • Journal of Information Processing Systems
    • /
    • v.17 no.1
    • /
    • pp.37-50
    • /
    • 2021
  • Query applications based on nested data, the most commonly used form of data representation on the web, especially precise query, is becoming more extensively used. MapReduce, a distributed architecture with parallel computing power, provides a good solution for big data processing. However, in practical application, query requests are usually concurrent, which causes bottlenecks in server processing. To solve this problem, this paper first combines a column storage structure and an inverted index to build index for nested data on MapReduce. On this basis, this paper puts forward an optimization strategy which combines query execution service tree and frequent sub-query trajectory to reduce the response time of frequent queries and further improve the efficiency of multi-user concurrent queries on large scale nested data. Experiments show that this method greatly improves the efficiency of nested data query.