• Title/Summary/Keyword: data pre-processing

Search Result 800, Processing Time 0.032 seconds

Pre-processing for IPC Classification of Patent Documents (특허문서의 IPC 분류를 위한 데이터 변환 및 통합)

  • Su-Hyun Park;Jin Kim
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2023.11a
    • /
    • pp.367-368
    • /
    • 2023
  • 4차 산업혁명으로 다양한 기술과 아이디어가 생겨나고 있고, 이를 보호하기 위한 특허는 그 등록 건수가 매년 증가하는 추세이다. 그러나 현재 특허문서를 분류하는 과정을 수동으로 진행하고 있기에 이를 자동으로 진행할 수 있는 분류기를 생성할 필요를 느꼈고, 본 논문에서는 특허문서를 분류기에 적용할 데이터의 전처리 과정 중 데이터 변환과 통합 과정을 다루었다.

Efficient Processing of Multidimensional Vessel USN Stream Data using Clustering Hash Table (클러스터링 해쉬 테이블을 이용한 다차원 선박 USN 스트림 데이터의 효율적인 처리)

  • Song, Byoung-Ho;Oh, Il-Whan;Lee, Seong-Ro
    • Journal of the Institute of Electronics Engineers of Korea SP
    • /
    • v.47 no.6
    • /
    • pp.137-145
    • /
    • 2010
  • Digital vessel have to accurate and efficient mange the digital data from various sensors in the digital vessel. But, In sensor network, it is difficult to transmit and analyze the entire stream data depending on limited networks, power and processor. Therefore it is suitable to use alternative stream data processing after classifying the continuous stream data. In this paper, We propose efficient processing method that arrange some sensors (temperature, humidity, lighting, voice) and process query based on sliding window for efficient input stream and pre-clustering using multiple Support Vector Machine(SVM) algorithm and manage hash table to summarized information. Processing performance improve as store and search and memory using hash table and usage reduced so maintain hash table in memory. We obtained to efficient result that accuracy rate and processing performance of proposal method using 35,912 data sets.

Application of Digital Signal Analysis Technique to Enhance the Quality of Tracer Gas Measurements in IAQ Model Tests

  • Lee, Hee-Kwan;Awbi, Hazim B.
    • Journal of Korean Society for Atmospheric Environment
    • /
    • v.23 no.E2
    • /
    • pp.66-73
    • /
    • 2007
  • The introduction of tracer gas techniques to ventilation studies in indoor environments provides valuable information that used to be unattainable from conventional testing environments. Data acquisition systems (DASs) containing analogue-to-digital (A/D) converters are usually used to function the key role that records signals to storage in digital format. In the testing process, there exist a number of components in the measuring equipment which may produce system-based inference to the monitored results. These unwanted fluctuations may cause significant error in data analysis, especially when non-linear algorithms are involved. In this study, a pre-processor is developed and applied to separate the unwanted fluctuations (noise or interference) in raw measurements and to reduce the uncertainty in the measurement. Moving average, notch filter, FIR (Finite Impulse Response) filters, and IIR (Infinite Impulse Response) filters are designed and applied to collect the desired information from the raw measurements. Tracer gas concentrations are monitored during leakage and ventilation tests in the model test room. The signal analysis functions are introduced to carry out the digital signal processing (DSP) work. Overall the FIR filters process the $CO_2$ measurement properly for ventilation rate and mean age of air calculations. It is found that, the Kaiser filter was the most applicable digital filter for pre-processing the tracer gas measurements. Although the IIR filters help to reduce the random noise in the data, they cause considerable changes to the filtered data, which is not desirable.

Parallelization of Genome Sequence Data Pre-Processing on Big Data and HPC Framework (빅데이터 및 고성능컴퓨팅 프레임워크를 활용한 유전체 데이터 전처리 과정의 병렬화)

  • Byun, Eun-Kyu;Kwak, Jae-Hyuck;Mun, Jihyeob
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.8 no.10
    • /
    • pp.231-238
    • /
    • 2019
  • Analyzing next-generation genome sequencing data in a conventional way using single server may take several tens of hours depending on the data size. However, in order to cope with emergency situations where the results need to be known within a few hours, it is required to improve the performance of a single genome analysis. In this paper, we propose a parallelized method for pre-processing genome sequence data which can reduce the analysis time by utilizing the big data technology and the highperformance computing cluster which is connected to the high-speed network and shares the parallel file system. For the reliability of analytical data, we have chosen a strategy to parallelize the existing analytical tools and algorithms to the new environment. Parallelized processing, data distribution, and parallel merging techniques have been developed and performance improvements have been confirmed through experiments.

Development Technique for Dynamic Node Management of Visual Modeler

  • Yoon, C.R.;Kim, K.O.
    • Proceedings of the KSRS Conference
    • /
    • 2003.11a
    • /
    • pp.1131-1133
    • /
    • 2003
  • Spatial image processing software requires various user interactions to make a plan, prepare necessary data such as images, vectors, ancillary data and user-defined data, execute functions according to pre-defined procedures, analyze and store the results. In this manner, overall processes are controlled by user interactions. In this paper, we propose visual modeler which has the automated spatial image processing technique to minimize user interactions and re -use repeatable procedure. The proposed visual modeler is designed to use inter-operable components proposed by OpenGIS consortium as well as conventional COM components.

  • PDF

Pre-screening technique for MT and GDS data processing based on the spectral power of Electromagnetic field (전자기장의 분광 에너지에 기반한 MT 및 GDS 자료의 전처리 기법 연구)

  • Yang, Jun-Mo;Kwon, Byung-Doo
    • 한국지구물리탐사학회:학술대회논문집
    • /
    • 2006.06a
    • /
    • pp.253-258
    • /
    • 2006
  • The Korean peninsula has been known to be very difficult to acquire clean MT and GDS data due to its highly industrialization and civilization. In this environment, a pre-screening step selecting data segments with a proper S/N ration is an essential one. This study modified the automatic pre-screening step based on the spectral power of electromagnetic field (RMP) taking account of the situation of the Korean Peninsula. The modified RMP technique was applied to MT data measured at seven sites located in middle part of the peninsula. In the whole sense, the RMP technique considerably improved the connectivity of apparent resistivity and phase curves around the period of 10 sec. In addition, the results processed by the RMP technique showed a very little difference with those derived from manual editing, and the superior performance of it is found especially in the connectivity of apparent resistivity curve.

  • PDF

An Automatic Inspection of the Surface Outlook of High Speed Moving Plate by Using One Dimensional CCD Camera

  • Hyun, Lim-Sung;Suck, Boo-Kwang
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 2001.10a
    • /
    • pp.118.5-118
    • /
    • 2001
  • This paper describes an image processing method for inspecting the surface outlook of high speed moving plates. Noise free image and a new real time processing methods are required to inspect the surface outlook of the high speed moving plates in real time. It is difficult to get a noise free image due to a signal noise, a light noise and background image in typical industrial factory. Thus, pre-processing techniques should be required to get a good image and produce so many time steps to proceed the image data. The objective of this research is to get image on the surface of the moving plates with a speed of 1m/sec and to detect some defaults on the surface image. So, the pre-processing techniques ...

  • PDF

An implementation of the high speed image processing board for contact image sensor (Contact image sensor를 위한 고속 영상 처리 보드 구현)

  • Kang, Hyun-Inn;Ju, Yong-Wan;Baek, Kwang-Ryul
    • Journal of Institute of Control, Robotics and Systems
    • /
    • v.5 no.6
    • /
    • pp.691-697
    • /
    • 1999
  • This paper describes the implementation of a high speed image processing board. This image processing board is consist of a image acquisition part and a image processing part. The image acquistion part is digitizing the image input data from CIS and save it to the dual port RAM. By putting on the dual port memory between two parts, during acquistion of image, the image processing part can be effectively processing of large-volume image data. Most of all image preprocessing part are integrated in a large-scaled FPGA. We arwe using ADSP-2181 of the Analog Device Inc., LTD. for a image processing part, and using the available all memory of DSP for the large-volume image data. Especially, using of IDMA exchanges the data with the external microprocessor or the external PC, and can watch the result of image processing and acquired image. Finally, we show that an implemented image processing board used for the simulation of image retreval by the one of the typical application.

  • PDF

Absolute Atmospheric Correction Procedure for the EO-1 Hyperion Data Using MODTRAN Code

  • Kim, Sun-Hwa;Kang, Sung-Jin;Chi, Jun-Hwa;Lee, Kyu-Sung
    • Korean Journal of Remote Sensing
    • /
    • v.23 no.1
    • /
    • pp.7-14
    • /
    • 2007
  • Atmospheric correction is one of critical procedures to extract quantitative information related to biophysical variables from hyperspectral imagery. Most atmospheric correction algorithms developed for hyperspectral data have been based upon atmospheric radiative transfer (RT) codes, such as MODTRAN. Because of the difficulty in acquisition of atmospheric data at the time of image capture, the complexity of RT model, and large volume of hyperspectral data, atmospheric correction can be very difficult and time-consuming processing. In this study, we attempted to develop an efficient method for the atmospheric correction of EO-1 Hyperion data. This method uses the pre-calculated look-up-table (LUT) for fast and simple processing. The pre-calculated LUT was generated by successive running of MODTRAN model with several input parameters related to solar and sensor geometry, radiometric specification of sensor, and atmospheric condition. Atmospheric water vapour contents image was generated directly from a few absorption bands of Hyperion data themselves and used one of input parameters. This new atmospheric correction method was tested on the Hyperion data acquired on June 3, 2001 over Seoul area. Reflectance spectra of several known targets corresponded with the typical pattern of spectral reflectance on the atmospherically corrected Hyperion image, although further improvement to reduce sensor noise is necessary.

A Study on GNSS Data Pre-processing for Analyzing Geodetic Effects on Crustal Deformation due to the Earthquake (지진에 의한 측지학적 지각변동 분석을 위한 GNSS 자료 전처리 연구)

  • Sohn, Dong Hyo;Kim, Du Sik;Park, Kwan Dong
    • Journal of Korean Society for Geospatial Information Science
    • /
    • v.23 no.1
    • /
    • pp.47-54
    • /
    • 2015
  • In this study, we developed strategies for pre-processing GNSS data for the purpose of separating geodetic factors from crustal deformation due to the earthquakes. Before interpreting GNSS data analysis results, we removed false signals from GNSS coordinate time series. Because permanent GNSS stations are located on a large tectonic plate, GNSS position estimates should be affected by the tectonic velocity of the plate. Also, stations with surrounding trees have seasonal signals in their three-dimensional coordinate estimates. Thus, we have estimated the location of an Euler pole and angular velocities to deduce the plate tectonic velocity and verified with geological models. Also, annual amplitudes and initial phases were estimated to get rid of those false annual signals showing up in the time series. By considering the two effects, truly geodetic analysis was possible and the result was used as preliminary data for analyzing post-seismic deformation of the Korean peninsula due to the Tohoku-oki earthquake.