• 제목/요약/키워드: data pre-processing

검색결과 801건 처리시간 0.03초

Application of recursive SSA as data pre-processing filter for stochastic subspace identification

  • Loh, Chin-Hsiung;Liu, Yi-Cheng
    • Smart Structures and Systems
    • /
    • 제11권1호
    • /
    • pp.19-34
    • /
    • 2013
  • The objective of this paper is to develop on-line system parameter estimation and damage detection technique from the response measurements through using the Recursive Covariance-Driven Stochastic Subspace identification (RSSI-COV) approach. To reduce the effect of noise on the results of identification, discussion on the pre-processing of data using recursive singular spectrum analysis (rSSA) is presented to remove the noise contaminant measurements so as to enhance the stability of data analysis. Through the application of rSSA-SSI-COV to the vibration measurement of bridge during scouring experiment, the ability of the proposed algorithm was proved to be robust to the noise perturbations and offers a very good online tracking capability. The accuracy and robustness offered by rSSA-SSI-COV provides a key to obtain the evidence of imminent bridge settlement and a very stable modal frequency tracking which makes it possible for early warning. The peak values of the identified $1^{st}$ mode shape slope ratio has shown to be a good indicator for damage location, meanwhile, the drastic movements of the peak of $2^{nd}$ mode slope ratio could be used as another feature to indicate imminent pier settlement.

Preprocessing of the Direct-broadcast Data from the Atmospheric Infared Sounder (AIRS) Sounding Suite on Aqua Satellite

  • 김성범;박혜숙;김금란;박승환;김문규;이종규
    • 대기
    • /
    • 제13권4호
    • /
    • pp.71-79
    • /
    • 2003
  • We present a pre processing system for the Atmospheric Infrared Sounder (AIRS) sounding suite onboard Aqua satellite. With its unprecedented 2378 channels in IR bands, AIRS aims at achieving the sounding accuracy [s1]of a radiosonde (1 K in 1-km layer for temperature and 10% in 2-km layer for humidity). The core of the pre p rocessor is the International MODIS/AIRS Processing Package (IMAPP) that performs the geometric and radiometric correction to compute the Earth's radiance. Then we remove spurious data and retrieve the brightness temperature (Tb). Since we process the direct-broadcast data almost for the first time among the AIRS directbroadcast community, special attention is needed to understand and verify the products. This includes the pixel-to-pixel verification of the direct-broadcast product with reference to the fullorbit product, which shows the difference of less than $10^{-3}$ K in IR Tb.

Crop Leaf Disease Identification Using Deep Transfer Learning

  • Changjian Zhou;Yutong Zhang;Wenzhong Zhao
    • Journal of Information Processing Systems
    • /
    • 제20권2호
    • /
    • pp.149-158
    • /
    • 2024
  • Traditional manual identification of crop leaf diseases is challenging. Owing to the limitations in manpower and resources, it is challenging to explore crop diseases on a large scale. The emergence of artificial intelligence technologies, particularly the extensive application of deep learning technologies, is expected to overcome these challenges and greatly improve the accuracy and efficiency of crop disease identification. Crop leaf disease identification models have been designed and trained using large-scale training data, enabling them to predict different categories of diseases from unlabeled crop leaves. However, these models, which possess strong feature representation capabilities, require substantial training data, and there is often a shortage of such datasets in practical farming scenarios. To address this issue and improve the feature learning abilities of models, this study proposes a deep transfer learning adaptation strategy. The novel proposed method aims to transfer the weights and parameters from pre-trained models in similar large-scale training datasets, such as ImageNet. ImageNet pre-trained weights are adopted and fine-tuned with the features of crop leaf diseases to improve prediction ability. In this study, we collected 16,060 crop leaf disease images, spanning 12 categories, for training. The experimental results demonstrate that an impressive accuracy of 98% is achieved using the proposed method on the transferred ResNet-50 model, thereby confirming the effectiveness of our transfer learning approach.

A CPU-GPU Hybrid System of Environment Perception and 3D Terrain Reconstruction for Unmanned Ground Vehicle

  • Song, Wei;Zou, Shuanghui;Tian, Yifei;Sun, Su;Fong, Simon;Cho, Kyungeun;Qiu, Lvyang
    • Journal of Information Processing Systems
    • /
    • 제14권6호
    • /
    • pp.1445-1456
    • /
    • 2018
  • Environment perception and three-dimensional (3D) reconstruction tasks are used to provide unmanned ground vehicle (UGV) with driving awareness interfaces. The speed of obstacle segmentation and surrounding terrain reconstruction crucially influences decision making in UGVs. To increase the processing speed of environment information analysis, we develop a CPU-GPU hybrid system of automatic environment perception and 3D terrain reconstruction based on the integration of multiple sensors. The system consists of three functional modules, namely, multi-sensor data collection and pre-processing, environment perception, and 3D reconstruction. To integrate individual datasets collected from different sensors, the pre-processing function registers the sensed LiDAR (light detection and ranging) point clouds, video sequences, and motion information into a global terrain model after filtering redundant and noise data according to the redundancy removal principle. In the environment perception module, the registered discrete points are clustered into ground surface and individual objects by using a ground segmentation method and a connected component labeling algorithm. The estimated ground surface and non-ground objects indicate the terrain to be traversed and obstacles in the environment, thus creating driving awareness. The 3D reconstruction module calibrates the projection matrix between the mounted LiDAR and cameras to map the local point clouds onto the captured video images. Texture meshes and color particle models are used to reconstruct the ground surface and objects of the 3D terrain model, respectively. To accelerate the proposed system, we apply the GPU parallel computation method to implement the applied computer graphics and image processing algorithms in parallel.

프로세서 노드 상황을 고려하는 효율적인 메시지 스캐터 및 개더 알고리즘 (Efficient Message Scattering and Gathering Based on Processing Node Status)

  • Park, Jongsu
    • 한국정보통신학회논문지
    • /
    • 제26권4호
    • /
    • pp.637-640
    • /
    • 2022
  • To maximize performance in a high-performance multicore processor system. it is essential to enable effective data communication between processing cores. Data communication between processor nodes can be broadly classified into collective and point-to-point communications. Collective communication comprises scattering and gathering. This paper presents a efficient message scattering and gathering based on processing node status. In the proposed algorithms, the transmission order is changed according to the data size of the pre-existing communication, to reduce the waiting time required until the collective communications begin. From the simulation, the performances of the proposed message scattering and gathering algorithms were improved by approximately 71.41% and 69.84%.

Number of sampling leaves for reflectance measurement of Chinese cabbage and kale

  • Chung, Sun-Ok;Ngo, Viet-Duc;Kabir, Md. Shaha Nur;Hong, Soon-Jung;Park, Sang-Un;Kim, Sun-Ju;Park, Jong-Tae
    • 농업과학연구
    • /
    • 제41권3호
    • /
    • pp.169-175
    • /
    • 2014
  • Objective of this study was to investigate effects of pre-processing method and number of sampling leaves on stability of the reflectance measurement for Chinese cabbage and kale leaves. Chinese cabbage and kale were transplanted and cultivated in a plant factory. Leaf samples of the kale and cabbage were collected at 4 weeks after transplanting of the seedlings. Spectra data were collected with an UV/VIS/NIR spectrometer in the wavelength region from 190 to 1130 nm. All leaves (mature and young leaves) were measured on 9 and 12 points in the blade part in the upper area for kale and cabbage leaves, respectively. To reduce the spectral noise, the raw spectral data were preprocessed by different methods: i) moving average, ii) Savitzky-Golay filter, iii) local regression using weighted linear least squares and a $1^{st}$ degree polynomial model (lowess), iv) local regression using weighted linear least squares and a $2^{nd}$ degree polynomial model (loess), v) a robust version of 'lowess', vi) a robust version of 'loess', with 7, 11, 15 smoothing points. Effects of number of sampling leaves were investigated by reflectance difference (RD) and cross-correlation (CC) methods. Results indicated that the contribution of the spectral data collected at 4 sampling leaves were good for both of the crops for reflectance measurement that does not change stability of measurement much. Furthermore, moving average method with 11 smoothing points was believed to provide reliable pre-processed data for further analysis.

LDA기반 토픽모델링을 활용한 공공데이터 기반의 교육용 데이터마이닝 연구 (A Study on Educational Data Mining for Public Data Portal through Topic Modeling Method with Latent Dirichlet Allocation)

  • 신승기
    • 정보교육학회논문지
    • /
    • 제26권5호
    • /
    • pp.439-448
    • /
    • 2022
  • 본 연구에서는 공공데이터포털에서 제공하는 교육관련 데이터를 검색하고 토픽모델링 기법을 활용한 분류를 통해 어떠한 데이터의 종류가 구축되어 있으며 활용이 가능한지를 살펴보고자 하였다. 공공데이터포털의 데이터에 대하여 분류체계를 기준으로 교육분야의 파일데이터는 3,072건이 수집되었으며, 검색어를 활용하여 '교육'을 검색하여 나타난 파일데이터 2,361건으로 나타났다. 각각의 데이터셋에 대하여 불용어처리를 실시하고 데이터 전처리를 수행하여 LDA기반 토픽모델링을 활용하여 텍스트마이닝 분석을 실시하였다. 사전에 교육으로 분류된 데이터셋에서는 현재 재학중인 학교급별 학생을 대상으로 지원하는 프로그램과 정보에 대한 내용이 제공되고 있었다. 한편, 교육으로 검색하여 수집된 데이터셋에서는 장애인, 학부모, 노인, 아동 등 평생교육의 관점으로 제공되는 교육 프로그램 및 지원현황이라는 특징이 나타났다. 데이터과학기반의 의사결정 및 문제해결력을 기르기 위해 공공데이터포털이 제공하는 데이터에서 교육과정 및 내용이 충분히 제공되는 것도 좋은 기회가 될 것이다.

도시하천관리 연계 플랫폼 개발(I) (Development of urban river data management platform(I))

  • 이성학;심규철;구본현
    • 한국수자원학회논문집
    • /
    • 제52권12호
    • /
    • pp.1087-1098
    • /
    • 2019
  • 본 연구에서는 도시하천관리에 있어 이용되는 데이터를 수집, 정제 및 제공하는 기능을 수행하는 도시하천 통합데이터 플랫폼 개발을 수행하였다. 도시하천 통합데이터 플랫폼은 Open API서비스를 활용하여 다양한 기관에서 제공되는 데이터를 수집하는 기능을 가지고 있으며, 수집된 데이터는 전처리 과정을 통하여 정제되어 데이터베이스에 적재된다. 수집된 데이터는 시각화 시스템을 활용하여 검토 및 분석이 가능하며, 단위 Open API 형태로 제공되므로 도시하천모형에서 이를 조합하여 개별적인 입력자료로 활용할 수 있도록 하였다. 또한 실시간 데이터에 대한 제공시스템을 개발하여 도시하천모형에 실시간 데이터를 적용할 수 있도록 하였다. 이를 통하여 사용자는 데이터의 수집과 전처리, 입력자료 구축에 필요한 시간과 노력을 절감하여 도시하천관리 모형과 시스템의 개발에 있어 효율성과 확장성이 증대될 것으로 판단된다.

머신러닝을 위한 온톨로지 기반의 Raw Data 전처리 기법 (Pre-processing Method of Raw Data Based on Ontology for Machine Learning)

  • 황치곤;윤창표
    • 한국정보통신학회논문지
    • /
    • 제24권5호
    • /
    • pp.600-608
    • /
    • 2020
  • 머신러닝은 학습 데이터로부터 목적함수를 구성하고, 테스트 데이터를 통해 목적함수의 확인함으로써 발생하는 데이터에 대한 예측을 수행한다. 머신러닝에서 입력데이터는 전처리 과정을 통해 정규화 과정을 거친다. 이런 정규화는 입력데이터의 평균과 표준편차를 이용하여 표준화하거나, 수치 데이터가 아닌 nominal value는 one-hot 코드 형태로 변환하는 방식을 이용한다. 그러나 이 전처리 과정만으로 문제를 해결할 수 없다. 이러한 이유로 본 논문에서 입력데이터의 정규화를 위해 온톨로지를 이용하는 방법을 제안한다. 이를 위한 테스트 데이터는 모바일 기기로부터 수집된 와이파이 장치의 RSSI값을 이용하고, 수집된 데이터의 노이즈와 이질적 문제는 온톨로지를 이용하여 정제하는 방법을 제시한다.

WWW에서 대화형 원격 한자학습 시스템 (Interactive chinese Character Distance Learning System on the WWW)

  • 강종규;박상우;김현숙;김계환;진성일
    • 한국정보처리학회논문지
    • /
    • 제4권3호
    • /
    • pp.698-708
    • /
    • 1997
  • WWW 상에서 원격교육 서버를 구축하고 서비스 하기 위해서 기존의 멀티미디어 데 이터의 전송 및 처리가 다운로드 방식이 아닌 실시간 처리 방식이 요구된다. 본 연구 에서는 이러한 요구를 만족하기 위해 원격교육을 위한 동영상(AVI)와 음성(WAV)같은 멀티미디어 데이터를 분산환경에서 실시간으로 처리할 수 있는 실시간처리 모듈을 개발하여 MuX(Mulitimeida I/O Server)에 추가하고 이를WWW와 접목시켜 하이퍼텍스트 및 동영상 데이터를 실시간으로 서비스 할 수 있는 실시간 WWW서버를 개발하였으며, 이에 기반한 실시간 대화형 한자학습을 위한 원격교육 시스템을 연구 개발하였다.

  • PDF