• Title/Summary/Keyword: data pre-processing

Search Result 801, Processing Time 0.029 seconds

Application of recursive SSA as data pre-processing filter for stochastic subspace identification

  • Loh, Chin-Hsiung;Liu, Yi-Cheng
    • Smart Structures and Systems
    • /
    • v.11 no.1
    • /
    • pp.19-34
    • /
    • 2013
  • The objective of this paper is to develop on-line system parameter estimation and damage detection technique from the response measurements through using the Recursive Covariance-Driven Stochastic Subspace identification (RSSI-COV) approach. To reduce the effect of noise on the results of identification, discussion on the pre-processing of data using recursive singular spectrum analysis (rSSA) is presented to remove the noise contaminant measurements so as to enhance the stability of data analysis. Through the application of rSSA-SSI-COV to the vibration measurement of bridge during scouring experiment, the ability of the proposed algorithm was proved to be robust to the noise perturbations and offers a very good online tracking capability. The accuracy and robustness offered by rSSA-SSI-COV provides a key to obtain the evidence of imminent bridge settlement and a very stable modal frequency tracking which makes it possible for early warning. The peak values of the identified $1^{st}$ mode shape slope ratio has shown to be a good indicator for damage location, meanwhile, the drastic movements of the peak of $2^{nd}$ mode slope ratio could be used as another feature to indicate imminent pier settlement.

Preprocessing of the Direct-broadcast Data from the Atmospheric Infared Sounder (AIRS) Sounding Suite on Aqua Satellite

  • Kim, Seungbum;Park, Hyesook;Kim, Kumlan;Park, Seunghwan;Kim, Moongyu;Lee, Jongju
    • Atmosphere
    • /
    • v.13 no.4
    • /
    • pp.71-79
    • /
    • 2003
  • We present a pre processing system for the Atmospheric Infrared Sounder (AIRS) sounding suite onboard Aqua satellite. With its unprecedented 2378 channels in IR bands, AIRS aims at achieving the sounding accuracy [s1]of a radiosonde (1 K in 1-km layer for temperature and 10% in 2-km layer for humidity). The core of the pre p rocessor is the International MODIS/AIRS Processing Package (IMAPP) that performs the geometric and radiometric correction to compute the Earth's radiance. Then we remove spurious data and retrieve the brightness temperature (Tb). Since we process the direct-broadcast data almost for the first time among the AIRS directbroadcast community, special attention is needed to understand and verify the products. This includes the pixel-to-pixel verification of the direct-broadcast product with reference to the fullorbit product, which shows the difference of less than $10^{-3}$ K in IR Tb.

Crop Leaf Disease Identification Using Deep Transfer Learning

  • Changjian Zhou;Yutong Zhang;Wenzhong Zhao
    • Journal of Information Processing Systems
    • /
    • v.20 no.2
    • /
    • pp.149-158
    • /
    • 2024
  • Traditional manual identification of crop leaf diseases is challenging. Owing to the limitations in manpower and resources, it is challenging to explore crop diseases on a large scale. The emergence of artificial intelligence technologies, particularly the extensive application of deep learning technologies, is expected to overcome these challenges and greatly improve the accuracy and efficiency of crop disease identification. Crop leaf disease identification models have been designed and trained using large-scale training data, enabling them to predict different categories of diseases from unlabeled crop leaves. However, these models, which possess strong feature representation capabilities, require substantial training data, and there is often a shortage of such datasets in practical farming scenarios. To address this issue and improve the feature learning abilities of models, this study proposes a deep transfer learning adaptation strategy. The novel proposed method aims to transfer the weights and parameters from pre-trained models in similar large-scale training datasets, such as ImageNet. ImageNet pre-trained weights are adopted and fine-tuned with the features of crop leaf diseases to improve prediction ability. In this study, we collected 16,060 crop leaf disease images, spanning 12 categories, for training. The experimental results demonstrate that an impressive accuracy of 98% is achieved using the proposed method on the transferred ResNet-50 model, thereby confirming the effectiveness of our transfer learning approach.

A CPU-GPU Hybrid System of Environment Perception and 3D Terrain Reconstruction for Unmanned Ground Vehicle

  • Song, Wei;Zou, Shuanghui;Tian, Yifei;Sun, Su;Fong, Simon;Cho, Kyungeun;Qiu, Lvyang
    • Journal of Information Processing Systems
    • /
    • v.14 no.6
    • /
    • pp.1445-1456
    • /
    • 2018
  • Environment perception and three-dimensional (3D) reconstruction tasks are used to provide unmanned ground vehicle (UGV) with driving awareness interfaces. The speed of obstacle segmentation and surrounding terrain reconstruction crucially influences decision making in UGVs. To increase the processing speed of environment information analysis, we develop a CPU-GPU hybrid system of automatic environment perception and 3D terrain reconstruction based on the integration of multiple sensors. The system consists of three functional modules, namely, multi-sensor data collection and pre-processing, environment perception, and 3D reconstruction. To integrate individual datasets collected from different sensors, the pre-processing function registers the sensed LiDAR (light detection and ranging) point clouds, video sequences, and motion information into a global terrain model after filtering redundant and noise data according to the redundancy removal principle. In the environment perception module, the registered discrete points are clustered into ground surface and individual objects by using a ground segmentation method and a connected component labeling algorithm. The estimated ground surface and non-ground objects indicate the terrain to be traversed and obstacles in the environment, thus creating driving awareness. The 3D reconstruction module calibrates the projection matrix between the mounted LiDAR and cameras to map the local point clouds onto the captured video images. Texture meshes and color particle models are used to reconstruct the ground surface and objects of the 3D terrain model, respectively. To accelerate the proposed system, we apply the GPU parallel computation method to implement the applied computer graphics and image processing algorithms in parallel.

Efficient Message Scattering and Gathering Based on Processing Node Status (프로세서 노드 상황을 고려하는 효율적인 메시지 스캐터 및 개더 알고리즘)

  • Park, Jongsu
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.26 no.4
    • /
    • pp.637-640
    • /
    • 2022
  • To maximize performance in a high-performance multicore processor system. it is essential to enable effective data communication between processing cores. Data communication between processor nodes can be broadly classified into collective and point-to-point communications. Collective communication comprises scattering and gathering. This paper presents a efficient message scattering and gathering based on processing node status. In the proposed algorithms, the transmission order is changed according to the data size of the pre-existing communication, to reduce the waiting time required until the collective communications begin. From the simulation, the performances of the proposed message scattering and gathering algorithms were improved by approximately 71.41% and 69.84%.

Number of sampling leaves for reflectance measurement of Chinese cabbage and kale

  • Chung, Sun-Ok;Ngo, Viet-Duc;Kabir, Md. Shaha Nur;Hong, Soon-Jung;Park, Sang-Un;Kim, Sun-Ju;Park, Jong-Tae
    • Korean Journal of Agricultural Science
    • /
    • v.41 no.3
    • /
    • pp.169-175
    • /
    • 2014
  • Objective of this study was to investigate effects of pre-processing method and number of sampling leaves on stability of the reflectance measurement for Chinese cabbage and kale leaves. Chinese cabbage and kale were transplanted and cultivated in a plant factory. Leaf samples of the kale and cabbage were collected at 4 weeks after transplanting of the seedlings. Spectra data were collected with an UV/VIS/NIR spectrometer in the wavelength region from 190 to 1130 nm. All leaves (mature and young leaves) were measured on 9 and 12 points in the blade part in the upper area for kale and cabbage leaves, respectively. To reduce the spectral noise, the raw spectral data were preprocessed by different methods: i) moving average, ii) Savitzky-Golay filter, iii) local regression using weighted linear least squares and a $1^{st}$ degree polynomial model (lowess), iv) local regression using weighted linear least squares and a $2^{nd}$ degree polynomial model (loess), v) a robust version of 'lowess', vi) a robust version of 'loess', with 7, 11, 15 smoothing points. Effects of number of sampling leaves were investigated by reflectance difference (RD) and cross-correlation (CC) methods. Results indicated that the contribution of the spectral data collected at 4 sampling leaves were good for both of the crops for reflectance measurement that does not change stability of measurement much. Furthermore, moving average method with 11 smoothing points was believed to provide reliable pre-processed data for further analysis.

A Study on Educational Data Mining for Public Data Portal through Topic Modeling Method with Latent Dirichlet Allocation (LDA기반 토픽모델링을 활용한 공공데이터 기반의 교육용 데이터마이닝 연구)

  • Seungki Shin
    • Journal of The Korean Association of Information Education
    • /
    • v.26 no.5
    • /
    • pp.439-448
    • /
    • 2022
  • This study aims to search for education-related datasets provided by public data portals and examine what data types are constructed through classification using topic modeling methods. Regarding the data of the public data portal, 3,072 cases of file data in the education field were collected based on the classification system. Text mining analysis was performed using the LDA-based topic modeling method with stopword processing and data pre-processing for each dataset. Program information and student-supporting notifications were usually provided in the pre-classified dataset for education from the data portal. On the other hand, the characteristics of educational programs and supporting information for the disabled, parents, the elderly, and children through the perspective of lifelong education were generally indicated in the dataset collected by searching for education. The results of data analysis through this study show that providing sufficient educational information through the public data portal would be better to help the students' data science-based decision-making and problem-solving skills.

Development of urban river data management platform(I) (도시하천관리 연계 플랫폼 개발(I))

  • Lee, Sunghack;Shim, Kyucheoul;Koo, Bonhyun
    • Journal of Korea Water Resources Association
    • /
    • v.52 no.12
    • /
    • pp.1087-1098
    • /
    • 2019
  • In this study, we developed an integrated urban river data platform that collects, cleans, and provides data for urban river management. The urban river integrated data platform has the function of collecting data provided by various institutions using the Open API service. The collected data is purified through pre-processing and loaded into a database. The collected data can be reviewed and analyzed using a visualization system and provided through the Open API, so that it can be used as individual input data by combining them in the urban river model. In addition, the development system for real-time data was developed to apply real-time data to urban river models. Through this, users will be able to reduce the time and effort required for data collection, pre-processing and input data construction, thereby increasing efficiency and scalability in the development of urban river models and systems.

Pre-processing Method of Raw Data Based on Ontology for Machine Learning (머신러닝을 위한 온톨로지 기반의 Raw Data 전처리 기법)

  • Hwang, Chi-Gon;Yoon, Chang-Pyo
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.24 no.5
    • /
    • pp.600-608
    • /
    • 2020
  • Machine learning constructs an objective function from learning data, and predicts the result of the data generated by checking the objective function through test data. In machine learning, input data is subjected to a normalisation process through a preprocessing. In the case of numerical data, normalization is standardized by using the average and standard deviation of the input data. In the case of nominal data, which is non-numerical data, it is converted into a one-hot code form. However, this preprocessing alone cannot solve the problem. For this reason, we propose a method that uses ontology to normalize input data in this paper. The test data for this uses the received signal strength indicator (RSSI) value of the Wi-Fi device collected from the mobile device. These data are solved through ontology because they includes noise and heterogeneous problems.

Interactive chinese Character Distance Learning System on the WWW (WWW에서 대화형 원격 한자학습 시스템)

  • Gang, Jong-Gyu;Park, Sang-U;Kim, Hyeon-Suk;Kim, Gye-Hwan;Jin, Seong-Il
    • The Transactions of the Korea Information Processing Society
    • /
    • v.4 no.3
    • /
    • pp.698-708
    • /
    • 1997
  • To construct distance learing servers and provide their service using the WWW(World Wide Web), it is necessary that we use a real-time processing mehtod rather than the processing after downloading method for multimedia data transmission and their processing.To fulfill such requirements, we developed a real-time processing muduloe for distance education which can process multimedia data in AVI and WAV formats in distrbuted eviroments.We in turn developede a real-time WWW server that can provide real-time services of hypertxt and motion poctures data in temsw of adding the real-time porcessing modute to the MuX framework and intergarting them with WWW. We frnally developed as distance lerming system for real-time interactive chinese character learming, bassed on the results from the pre-vious steps.

  • PDF