• Title/Summary/Keyword: Distributed Data pipeline

Search Result 23, Processing Time 0.018 seconds

Investigating Data Preprocessing Algorithms of a Deep Learning Postprocessing Model for the Improvement of Sub-Seasonal to Seasonal Climate Predictions (계절내-계절 기후예측의 딥러닝 기반 후보정을 위한 입력자료 전처리 기법 평가)

  • Uran Chung;Jinyoung Rhee;Miae Kim;Soo-Jin Sohn
    • Korean Journal of Agricultural and Forest Meteorology
    • /
    • v.25 no.2
    • /
    • pp.80-98
    • /
    • 2023
  • This study explores the effectiveness of various data preprocessing algorithms for improving subseasonal to seasonal (S2S) climate predictions from six climate forecast models and their Multi-Model Ensemble (MME) using a deep learning-based postprocessing model. A pipeline of data transformation algorithms was constructed to convert raw S2S prediction data into the training data processed with several statistical distribution. A dimensionality reduction algorithm for selecting features through rankings of correlation coefficients between the observed and the input data. The training model in the study was designed with TimeDistributed wrapper applied to all convolutional layers of U-Net: The TimeDistributed wrapper allows a U-Net convolutional layer to be directly applied to 5-dimensional time series data while maintaining the time axis of data, but every input should be at least 3D in U-Net. We found that Robust and Standard transformation algorithms are most suitable for improving S2S predictions. The dimensionality reduction based on feature selections did not significantly improve predictions of daily precipitation for six climate models and even worsened predictions of daily maximum and minimum temperatures. While deep learning-based postprocessing was also improved MME S2S precipitation predictions, it did not have a significant effect on temperature predictions, particularly for the lead time of weeks 1 and 2. Further research is needed to develop an optimal deep learning model for improving S2S temperature predictions by testing various models and parameters.

Development of SNP marker set for marker-assisted backcrossing (MABC) in cultivating tomato varieties

  • Park, GiRim;Jang, Hyun A;Jo, Sung-Hwan;Park, Younghoon;Oh, Sang-Keun;Nam, Moon
    • Korean Journal of Agricultural Science
    • /
    • v.45 no.3
    • /
    • pp.385-400
    • /
    • 2018
  • Marker-assisted backcrossing (MABC) is useful for selecting offspring with a highly recovered genetic background for a recurrent parent at early generation unlike rice and other field crops. Molecular marker sets applicable to practical MABC are scarce in vegetable crops including tomatoes. In this study, we used the National Center for Biotechnology Information- short read archive (NCBI-SRA) database that provided the whole genome sequences of 234 tomato accessions and selected 27,680 tag-single nucleotide polymorphisms (tag-SNPs) that can identify haplotypes in the tomato genome. From this SNP dataset, a total of 143 tag-SNPs that have a high polymorphism information content (PIC) value (> 0.3) and are physically evenly distributed on each chromosome were selected as a MABC marker set. This marker set was tested for its polymorphism in each pairwise cross combination constructed with 124 of the 234 tomato accessions, and a relatively high number of SNP markers polymorphic for the cross combination was observed. The reliability of the MABC SNP set was assessed by converting 18 SNPs into Luna probe-based high-resolution melting (HRM) markers and genotyping nine tomato accessions. The results show that the SNP information and HRM marker genotype matched in 98.6% of the experiment data points, indicating that our sequence analysis pipeline for SNP mining worked successfully. The tag-SNP set for the MABC developed in this study can be useful for not only a practical backcrossing program but also for cultivar identification and F1 seed purity test in tomatoes.

Model Analysis of AI-Based Water Pipeline Improved Decision (AI기반 상수도시설 개량 의사결정 모델 분석)

  • Kim, Gi-Tae;Min, Byung-Won;Oh, Yong-Sun
    • Journal of Internet of Things and Convergence
    • /
    • v.8 no.5
    • /
    • pp.11-16
    • /
    • 2022
  • As an interest in the development of artificial intelligence(AI) technology in the water supply sector increases, we have developed an AI algorithm that can predict improvement decision-making ratings through repetitive learning using the data of pipe condition evaluation results, and present the most reliable prediction model through a verification process. We have developed the algorithm that can predict pipe ratings by pre-processing 12 indirect evaluation items based on the 2020 Han River Basin's basic plan and applying the AI algorithm to update weighting factors through backpropagation. This method ensured that the concordance rate between the direct evaluation result value and the calculated result value through repetitive learning and verification was more than 90%. As a result of the algorithm accuracy verification process, it was confirmed that all water pipe type data were evenly distributed, and the more learning data, the higher prediction accuracy. If data from all across the country is collected, the reliability of the prediction technique for pipe ratings using AI algorithm will be improved, and therefore, it is expected that the AI algorithm will play a role in supporting decision-making in the objective evaluation of the condition of aging pipes.