• Title/Summary/Keyword: data pre-processing

Search Result 800, Processing Time 0.025 seconds

Practical issues in signal processing for structural flexibility identification

  • Zhang, J.;Zhou, Y.;Li, P.J.
    • Smart Structures and Systems
    • /
    • v.15 no.1
    • /
    • pp.209-225
    • /
    • 2015
  • Compared to ambient vibration testing, impact testing has the merit to extract not only structural modal parameters but also structural flexibility. Therefore, structural deflections under any static load can be predicted from the identified results of the impact test data. In this article, a signal processing procedure for structural flexibility identification is first presented. Especially, practical issues in applying the proposed procedure for structural flexibility identification are investigated, which include sensitivity analyses of three pre-defined parameters required in the data pre-processing stage to investigate how they affect the accuracy of the identified structural flexibility. Finally, multiple-reference impact test data of a three-span reinforced concrete T-beam bridge are simulated by the FE analysis, and they are used as a benchmark structure to investigate the practical issues in the proposed signal processing procedure for structural flexibility identification.

Korean Machine Reading Comprehension for Patent Consultation Using BERT (BERT를 이용한 한국어 특허상담 기계독해)

  • Min, Jae-Ok;Park, Jin-Woo;Jo, Yu-Jeong;Lee, Bong-Gun
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.9 no.4
    • /
    • pp.145-152
    • /
    • 2020
  • MRC (Machine reading comprehension) is the AI NLP task that predict the answer for user's query by understanding of the relevant document and which can be used in automated consult services such as chatbots. Recently, the BERT (Pre-training of Deep Bidirectional Transformers for Language Understanding) model, which shows high performance in various fields of natural language processing, have two phases. First phase is Pre-training the big data of each domain. And second phase is fine-tuning the model for solving each NLP tasks as a prediction. In this paper, we have made the Patent MRC dataset and shown that how to build the patent consultation training data for MRC task. And we propose the method to improve the performance of the MRC task using the Pre-trained Patent-BERT model by the patent consultation corpus and the language processing algorithm suitable for the machine learning of the patent counseling data. As a result of experiment, we show that the performance of the method proposed in this paper is improved to answer the patent counseling query.

Design of Low Complexity Human Anxiety Classification Model based on Machine Learning (기계학습 기반 저 복잡도 긴장 상태 분류 모델)

  • Hong, Eunjae;Park, Hyunggon
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.66 no.9
    • /
    • pp.1402-1408
    • /
    • 2017
  • Recently, services for personal biometric data analysis based on real-time monitoring systems has been increasing and many of them have focused on recognition of emotions. In this paper, we propose a classification model to classify anxiety emotion using biometric data actually collected from people. We propose to deploy the support vector machine to build a classification model. In order to improve the classification accuracy, we propose two data pre-processing procedures, which are normalization and data deletion. The proposed algorithms are actually implemented based on Real-time Traffic Flow Measurement structure, which consists of data collection module, data preprocessing module, and creating classification model module. Our experiment results show that the proposed classification model can infers anxiety emotions of people with the accuracy of 65.18%. Moreover, the proposed model with the proposed pre-processing techniques shows the improved accuracy, which is 78.77%. Therefore, we can conclude that the proposed classification model based on the pre-processing process can improve the classification accuracy with lower computation complexity.

Comparison of Pre-processed Brain Tumor MR Images Using Deep Learning Detection Algorithms

  • Kwon, Hee Jae;Lee, Gi Pyo;Kim, Young Jae;Kim, Kwang Gi
    • Journal of Multimedia Information System
    • /
    • v.8 no.2
    • /
    • pp.79-84
    • /
    • 2021
  • Detecting brain tumors of different sizes is a challenging task. This study aimed to identify brain tumors using detection algorithms. Most studies in this area use segmentation; however, we utilized detection owing to its advantages. Data were obtained from 64 patients and 11,200 MR images. The deep learning model used was RetinaNet, which is based on ResNet152. The model learned three different types of pre-processing images: normal, general histogram equalization, and contrast-limited adaptive histogram equalization (CLAHE). The three types of images were compared to determine the pre-processing technique that exhibits the best performance in the deep learning algorithms. During pre-processing, we converted the MR images from DICOM to JPG format. Additionally, we regulated the window level and width. The model compared the pre-processed images to determine which images showed adequate performance; CLAHE showed the best performance, with a sensitivity of 81.79%. The RetinaNet model for detecting brain tumors through deep learning algorithms demonstrated satisfactory performance in finding lesions. In future, we plan to develop a new model for improving the detection performance using well-processed data. This study lays the groundwork for future detection technologies that can help doctors find lesions more easily in clinical tasks.

Study on the efficiency improvement of wind turbine load analysis by using automatic generation for wind load condition data (풍황 하중조건 데이터 자동생성화를 이용한 풍력터빈 하중해석의 효율 향상에 관한 연구)

  • Ahn, Kyoung-Min;Lim, Dong-Soo;Lee, Hyun-Joo;Choi, Won-Ho;Lee, Seung-Kuh
    • 한국신재생에너지학회:학술대회논문집
    • /
    • 2006.11a
    • /
    • pp.269-272
    • /
    • 2006
  • Load analysis software enables to design wind turbines effectively and exactly. In this paper, Bladed software developed by Garrad Hassan and Partners is used for load analysis. When using Bladed software, many time is requested to input data which is called by pre-processing. So in this paper, pre-processing Is automated by in-house software(BX) With this BX software, we can reduce the total time for pre-processing about 90%.

  • PDF

A BERT-Based Automatic Scoring Model of Korean Language Learners' Essay

  • Lee, Jung Hee;Park, Ji Su;Shon, Jin Gon
    • Journal of Information Processing Systems
    • /
    • v.18 no.2
    • /
    • pp.282-291
    • /
    • 2022
  • This research applies a pre-trained bidirectional encoder representations from transformers (BERT) handwriting recognition model to predict foreign Korean-language learners' writing scores. A corpus of 586 answers to midterm and final exams written by foreign learners at the Intermediate 1 level was acquired and used for pre-training, resulting in consistent performance, even with small datasets. The test data were pre-processed and fine-tuned, and the results were calculated in the form of a score prediction. The difference between the prediction and actual score was then calculated. An accuracy of 95.8% was demonstrated, indicating that the prediction results were strong overall; hence, the tool is suitable for the automatic scoring of Korean written test answers, including grammatical errors, written by foreigners. These results are particularly meaningful in that the data included written language text produced by foreign learners, not native speakers.

Mobile Transaction Processing in Hybrid Broadcasting Environment (복합 브로드캐스팅 환경에서 이동 트랜잭션 처리)

  • 김성석;양순옥
    • Journal of KIISE:Databases
    • /
    • v.31 no.4
    • /
    • pp.422-431
    • /
    • 2004
  • In recent years, different models in data delivery have been explored in mobile computing systems. Particularly, there were a lot of research efforts in the periodic push model where the server repetitively disseminates information without explicit request. However, average waiting time per data operation highly depends on the length of a broadcast cycle and different access pattern among clients may deteriorate the response time considerably. In this case, clients are preferably willing to send a data request to the server explicitly through backchannel in order to obtain optimal response time. We call the broadcast model supporting backchannel as hybrid broadcast. In this paper, we devise a new transaction processing algorithm(O-PreH) in hybrid broadcast environments. The data objects which the server maintains are divided into Push_Data for periodic broadcasting and Pull_Data for on-demand processing. Clients tune in broadcast channel or demand the data of interests according to the data type. Periodic invalidation reports from the server support maintaining transactional consistency. If one or more conflicts are found, conflict orders are determined not to violate the consistency(pre-reordering) and then the remaining operations have to be executed pessimistically. Through extensive simulations, we demonstrate the improved throughput of the proposed algorithm.

A Study on Water Network Modeling System Based Upon GIS (지리정보시스템 기반의 상수관망 모델링 시스템 연구)

  • Kim, Joon-Hyun;Yakunina, Natalia
    • Journal of Environmental Impact Assessment
    • /
    • v.19 no.3
    • /
    • pp.315-321
    • /
    • 2010
  • ArcView and water network models have been integrated to develop the water network modeling system based upon GIS. To develop this system, pre, main, and post processing systems are required. GIS programming technique was adopted by using the ArcView's script language Avenue. The input data of models have been prepared by using the AutoCAD Map3D through the conversion of modeling input data to GIS data for A city. The modeling has been implemented by using EPANET, WaterCAD, InfoWorks. To develop the post processing system, the modeling results of the water network models have been analyzed by using GIS. During the application process of the developed system to B city with 300,000 population, main problems were found in the constructed GIS DB of that city. Thus, pilot study area of B city has been constructed, and pre-, main, and post-processing techniques were invented based upon GIS. Finally, the problems related to waterworks GIS projects in Korea were discussed and solutions were suggested.

Efficient Implementation of Single Error Correction and Double Error Detection Code with Check Bit Pre-computation for Memories

  • Cha, Sanguhn;Yoon, Hongil
    • JSTS:Journal of Semiconductor Technology and Science
    • /
    • v.12 no.4
    • /
    • pp.418-425
    • /
    • 2012
  • In this paper, efficient implementation of error correction code (ECC) processing circuits based on single error correction and double error detection (SEC-DED) code with check bit pre-computation is proposed for memories. During the write operation of memory, check bit pre-computation eliminates the overall bits computation required to detect a double error, thereby reducing the complexity of the ECC processing circuits. In order to implement the ECC processing circuits using the check bit pre-computation more efficiently, the proper SEC-DED codes are proposed. The H-matrix of the proposed SEC-DED code is the same as that of the odd-weight-column code during the write operation and is designed by replacing 0's with 1's at the last row of the H-matrix of the odd-weight-column code during the read operation. When compared with a conventional implementation utilizing the odd-weight- column code, the implementation based on the proposed SEC-DED code with check bit pre-computation achieves reductions in the number of gates, latency, and power consumption of the ECC processing circuits by up to 9.3%, 18.4%, and 14.1% for 64 data bits in a word.

Development of Pre-Processing and Bias Correction Modules for AMSU-A Satellite Data in the KIAPS Observation Processing System (KIAPS 관측자료 처리시스템에서의 AMSU-A 위성자료 초기 전처리와 편향보정 모듈 개발)

  • Lee, Sihye;Kim, Ju-Hye;Kang, Jeon-Ho;Chun, Hyoung-Wook
    • Atmosphere
    • /
    • v.23 no.4
    • /
    • pp.453-470
    • /
    • 2013
  • As a part of the KIAPS Observation Processing System (KOPS), we have developed the modules of satellite radiance data pre-processing and quality control, which include observation operators to interpolate model state variables into radiances in observation space. AMSU-A (Advanced Microwave Sounding Unit-A) level-1d radiance data have been extracted using the BUFR (Binary Universal Form for the Representation of meteorological data) decoder and a first guess has been calculated with RTTOV (Radiative Transfer for TIROS Operational Vertical Sounder) version 10.2. For initial quality checks, the pixels contaminated by large amounts of cloud liquid water, heavy precipitation, and sea ice have been removed. Channels for assimilation, rejection, or monitoring have been respectively selected for different surface types since the errors from the skin temperature are caused by inaccurate surface emissivity. Correcting the bias caused by errors in the instruments and radiative transfer model is crucial in radiance data pre-processing. We have developed bias correction modules in two steps based on 30-day innovation statistics (observed radiance minus background; O-B). The scan bias correction has been calculated individually for each channel, satellite, and scan position. Then a multiple linear regression of the scan-bias-corrected innovations with several predictors has been employed to correct the airmass bias.