• Title/Summary/Keyword: data pre-processing

Search Result 809, Processing Time 0.027 seconds

Energy Big Data Pre-processing System for Energy New Industries (에너지신산업을 위한 에너지 빅데이터 전처리 시스템)

  • Yang, Soo-Young;Kim, Yo-Han;Kim, Sang-Hyun;Kim, Won-Jung
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.16 no.5
    • /
    • pp.851-858
    • /
    • 2021
  • Due to the increase in renewable energy and distributed resources, not only traditional data but also various energy-related data are being generated in the new energy industry. In other words, there are various renewable energy facilities and power generation data, system operation data, metering and rate-related data, as well as weather and energy efficiency data necessary for new services and analysis. Energy big data processing technology can systematically analyze and diagnose data generated in the first half of the power production and consumption infrastructure, including distributed resources, systems, and AMI. Through this, it will be a technology that supports the creation of new businesses in convergence between the ICT industry and the energy industry. To this end, research on the data analysis system, such as itemized characteristic analysis of the collected data, correlation sampling, categorization of each feature, and element definition, is needed. In addition, research on data purification technology for data loss and abnormal state processing should be conducted. In addition, it is necessary to develop and structure NIFI, Spark, and HDFS systems so that energy data can be stored and managed in real time. In this study, the overall energy data processing technology and system for various power transactions as described above were proposed.

Study on the Big Data Platform Construction of Fisheries (수산업 빅데이터 플랫폼 구축 방안에 대한 연구)

  • Choi, Joowon;Jung, Jaewook;Kim, Youngae;Shin, Yongtae
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.8
    • /
    • pp.181-188
    • /
    • 2020
  • The fisheries industry is rapidly shifting from a traditional fishery to aquaculture paradigm and it faces various problems such as depletion of fishery resources and aging of fishing villages. We need the establishment of a fisheries big data platform that includes both the data of the central and surrounding industries of the fisheries industry for enhancement of establishment of a fisheries, 6th industrialization of fishing villages, establishment of related technical standards, and discovery of the new industries to overcome this. Data center agencies should collect, link, and pre-processing, and the platform organizer should create a water industry data virtuous circle through the establishment, operation, and data market of big data platforms to help overcome the current crisis, secure smart fisheries hegemony, and use it as a key to value transfer. Through this study, I would like to propose a policy and technical big data platform construction plan to successfully promote it.

Artificial Intelligence-based Classification Scheme to improve Time Series Data Accuracy of IoT Sensors (IoT 센서의 시계열 데이터 정확도 향상을 위한 인공지능 기반 분류 기법)

  • Kim, Jin-Young;Sim, Isaac;Yoon, Sung-Hoon
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.21 no.4
    • /
    • pp.57-62
    • /
    • 2021
  • As the parallel computing capability for artificial intelligence improves, the field of artificial intelligence technology is expanding in various industries. In particular, artificial intelligence is being introduced to process data generated from IoT sensors that have enoumous data. However, the limitation exists when applying the AI techniques on IoT network because IoT has time series data, where the importance of data changes over time. In this paper, we propose time-weighted and user-state based artificial intelligence processing techniques to effectively process IoT sensor data. This technique aims to effectively classify IoT sensor data through a data pre-processing process that personalizes time series data and places a weight on the time series data before artificial intelligence learning and use status of personal data. Based on the research, it is possible to propose a method of applying artificial intelligence learning in various fields.

Recognition of Individual Holstein Cattle by Imaging Body Patterns

  • Kim, Hyeon T.;Choi, Hong L.;Lee, Dae W.;Yoon, Yong C.
    • Asian-Australasian Journal of Animal Sciences
    • /
    • v.18 no.8
    • /
    • pp.1194-1198
    • /
    • 2005
  • A computer vision system was designed and validated to recognize an individual Holstein cattle by processing images of their body patterns. This system involves image capture, image pre-processing, algorithm processing, and an artificial neural network recognition algorithm. Optimum management of individuals is one of the most important factors in keeping cattle healthy and productive. In this study, an image-processing system was used to recognize individual Holstein cattle by identifying the body-pattern images captured by a charge-coupled device (CCD). A recognition system was developed and applied to acquire images of 49 cattles. The pixel values of the body images were transformed into input data comprising binary signals for the neural network. Images of the 49 cattle were analyzed to learn input layer elements, and ten cattles were used to verify the output layer elements in the neural network by using an individual recognition program. The system proved to be reliable for the individual recognition of cattles in natural light.

Depth Image Distortion Correction Method according to the Position and Angle of Depth Sensor and Its Hardware Implementation (거리 측정 센서의 위치와 각도에 따른 깊이 영상 왜곡 보정 방법 및 하드웨어 구현)

  • Jang, Kyounghoon;Cho, Hosang;Kim, Geun-Jun;Kang, Bongsoon
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.18 no.5
    • /
    • pp.1103-1109
    • /
    • 2014
  • The motion recognition system has been broadly studied in digital image and video processing fields. Recently, method using th depth image is used very useful. However, recognition accuracy of depth image based method will be loss caused by size and shape of object distorted for angle of the depth sensor. Therefore, distortion correction of depth sensor is positively necessary for distinguished performance of the recognition system. In this paper, we propose a pre-processing algorithm to improve the motion recognition system. Depth data from depth sensor converted to real world, performed the corrected angle, and then inverse converted to projective world. The proposed system make progress using the OpenCV and the window program, and we test a system using the Kinect in real time. In addition, designed using Verilog-HDL and verified through the Zynq-7000 FPGA Board of Xilinx.

System Implementation of Paper Currency Discrimination by Using Integrated Image Features (통합 영상 특징에 의한 지폐 분류 시스템의 구현)

  • Gang, Hyeon-In;Choe, Tae-Wan
    • The KIPS Transactions:PartB
    • /
    • v.9B no.4
    • /
    • pp.471-480
    • /
    • 2002
  • In this paper, we implemented a real-time system improving the performance of the paper currency discrimination by integrating a weighted region of interest matching algorithm with a weighted shape feature matching algorithm of the blocked image. The system classifies the paper currency by comparing a query image with compared images based on the database that contain images of paper currency. Especially, the system has good efficiency at the contaminated, rotated, and translated paper currency. The system hardware consists of three parts as follows : the paper currency image acquired by CIS(contact image sensor) is applied to the pre-processing part with A/D converter and PLD. Finally the pre-processed image data are classified by the main image processing part with a high-speed DSP based on the proposed algorithm.

Pre-processing and Bias Correction for AMSU-A Radiance Data Based on Statistical Methods (통계적 방법에 근거한 AMSU-A 복사자료의 전처리 및 편향보정)

  • Lee, Sihye;Kim, Sangil;Chun, Hyoung-Wook;Kim, Ju-Hye;Kang, Jeon-Ho
    • Atmosphere
    • /
    • v.24 no.4
    • /
    • pp.491-502
    • /
    • 2014
  • As a part of the KIAPS (Korea Institute of Atmospheric Prediction Systems) Package for Observation Processing (KPOP), we have developed the modules for Advanced Microwave Sounding Unit-A (AMSU-A) pre-processing and its bias correction. The KPOP system calculates the airmass bias correction coefficients via the method of multiple linear regression in which the scan-corrected innovation and the thicknesses of 850~300, 200~50, 50~5, and 10~1 hPa are respectively used for dependent and independent variables. Among the four airmass predictors, the multicollinearity has been shown by the Variance Inflation Factor (VIF) that quantifies the severity of multicollinearity in a least square regression. To resolve the multicollinearity, we adopted simple linear regression and Principal Component Regression (PCR) to calculate the airmass bias correction coefficients and compared the results with those from the multiple linear regression. The analysis shows that the order of performances is multiple linear, principal component, and simple linear regressions. For bias correction for the AMSU-A channel 4 which is the most sensitive to the lower troposphere, the multiple linear regression with all four airmass predictors is superior to the simple linear regression with one airmass predictor of 850~300 hPa. The results of PCR with 95% accumulated variances accounted for eigenvalues showed the similar results of the multiple linear regression.

Web based CFD Simulation Service Improvement and Utilization (웹기반 열유체 시뮬레이션 서비스의 개선 및 활용)

  • Jung, Young Jin;Jin, Du-Seok
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.17 no.5
    • /
    • pp.1160-1167
    • /
    • 2013
  • Web based simulation service is utilized to computationally analyze various phenomena in real world according to the progress of network and computing technology. In this paper, we present an improvement and utilization of e-AIRS (e-Science Aerospace Integrated Research System). e-AIRS, has been utilized to support web based CFD simulation service since 2008. has some problems such as stable system, pre processing, post processing. To solver this problem, we improved e-AIRS such as distributed service processing, personal simulation job assignment control, and faster data loading. After improvement, although users increase from 110 to 606, the priority of user requirements is changed from stable system to pre/post processor. User requirements and statistics about e-AIRS simulation service for each semester is analyzed to support more stable and comfortable service.

A Case Study on the Data Processing to Enhance the Resolution of Chirp SBP Data (Chirp SBP 자료 해상도 향상을 위한 전산처리연구)

  • Kim, Young-Jun;Kim, Won-Sik;Shin, Sung-Ryul;Kim, Jin-Ho
    • Geophysics and Geophysical Exploration
    • /
    • v.14 no.4
    • /
    • pp.289-297
    • /
    • 2011
  • Chirp sub-bottom profilers (SBP) data are comparatively higher-resolution data than other seismic data and it's raw signal can be used as a final section after conducting basic filtering. However, Chirp SBP signal has possibility to include various noise in high-frequency band and to provide the distorted image for the complex geological structure in time domain. This study aims at the goal to establish the workflow of Chirp SBP data processing for enhanced image and to analyze the proper parameters for the domestic continental shelf. After pre-processing, we include the dynamic S/N filtering to eliminate the high-frequency component noise, the dip scan stack to enhance the continuity of reflection events and finally the post-stack depth migration to correct the distorted structure on the time domain sections. We demonstrated our workflow on the data acquired by domestically widely used equipments and then we could obtain the improved seismic sections of depth domain. This workflow seems to provide the proper seismic section to interpretation when applied to data processing of Chirp SBP that are largely used for domestic acquisition.

Pre-evaluation of Non-alcoholic Fatty Liver Disease Model Using Micro-MRI: For Big Data Application (비알콜성 간 질환 동물모델 영상 빅 데이터 구축을 위한 영상데이터 수집 및 사전평가)

  • Lee, Gi-Taek;Jun, Hong Young;Kim, Tae-Hoon;Jang, Mi Yeon;Kim, Dae Won;Yoon, Kwon-Ha
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2017.04a
    • /
    • pp.982-983
    • /
    • 2017
  • 본 연구는 최근 문제가 되고 있는 비알콜성 간 질환에 대한 빅 데이터의 사전 데이터를 만들기 위해 마우스에서 고지방 식이와 Streptozotocin ((STZ)로 모델을 제작하였고, 당뇨와 비만 정도를 측정하여 질환발생 정도를 확인하였다. 또한, MR영상의 지속적인 촬용으로 질환발생과정에 대해 3D분석 소프트웨어로 평가되었다.