• Title/Summary/Keyword: real-time preprocessing

Search Result 195, Processing Time 0.027 seconds

A Design on Informal Big Data Topic Extraction System Based on Spark Framework (Spark 프레임워크 기반 비정형 빅데이터 토픽 추출 시스템 설계)

  • Park, Kiejin
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.11
    • /
    • pp.521-526
    • /
    • 2016
  • As on-line informal text data have massive in its volume and have unstructured characteristics in nature, there are limitations in applying traditional relational data model technologies for data storage and data analysis jobs. Moreover, using dynamically generating massive social data, social user's real-time reaction analysis tasks is hard to accomplish. In the paper, to capture easily the semantics of massive and informal on-line documents with unsupervised learning mechanism, we design and implement automatic topic extraction systems according to the mass of the words that consists a document. The input data set to the proposed system are generated first, using N-gram algorithm to build multiple words to capture the meaning of the sentences precisely, and Hadoop and Spark (In-memory distributed computing framework) are adopted to run topic model. In the experiment phases, TB level input data are processed for data preprocessing and proposed topic extraction steps are applied. We conclude that the proposed system shows good performance in extracting meaningful topics in time as the intermediate results come from main memories directly instead of an HDD reading.

A semi-supervised interpretable machine learning framework for sensor fault detection

  • Martakis, Panagiotis;Movsessian, Artur;Reuland, Yves;Pai, Sai G.S.;Quqa, Said;Cava, David Garcia;Tcherniak, Dmitri;Chatzi, Eleni
    • Smart Structures and Systems
    • /
    • v.29 no.1
    • /
    • pp.251-266
    • /
    • 2022
  • Structural Health Monitoring (SHM) of critical infrastructure comprises a major pillar of maintenance management, shielding public safety and economic sustainability. Although SHM is usually associated with data-driven metrics and thresholds, expert judgement is essential, especially in cases where erroneous predictions can bear casualties or substantial economic loss. Considering that visual inspections are time consuming and potentially subjective, artificial-intelligence tools may be leveraged in order to minimize the inspection effort and provide objective outcomes. In this context, timely detection of sensor malfunctioning is crucial in preventing inaccurate assessment and false alarms. The present work introduces a sensor-fault detection and interpretation framework, based on the well-established support-vector machine scheme for anomaly detection, combined with a coalitional game-theory approach. The proposed framework is implemented in two datasets, provided along the 1st International Project Competition for Structural Health Monitoring (IPC-SHM 2020), comprising acceleration and cable-load measurements from two real cable-stayed bridges. The results demonstrate good predictive performance and highlight the potential for seamless adaption of the algorithm to intrinsically different data domains. For the first time, the term "decision trajectories", originating from the field of cognitive sciences, is introduced and applied in the context of SHM. This provides an intuitive and comprehensive illustration of the impact of individual features, along with an elaboration on feature dependencies that drive individual model predictions. Overall, the proposed framework provides an easy-to-train, application-agnostic and interpretable anomaly detector, which can be integrated into the preprocessing part of various SHM and condition-monitoring applications, offering a first screening of the sensor health prior to further analysis.

THE NONDESTRUCTIVE MEASUREMENT OF THE SOLUBLE SOLID AND ACID CONTENTS OF INTACT PEACH USING VIS/NIR TRANSMITTANCE SPECTRA

  • Hwang, I.G.;Noh, S.H.;Lee, H.Y.;Yang, S.B.
    • Proceedings of the Korean Society for Agricultural Machinery Conference
    • /
    • 2000.11b
    • /
    • pp.210-218
    • /
    • 2000
  • Since the SSC(soluble solid contents) and titratable acidity of fruit are highly concerned to the taste, the need for measuring them by non-destructive technology such as NIR(Visual and Near-infrared) spectroscopy is increasing. Specially, in order to grade the quality of each fruit with a sorter at sorting and packing facilities, technologies for online measurement satisfying the tolerance in terms of accuracy and speed should be developed. Many researches have been done to develop devices to measure the internal qualities of fruit such as SSC, titratable acidity, firmness, etc. with the VIS(Visual)/NIR(Near Infrared) reflectance spectra. The distributions of the SSC, titratable acidity, firmness, etc. are different with respect to the position and depth of fruit, and generally the VIS/NIR light can interact with fruit in a few millimeters of pathlength, and it is very difficult to measure the qualities of inner flesh of fruit. Therefore, to measure the average concentrations of each quality factor such as SSC and titratable acidity with the reflectance-type NIR devices, the spectra of fruit at several positions should be measured. Recently, the interest about the transmittance-type VIS/NIR devices is increasing. NIR light can penetrate through the fruit about 1/10-1/1,000,000 %. Therefore, very intensive light source and very sensitive sensor should be adopted to measure the transmitted light spectra of intact fruit. The ultimate purpose of this study was to develop a device to measure the transmitted light spectra of intact fruit such as apple, pear, peach, etc. With the transmittance-type VIS/NIR device, the feasibility of measurement of the SSC and titratable acidity in intact fruit cultivated in Korea was tested. The results are summarized as follows; A simple measurement device which can measure the transmitted light spectra of intact fruit was constructed with sample holder, two 500W-tungsten halogen lamps, a real-time spectrometer having a very sensitive CCD array sensor and optical fiber probe. With the device, it was possible to measure the transmitted light spectra of intact fruit such as apple, pear and peach. Main factors affecting the intensity of transmitted light spectra were the size of sample, the radiation intensity of light source and the integration time of the detector. Sample holder should be designed so that direct light leakage to the probe could be protected. Preprocessing method to the raw spectrum data significantly influenced the performance of the nondestructive measurement of SSC and titratable acidity of intact fruit. Representative results of PLS models in predicting the SSC of peach were SEP of 0.558 Brix% and R2 of 0.819, and those in predicting titratable acidity were SEP of 0.056% and R2 of 0.655.

  • PDF

Smart Emotion Management System based on multi-biosignal Analysis using Artificial Intelligence (인공지능을 활용한 다중 생체신호 분석 기반 스마트 감정 관리 시스템)

  • Noh, Ayoung;Kim, Youngjoon;Kim, Hyeong-Su;Kim, Won-Tae
    • Journal of IKEEE
    • /
    • v.21 no.4
    • /
    • pp.397-403
    • /
    • 2017
  • In the modern society, psychological diseases and impulsive crimes due to stress are occurring. In order to reduce the stress, the existing treatment methods consisted of continuous visit counseling to determine the psychological state and prescribe medication or psychotherapy. Although this face-to-face counseling method is effective, it takes much time to determine the state of the patient, and there is a problem of treatment efficiency that is difficult to be continuously managed depending on the individual situation. In this paper, we propose an artificial intelligence emotion management system that emotions of user monitor in real time and induced to a table state. The system measures multiple bio-signals based on the PPG and the GSR sensors, preprocesses the data into appropriate data types, and classifies four typical emotional states such as pleasure, relax, sadness, and horror through the SVM algorithm. We verify that the emotion of the user is guided to a stable state by providing a real-time emotion management service when the classification result is judged to be a negative state such as sadness or fear through experiments.

Understanding Public Opinion by Analyzing Twitter Posts Related to Real Estate Policy (부동산 정책 관련 트위터 게시물 분석을 통한 대중 여론 이해)

  • Kim, Kyuli;Oh, Chanhee;Zhu, Yongjun
    • Journal of the Korean Society for Library and Information Science
    • /
    • v.56 no.3
    • /
    • pp.47-72
    • /
    • 2022
  • This study aims to understand the trends of subjects related to real estate policies and public's emotional opinion on the policies. Two keywords related to real estate policies such as "real estate policy" and "real estate measure" were used to collect tweets created from February 25, 2008 to August 31, 2021. A total of 91,740 tweets were collected and we applied sentiment analysis and dynamic topic modeling to the final preprocessed and categorized data of 18,925 tweets. Sentiment analysis and dynamic topic model analysis were conducted for a total of 18,925 posts after preprocessing data and categorizing them into supply, real estate tax, interest rate, and population variance. Keywords of each category are as follows: the supply categories (rental housing, greenbelt, newlyweds, homeless, supply, reconstruction, sale), real estate tax categories (comprehensive real estate tax, acquisition tax, holding tax, multiple homeowners, speculation), interest rate categories (interest rate), and population variance categories (Sejong, new city). The results of the sentiment analysis showed that one person posted on average one or two positive tweets whereas in the case of negative and neutral tweets, one person posted two or three. In addition, we found that part of people have both positive as well as negative and neutral opinions towards real estate policies. As the results of dynamic topic modeling analysis, negative reactions to real estate speculative forces and unearned income were identified as major negative topics and as for positive topics, expectation on increasing supply of housing and benefits for homeless people who purchase houses were identified. Unlike previous studies, which focused on changes and evaluations of specific real estate policies, this study has academic significance in that it collected posts from Twitter, one of the social media platforms, used emotional analysis, dynamic topic modeling analysis, and identified potential topics and trends of real estate policy over time. The results of the study can help create new policies that take public opinion on real estate policies into consideration.

Research on artificial intelligence based battery analysis and evaluation methods using electric vehicle operation data (전기 차 운행 데이터를 활용한 인공지능 기반의 배터리 분석 및 평가 방법 연구)

  • SeungMo Hong
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.16 no.6
    • /
    • pp.385-391
    • /
    • 2023
  • As the use of electric vehicles has increased to minimize carbon emissions, the analyzing the state and performance of lithium-ion batteries that is instrumental in electric vehicles have been important. Comprehensive analysis using not only the voltage, current and temperature of the battery pack, which can affect the condition and performance of the battery, but also the driving data and charging pattern data of the electric vehicle is required. Therefore, a thorough analysis is imperative, utilizing electric vehicle operation data, charging pattern data, as well as battery pack voltage, current, and temperature data, which collectively influence the condition and performance of the battery. Therefore, collection and preprocessing of battery data collected from electric vehicles, collection and preprocessing of data on driver driving habits in addition to simple battery data, detailed design and modification of artificial intelligence algorithm based on the analyzed influencing factors, and A battery analysis and evaluation model was designed. In this paper, we gathered operational data and battery data from real-time electric buses. These data sets were then utilized to train a Random Forest algorithm. Furthermore, a comprehensive assessment of battery status, operation, and charging patterns was conducted using the explainable Artificial Intelligence (XAI) algorithm. The study identified crucial influencing factors on battery status, including rapid acceleration, rapid deceleration, sudden stops in driving patterns, the number of drives per day in the charging and discharging pattern, daily accumulated Depth of Discharge (DOD), cell voltage differences during discharge, maximum cell temperature, and minimum cell temperature. These factors were confirmed to significantly impact the battery condition. Based on the identified influencing factors, a battery analysis and evaluation model was designed and assessed using the Random Forest algorithm. The results contribute to the understanding of battery health and lay the foundation for effective battery management in electric vehicles.

Research of Water-related Disaster Monitoring Using Satellite Bigdata Based on Google Earth Engine Cloud Computing Platform (구글어스엔진 클라우드 컴퓨팅 플랫폼 기반 위성 빅데이터를 활용한 수재해 모니터링 연구)

  • Park, Jongsoo;Kang, Ki-mook
    • Korean Journal of Remote Sensing
    • /
    • v.38 no.6_3
    • /
    • pp.1761-1775
    • /
    • 2022
  • Due to unpredictable climate change, the frequency of occurrence of water-related disasters and the scale of damage are also continuously increasing. In terms of disaster management, it is essential to identify the damaged area in a wide area and monitor for mid-term and long-term forecasting. In the field of water disasters, research on remote sensing technology using Synthetic Aperture Radar (SAR) satellite images for wide-area monitoring is being actively conducted. Time-series analysis for monitoring requires a complex preprocessing process that collects a large amount of images and considers the noisy radar characteristics, and for this, a considerable amount of time is required. With the recent development of cloud computing technology, many platforms capable of performing spatiotemporal analysis using satellite big data have been proposed. Google Earth Engine (GEE)is a representative platform that provides about 600 satellite data for free and enables semi real time space time analysis based on the analysis preparation data of satellite images. Therefore, in this study, immediate water disaster damage detection and mid to long term time series observation studies were conducted using GEE. Through the Otsu technique, which is mainly used for change detection, changes in river width and flood area due to river flooding were confirmed, centered on the torrential rains that occurred in 2020. In addition, in terms of disaster management, the change trend of the time series waterbody from 2018 to 2022 was confirmed. The short processing time through javascript based coding, and the strength of spatiotemporal analysis and result expression, are expected to enable use in the field of water disasters. In addition, it is expected that the field of application will be expanded through connection with various satellite bigdata in the future.

An Electric Load Forecasting Scheme for University Campus Buildings Using Artificial Neural Network and Support Vector Regression (인공 신경망과 지지 벡터 회귀분석을 이용한 대학 캠퍼스 건물의 전력 사용량 예측 기법)

  • Moon, Jihoon;Jun, Sanghoon;Park, Jinwoong;Choi, Young-Hwan;Hwang, Eenjun
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.5 no.10
    • /
    • pp.293-302
    • /
    • 2016
  • Since the electricity is produced and consumed simultaneously, predicting the electric load and securing affordable electric power are necessary for reliable electric power supply. In particular, a university campus is one of the highest power consuming institutions and tends to have a wide variation of electric load depending on time and environment. For these reasons, an accurate electric load forecasting method that can predict power consumption in real-time is required for efficient power supply and management. Even though various influencing factors of power consumption have been discovered for the educational institutions by analyzing power consumption patterns and usage cases, further studies are required for the quantitative prediction of electric load. In this paper, we build an electric load forecasting model by implementing and evaluating various machine learning algorithms. To do that, we consider three building clusters in a campus and collect their power consumption every 15 minutes for more than one year. In the preprocessing, features are represented by considering periodic characteristic of the data and principal component analysis is performed for the features. In order to train the electric load forecasting model, we employ both artificial neural network and support vector machine. We evaluate the prediction performance of each forecasting model by 5-fold cross-validation and compare the prediction result to real electric load.

CNN-LSTM-based Upper Extremity Rehabilitation Exercise Real-time Monitoring System (CNN-LSTM 기반의 상지 재활운동 실시간 모니터링 시스템)

  • Jae-Jung Kim;Jung-Hyun Kim;Sol Lee;Ji-Yun Seo;Do-Un Jeong
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.24 no.3
    • /
    • pp.134-139
    • /
    • 2023
  • Rehabilitators perform outpatient treatment and daily rehabilitation exercises to recover physical function with the aim of quickly returning to society after surgical treatment. Unlike performing exercises in a hospital with the help of a professional therapist, there are many difficulties in performing rehabilitation exercises by the patient on a daily basis. In this paper, we propose a CNN-LSTM-based upper limb rehabilitation real-time monitoring system so that patients can perform rehabilitation efficiently and with correct posture on a daily basis. The proposed system measures biological signals through shoulder-mounted hardware equipped with EMG and IMU, performs preprocessing and normalization for learning, and uses them as a learning dataset. The implemented model consists of three polling layers of three synthetic stacks for feature detection and two LSTM layers for classification, and we were able to confirm a learning result of 97.44% on the validation data. After that, we conducted a comparative evaluation with the Teachable machine, and as a result of the comparative evaluation, we confirmed that the model was implemented at 93.6% and the Teachable machine at 94.4%, and both models showed similar classification performance.

A Feature-Based Retrieval Technique for Image Database (특징기반 영상 데이터베이스 검색 기법)

  • Kim, Bong-Gi;Oh, Hae-Seok
    • The Transactions of the Korea Information Processing Society
    • /
    • v.5 no.11
    • /
    • pp.2776-2785
    • /
    • 1998
  • An image retrieval system based on image content is a key issue for building and managing large multimedia database, such as art galleries and museums, trademarks and copyrights, and picture archiving and communication system. Therefore, the interest on the subject of content-based image retrieval has been greatly increased for the last few years. This paper proposes a feature-based image retrieval technique which uses a compound feature vector representing both of color and shape of an image. Color information for the feature vector is obtained using the algebraic moment of each pixel of an image based on the property of regional color distribution. Shape information for the feature vector is obtained using the Improved Moment Invariant(IMI) which reduces the quantity of computation and increases retrieval efficiency. In the preprocessing phase for extracting shape feature, we transform a color image into a gray image. Since we make use of the modified DCT algorithm, it is implemented easily and can extract contour in real time. As an experiment, we have compared our method with previous methods using a database consisting of 150 automobile images, and the results of the experiment have shown that our method has the better performance on retrieval effectiveness.

  • PDF