• Title/Summary/Keyword: Data Processing Time

Search Result 5,280, Processing Time 0.031 seconds

Implement of MapReduce-based Big Data Processing Scheme for Reducing Big Data Processing Delay Time and Store Data (빅데이터 처리시간 감소와 저장 효율성이 향상을 위한 맵리듀스 기반 빅데이터 처리 기법 구현)

  • Lee, Hyeopgeon;Kim, Young-Woon;Kim, Ki-Young
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.10
    • /
    • pp.13-19
    • /
    • 2018
  • MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

An Efficient data management Scheme for Hierarchical Multi-processing using Double Hash Chain (이중 해쉬체인을 이용한 계층적 다중 처리를 위한 효율적인 데이터 관리 기법)

  • Jeong, Yoon-Su;Kim, Yong-Tae;Park, Gil-Cheol
    • Journal of Digital Convergence
    • /
    • v.13 no.10
    • /
    • pp.271-278
    • /
    • 2015
  • Recently, bit data is difficult to easily collect the desired data because big data is collected via the Internet. Big data is higher than the rate at which the data type and the period of time for which data is collected depending on the size of data increases. In particular, since the data of all different by the intended use and the type of data processing accuracy and computational cost is one of the important items. In this paper, we propose data processing method using a dual-chain in a manner to minimize the computational cost of the data when data is correctly extracted at the same time a multi-layered process through the desired number of the user and different kinds of data on the Internet. The proposed scheme is classified into a hierarchical data in accordance with the intended use and method to extract various kinds of data. At this time, multi-processing and tie the data hash with the double chain to enhance the accuracy of the reading. In addition, the proposed method is to organize the data in the hash chain for easy access to the hierarchically classified data and reduced the cost of processing the data. Experimental results, the proposed method is the accuracy of the data on average 7.8% higher than conventional techniques, processing costs were reduced by 4.9% of the data.

Software-based Real-time GNSS Signal Generation and Processing Using a Graphic Processing Unit (GPU)

  • Im, Sung-Hyuck;Jee, Gyu-In
    • Journal of Positioning, Navigation, and Timing
    • /
    • v.3 no.3
    • /
    • pp.99-105
    • /
    • 2014
  • A graphic processing unit (GPU) can perform the same calculation on multiple data (SIMD: single instruction multiple data) using hundreds of to thousands of special purpose processors for graphic processing. Thus, high efficiency is expected when GPU is used for the generation and correlation of satellite navigation signals, which perform generation and processing by applying the same calculation procedure to tens of millions of discrete signal samples per second. In this study, the structure of a GPU-based GNSS simulator for the generation and processing of satellite navigation signals was designed, developed, and verified. To verify the developed satellite navigation signal generator, generated signals were applied to the OEM-V3 receiver of Novatel Inc., and the measured values were examined. To verify the satellite navigation signal processor, the performance was examined by collecting and processing actual GNSS intermediate frequency signals. The results of the verification indicated that satellite navigation signals could be generated and processed in real time using two GPUs.

Spatio-temporal Query Processing Systems for Ubiquitous Environments (유비쿼터스 환경을 위한 시공간 질의 처리 시스템)

  • Lee, Ki-Young;Lim, Myung-Jae;Kim, Kyu-Ho;Kim, Joung-Joon
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.10 no.3
    • /
    • pp.145-152
    • /
    • 2010
  • With the recent development of the ubiquitous computing technology, there are increasing interest and research in technologies such as sensors and RFID related to information recognition and location positioning in various ubiquitous fields. Especially, RTLS(Real-Time Locating Services) dealing with spatio-temporal data is emerging as a promising technology. For these reasons, the ISO/IEC published the RTLS standard specification for compatibility and interoperability in RTLS. Therefore, in this paper, we designed and implemented Spatio-temporal Query Processing Systems for efficiently managing and searching the incoming Spatio-temporal data stream of moving objects. Spatio-temporal Query Processing Systems's spatio-temporal middleware maintains interoperability among heterogeneous devices and guarantees data integrity in query processing through real time processing of unceasing spatio-temporal data streams and two way synchronization of spatio-temporal DBMSs. Web Server uses the SOAP(Simple Object Access Protocol) message between client and server for interoperability and translates client's SOAP message into CQL(Continuous Query Language) of the spatio-temporal middleware. Finally, this thesis proved the utility of the system by applying the spatio-temporal Query Processing Systems to a real-time Locating Services.

Spatiotemporal Data Model and Extension of their Operations for a Layered Temporal Geographic Information System (계층적 시간지원 지리정보 시스템을 위한 시공간 데이터 모델과 그 연산자 확장)

  • Kim, Dong-Ho;Lee, Jong-Yun;Joo, Young-Do;Ryu, Keun-Ho
    • The Transactions of the Korea Information Processing Society
    • /
    • v.5 no.5
    • /
    • pp.1083-1097
    • /
    • 1998
  • The conventional geographic information systems(GIS) is a software which handles spatial and aspatial information of objects in the real world. The system can not support users time-varying information because it manipulates their snapshot data in the spatial database. Also even though it supports time-varying information, it is very limited and hs many difficulties in presenting and processing queries. This paper therefore describes an integrated spatiotemporal data model using loosely-coupled approach which is extended a time dimension for the previous spatial database and which handles time-varying historical information of spatial objects. Conclusionally this paper not only designed a data structure for spatiotemporal database, but also implemented spatial comparison operations varying over time.

  • PDF

Instance-Level Subsequence Matching Method based on a Virtual Window (가상 윈도우 기반 인스턴스 레벨 서브시퀀스 매칭 방안)

  • Ihm, Sun-Young;Park, Young-Ho
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.3 no.2
    • /
    • pp.43-46
    • /
    • 2014
  • A time-series data is the collection of real numbers over the time intervals. One of the main tasks in time-series data is efficiently to find subsequences similar to a given query sequence. In this paper, we propose an efficient subsequence matching method, which is called Instance-Match (I-Match). I-Match constructs a virtual window in order to reduce false alarms. Through the experiment with real data set and query sets, we show that I-Match improves query processing time by up to 2.95 times and significantly reduces the number of candidates comparing to Dual Match.

Plurality Rule-based Density and Correlation Coefficient-based Clustering for K-NN

  • Aung, Swe Swe;Nagayama, Itaru;Tamaki, Shiro
    • IEIE Transactions on Smart Processing and Computing
    • /
    • v.6 no.3
    • /
    • pp.183-192
    • /
    • 2017
  • k-nearest neighbor (K-NN) is a well-known classification algorithm, being feature space-based on nearest-neighbor training examples in machine learning. However, K-NN, as we know, is a lazy learning method. Therefore, if a K-NN-based system very much depends on a huge amount of history data to achieve an accurate prediction result for a particular task, it gradually faces a processing-time performance-degradation problem. We have noticed that many researchers usually contemplate only classification accuracy. But estimation speed also plays an essential role in real-time prediction systems. To compensate for this weakness, this paper proposes correlation coefficient-based clustering (CCC) aimed at upgrading the performance of K-NN by leveraging processing-time speed and plurality rule-based density (PRD) to improve estimation accuracy. For experiments, we used real datasets (on breast cancer, breast tissue, heart, and the iris) from the University of California, Irvine (UCI) machine learning repository. Moreover, real traffic data collected from Ojana Junction, Route 58, Okinawa, Japan, was also utilized to lay bare the efficiency of this method. By using these datasets, we proved better processing-time performance with the new approach by comparing it with classical K-NN. Besides, via experiments on real-world datasets, we compared the prediction accuracy of our approach with density peaks clustering based on K-NN and principal component analysis (DPC-KNN-PCA).

Fault Detection in the Semiconductor Etch Process Using the Seasonal Autoregressive Integrated Moving Average Modeling

  • Arshad, Muhammad Zeeshan;Nawaz, Javeria Muhammad;Hong, Sang Jeen
    • Journal of Information Processing Systems
    • /
    • v.10 no.3
    • /
    • pp.429-442
    • /
    • 2014
  • In this paper, we investigated the use of seasonal autoregressive integrated moving average (SARIMA) time series models for fault detection in semiconductor etch equipment data. The derivative dynamic time warping algorithm was employed for the synchronization of data. The models were generated using a set of data from healthy runs, and the established models were compared with the experimental runs to find the faulty runs. It has been shown that the SARIMA modeling for this data can detect faults in the etch tool data from the semiconductor industry with an accuracy of 80% and 90% using the parameter-wise error computation and the step-wise error computation, respectively. We found that SARIMA is useful to detect incipient faults in semiconductor fabrication.

An event-based temporal reasoning method (사건 기반 시간 추론 기법)

  • 이종현;이민석;우영운;박충식;김재희
    • Journal of the Korean Institute of Telematics and Electronics C
    • /
    • v.34C no.5
    • /
    • pp.93-102
    • /
    • 1997
  • Conventional expert systems have difficulties in the inference on time-varing situations because they don't have the structure for processing time related informations and rule representation method to describe time explicitely. Some expert systems capable of temporal reasoning are not applicable to the domain in which state changes happen by unpredictble events that cannot be represented by periodic changes of data. In this paper, an event based temporal reasoning method is proposed. It is capable of processing te unpredictable events, representing the knowledge related to event and time, and infering by that knowledge as well as infering by periodically time-varing data. The NEO/temporal, an temporal inference engine, is implemented by applying the proposed temporal reasoning situation assessment and decision supporting system is implemented to show the benefits of the proposed temporal information processing model.

  • PDF

Implementation of Slaving Data Processing Function for Mission Control System in Space Center (우주센터 발사통제시스템의 추적연동정보 처리기능 구현)

  • Choi, Yong-Tae;Ra, Sung-Woong
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.19 no.3
    • /
    • pp.31-39
    • /
    • 2014
  • In KSLV-I launch mission, real-time data from the tracking stations are acquired, processed and distributed by the Mission Control System to the user group who needed to monitor processed data for safety and flight monitoring purposes. The processed trajectory data by the mission control system is sent to each tracking system for target designation in case of tracking failure. Also, the processed data are used for decision making for flight termination when anomalies occur during flight of the launch vehicle. In this paper, we propose the processing mechanism of slaving data which plays a key role of launch vehicle tracking mission. The best position data is selected by predefined logic and current status after every available position data are acquired and pre-processed. And, the slaving data is distributed to each tracking stations through time delay is compensated by extrapolation. For the accurate processing, operation timing of every procesing modules are triggered by time-tick signal(25ms period) which is driven from UTC(Universial Time Coordinates) time. To evaluate the proposed method, we compared slaving data to the position data which received by tracking radar. The experiments show the average difference value is below 0.01 degree.