• Title/Summary/Keyword: Very large real-time data

Search Result 130, Processing Time 0.03 seconds

Real-Time Hybrid Broadcasting Algorithm Considering Data Property in Mobile Computing Environments (이동 컴퓨팅 환경에서 데이타 특성을 고려한 실시간 혼성 방송 알고리즘)

  • Yoon Hyesook;Kim Young-Kuk
    • Journal of KIISE:Information Networking
    • /
    • v.32 no.3
    • /
    • pp.339-349
    • /
    • 2005
  • For recent years, data broadcast technology has been recognized as a very effective data delivery mechanism in mobile computing environment with a large number of cli;ents. Especially, a hybrid broadcast algorithm in real-time environment, which integrates one-way broadcast and on-demand broadcast, has an advantage of adapting the requests of clients to a limited up-link bandwidth and following the change of data access pattern. However, previous hybrid broadcasting algorithms has a problem in the methods to get a grip on the change of data access Pattern. It is caused by the diminution of requests for the data items which are contained in periodic broadcasting schedule because they are already broadcasted. To solve this problem, existing researches may remove data items in periodic broadcasting schedule over a few cycles multiplying cooling factor or find out the requests of data items with extracting them on purpose. Both of them are the artificial methods not considering the property of data. In this paper, we propose a real-time adaptive hybrid broadcasting based on data type(RTAHB-DT) to broadcast considering data property and analysis the performance of our aigorithm through simulation study.

A Subsequence Matching Technique that Supports Time Warping Efficiently (타임 워핑을 지원하는 효율적인 서브시퀀스 매칭 기법)

  • Park, Sang-Hyun;Kim, Sang-Wook;Cho, June-Suh;Lee, Hoen-Gil
    • Journal of Industrial Technology
    • /
    • v.21 no.A
    • /
    • pp.167-179
    • /
    • 2001
  • This paper discusses an index-based subsequence matching that supports time warping in large sequence databases. Time warping enables finding sequences with similar patterns even when they are of different lengths. In earlier work, we suggested an efficient method for whole matching under time warping. This method constructs a multidimensional index on a set of feature vectors, which are invariant to time warping, from data sequences. For filtering at feature space, it also applies a lower-bound function, which consistently underestimates the time warping distance as well as satisfies the triangular inequality. In this paper, we incorporate the prefix-querying approach based on sliding windows into the earlier approach. For indexing, we extract a feature vector from every subsequence inside a sliding window and construct a multi-dimensional index using a feature vector as indexing attributes. For query precessing, we perform a series of index searches using the feature vectors of qualifying query prefixes. Our approach provides effective and scalable subsequence matching even with a large volume of a database. We also prove that our approach does not incur false dismissal. To verily the superiority of our method, we perform extensive experiments. The results reseal that our method achieves significant speedup with real-world S&P 500 stock data and with very large synthetic data.

  • PDF

Classification of large-scale data and data batch stream with forward stagewise algorithm (전진적 단계 알고리즘을 이용한 대용량 데이터와 순차적 배치 데이터의 분류)

  • Yoon, Young Joo
    • Journal of the Korean Data and Information Science Society
    • /
    • v.25 no.6
    • /
    • pp.1283-1291
    • /
    • 2014
  • In this paper, we propose forward stagewise algorithm when data are very large or coming in batches sequentially over time. In this situation, ordinary boosting algorithm for large scale data and data batch stream may be greedy and have worse performance with class noise situations. To overcome those and apply to large scale data or data batch stream, we modify the forward stagewise algorithm. This algorithm has better results for both large scale data and data batch stream with or without concept drift on simulated data and real data sets than boosting algorithms.

Design of a Disaster Big Data Platform for Collecting and Analyzing Social Media (소셜미디어 수집과 분석을 위한 재난 빅 데이터 플랫폼의 설계)

  • Nguyen, Van-Quyet;Nguyen, Sinh-Ngoc;Nguyen, Giang-Truong;Kim, Kyungbaek
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2017.04a
    • /
    • pp.661-664
    • /
    • 2017
  • Recently, during disasters occurrence, dealing with emergencies has been handled well by the early transmission of disaster relating notifications on social media networks (e.g., Twitter or Facebook). Intuitively, with their characteristics (e.g., real-time, mobility) and big communities whose users could be regarded as volunteers, social networks are proved to be a crucial role for disasters response. However, the amount of data transmitted during disasters is an obstacle for filtering informative messages; because the messages are diversity, large and very noise. This large volume of data could be seen as Social Big Data (SBD). In this paper, we proposed a big data platform for collecting and analyzing disasters' data from SBD. Firstly, we designed a collecting module; which could rapidly extract disasters' information from the Twitter; by big data frameworks supporting streaming data on distributed system; such as Kafka and Spark. Secondly, we developed an analyzing module which learned from SBD to distinguish the useful information from the irrelevant one. Finally, we also designed a real-time visualization on the web interface for displaying the results of analysis phase. To show the viability of our platform, we conducted experiments of the collecting and analyzing phases in 10 days for both real-time and historical tweets, which were about disasters happened in South Korea. The results prove that our big data platform could be applied to disaster information based systems, by providing a huge relevant data; which can be used for inferring affected regions and victims in disaster situations, from 21.000 collected tweets.

Efficient Processing of Continuous Join Queries between a Data Stream and Multiple Relations for Real-Time Analysis of E-Commerce Data (전자상거래 데이터의 실시간 분석을 위한 데이터 스트림과 다수 릴레이션 간의 효율적인 연속 조인 처리 기법)

  • Kim, Haeri;Lee, Ki Yong
    • The Journal of Society for e-Business Studies
    • /
    • v.18 no.3
    • /
    • pp.159-175
    • /
    • 2013
  • Recently, as real-time availability of e-commerce data becomes possible, the requirement of real-time analysis of e-commerce increases significantly. In the real-time analysis of e-commerce data, it is very important to efficiently process continuous join queries between an e-commerce data stream and disk-based large relations. In this paper, we propose an efficient method for processing a continuous join query between an e-commerce data stream and multiple disk-based relations. The proposed method improves the service rate significantly, while reducing the amount of required memory substantially. Through analysis and various experiments, we show the efficiency of the proposed method compared with the previous one in terms of service rate and memory usage.

Web-based Application Service Management System for Fault Monitoring

  • Min, Sang-Cheol;Chung, Tai-Myoung;Park, Hyoung-Woo;Lee, Kyung-Ha;Pang, Kee-Hong
    • Journal of Electrical Engineering and information Science
    • /
    • v.2 no.6
    • /
    • pp.64-73
    • /
    • 1997
  • Network technology has been developed for very high-speed networking and multimedia data whose characteristics are the continuous and bursty transmission as well as a large amount of data. With this trend users wish to view the information about the application services as well as network devices and system hardware. However, it is rarely available for the users the information of performance or faults of the application services. Most of information is limited to the information related network devices or system hardware. Furthermore, users expect the best services without knowing the service environments in the network and there is no good way of delivering the service related problems and fault information of application services in a high speed network yet. In this paper we present a web-based application management system that we have developed for the past year. It includes a method to build an agent system that uses an existing network management standards, SNMP MIB and SNMP protocols. The user interface of the system is also developed to support visualization effects with web-based Java interface which offers a convenient way not only to access management information but also to control networked applications.

  • PDF

Smart Roll Forming Based on Real-Time Process Data (실시간 공정데이터 기반의 스마트 롤포밍에 관한 연구)

  • Son, Jae-Hwan;Cho, Dong-Hyun;Kim, Chul-Hong
    • Journal of the Korean Society of Manufacturing Process Engineers
    • /
    • v.17 no.5
    • /
    • pp.45-51
    • /
    • 2018
  • Roll forming refers to the production of long plate-molded products, such as panels, pipes, tubes, channels, and frames, by continuously causing the bending deformation to thin plates using rotating rolls. As the roll forming method has advantages in terms of mass production because of its excellent productivity, the size of the roll forming industry has been continuously increasing and the roll forming method is increasingly being used in diverse industrial fields as a very important processing method. Furthermore, as the roll forming method mainly depends on the continuous bending deformation of the plate materials, the time and the cost of the heterogeneous materials developed in the process are relatively large when considered from the viewpoint of plastic working because many processes are continuously implemented. The existing studies on roll forming manufacturing have reported the loss of large amounts of time and materials when the raw materials or product types were changed; further, they have stated that the use of this method can hardly guarantee the uniformity of the formed shapes and the consistency in terms of size and cannot detect all the defects occurring during the mass production and related to the dimensions. Therefore, in this research, a real-time process data-based smart roll forming method that can be applied to multiple products was studied. As a result, a roll forming system was implemented that remembers and automatically sets the changes in the finely adjusted values of the supplied quantities of individual heterogeneous materials so that the equipment setting changing time for heterogeneous material replacements or changes in the products being produced can be shortened. It also secures the uniformity of the products so that more competitive and precise slide-rail products can be mass-produced with improvements in the quality, price, and productivity of the products.

Compression and Visualization Techniques for Time-Varying Volume Data (시변 볼륨 데이터의 압축과 가시화 기법)

  • Sohn, Bong-Soo
    • Journal of the Korea Society of Computer and Information
    • /
    • v.12 no.3
    • /
    • pp.85-93
    • /
    • 2007
  • This paper describes a compression scheme for volumetric video data(3D space X 1D time) there each frame of the volume is decompressed and rendered in real-time. Since even one frame size of volume is very large, runtime decompression can be a bottleneck for real-time playback of time-varying volume data. To increase the run-time decompression speed and compression ratio, we decompose the volume into small blocks and only update significantly changing blocks. The results show that our compression scheme compromises decompression speed and image quality well enough for interactive time-varying visualization.

  • PDF

Development of Precision Vision Inspection System for Micro Optical Parts using a New Optical Probe Implemented to have Multiple Fields of Views (다중광학창을 가진 광학소자 자동 검사 시스템 개발)

  • 이일환;이기수;박희재
    • Proceedings of the Korean Society of Precision Engineering Conference
    • /
    • 2001.04a
    • /
    • pp.105-109
    • /
    • 2001
  • The micro optical parts such as ferrules are required to be manufactured within very small tolerances, as the slight deviation of the tolerance would give very large amount of loss in communication efficiency. For efficient optical communication, outer diameter, fiber diameter, fiber separation and eccentricity are significant parameters to be inspected., Thus we developed an automatic inspection system to evaluate shape parameters of the optical fiber connectors(ferrule) upto submicron accuracy using machine vision. new optical probe of multi fields of views has been developed and the image processing and data analysis algorithms have been complemented in real time basis. The developed system is successfully used in the practical ferrule manufacturing industry, and about 0.1$\mu\textrm{m}$ accuracy can be obtained with very fast inspection time.

  • PDF

Product Recommendation System on VLDB using k-means Clustering and Sequential Pattern Technique (k-means 클러스터링과 순차 패턴 기법을 이용한 VLDB 기반의 상품 추천시스템)

  • Shim, Jang-Sup;Woo, Seon-Mi;Lee, Dong-Ha;Kim, Yong-Sung;Chung, Soon-Key
    • The KIPS Transactions:PartD
    • /
    • v.13D no.7 s.110
    • /
    • pp.1027-1038
    • /
    • 2006
  • There are many technical problems in the recommendation system based on very large database(VLDB). So, it is necessary to study the recommendation system' structure and the data-mining technique suitable for the large scale Internet shopping mail. Thus we design and implement the product recommendation system using k-means clustering algorithm and sequential pattern technique which can be used in large scale Internet shopping mall. This paper processes user information by batch processing, defines the various categories by hierarchical structure, and uses a sequential pattern mining technique for the search engine. For predictive modeling and experiment, we use the real data(user's interest and preference of given category) extracted from log file of the major Internet shopping mall in Korea during 30 days. And we define PRP(Predictive Recommend Precision), PRR(Predictive Recommend Recall), and PF1(Predictive Factor One-measure) for evaluation. In the result of experiments, the best recommendation time and the best learning time of our system are much as O(N) and the values of measures are very excellent.