• Title/Summary/Keyword: Big Stream

Search Result 142, Processing Time 0.037 seconds

Change characteristic of basin topographical parameters according to the threshold area of minimum order stream (최소차 하천의 임계면적에 따른 유역 지형매개변수의 변화특성)

  • Ahn Seung-Seop;Park Ro-Sam;Kim Jong-Ho;Lim Ki-Seok;Song Si-Hoon
    • Journal of Environmental Science International
    • /
    • v.14 no.1
    • /
    • pp.33-40
    • /
    • 2005
  • The size of minimum order stream has a very sensitive effect on runoff analysis model using the divergence characteristic of stream. Therefore, in this study, the threshold area of minimum order stream has been examined the change characteristic of topographical parameters. The subject basin of the research was the upper basin of the Kumho water gage station which is located in the middle of the Kumho river. The 1:25,000 numerical geography which was constructed $10{\times}10m$ mesh was used. The range of investigation of topographical parameters are number of stream order, length, area, slope, basin relief, sinuosity ratio, drainage density and total stream length etc. It was found from the result of analysis that the threshold value of minimum order stream has a very big effect on topographical parameters of basin. It was found that the threshold area of minimum ord er stream revealed under $0.10km^{2}.$ Furthermore, the parameters showed a serious change except for over $0.10km^{2}.$

Performance Evaluation of Big Stream based High Speed Data Storage (빅 스트림 기반 초고속 데이터 스토리지 성능 평가)

  • Song, Min-Gyu;Kang, Yong-Woo;Kim, Hyo-Ryoung
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.12 no.5
    • /
    • pp.817-828
    • /
    • 2017
  • It is very hard to find the system which processes single 10Gbps stream, and the related application is also rare. But in the field of science such as physics and astronomy, these high speed systems have been widely used and now more upgraded performance is expected. For this reason, high speed network based storage which captures and records 10Gbps level of packets was developed for the support of small astronomical company in KASI. But for the use of the system in research, system performance should be not only evaluated but also optimized. In this paper, we first implement system environment for the performance evaluation and discuss the experiment procedure and solution to acquire numerical results.

Efficient Processing of an Aggregate Query Stream in MapReduce (맵리듀스에서 집계 질의 스트림의 효율적인 처리 기법)

  • Choi, Hyunjean;Lee, Ki Yong
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.3 no.2
    • /
    • pp.73-80
    • /
    • 2014
  • MapReduce is a widely used programming model for analyzing and processing Big data. Aggregate queries are one of the most common types of queries used for analyzing Big data. In this paper, we propose an efficient method for processing an aggregate query stream, where many concurrent users continuously issue different aggregate queries on the same data. Instead of processing each aggregate query separately, the proposed method processes multiple aggregate queries together in a batch by a single, optimized MapReduce job. As a result, the number of queries processed per unit time increases significantly. Through various experiments, we show that the proposed method improves the performance significantly compared to a naive method.

Scalable Big Data Pipeline for Video Stream Analytics Over Commodity Hardware

  • Ayub, Umer;Ahsan, Syed M.;Qureshi, Shavez M.
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.4
    • /
    • pp.1146-1165
    • /
    • 2022
  • A huge amount of data in the form of videos and images is being produced owning to advancements in sensor technology. Use of low performance commodity hardware coupled with resource heavy image processing and analyzing approaches to infer and extract actionable insights from this data poses a bottleneck for timely decision making. Current approach of GPU assisted and cloud-based architecture video analysis techniques give significant performance gain, but its usage is constrained by financial considerations and extremely complex architecture level details. In this paper we propose a data pipeline system that uses open-source tools such as Apache Spark, Kafka and OpenCV running over commodity hardware for video stream processing and image processing in a distributed environment. Experimental results show that our proposed approach eliminates the need of GPU based hardware and cloud computing infrastructure to achieve efficient video steam processing for face detection with increased throughput, scalability and better performance.

Detection of Malicious PDF based on Document Structure Features and Stream Objects

  • Kang, Ah Reum;Jeong, Young-Seob;Kim, Se Lyeong;Kim, Jonghyun;Woo, Jiyoung;Choi, Sunoh
    • Journal of the Korea Society of Computer and Information
    • /
    • v.23 no.11
    • /
    • pp.85-93
    • /
    • 2018
  • In recent years, there has been an increasing number of ways to distribute document-based malicious code using vulnerabilities in document files. Because document type malware is not an executable file itself, it is easy to bypass existing security programs, so research on a model to detect it is necessary. In this study, we extract main features from the document structure and the JavaScript contained in the stream object In addition, when JavaScript is inserted, keywords with high occurrence frequency in malicious code such as function name, reserved word and the readable string in the script are extracted. Then, we generate a machine learning model that can distinguish between normal and malicious. In order to make it difficult to bypass, we try to achieve good performance in a black box type algorithm. For an experiment, a large amount of documents compared to previous studies is analyzed. Experimental results show 98.9% detection rate from three different type algorithms. SVM, which is a black box type algorithm and makes obfuscation difficult, shows much higher performance than in previous studies.

In-Memory Based Incremental Processing Method for Stream Query Processing in Big Data Environments (빅데이터 환경에서 스트림 질의 처리를 위한 인메모리 기반 점진적 처리 기법)

  • Bok, Kyoungsoo;Yook, Misun;Noh, Yeonwoo;Han, Jieun;Kim, Yeonwoo;Lim, Jongtae;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.16 no.2
    • /
    • pp.163-173
    • /
    • 2016
  • Recently, massive amounts of stream data have been studied for distributed processing. In this paper, we propose an incremental stream data processing method based on in-memory in big data environments. The proposed method stores input data in a temporary queue and compare them with data in a master node. If the data is in the master node, the proposed method reuses the previous processing results located in the node chosen by the master node. If there are no previous results of data in the node, the proposed method processes the data and stores the result in a separate node. We also propose a job scheduling technique considering the load and performance of a node. In order to show the superiority of the proposed method, we compare it with the existing method in terms of query processing time. Our experimental results show that our method outperforms the existing method in terms of query processing time.

Estimating Critical Stream Power by the Distribution of Gravel-bed Materials in the Meandering River (만곡하천의 자갈하상재료 분포에 따른 한계수류력 평가)

  • Shin, Seung-Sook;Park, Sang-Deok;Lee, Seung-Kyu;Ji, Min-Gyu
    • Journal of Korea Water Resources Association
    • /
    • v.45 no.2
    • /
    • pp.151-163
    • /
    • 2012
  • The distribution of gravel-bed materials in mountainous river is formed by the process of deposition and transportation of sediment responding to stream power of the latest flood that is over the certain scale. The particle size of bed material was surveyed in the longitudinal points of river and detail points of a specific meandering section and used to estimate the critical velocity and stream power. Yang's critical unit stream power and Bagnold's critical stream power for gravel-bed materials increased with the distance from downstream to upstream. Dimensionless shear stress based on the designed flood discharge in Shields diagram was evaluated that the gravel-bed materials in most survey points may be transported as form of bedload. The mean diameter in the meandering section was the biggest size in first water impingement point of inflow water from upstream and the second big size in second water impingement point by reflection flow. The mean diameters were relatively the small sizes in points right after water impingement. The range of mean critical velocity was 0.77~2.60 m/s and critical unit stream power was big greatly in first water impingement point. The distribution of critical stream power, range of 7~171 $W/m^2$, was shown that variation in longitudinal section was more obvious than that of cross section and estimated that critical stream power may be affected greatly in first and second water impingement point.

A Study for the Calculation of Instream Flow in the Rehabilitation of Urban stream (도심하천 복원에 따른 하천유지유량 산정 연구)

  • Choi, Gye Woon;Chang, Yun Gyu;Han, Man Sin
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2004.05b
    • /
    • pp.448-452
    • /
    • 2004
  • Instream flow is necessary to manage the basic function of the rivers. The evaluation method of instream flow in a big river has been studied widely. However, the study in a urban hasn't researched. In theses days the environmental function of a river becomes very important. The evaluation method and secure of instream flow are basic conditions to perform it. Especially view of stream, recreation, protection of ecology are highly demanded In a urban according to the multipurpose of river spaces. In this paper the evaluation methods of instream flow were compared and investigated with many papers. This paper presents a proper evaluation procedure of instream flow in a urban stream through comparison and examination. According to the demanded hydraulic conditions the method can be considered the environmental function depend on the purpose of river restoration. The relationship of the coefficient of roughness and the slope of the river bed were examinated in order to estimate the minimum instream flow corresponding to proper Hydraulic conditions. Also calculate the instream flow of Sueng-gi stream in In-cheon.

  • PDF

Load Balancing for Distributed Processing of Real-time Spatial Big Data Stream (실시간 공간 빅데이터 스트림 분산 처리를 위한 부하 균형화 방법)

  • Yoon, Susik;Lee, Jae-Gil
    • Journal of KIISE
    • /
    • v.44 no.11
    • /
    • pp.1209-1218
    • /
    • 2017
  • A variety of sensors is widely used these days, and it has become much easier to acquire spatial big data streams from various sources. Since spatial data streams have inherently skewed and dynamically changing distributions, the system must effectively distribute the load among workers. Previous studies to solve this load imbalance problem are not directly applicable to processing spatial data. In this research, we propose Adaptive Spatial Key Grouping (ASKG). The main idea of ASKG is, by utilizing the previous distribution of the data streams, to adaptively suggest a new grouping scheme that evenly distributes the future load among workers. We evaluate the validity of the proposed algorithm in various environments, by conducting an experiment with real datasets while varying the number of workers, input rate, and processing overhead. Compared to two other alternative algorithms, ASKG improves the system performance in terms of load imbalance, throughput, and latency.

Estimate of Roughness Coefficient for Stream at Island of Incheon (인천연안도서지역 소하천의 조도계수 산정(영흥도를 중심으로))

  • Choi, Gye-Woon;Chung, Yeoun-Jung;Han, Man-Shin
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2007.05a
    • /
    • pp.935-940
    • /
    • 2007
  • The roughness coefficients at stage gaging station were calculated and suggested based upon figures and formulae. The results shows that roughness coefficient decreases by increasing the discharge with the exception of no change or increasing in the cases having the tall vegetation and big influence from the stream bank. As a result of the research about the water quality in streams, vegetation and aquatic organisms, the paper suggests the need for management system to protect the organisms in danger of extinction and the need for preserving the diverse species at the estuary that is located in the downstream.

  • PDF