• Journal of Computing Science and Engineering
• /
• v.3 no.1
• /
• pp.27-58
• /
• 2009

Aggregation join queries are an important class of queries over data streams. These queries involve both join and aggregation operations, with window-based joins followed by an aggregation on the join output. All existing research address join query optimization and aggregation query optimization as separate problems. We observe that, by putting them within the same scope of query optimization, more efficient query execution plans are possible through more versatile query transformations. The enabling idea is to perform aggregation before join so that the join execution time may be reduced. There has been some research done on such query transformations in relational databases, but none has been done in data streams. Doing it in data streams brings new challenges due to the incremental and continuous arrival of tuples. These challenges are addressed in this paper. Specifically, we first present a query processing model geared to facilitate query transformations and propose a query transformation rule specialized to work with streams. The rule is simple and yet covers all possible cases of transformation. Then we present a generic query processing algorithm that works with all alternative query execution plans possible with the transformation, and develop the cost formulas of the query execution plans. Based on the processing algorithm, we validate the rule theoretically by proving the equivalence of query execution plans. Finally, through extensive experiments, we validate the cost formulas and study the performances of alternative query execution plans.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2015.04a
• /
• pp.132-135
• /
• 2015

맵리듀스(MapReduce)는 분산 환경에서의 빅데이터(Big Data), 즉 대용량 데이터를 처리하는 프로그래밍 모델이다. 대용량의 데이터를 분석하기 위해서 집계 함수(Aggregation function)로 데이터를 처리할 수 있다. 본 논문에서는 맵리듀스 환경을 기반으로 SQL 쿼리에서 집계 함수를 더 적은 비용으로 수행하며 효율적으로 처리할 수 있는 두 가지 전략을 제안한다. 두 가지 전략 중 더 높은 성능을 보이는 전략을 더 효율적인 처리 방법으로 판단한다. 첫 번째 전략은 두 테이블을 Join하여 집계 함수를 처리하는 방법이다. 두 번째 전략은 집계 함수를 처리하여 Join에 참여할 튜플의 수를 최소로 줄인 후 Join을 수행하고 다시 집계 함수를 처리하는 방법이다. 두 제안 방법을 비교하기 위하여 실험을 한 결과 두 번째 전략이 더 적은 비용이 드므로 더 효율적인 처리 방법인 것으로 보인다.

• The Journal of the Korea Contents Association
• /
• v.11 no.2
• /
• pp.126-134
• /
• 2011

Recently, many researchers are interested in using join queries to correlate sensor readings stored in different regions. In the conventional algorithm, the preliminary join coordinator collects the synopsis from sensor nodes and determines a set of sensor readings that are required for processing the join query. Then, the base station collects only a part of sensor readings instead of whole readings and performs the final join process. However, it has a problem that incurs communication overhead for processing the preliminary join. In this paper, we propose a novel energy-efficient in-network join scheme that solves such a problem. The proposed scheme determines a preliminary join coordinator located to minimize the communication cost for the preliminary join. The coordinator prunes data that do not contribute to the join result and performs the compression of sensor readings in the early stage of the join processing. Therefore, the base station just collects a part of compressed sensor readings with the decompression table and determines the join result from them. In the result, the proposed scheme reduces communication costs for the preliminary join processing and prolongs the network lifetime.

• Journal of Computing Science and Engineering
• /
• v.7 no.4
• /
• pp.220-230
• /
• 2013

In recent years, progress in hardware technology has resulted in the possibility of monitoring many events in real time. The volume of incoming data may be so large, that monitoring all individual data might be intractable. Revisiting any particular record can also be impossible in this environment. Therefore, many database schemes, such as aggregation, join, frequent pattern mining, and indexing, become more challenging in this context. This paper surveys the previous efforts to resolve these issues in processing data streams. The emphasis is on specifying and processing sliding window queries, which are supported in many stream processing engines. We also review the related work on stream query processing, including synopsis structures, plan sharing, operator scheduling, load shedding, and disorder control.

• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.10
• /
• pp.477-482
• /
• 2016

As being the advanced analytics indispensable on big data for agile decision-making and tactical planning in enterprises, distributed processing platforms, such as Hadoop and Spark which distribute and handle the large volume of data on multiple nodes, receive great attention in the field. In Spark platform stack, Spark SQL unveiled recently to make Spark able to support distributed processing framework based on SQL. However, Spark SQL cannot effectively handle advanced analytics that involves machine learning and graph processing in terms of iterative tasks and task allocations. Motivated by these issues, this paper proposes the design of SQL-based big data optimal processing engine and processing framework to support advanced analytics in Spark environments. Big data optimal processing engines copes with complex SQL queries that involves multiple parameters and join, aggregation and sorting operations in distributed/parallel manner and the proposing framework optimizes machine learning process in terms of relational operations.