Browse > Article
http://dx.doi.org/10.5762/KAIS.2018.19.8.535

Dynamic Load Management Method for Spatial Data Stream Processing on MapReduce Online Frameworks  

Jeong, Weonil (Division of Computer and Information Engineering, Hose University)
Publication Information
Journal of the Korea Academia-Industrial cooperation Society / v.19, no.8, 2018 , pp. 535-544 More about this Journal
Abstract
As the spread of mobile devices equipped with various sensors and high-quality wireless network communications functionsexpands, the amount of spatio-temporal data generated from mobile devices in various service fields is rapidly increasing. In conventional research into processing a large amount of real-time spatio-temporal streams, it is very difficult to apply a Hadoop-based spatial big data system, designed to be a batch processing platform, to a real-time service for spatio-temporal data streams. This paper extends the MapReduce online framework to support real-time query processing for continuous-input, spatio-temporal data streams, and proposes a load management method to distribute overloads for efficient query processing. The proposed scheme shows a dynamic load balancing method for the nodes based on the inflow rate and the load factor of the input data based on the space partition. Experiments show that it is possible to support efficient query processing by distributing the spatial data stream in the corresponding area to the shared resources when load management in a specific area is required.
Keywords
Spatial Big Data; Spatial Data Stream Processing; Load Management; Load Balance; MapReduce Online;
Citations & Related Records
Times Cited By KSCI : 4  (Citation Analysis)
연도 인용수 순위
1 J. Abdul, M. Alkathiri and M. B. Potdar, "Geospatial Hadoop (GS-Hadoop) an efficient mapreduce based engine for distributed processing of shapefiles", Advances in Computing, Communication, & Automation (ICACCA) (Fall), International Conference, pp. 1-7, 2016. DOI: https://doi.org/10.1109/ICACCAF.2016.7748956   DOI
2 J. M. Park, M. H. Lee, D. B. Shin and J. W. Ahn, "Deduction of the Policy Issues for Activating the Geo-Spatial Big Data Services", Journal of Korea Spatial Information Society, vol. 23, no. 6, pp. 19-29, 2015. DOI: https://doi.org/10.12672/ksis.2015.23.6.019   DOI
3 A. Aji, H. Vo, W. Fusheng, R. Lee, X. Zhang and J. Saltz, "Hadoop-GIS: A High Performance Spatial Data Warehousing System over MapReduce", VLDB Endowment, vol. 6, no. 11, pp. 1009-1020, 2013. DOI: https://doi.org/10.14778/2536222.2536227   DOI
4 A. Eldawy, Alarabi, and M. F. Mokbel, "SpatialHaddop: A MapReduce Framework for Spatial Data", Data Engineering (ICDE), 2015 IEEE 31st International Conference on 2015, pp. 1352-1363, 2015. DOI: https://doi.org/10.1109/ICDE.2015.7113382   DOI
5 In-Hak Joo, "Spatial Big Data Query Processing System Supporting SQL-based Query Language in Hadoop", Journal of Korea institute of information, electronics, and communication technology vol. 10, no. 1, pp. 1-8, 2017.   DOI
6 G. H. Kim, J. H. Yoon, C. M. Jun and H. C. Jung, "Providing Service Model Based on Concept and Requirements of Spatial Big Data", Journal of the Korean Society for Geospatial Information Science. vol. 24, no. 4, pp. 89-96, 2016. DOI: https://doi.org/10.7319/kogsis.2016.24.4.089   DOI
7 Apache Hadoop, http://hadoop.apache.org/
8 D. Jeong, S. Jeon and B. Hong, "A Study on MapReduce Processing for Multi-dimensional Continuous Query", Lecture notes in computer science, vol. 7827, pp. 74-78, 2013. DOI: https://doi.org/10.1007/978-3-642-40270-8_6   DOI
9 J. Dean, S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters", Proc. of the 6th Symposium on Operating Systems Design and Implementation, pp. 137-150, 2004.
10 A. M. Aly, A. Sallam, B. M. Gnanasekaran, L. Nguyen-Dinh, W. G. Aref, M. Ouzzani and A. Ghafoor, "M3: Stream Processing on Main-Memory MapReduce", Data Engineering (ICDE), IEEE 28th International Conference, pp. 1253-1256, 2012. DOI: https://doi.org/10.1109/ICDE.2012.120   DOI
11 D. Yang, J. Cao, S. Wu and J. Wang, "Progressive online aggregation in a distributed stream system", The Journal of systems and software, vol. 102, pp. 146-157, 2015. DOI: https://doi.org/10.1007/978-3-642-40270-8_6   DOI
12 X. Song, J. Gao, J. Ma, S. Niu and H. He, "HTME: A data streams processing strategy based on Hoeffding tree in MapReduce environment", Intelligent Control and Automation(WCICA), pp. 1042-1045, 2016.
13 S. Park, W. Ryu, B. Hong and J Kwon, "MapReduce-based Stream Assigning and Splitting Technique for Stream Data Processing", Journal of KIISE, vol. 19, no. 8, pp. 439-443, 2013.
14 R. A. Finkel and J. L. Bentley, "Quad trees a data structure for retrieval on composite keys", Acta informatica, vol. 4, no. 1, pp. 1-9, 1974. DOI: https://doi.org/10.1007/BF00288933   DOI
15 K. Madsen and Y. Zhou, "Dynamic resource management in a MapReduce-style platform for fast data processing", Data Engineering Workshops(ICDEW), 31st IEEE International Conference, pp. 10-13, 2015. DOI: https://doi.org/10.1109/ICDEW.2015.7129537   DOI
16 T. Condie, N. Conway, P. ALvaro and J.M. Hellerstein, "MapReduce Online", NSDI'10, 2010.
17 S. Baek, D. Lee, G. Kim, W. Chung and H. Bae, "Load Shedding Method based on Grid Hash to Improve Accuracy of Spatial Sliding Window Aggregate Queries", Journal of KSIS, vol. 11, no. 2, pp. 89-98, 2009.
18 H. Kim, S. Baek, D. Lee, G. Kim, H. Bae, "Pre-filtering based Post-Load Shedding Method for Improving Spatial Query Accuracy in GeoSensor Environment", Journal of KSISS, vol. 12, no. 1, pp. 18-27, 2010.
19 W. Jeong, "Dynamic Load Shedding Scheme based on Input Rate of Spatial Data Stream and Data Density", Journal of KAIS, vol. 16, no. 3, pp. 2158-2164, 2015. DOI: https://doi.org/10.5762/KAIS.2015.16.3.2158   DOI