[KSCI] Korea Science Citation Index Service

http://dx.doi.org/10.14372/IEMEK.2021.16.6.259

Implementation and Performance Aanalysis of Efficient Big Data Processing System Through Dynamic Configuration of Edge Server Computing and Storage Modules

Kim, Yongyeon (ETRI)
Jeon, Jaeho (ETRI)
Kang, Sungjoo (ETRI)

Publication Information

IEMEK Journal of Embedded Systems and Applications / v.16, no.6, 2021 , pp. 259-266 More about this Journal

Abstract

Edge Computing enables real-time big data processing by performing computing close to the physical location of the user or data source. However, in an edge computing environment, various situations that affect big data processing performance may occur depending on temporary service requirements or changes of physical resources in the field. In this paper, we proposed a BigCrawler system that dynamically configures the computing module and storage module according to the big data collection status and computing resource usage status in the edge computing environment. And the feature of big data processing workload according to the arrangement of computing module and storage module were analyzed.

Keywords

Bigdata Analytics; Edge Computing Server; Data Crawling; Kubernetes; Elastic Search;

Citations & Related Records

Reference

1	D. Kim, Y. Park, T. Chung, "Development of Big-data Management Platform Considering Docker Based Real Time Data Connecting and Processing Environments," IEMEK J. Embed. Sys. Appl., Vol. 16, No. 4, pp. 153-161, June, 2021 (in Korean). DOI
2	Kubernetes, https://kubernetes.io/
3	J. William, "Web Data Crawling vs Web Data Scraping", Promptcloud, https://www.promptcloud.com/blog/data-scraping-vs-data-crawling.
4	AWS serverless data analytics pipeline reference architecture, https://aws.amazon.com/ko/blogs/big-data/aws-serverless-data-analytics-pipeline-reference-architecture/
5	"Technology Roadmap of SME", 2018-2020, https://www.smtech.go.kr
6	Dell, "Dell EMC PowerEdge XE Servers", 2021
7	J. Wang, W. Zhang, Y. Shi, S. Duan, J. Liu, "Industrial Big Data Analytics: Challenges, Methodologies, and Applications", CoRR, arXiv:1807.01016, 2018
8	B. Duncan, M. Whittington, V. Chang, "Enterprise security and privacy: Why adding IoT and big data makes it so much more difficult," International Conference on Engineering and Technology (ICET), 2017, pp. 1-7
9	M. Caprolu, R. Di Pietro, F. Lombardi and S. Raponi, "Edge Computing Perspectives: Architectures, Technologies, and Open Security Issues," 2019 IEEE International Conference on Edge Computing (EDGE), 2019, pp. 116-123
10	Gigabyte, "Edge Server", 2021
11	S. J. Shin, J. Woo, W. Seo, "Developing a Big Data Analytics Platform Architecture for Smart Factory," Journal of Korea Multimedia Society, Vol. 19, No. 8, pp. 1516-1529, Aug. 2016 (in Korean). DOI
12	Apache Lucene, https://lucene.apache.org/
13	Elastic(ELK) Stack, https://www.elastic.co/
14	Prometheus, https://prometheus.io/
15	Analytics end-to-end with Azure Synapse, https://docs.microsoft.com/en-us/azure/architecture/example-scenario/dataplate2e/data-platform-end-to-end
16	"Power facility energy pattern and failure analysis sensor," https://aihub.or.kr/aidata/30759 (in Korean).
17	T. Kim, T, Kim, S. Jin, "Multi-access Edge Computing Scheduler for Low Latency Services," IEMEK J. Embed. Sys. Appl., Vol. 15, No. 6, pp. 299-305, December, 2020 (in Korean). DOI