Browse > Article
http://dx.doi.org/10.6109/jkiice.2019.23.10.1179

Application Of Open Data Framework For Real-Time Data Processing  

Park, Sun-ho (KL-Net)
Kim, Young-kil (Department of BioMedical Engineering, Ajou University)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.23, no.10, 2019 , pp. 1179-1187 More about this Journal
Abstract
In today's technology environment, most big data-based applications and solutions are based on real-time processing of streaming data. Real-time processing and analysis of big data streams plays an important role in the development of big data-based applications and solutions. In particular, in the maritime data processing environment, the necessity of developing a technology capable of rapidly processing and analyzing a large amount of real-time data due to the explosion of data is accelerating. Therefore, this paper analyzes the characteristics of NiFi, Kafka, and Druid as suitable open source among various open data technologies for processing big data, and provides the latest information on external linkage necessary for maritime service analysis in Korean e-Navigation service. To this end, we will lay the foundation for applying open data framework technology for real-time data processing.
Keywords
e-Navigation; Open Data Framework; Big Data; Streaming;
Citations & Related Records
연도 인용수 순위
  • Reference
1 F. Gurcan, and M. Berigel, "Real-Time Processing of Big Data Stream: Lifecycle, Tools, Tasks, and Challengs," 2nd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Turkey, pp. 1-6, 2018.
2 H. S. Jung, C. S. Yoon, and Y. W. Lee, "Cloud computing platform based real-time processing for stream reasoning," Sixth International Conference on Future Generation Communication Technologies (FGCT), Doublin, pp. 1-5, 2017.
3 How to analyze real-time big-data [Internet]. Available: https://d2.naver.com/helloworld/694050.
4 Real-time data feed processing and Apache Kafka for it [Internet]. Available: https://m.blog.naver.com/sundooedu/21230385470.
5 K. S. Paik, "Multi-channel data connection and Real-time processing system desinged for Big Data collection," The Korea Contents Society, 2016.
6 S. Han, H. Chung, D. Ok, and Y. N, "Impact Analysis of Data Volume on Spark Performance," The Korean Institute of Information Scientists and Engineers, 2016.
7 Apache NiFi [Internet]. Available: https://en.wikipedia.org/wiki/Apache_NiFi.
8 The Core concepts of NiFi [Internet]. Available: https://nifi.apache.org/docs.html.
9 Kafka Introduction [Internet]. Available: https://kafka.apache.org/intro.
10 What is Druid [Internet]. Available: https://druid.apache.org/docs/latest/design.
11 Druid Coordinator Process [Internet]. Available: https://druid.apache.org/docs/latest/design/coordinator.html.
12 Druid Overlord Process [Internet]. Available: https://druid.apache.org/docs/latest/design/overload.html.
13 Druid Broker [Internet]. Available: https://druid.apache.org/docs/latest/design/broker.html.
14 Druid Historical Process [Internet]. Available: https://druid.apache.org/docs/latest/design/historical.html.
15 Druid MiddleManager Process [Internet]. Available: https://druid.apache.org/docs/latest/design/middlemanager.html.
16 Druid Zookeeper [Internet]. Available: https://druid.apache.org/ocs/latest/dependencies/zookeeper.html.
17 Druid Deep Storage [Internet]. Available: https://druid.pache.org/docs/latest/dependencies/deep-storage.html.
18 F. Yang, (2016, June). Building a Streaming Analytics Stack with ApacheKafka and Druid. Confluent [Online]. Available:https://www.confluent.io/blog/building-a-streaming-analytics-stack-with-apache-kafka-and-druid.
19 Druid Metadata Storage [Internet]. Available: https://druid.pache.org/docs/latest/dependencies/metadata-storage.html.
20 H. Isah, and F. Zulkernine, "A Scalable and Roburst Framework for Data Stream Ingestion," Cornell University, arXiv: 812.04197, 2018.
21 F. Yang, E. Tschetter, and X. Leaute, "Druid-A Real time Analytical Data Store," 2014.