KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Handling Streaming Data by Using Open Source Framework Storm in IoT Environment

오픈소스 프레임워크 Storm을 활용한 IoT 환경 스트리밍 데이터 처리

  • Received : 2016.04.22
  • Accepted : 2016.05.25
  • Published : 2016.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

To utilize sensory data, it is necessary to design architecture for processing and handling data generated from sensors in an IoT environment. Especially in the IoT environment, a thing connects to the Internet and efficiently enables to communicate a device with diverse sensors. But Hadoop and Twister based on MapReduce are good at handling data in a batch processing. It has a limitation for processing stream data from a sensor in a motion. Traditional streaming data processing has been mainly applied a MoM based message queuing system. It has maintainability and scalability problems because a programmer should consider details related with complex messaging flow. In this paper architecture is designed to handle sensory data aggregated The designed software architecture is used to operate an application on the open source framework Storm. The application is conceptually used to transform streaming data which aggregated via sensor gateway by pipe-filter style.

IoT 환경에서 센싱 정보의 활용을 위해서는 센서로 부터 생성된 정보의 가공 및 처리를 위한 효율적인 소프트웨어 아키텍처 설계가 필수적이다. 특히 IoT 환경에서 사물은 인터넷에 연결되고 각종 센서를 탑재한 디바이스간 통신이 가능하여야 한다. 그러나 MapReduce 기반의 Hadoop과 Twister은 데이터의 배치 처리에 적합하지만, 스트리밍 센서 자료의 이동중 처리에는 제약점을 갖는다. 전통적인 스트리밍 데이터 처리 방법인 MOM 기반의 메시지 큐 시스템을 이용해 메시지 스트림을 처리하는 방식은 프로그래머가 메시지 흐름의 복잡도를 고려해야 상세한 처리를 프로그램 함으로써 유지보수 및 확장성을 갖기 어려움이 있다. 이 논문에서는 IoT 환경에서 수집된 센싱 자료의 처리를 위한 소프트 웨어 아키텍처를 설계하였다. 또한 설계된 소프트아키텍처를 기반으로 오픈소스 프레임워크인 Storm의 응용 구성을 기술한다. 구성응용은 센서게이트웨이(Sensor Gateway)를 통해 자료를 수집한 후 실시간 스트리밍 데이터를 파이프-필터 스타일로서 변환한다.

Keywords

References

  1. Luigi Atzori, Antonio Iera, and Giacomo Morabito, "The Internet of Things: A survey," Computer Networks: The International Journal of Computer and Telecommunications Networking, Vol.54, No.15, pp.2787-2805, 2010. https://doi.org/10.1016/j.comnet.2010.05.010
  2. Yunhee Kang, "New Approach to the Platform for the Application Development on the Internet of Things Environment," Journal of Platform Technology, Vol.3, No.1, pp.21-27, 2015.
  3. Z.(G.) Guo, R. Singh, and M. Pierce, "Building the PolarGrid Portal Using Web 2.0 and OpenSocial," The International Conference for High Performance Computing, Networking, Storage and Analysis (SC'09), Porland, OR, ACM Press, p.5, 2009.
  4. Yunhee Kang, "Enabling BigData Platform for MapReduce Applications in the Science Cloud," Journal of Platform Technology, Vol.2, No.1, pp.42-52, 2014.
  5. W. Gropp and E. Lusk, "Goals guiding design: PVM and MPI," Cluster Computing, 2002. Proceedings. 2002 IEEE International Conference on, pp.257-265, 2002, doi: 10.1109/CLUSTR.2002.1137754.
  6. J. Dean and S. Ghemawat, "Mapreduce: Simplified Data Processing on Large Clusters," Communications of the Acm, Vol.51, pp.107-113, Jan., 2008.
  7. J. Dean and S. Ghemawat, "MapReduce: A Flexible Data Processing Tool," Communications of the Acm, Vol.53, pp. 72-77, Jan., 2010.
  8. Hadoop [Internet], http://hadoop.apache.org/.
  9. J. Ekanayake et al., "Twister: A Runtime for Iterative MapReduce," The First International Workshop on MapReduce and its Applications (MAPREDUCE'10) - HPDC2010, 2010.
  10. Yunhee Kang and Y. B. Park, "The performance evaluation of k-means by two MapReduce frameworks, Hadoop vs. Twister," 2015 International Conference on Information Networking (ICOIN), Cambodia, pp.405-406, 2015.
  11. Yunhee Kang and Geoffrey C. Fox, "Performance Evaluation of MapReduce Applications on Cloud Computing Environment, FutureGrid," FGIT-GDC, pp.77-86, 2011.
  12. Curry Edward, "Message-Oriented Middleware," in Middleware for Communications, ed. Qusay H Mahmoud, 1-28. Chichester, England: John Wiley and Sons, 2004.
  13. S. Pallickara and G. Fox, "NaradaBrokering: a distributed middleware framework and architecture for enabling durable peer-to-peer grids," the Proceedings of the ACM/IFIP/USENIX 2003 International Conference on Middleware, Rio de Janeiro, Brazil, 2003.
  14. Apache Storm [Internet], http://storm.incubator.apache.org.
  15. J. Carlson, "Redis in Action," 1st ed., New York: Manning, 2013.
  16. J. Pokorny, "NoSQL databases: a step to database scalability in web environment," in Proceedings of the 13th International Conference on Information Integration and Web-based Applications and Services, 2011, pp.278-283.
  17. M. Shaw and D. Garlan, "Software Architecture: Perspectives on an Emerging Discipline," Prentice-Hall, 1996.