Browse > Article
http://dx.doi.org/10.6109/jkiice.2017.21.8.1531

Log processing using messaging system in SSD Storage Tester  

Nam, Ki-ahn (Department of Computer Science and Engineering, Korea University of Technology and Education)
Kwon, Oh-young (Department of Computer Science and Engineering, Korea University of Technology and Education)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.21, no.8, 2017 , pp. 1531-1539 More about this Journal
Abstract
The existing SSD storage tester processed logs in a 1-N structure between server and client using TCP and network file system. This method causes some problems for example, an increase in CPU usage and difficulty in exception handling, etc. In this paper, we implement a log processing message layer that can deal with asynchronous distributed processing using open source messaging system such as kafka, RabbitMQ and compare this layer with existing log transmission method. A log simulator was implemented to compare the transmission bandwidth and CPU usage. Test results show that the transmission using the message layer has higher performance than the transmission using the message layer, and the CPU usage does not show any significant difference The message layer can be implemented more easily than the conventional method and the efficiency is higher than that of the conventional method.
Keywords
Log processing; Messaging system; Kafka; RabbitMQ; Message layer;
Citations & Related Records
연도 인용수 순위
  • Reference
1 B. Parhami, Computer Architecture, New York, NY: Oxford University Press, 2005.
2 J. Kreps, N. Narkhede, and J. Rao, "Kafka: A distributed messaging system for Log processing," In Proceedings of the NetDB : 6th Workshop on Networking meetsDatabases, Athens, pp. 1-7, 2011.
3 K. Joshua, "Advanced message queuing protocol (AMQP)," Linux Journal, vol. 3, no.187, pp. 54-61, Nov. 2009.
4 M. Rostanski, K. Grochla, and A. Seman, "Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ," In Proceedings of Computer Science and Information Systems (FedCSIS), 2014 Federated Conference on. IEEE, New York: NY, pp. 879-884, 2014.
5 P. Dobbelaere and K. S. Esmaili, "Kafka versus RabbitMQ: A comparative study of two industry reference publish/ subscribe implementations: Industry Paper," In Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, Barcelona, pp. 227- 238, 2017.
6 B. Snyder, D. Bosanac and R. Davies, Introduction to apache activemq, New York, NY: manning publications Co, 2017.
7 P. Hintjens, ZeroMQ: Messaging for Many Applications, California, O'Reilly Media Inc, 2013.
8 S. Boschi, and G. Santomaggio, RabbitMQ cookbook, Birmingham, Packt Publishing Ltd, 2013.
9 S. Minni, Apache Kafka Cookbook, Birmingham, Packt Publishing Ltd, 2015.