Browse > Article
http://dx.doi.org/10.3837/tiis.2022.04.004

Scalable Big Data Pipeline for Video Stream Analytics Over Commodity Hardware  

Ayub, Umer (Department of Computer Science, Qarshi University)
Ahsan, Syed M. (Department of Computer Science, Qarshi University)
Qureshi, Shavez M. (Department of Computer Science, Qarshi University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.16, no.4, 2022 , pp. 1146-1165 More about this Journal
Abstract
A huge amount of data in the form of videos and images is being produced owning to advancements in sensor technology. Use of low performance commodity hardware coupled with resource heavy image processing and analyzing approaches to infer and extract actionable insights from this data poses a bottleneck for timely decision making. Current approach of GPU assisted and cloud-based architecture video analysis techniques give significant performance gain, but its usage is constrained by financial considerations and extremely complex architecture level details. In this paper we propose a data pipeline system that uses open-source tools such as Apache Spark, Kafka and OpenCV running over commodity hardware for video stream processing and image processing in a distributed environment. Experimental results show that our proposed approach eliminates the need of GPU based hardware and cloud computing infrastructure to achieve efficient video steam processing for face detection with increased throughput, scalability and better performance.
Keywords
Video Analytics; Big Data; Data Pipeline; Spark; Kafka; OpenCV;
Citations & Related Records
연도 인용수 순위
  • Reference
1 J. Fung and S. Mann, "Using graphics devices in reverse: GPU-based image processing and computer vision," in Proc. of 2008 IEEE International Conference on Multimedia and Expo, pp. 9-12, 2008.
2 Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica, "Spark: Cluster computing with working sets," in Proc. of the 2Nd USENIX Conference on Hot Top- ics in Cloud Computing (Berkeley, CA, USA), HotCloud'10, pp. 10-10, 2010.
3 Chun-Wei Tsai, Chin-Feng Lai, Han-Chieh Chao, and Athanasios V. Vasilakos, "Big data analytics: A Survey," Journal of Big Data, vol. 2, no. 21, pp. 13-52, 2015.   DOI
4 M. Cao, L. Zheng, W. Jia, and X. Liu, "Joint 3D reconstruction and object tracking for traffic video analysis under iov environment," IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 6, pp. 3577-3591, 2021.   DOI
5 Nan Zhang, Yun-shan Chen, and Jian-li Wang, "Image parallel processing based on GPU," in Proc. of 2010 2nd International Conference on Advanced Computer Control, vol. 3, pp. 367-370, 2010.
6 Suramya Tomar, "Converting video formats with ffmpeg," Linux Jour- nal, vol. 2006, no. 146, p. 10, 2006.
7 J. Choi, B. Kim, J. Jeon, H. Lee, E. Lim, and C. E. Rhee, "Poster: Gpu based near data processing for image processing with pattern aware data allocation and prefetching," in Proc. of 2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT), pp. 469-470, 2019.
8 Lei Huang Yuzhong Yan, "Large-scale image processing research cloud,".
9 Chris Sweeney, Liu Liu, Sean Arietta, and Jason Lawrence, "Hipi: a hadoop image processing interface for image-based mapreduce tasks," Chris. University of Virginiam, vol. 2, no. 1, pp. 1-5, 2011.
10 Tingxi Wen, Haotian Liu, Luxin Lin, Bin Wang, Jigong Hou, Chuanbo Huang, Ting Pan, and Yu Du, "Multiswarm artificial bee colony algorithm based on spark cloud computing platform for medical image registration," Computer Methods and Programs in Biomedicine, vol. 192, 105432, 2020.   DOI
11 J. Kreps, N. Narkhede, and J. Rao, Kafka: "A distributed messaging system for log processing," in Proc. of 6th International Work- shop on Networking Meets Databases (NetDB), Athens, Greece, 2011.
12 Ashiq Anjum, Tariq Abdullah, Muhammad Tariq, Yusuf Baltaci, and Nick Antonopoulos, "Video stream analysis in clouds: An object detection and classification framework for high performance video analytics," IEEE Transactions on Cloud Computing, vol. 7, no. 4, pp. 1152-1167, 2019.   DOI
13 Samvit Jain, Ganesh Ananthanarayanan, Junchen Jiang, Yuanchao Shu, and Joseph Gonzalez, "Scaling video analytics systems to large camera deployments," in Proc. of the 20th International Workshop on Mobile Computing Systems and Applications, pp. 9-14, 2019.
14 Martin Gill and Angela Spriggs, Assessing the impact of CCTV, vol. 292, Home Office Research, Development and Statistics Directorate London, 2005.
15 Michael Zink, Ramesh Sitaraman, and Klara Nahrstedt, "Scalable 360° video stream delivery: Challenges, solutions, and opportunities," Proceedings of the IEEE, vol. 107, no. 4, pp. 639-650, 2019.   DOI
16 Qingyang Zhang, Hui Sun, Xiaopei Wu, and Hong Zhong, "Edge video analytics for public safety: A review," Proceedings of the IEEE, vol. 107, no. 8, pp. 1675-1696, 2019.   DOI
17 I. K. Park, N. Singhal, M. H. Lee, S. Cho, and C. Kim, "Design and performance evaluation of image processing algorithms on GPUs," IEEE Transactions on Parallel and Distributed Systems, vol. 22, no. 1, pp. 91-104, 2011.   DOI
18 Helly M Patel, Krunal Panchal, Prashant Chauhan, and MB Potdar, "Large scale image processing using distributed and parallel architecture," International Journal of Computer Science and Information Technologies, Gujrat, India, vol. 6, no. 6, pp. 5531-5535, 2015.
19 Bilal Iqbal, Waheed Iqbal, Nazar Khan, Arif Mahmood, and Abdelkarim Erradi, "Canny edge detection and hough transform for high resolution video streams using hadoop and spark," Cluster Computing, vol. 23, no. 1, pp. 397-408, 2020.   DOI
20 A Xuggler, Xuggle, "xuggler api," 2012. [Online]. Available: http://www.xuggle.com/xuggler/