Browse > Article
http://dx.doi.org/10.9717/kmms.2020.24.4.538

Real-Time Panoramic Video Streaming Technique with Multiple Virtual Cameras  

Ok, Sooyol (Dept. of Computer Engineering, Dong-A University)
Lee, Suk-Hwan (Dept. of Computer Engineering, Dong-A University)
Publication Information
Journal of Korea Multimedia Society / v.24, no.4, 2021 , pp. 538-549 More about this Journal
Abstract
In this paper, we introduce a technique for 360-degree panoramic video streaming with multiple virtual cameras in real-time. The proposed technique consists of generating 360-degree panoramic video data by ORB feature point detection, texture transformation, panoramic video data compression, and RTSP-based video streaming transmission. Especially, the generating process of 360-degree panoramic video data and texture transformation are accelerated by CUDA for complex processing such as camera calibration, stitching, blending, encoding. Our experiment evaluated the frames per second (fps) of the transmitted 360-degree panoramic video. Experimental results verified that our technique takes at least 30fps at 4K output resolution, which indicates that it can both generates and transmits 360-degree panoramic video data in real time.
Keywords
Virtual Camera; Panoramic Video Streaming; RTSP Streaming; 360-degree VR;
Citations & Related Records
연도 인용수 순위
  • Reference
1 M. Hosseini and V. Swaminathan, "Adaptive 360 VR Video Streaming: Divide and Conquer," IEEE International Symposium on Multimedia (ISM), pp. 107-110, 2016.
2 A.T. Nasrabadi, A. Mahzari, J.D. Beshay, and R. Prakash, "Adaptive 360-Degree Video Streaming using Scalable Video Coding," Proceedings of ACM on Multimedia Conference, pp. 1689-1697, 2017.
3 D.V. Nguyen, H.T.T. Tran, and T.C. Thang, "A Client-based Adaptation Framework for 360-Degree Video Streaming," Journal of Visual Communication and Image Representation, Vol. 59, pp. 231-243, 2019.   DOI
4 J. Jeong and K. Jun, "High Resolution 360 degree Video Generation System using Multiple Cameras," Journal of Korea Multimedia Society, Vol. 19, No. 8, pp. 1329-1336, 2016.   DOI
5 D.V. Nguyen, T.T. Le, S. Lee, and E.-S. Ryu, "SHVC Tile-Based 360-Degree Video Streaming for Mobile VR: PC Offloading Over mmWave," Sensors, Vol. 18, Issue. 11, 3728, 2018.   DOI
6 W. Ye, K. Yu, Y. Yu, and J. Li, "Logical Stitching: A Panoramic Image Stitching Method Based on Color Calibration Box," 14th IEEE International Conference on Signal Processing (ICSP), pp. 1139-1143, 2018.
7 M. Han, S.-H. Lee, and S. Ok, "A Real-Time Architecture of 360-Degree Panoramic Video Streaming System," 2nd IEEE International Conference on Knowledge Innovation and Invention, pp. 477-480, 2019.
8 T. Dendale, Improving 360-Degrees Panoramic Video Stitching, Master Thesis, Universiteit Hasselt, 2016.
9 M. Brown and D.G. Lowe, "Automatic Panoramic Image Stitching using Invariant Features," International Journal of Computer Vision, Vol. 74, Issue 1, pp. 59-73, 2007.   DOI
10 Y. Lu, K. Wang, and G. Fan, "Photometric Calibration and Image Stitching for a Large Field of View Multi-Camera System," Sensors, Vol. 16, Issue 4, E516, 2016.
11 A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield, "Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data," arXiv Preprint, arXiv: 1810.10093, 2018.
12 R. Girshick, "Faster R-CNN," IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 2015.
13 M. Wang, S. Niu, and X. Yang, "A Novel Panoramic Image Stitching Algorithm Based on ORB," International Conference on Applied System Innovation (ICASI), 2017.
14 G.W. Hyun, Real-time Iimage Generation Method through Multiple Virtual Camera, Master Thesis, Tongmyong University, 2017.
15 NVIDIA Video Codec SDK, https://developer.nvidia.com/nvidia-video-codec-sdk (accessed on July 1, 2019).
16 Ultra Video Group, http://ultravideo.cs.tut.fi/ (accessed on July 1, 2019).
17 VRWorks-360Video-Multiband Blending. https://developer.nvidia.com/vrworks/vrworks-360video/Multiband-Blending (accessed on July 1, 2019).
18 E. Rublee, V. Rabaud, K. Konolige and G. Bradski, "ORB: an efficient alternative to SIFT or SURF," IEEE International Conference Computer Vision (ICCV), pp. 24-33, 2011.
19 L. Yu, Z. Yu, and Y. Gong, "An Improved ORB Algorithm of Extracting and Matching Features," International Journal of Signal Processing, Image Processing and Pattern Recognition, Vol. 8, No. 5, pp. 117-126, 2015.   DOI
20 F. Wu and X. Fang, "An Improved RANSAC Homography Algorithm for Feature Based Image Mosaic," 7th WSEAS International Conference on Signal Processing, Computational Geometry & Artificial Vision, pp. 202-207, August 2007.
21 T. Moriyoshi, F. Takano and Y. Nakamura, "GPU Acceleration of H.264 / MPEG-4 AVC Software Video Encoder," Asia Pacific Signal and Information Processing Association, Annual Summit and Conference (AP SIPA ASC), 2011.
22 MS-RTSP, Real-Time Streaming Protocol (RTSP ) Windows Media Extensions, v20180 912, Sept. 2018.
23 NVIDIA DevTech and A. Obukhov, "GPU-Accelerated Video Encoding," GPU Technology Conference, 2010.
24 Internet Streaming Media, Wireless, and Multicast technology, services & standards. https://www.live555.com (accessed on July 1, 2019).
25 I. Santos-Gonzalez, A. Rivero-Garcia, J. Molina-Gil, and P. Caballero-Gil, "Implementation and Analysis of Real-Time Streaming Protocols," Sensors, Vol. 17, Issue 4, 846, 2017.   DOI