Browse > Article
http://dx.doi.org/10.5909/JBE.2019.24.6.1053

Tile-Based 360 Degree Video Streaming System with User's gaze Prediction  

Lee, Soonbin (Gachon University, Department of Computer Engineering)
Jang, Dongmin (Sungkyunkwan University (SKKU), Department of Computer Education)
Jeong, Jong-Beom (Sungkyunkwan University (SKKU), Department of Computer Education)
Lee, Sangsoon (Gachon University, Department of Computer Engineering)
Ryu, Eun-Seok (Sungkyunkwan University (SKKU), Department of Computer Education)
Publication Information
Journal of Broadcast Engineering / v.24, no.6, 2019 , pp. 1053-1063 More about this Journal
Abstract
Recently, tile-based streaming that transmits one 360 video in several tiles, is actively being studied in order to transmit these 360 video more efficiently. In this paper, for the transmission of high-definition 360 video corresponding to user's viewport in tile-based streaming scenarios, a system of assigning the quality of tiles at each tile by applying the saliency map generated by existing network models is proposed. As a result of usage of Motion-Constrained Tile Set (MCTS) technique to encode each tile independently, the user's viewport was rendered and tested based on Salient360! dataset, streaming 360 video based on the proposed system results in gain to 23% of the user's viewport compared to using the existing high-efficiency video coding (HEVC).
Keywords
360 Video; HEVC; Saliency Map; Motion-Constrained Tile Set(MCTS);
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 Mary-Luc Champel, Thomas Stockhammer, Thierry Fautier, Emmanuel Thomas, Rob Koenen. 2016. Quality Requirements for VR. 116th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG 116/m39532.
2 Seehwan Yoo and Eun-Seok Ryu. 2017. Parallel HEVC decoding with asymmetric mobile multicores. Multimedia Tools and Applications 76, 16 (2017), 17337-17352.   DOI
3 Hyun-Joon Roh, SungWon Han, and Eun-Seok Ryu. 2017. Prediction complexity based HEVC parallel processing for asymmetric multicores. Multimedia Tools and Applications 76, 23 (2017), 25271-25284.   DOI
4 Tuan Thanh Le, Dien Van Nguyen, and Eun-Seok Ryu. 2018. Computing Offloading Over mmWave for Mobile VR: Make 360 Video Streaming Alive. IEEE Access (2018).
5 Dien Nguyen, Tuan Le, Sangsoon Lee, and Eun-Seok Ryu. 2018. SHVC Tile-Based 360-Degree Video Streaming for Mobile VR: PC Offloading Over mmWave. Sensors 18, 11 (2018), 3728.   DOI
6 Stefano Petrangeli, Viswanathan Swaminathan, Mohammad Hosseini, and Filip Turck. (2017). An HTTP/2-Based Adaptive Streaming Framework for $360^{\circ}$ Virtual Reality Videos. In Proceedings of the 25th ACM International Conference on Multimedia (MM '17). ACM, New York, NY, USA, 306-314.
7 Dongmin Jang, Jang-Woo Son, JongBeom Jeong, Eun-Seok Ryu, "Implementing Renderer for Viewport Dependent 360 Video", Journal of Broadcast Engineering (JBE) Vol. 23, No. 6, Nov. 2018.
8 Cagri Ozcinar, Ana De Abreu, and Aljoscha Smolic. "Viewport-aware adaptive $360^{\circ}$ video streaming using tiles for virtual reality." 2017 IEEE International Conference on Image Processing (ICIP) (2017): 2174-2178.
9 Jang-Woo Son, Dongmin Jang, Eun-Seok Ryu. 2018. Implementing Motion-Constrained Tile and Viewport Extraction for VR Streaming. ACM Network and Operating System Support for Digital Audio and Video 2018 (NOSSDAV2018).
10 Robert Skupin, Yago Sanchez, Karsten SAijhring, Thomas Schierl, Eun-Seok Ryu, and Jangwoo Son. 2018. Temporal MCTS Coding Constraints Implementation. 122th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG 122/m42423.
11 Jang-Woo Son, Dongmin Jang, JongBeom Jeong, Eun-Seok Ryu, "Viewport-Based 360 Degree Video Streaming using Motion-Constrained Tile Set", The Korean Institute of Broadcast and Media Engineers (KIBME) Summer Conference, pp.092-095, Jun. 21-23, 2018.
12 Robert Skupin, Yago Sanchez, Dimitri Podborski, Cornelius Hellge, and Thomas Schierl, "Viewport-dependent 360 degree video streaming based on the emerging Omnidirectional Media Format (OMAF) standard," Image Processing (ICIP), 2017 IEEE International Conference on. IEEE, Beijing, China, pp. 4592-4592, 2017.
13 Mikhail Startsev and Michael Dorr,"360-aware saliency estimation with conventional image saliency predictors.", Signal Processing: Image Communication 69 (2018), 43-52, 2018.   DOI
14 Matthias Kummerer, Lucas Thei and Matthias Bethge, (2014). "Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet."
15 Laurent itti, Christof Koch and Ernst Niebur, "A model of saliency-based visual attention for rapid scene analysis," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, Nov. 1998.   DOI
16 Borji, Ali. (2018). "Saliency Prediction in the Deep Learning Era: An Empirical Investigation."
17 Junting Pan, Cristian Canton-Ferrer, Kevin McGuinness, Noel E. O'Connor, Jordi Torres, Elisa Sayrol and Xavier Giro. "SalGAN: Visual Saliency Prediction with Generative Adversarial Networks." ArXiv abs/1701.01081 (2017).
18 Jesus Gutierrez, Erwan David, Antoine Coutrot, Matthieu Perreira Da Silva and Patrick Le Callet, "Introducing UN Salient360! Benchmark: A platform for evaluating visual attention models for 360 contents", International Conference on Quality of Multimedia Experience (QoMEX), Sardinia, Italy, May. 2018.
19 Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara, "Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model", IEEE Transactions on Image Processing 27, 10 (Oct 2018), 5142-5154.   DOI
20 Jonathan Harel, Christof Koch and Pietro Perona, "Graph-Based Visual Saliency." Adv. Neural Inform. Process. Syst. 19. 545-552, 2006.
21 Eleonora Vig, Michael Dorr and David Cox, "Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images." Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2798-2805. 10.1109/CVPR.2014.358, 2014.
22 Heinrich Hertz Institute Fraunhofer Institute for Telecommunications. 2018. High Efficiency Video Coding (HEVC) reference software HM. https://hevc.hhi.fraunhofer.de/.
23 Mohammad Hosseini, Viswanathan Swaminathan. "Adaptive 360 VR Video Streaming: Divide and Conquer.", In 2016 IEEE International Symposium on Multimedia (ISM). 107-110, 2016.
24 Ramin Ghaznavi-Youvalari, Alireza Zare, Alireza Aminlou, Miska M. Hannuksela and Moncef Gabbouj, "Shared Coded Picture Technique for Tile-Based Viewport-Adaptive Streaming of Omnidirectional Video," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 10, pp. 3106-3120, Oct. 2019.   DOI
25 Yashas Rai, Patrick Le Callet, and Philippe Guillotel. 2017. "Which saliency weighting for omnidirectional image quality assessment?", In 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX). 1-6, 2017.