Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Tile-Based 360 Degree Video Streaming System with User's gaze Prediction

사용자 시선 예측을 통한 360 영상 타일 기반 스트리밍 시스템

  • Lee, Soonbin (Gachon University, Department of Computer Engineering) ;
  • Jang, Dongmin (Sungkyunkwan University (SKKU), Department of Computer Education) ;
  • Jeong, Jong-Beom (Sungkyunkwan University (SKKU), Department of Computer Education) ;
  • Lee, Sangsoon (Gachon University, Department of Computer Engineering) ;
  • Ryu, Eun-Seok (Sungkyunkwan University (SKKU), Department of Computer Education)
  • 이순빈 (가천대학교 컴퓨터공학과) ;
  • 장동민 (성균관대학교 컴퓨터교육과) ;
  • 정종범 (성균관대학교 컴퓨터교육과) ;
  • 이상순 (가천대학교 컴퓨터공학과) ;
  • 류은석 (성균관대학교 컴퓨터교육과)
  • Received : 2019.10.07
  • Accepted : 2019.11.04
  • Published : 2019.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, tile-based streaming that transmits one 360 video in several tiles, is actively being studied in order to transmit these 360 video more efficiently. In this paper, for the transmission of high-definition 360 video corresponding to user's viewport in tile-based streaming scenarios, a system of assigning the quality of tiles at each tile by applying the saliency map generated by existing network models is proposed. As a result of usage of Motion-Constrained Tile Set (MCTS) technique to encode each tile independently, the user's viewport was rendered and tested based on Salient360! dataset, streaming 360 video based on the proposed system results in gain to 23% of the user's viewport compared to using the existing high-efficiency video coding (HEVC).

최근 360 영상에 대한 관심이 증대됨에 따라, 이러한 360 영상을 보다 효율적으로 전송하기 위해 하나의 360 영상을 여러 개의 타일로 나누어 전송하는 타일 기반 스트리밍이 활발히 연구되고 있다. 본 논문에서는 타일 기반 스트리밍 시나리오에서 사용자 시점에 대응하는 고화질 360 영상 전송을 위해, 기존 네트워크 모델로 생성된 중요도 맵(Saliency map)을 타일 기반 스트리밍에 적용하여 각 위치의 타일의 품질을 할당하는 시스템을 제안한다. 각 타일들을 독립적으로 부호화하기 위해 motion constrained tile set (MCTS) 기법을 적용함과 동시에 Salient360! 데이터셋으로 사용자 시선 시나리오를 토대로 사용자 시점 영상으로 복원하여 검증한 결과, 제안된 시스템을 기반으로 360 비디오 영상을 전송하면 기존 high-efficiency video coding (HEVC)을 사용하여 전송했을 때보다 사용자 시점의 영상은 큰 손실 없이 최대 23%의 BD-rate 효율을 보임을 확인하였다.

Keywords

References

  1. Mary-Luc Champel, Thomas Stockhammer, Thierry Fautier, Emmanuel Thomas, Rob Koenen. 2016. Quality Requirements for VR. 116th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG 116/m39532.
  2. Seehwan Yoo and Eun-Seok Ryu. 2017. Parallel HEVC decoding with asymmetric mobile multicores. Multimedia Tools and Applications 76, 16 (2017), 17337-17352. https://doi.org/10.1007/s11042-016-4269-2
  3. Hyun-Joon Roh, SungWon Han, and Eun-Seok Ryu. 2017. Prediction complexity based HEVC parallel processing for asymmetric multicores. Multimedia Tools and Applications 76, 23 (2017), 25271-25284. https://doi.org/10.1007/s11042-017-4413-7
  4. Tuan Thanh Le, Dien Van Nguyen, and Eun-Seok Ryu. 2018. Computing Offloading Over mmWave for Mobile VR: Make 360 Video Streaming Alive. IEEE Access (2018).
  5. Dien Nguyen, Tuan Le, Sangsoon Lee, and Eun-Seok Ryu. 2018. SHVC Tile-Based 360-Degree Video Streaming for Mobile VR: PC Offloading Over mmWave. Sensors 18, 11 (2018), 3728. https://doi.org/10.3390/s18113728
  6. Stefano Petrangeli, Viswanathan Swaminathan, Mohammad Hosseini, and Filip Turck. (2017). An HTTP/2-Based Adaptive Streaming Framework for $360^{\circ}$ Virtual Reality Videos. In Proceedings of the 25th ACM International Conference on Multimedia (MM '17). ACM, New York, NY, USA, 306-314.
  7. Cagri Ozcinar, Ana De Abreu, and Aljoscha Smolic. "Viewport-aware adaptive $360^{\circ}$ video streaming using tiles for virtual reality." 2017 IEEE International Conference on Image Processing (ICIP) (2017): 2174-2178.
  8. Jang-Woo Son, Dongmin Jang, Eun-Seok Ryu. 2018. Implementing Motion-Constrained Tile and Viewport Extraction for VR Streaming. ACM Network and Operating System Support for Digital Audio and Video 2018 (NOSSDAV2018).
  9. Robert Skupin, Yago Sanchez, Karsten SAijhring, Thomas Schierl, Eun-Seok Ryu, and Jangwoo Son. 2018. Temporal MCTS Coding Constraints Implementation. 122th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG 122/m42423.
  10. Dongmin Jang, Jang-Woo Son, JongBeom Jeong, Eun-Seok Ryu, "Implementing Renderer for Viewport Dependent 360 Video", Journal of Broadcast Engineering (JBE) Vol. 23, No. 6, Nov. 2018.
  11. Jang-Woo Son, Dongmin Jang, JongBeom Jeong, Eun-Seok Ryu, "Viewport-Based 360 Degree Video Streaming using Motion-Constrained Tile Set", The Korean Institute of Broadcast and Media Engineers (KIBME) Summer Conference, pp.092-095, Jun. 21-23, 2018.
  12. Robert Skupin, Yago Sanchez, Dimitri Podborski, Cornelius Hellge, and Thomas Schierl, "Viewport-dependent 360 degree video streaming based on the emerging Omnidirectional Media Format (OMAF) standard," Image Processing (ICIP), 2017 IEEE International Conference on. IEEE, Beijing, China, pp. 4592-4592, 2017.
  13. Matthias Kummerer, Lucas Thei and Matthias Bethge, (2014). "Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet."
  14. Laurent itti, Christof Koch and Ernst Niebur, "A model of saliency-based visual attention for rapid scene analysis," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, Nov. 1998. https://doi.org/10.1109/34.730558
  15. Borji, Ali. (2018). "Saliency Prediction in the Deep Learning Era: An Empirical Investigation."
  16. Junting Pan, Cristian Canton-Ferrer, Kevin McGuinness, Noel E. O'Connor, Jordi Torres, Elisa Sayrol and Xavier Giro. "SalGAN: Visual Saliency Prediction with Generative Adversarial Networks." ArXiv abs/1701.01081 (2017).
  17. Jesus Gutierrez, Erwan David, Antoine Coutrot, Matthieu Perreira Da Silva and Patrick Le Callet, "Introducing UN Salient360! Benchmark: A platform for evaluating visual attention models for 360 contents", International Conference on Quality of Multimedia Experience (QoMEX), Sardinia, Italy, May. 2018.
  18. Mikhail Startsev and Michael Dorr,"360-aware saliency estimation with conventional image saliency predictors.", Signal Processing: Image Communication 69 (2018), 43-52, 2018. https://doi.org/10.1016/j.image.2018.03.013
  19. Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara, "Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model", IEEE Transactions on Image Processing 27, 10 (Oct 2018), 5142-5154. https://doi.org/10.1109/tip.2018.2851672
  20. Jonathan Harel, Christof Koch and Pietro Perona, "Graph-Based Visual Saliency." Adv. Neural Inform. Process. Syst. 19. 545-552, 2006.
  21. Eleonora Vig, Michael Dorr and David Cox, "Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images." Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2798-2805. 10.1109/CVPR.2014.358, 2014.
  22. Mohammad Hosseini, Viswanathan Swaminathan. "Adaptive 360 VR Video Streaming: Divide and Conquer.", In 2016 IEEE International Symposium on Multimedia (ISM). 107-110, 2016.
  23. Ramin Ghaznavi-Youvalari, Alireza Zare, Alireza Aminlou, Miska M. Hannuksela and Moncef Gabbouj, "Shared Coded Picture Technique for Tile-Based Viewport-Adaptive Streaming of Omnidirectional Video," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 10, pp. 3106-3120, Oct. 2019. https://doi.org/10.1109/tcsvt.2018.2874179
  24. Yashas Rai, Patrick Le Callet, and Philippe Guillotel. 2017. "Which saliency weighting for omnidirectional image quality assessment?", In 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX). 1-6, 2017.
  25. Heinrich Hertz Institute Fraunhofer Institute for Telecommunications. 2018. High Efficiency Video Coding (HEVC) reference software HM. https://hevc.hhi.fraunhofer.de/.