Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Group-based Adaptive Rendering for 6DoF Immersive Video Streaming

6DoF 몰입형 비디오 스트리밍을 위한 그룹 분할 기반 적응적 렌더링 기법

  • Lee, Soonbin (Department of Computer Science Education, Sungkyunkwan University) ;
  • Jeong, Jong-Beom (Department of Computer Science Education, Sungkyunkwan University) ;
  • Ryu, Eun-Seok (Department of Computer Science Education, Sungkyunkwan University)
  • 이순빈 (성균관대학교 컴퓨터교육학과) ;
  • 정종범 (성균관대학교 컴퓨터교육학과) ;
  • 류은석 (성균관대학교 컴퓨터교육학과)
  • Received : 2022.01.27
  • Accepted : 2022.03.18
  • Published : 2022.03.30
Download PDF
⟨ Previous Next ⟩

Abstract

The MPEG-I (Immersive) group is working on a standardization project for immersive video that provides 6 degrees of freedom (6DoF). The MPEG Immersion Video (MIV) standard technology is intended to provide limited 6DoF based on depth map-based image rendering (DIBR) technique. Many efficient coding methods have been suggested for MIV, but efficient transmission strategies have received little attention in MPEG-I. This paper proposes group-based adaptive rendering method for immersive video streaming. Each group can be transmitted independently using group-based encoding, enabling adaptive transmission depending on the user's viewport. In the rendering process, the proposed method derives weights of group for view synthesis and allocate high quality bitstream according to a given viewport. The proposed method is implemented through the Test Model for Immersive Video (TMIV) test model. The proposed method demonstrates 17.0% Bjontegaard-delta rate (BD-rate) savings on the peak signalto-noise ratio (PSNR) and 14.6% on the Immersive Video PSNR(IV-PSNR) in terms of various end-to-end evaluation metrics in the experiment.

MPEG-I (Immersive) 그룹에서는 6자유도(DoF: degrees of freedom)를 제공하는 몰입형 비디오의 표준화 프로젝트를 진행 중에 있다. MPEG Immersive Video (MIV) 표준화 기술에서는 사용자에게 움직임 시차(parallax)를 제공하기 위해 취득한 다수의 영상을 깊이 맵 기반 이미지 렌더링(depth map-based image rendering, DIBR)을 바탕으로 임의의 사용자 시점의 뷰를 렌더링하게 된다. 현재 MIV에서는 효율적인 부호화를 위한 기술들이 많이 논의된 바 있지만, 전송 측면에 대해서는 여전히 논의가 필요하다. 본 논문은 사용자 시점에 적응적인 몰입형 비디오 스트리밍을 위한 품질 할당 기법을 제안한다. 현재 MIV에서 지원하고 있는 그룹 분할 기법을 통하여 독립적으로 전송, 복원이 가능한 시점 그룹 단위를 생성하여 이를 사용자 시점에 기반한 품질 할당 기법을 통해 효율적인 전송이 가능하도록 한다. 제안하는 적응적 전송 기법은 Test Model for Immersive Video (TMIV) 시험모델을 통해 구현되었으며, 주어진 합성 시점 위치에 따라 렌더링 과정에서의 기여도를 그룹별로 계산하고 우선 시점 그룹을 판단하여 고품질로 전송한다. 사용자 시점에 대한 렌더링 비교 결과를 통해 제안하는 기법이 기존 TMIV 대비 PSNR 지표에서 평균 17.0%, IV-PSNR 지표에서 14.6%의 BD-rate 감소율을 보여 효율적인 전송이 가능함을 보였다.

Keywords

Acknowledgement

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2020-0-00920, Development of ultra high resolution unstructured plenoptic video storage/compression/streaming technology for medium to large space).

References

  1. ISO/IEC JTC1/SC29/WG11, "New Work Item Proposal on Coded Representation of Immersive Media," MPEG 117, N16541, Jan.2017.
  2. ISO/IEC JTC1/SC29/WG11/N18145, "Call for Proposals on 3DoF+Visual", MPEG 125, Jan. 2019.
  3. ISO/IEC JTC1/SC29/WG11/M41555, "Proposed architectures for supporting Windowed 6DoF, Omnidirectional 6DoF and 6DoF media", MPEG 120, Oct. 2017.
  4. ISO/IEC JTC1/SC29/WG11/N19492, "Exploration Experiments on Processing for Future MPEG Immersive Video", MPEG 131, Jul. 2020.
  5. ISO/IEC JTC1/SC29/WG11/N19485, "Description of MPEG Immersive Video Core Experiments 1", MPEG 131, Jul. 2020.
  6. ISO/IEC JTC1/SC29/WG11/N19491, "Exploration Experiments on Coding for Future MPEG Immersive Video", MPEG 130, Apr. 2020.
  7. ISO/IEC JTC1/SC29/WG11/M49590, Video-based Point Cloud Coding High Level Syntax: Updates and Unification with the Working Draft on Metadata for Immersive Video", MPEG 127, Jul. 2019.
  8. ISO/IEC JTC1/SC29/WG11/M49406, "Group-Based TMIV", MPEG 127, Jul. 2019.
  9. ISO/IEC JTC1/SC29/WG11/M54151, "Grouping and anchor study on MIV content", MPEG 131, Jun, 2020.
  10. X. Corbillon, G. Simon, A. Devlic and J. Chakareski, "Viewportadaptive navigable 360-degree video delivery," 2017 IEEE International Conference on Communications (ICC), 2017, pp. 1-7. doi: https://doi.org10.1109/ICC.2017.7996611.
  11. J. Son, D. Jang, E-S. Ryu. "Implementing Motion-Constrained Tile and Viewport Extraction for VR Streaming." In Proceedings of the 28th ACM Network and Operating System Support for Digital Audio and Video (NOSSDAV'18), pp. 61-66, 2018. doi: https://doi.org/10.1145/3210445.3210455
  12. S. Lee, D. Jang, J-B. Jeong, E-S. Ryu. "Motion-constrained tile set based 360-degree video streaming using saliency map prediction." In Proceedings of the 29th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'19), pp. 20-24, 2019. doi: https://doi.org/10.1145/3304112.3325614
  13. ISO/IEC JTC1/SC29/WG11/M47749, "[V-PCC] A tile group design for V-PCC", A. M. Tourapis, J. Kim, K. Mammou, MPEG 126, Mar. 2019.
  14. ISO/IEC JTC1/SC29/WG11/N18654, ,"Requirements for Immersive Media Access and Delivery", MPEG 127, Jul. 2019.
  15. J. Jeong, S. Lee, I. Ryu, T. T. Le, and E. Ryu. "Towards Viewportdependent 6DoF 360 Video Tiled Streaming for Virtual Reality Systems." In Proceedings of the 28th ACM International Conference on Multimedia (MM '20), pp. 3687-3695, 2020. doi: https://doi.org/10.1145/3394171.3413712
  16. ISO/IEC JTC1/SC29/WG11/M54409, "[MPEG-I Visual] Ray-based blending weight for 6DoF view synthesis," MPEG 131, 2020.
  17. ISO/IEC JTC1/SC29/WG11/N19483, "Test Model 6 for MPEG Immersive Video," MPEG 131, Jul. 2020.
  18. ISO/IEC JTC1/SC29/WG11/N19484, "Common Test Conditions for MPEG Immersive Video," MPEG 131, Jul. 2020.
  19. ISO/IEC JTC1/SC29/WG4/M57726, "AhG on MPEG Immersive Video", MPEG 136, Oct. 2021.
  20. ISO/IEC JTC1/SC29/WG4/N0145, "Exploration Experiments on Future MPEG Immersive video", MPEG 136, Oct. 2021.
  21. ISO/IEC JTC1/SC29/WG4/N0013, "Software manual of IV-PSNR for Immersive Video," MPEG 132, Oct. 2020.
  22. J. Jeong, S. Lee, D. Jang and E. Ryu, "Towards 3DoF+ 360 Video Streaming System for Immersive Media," in IEEE Access, vol. 7, pp. 136399-136408, 2019. doi: https://doi.org/10.1109/ACCESS.2019.2942771.
  23. S. Lee, J. -B. Jeong and E. -S. Ryu, "Efficient Group-Based Packing Strategy for 6DoF Immersive Video Streaming," 2022 International Conference on Information Networking (ICOIN), 2022, pp. 310-314. doi: https://doi.org/10.1109/ICOIN53446.2022.9687139.
  24. Kwak, S., Yun, J., Jeong, J.-Y., Kim, Y., Ihm, I., Cheong, W.-S., and Seo, J., View synthesis with sparse light field for 6DoF immersive video, ETRI Journal 44 (2022), 24-37. doi: https://doi.org/10.4218/etrij.2021-0205
  25. Jeong, J.-B., Lee, S. and Ryu, E.-S. (2021), Sub-bitstream packing based lightweight tiled streaming for 6 degree of freedom immersive video. Electron. Lett., 57: 973-976. doi: https://doi.org/10.1049/ell2.12329