Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Performance Analysis of 6DoF Video Streaming Based on MPEG Immersive Video

MPEG 몰입형 비디오 기반 6DoF 영상 스트리밍 성능 분석

  • Jeong, Jong-Beom (Department of Computer Science Education, Sungkyunkwan University) ;
  • Lee, Soonbin (Department of Computer Science Education, Sungkyunkwan University) ;
  • Kim, Inae (Department of Computer Science Education, Sungkyunkwan University) ;
  • Ryu, Eun-Seok (Department of Computer Science Education, Sungkyunkwan University)
  • 정종범 (성균관대학교 컴퓨터교육학과) ;
  • 이순빈 (성균관대학교 컴퓨터교육학과) ;
  • 김인애 (성균관대학교 컴퓨터교육학과) ;
  • 류은석 (성균관대학교 컴퓨터교육학과)
  • Received : 2022.06.29
  • Accepted : 2022.08.16
  • Published : 2022.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

The moving picture experts group (MPEG) immersive video (MIV) coding standard has been established to support six degrees of freedom (6DoF) in virtual reality (VR) by transmitting high-quality multiple immersive videos. The MIV exploits two approaches considering tradeoff between bandwidth and computational complexity: 1) eliminating correlation between multi-view videos or 2) selecting representative videos. This paper presents performance analysis on intermediate synthesized views on source view positions and synthesized pose traces using high-efficiency video coding (HEVC) and versatile video coding (VVC) for above-mentioned two approaches.

다수의 고품질 몰입형 영상 전송을 통해 가상 현실에서 six degrees of freedom(6DoF)를 지원하기 위해 the moving picture experts group (MPEG) immersive video (MIV) 압축 표준이 설립되었다. MIV는 비트율과 연산 복잡도 간 상충관계를 고려하여 1) 시점 간 연관성 제거 또는 2) 대표 시점을 선택하여 전송하는 2가지 압축 방식을 제공한다. 본 논문은 전술한 두 가지 방식에 대해 high-efficiency video coding (HEVC), versatile video coding (VVC) 기반 성능 분석 결과를 입력 영상 위치에 합성한 가상 영상 및 사용자 시점 영상 중심으로 제시한다.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2022R1F1A1074935). This research was also supported by the SungKyunKwan University and the BK21 FOUR(Graduate School Innovation) funded by the Ministry of Education(MOE, Korea) and National Research Foundation of Korea(NRF).

References

  1. M.-L. Champel, T. Stockhammer, T. Fautier, E. Thomas, R. Koenen. 2016. Quality Requirements for VR. 116th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG 116/m39532.\
  2. T. T. Le, J. B. Jeong, E. -S. Ryu. 2019. Efficient transcoding and encryption for live 360 CCTV system. Applied Sciences, Vol. 9, No. 4. doi: https://doi.org/10.3390/app9040760
  3. J. -B. Jeong, S. Lee, I. Kim, S. Lee, E. -S. Ryu. 2020. Implementing VVC Tile Extractor for 360-degree Video Streaming Using Motion-Constrained Tile Set. Journal of Broadcast Engineering (JBE), Vol. 25, No. 7, pp. 1073-1080. doi: https://doi.org/10.5909/JBE.2020.25.7.1073
  4. T. T. Le, J. -B. Jeong, S. S. Lee, J. Kim, E. -S. Ryu. 2021. An Efficient Viewport-Dependent 360 VR System Based on Adaptive Tiled Streaming. CMC-Computers, Materials & Continua, Vol. 66, No. 3, pp. 2627-2643. doi: https://doi.org/10.32604/cmc.2021.013399
  5. J. -B. Jeong, S. Lee, I. Kim, E. -S. Ryu. 2021. Implementing Viewport Tile Extractor for Viewport-Adaptive 360-Degree Video Tiled Streaming. International Conference on Information Networking 2022 (ICOIN2022), pp. 8-12. doi: https://doi.org/10.1109/ICOIN50884.2021.9333964
  6. S. Lee, D. Jang, J. B. Jeong, E. -S. Ryu. 2019. Motion-constrained tile set based 360-degree video streaming using saliency map prediction. ACM Network and Operating System Support for Digital Audio and Video 2019 (NOSSDAV2019), pp. 20-24. doi: https://doi.org/10.1145/3304112.3325614
  7. A. Vetro, T. Wiegand, G. J. Sullivan. 2011. Overview of the stereo and multiview video coding extensions of the H. 264/MPEG-4 AVC standard. Proceedings of the IEEE, Vol. 99, No. 4, pp. 626-642. doi: https://doi.org/10.1109/JPROC.2010.2098830
  8. M. M. Hannuksela, Y. Yan, X. Huang, H. Li. 2015. Overview of the multiview high efficiency video coding (MV-HEVC) standard. IEEE International Conference on Image Processing 2015 (ICIP2015), pp. 2154-2158. doi: https://doi.org/10.1109/ICIP.2015.7351182
  9. R. Dore, F. Thudor. 2018. Outperforming 3DoF+ Anchors: first evidence. 123th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2018/m43504.
  10. J. -B. Jeong, S. Lee, D. Jang, E. -S. Ryu. 2019. Towards 3DoF+ 360 Video Streaming System for Immersive Media. IEEE Access, Vol. 7, pp. 136399-136408. doi: https://doi.org/10.1109/ACCESS.2019.2942771
  11. ISO/IEC JTC1/SC29/WG11. 2019. Call for Proposals on 3DoF+ Visual. 125th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG/n18145.
  12. J. Fleureau, F. Thudor, R. Dore, M. Dmitrichenko, B. Salahieh, J. Boyce. 2019. Description of Technicolor Intel response to MPEG-I 3DoF+ Call for Proposal. 126th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2019/m47445.
  13. M. Domanski, A. Dziembowski, D. Mieloch, O. Stankiewicz, J. Stankowski, A. Grzelka, G. Lee, J. Y. Jeong, J. Seo. 2019. Technical description of proposal for Call for Proposals on 3DoF+ Visual prepared by Poznan University of Technology (PUT) and Electronics and Telecommunications Research Institute (ETRI). 126th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2019/m47407.
  14. B. Kroon, B. Sonneveldt. 2019. Philips response to CfP on 3DoF+. 126th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2019/m47179.
  15. V. K. M. Vadakital, K. Roimela, L. Ilola, J. Keranen, M. Pesonen, S. Schwarz, J. lainema, M. Hannuksela. 2019. Description of Nokia's response to CFP for 3DOF+ visual. 126th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2019/m47372.
  16. B. Wang, Y. Sun, L. Yu. 2019. Description of Zhejiang University's response to 3DoF+ Visual CfP. 126th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2019/m47684.
  17. ISO/IEC JTC 1/SC 29/WG 4. 2021. Text of ISO/IEC FDIS 23090-12 MPEG Immersive Video. 135th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00111.
  18. B. Bross, Y. -K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, J. -R. Ohm. 2021. Overview of the versatile video coding (VVC) standard and its applications. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, pp. 3736-3764. doi: https://doi.org/10.1109/TCSVT.2021.3101953
  19. Jongseok Lee, Juntaek Park, Hansol Choi, Joohyung Byeon, Donggyu Sim, "Overview of VVC", Broadcast and Media Magazine, pp.10-25. Vol. 24, No. 3, Oct. 2019.
  20. E. S. Jang, M. Preda, K. Mammou, A. M. Tourapis, J. Kim, D. B. Graziosi, S. Rhyu, M. Budagavi. 2019. Video-based point-cloud-compression standard in MPEG: From evidence collection to committee draft [standards in a nutshell]. IEEE Signal Processing Magazine, Vol. 36, No. 3, pp. 118-123. doi: https://doi.org/10.1109/MSP.2019.2900721
  21. D. B. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, A. Tabatabai. 2020. An overview of ongoing point cloud compression standardization activities: Video-based (V-PCC) and geometry-based (G-PCC). APSIPA Transactions on Signal and Information Processing, Vol. 9, pp. 1-17. doi: https://doi.org/10.1017/ATSIP.2020.12
  22. DaeHyeok Gwon, Heeji Han, Haechul Choi , "Video based Point Cloud Compression with Versatile Video Coding" 2020 The Korean Institute of Broadcast and Media Engineers Summer Conference, pp. 375-377, July, 2020.
  23. ISO/IEC JTC 1/SC 29/WG 7. 2021. Text of ISO/IEC DIS 23090-5 Visual Volumetric Video-based Coding and Video-based Point Cloud Compression 2nd Edition. 135th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 7, MPEG/n00188.
  24. J. -B. Jeong, S. Lee, I. -W. Ryu, T. T. Le, E. -S. Ryu. 2020. Towards Viewport-dependent 6dof 360 Video Tiled Streaming for Virtual Reality Systems. ACM International Conference on Multimedia 2020 (MM 2020), pp. 3687-3695. doi: https://doi.org/10.1145/3394171.3413712
  25. S. Lee, J. -B. Jeong, I. Kim, S. Lee, E. -S. Ryu. 2020. Screen Content Coding Analysis to Improve Coding Efficiency for Immersive Video. Journal of Broadcast Engineering (JBE), Vol. 25, No. 6, pp. 911-921. doi: https://doi.org/10.5909/JBE.2020.25.6.911
  26. L. Kondrad, V. K. M. Vadakital, L. Ilola. 2020. .CE-1.3: frame packed video sub-bitstream type. 131th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 11, MPEG/m54274.
  27. B. Salahieh, G. Naf, J. Boyce. 2021. Frame Packing Implementation in TMIV. 134th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/m56827.
  28. J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. [MIV] Extraction and Merging on Frame Packed Video. 134th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/m56591.
  29. J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. Sub-bitstream packing based lightweight tiled streaming for 6 degree of freedom immersive video. Electronics Letters, Vol. 57, No. 25, pp. 973-976. doi: https://doi.org/10.1049/ell2.12329
  30. J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. DWS-BEAM: Decoder-Wise Subpicture Bitstream Extracting and Merging for MPEG Immersive Video. International Conference on Visual Communications and Image Processing 2021 (VCIP2021), pp. 1-5. doi: https://doi.org/10.1109/VCIP53242.2021.9675419
  31. ISO/IEC JTC 1/SC 29/WG 11. 2018. Reference View Synthesizer (RVS) manual. 124th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 11, MPEG/n18068.
  32. S. Facheada, D. Bonatto, A. Schenkel, G. Lafruit. 2018. Depth image based view synthesis with multiple reference views for virtual reality. The True Vision-Capture, Transmission and Display of 3D Video 2018 (3DTV-CON 2018), pp. 1-4. doi: https://doi.org/10.1109/3DTV.2018.8478484
  33. S. Kwak, J. Yun, J. Jeong, W. -S. Cheong, J. Seo. 2020. [MPEG-I Visual] Ray-based blending weight for 6DoF view synthesis. 131th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 11, MPEG/m54409.
  34. J. -B. Jeong, S. Lee, E. -S. Ryu. 2021. On OpenGL-based View Synthesizer Optimization for Real-time 6DoF Immersive Video Streaming. IEEE P3079.1 Contribution (DCN: 3079-21-0057-00-0001).
  35. D. Bonatto, S. Fachada, G. Lafruit. 2020. RaViS: Real-time accelerated View Synthesizer for immersive video 6DoF VR. Electronic Imaging, pp. 382-1-382-9. doi: https://doi.org/10.2352/ISSN.2470-1173.2020.13.ERVR-381
  36. H. Ahn, M. -j. Lee. 2020. MPEG-I RVS Software Speed-up for Real-time Application. Journal of Broadcast Engineering (JBE), Vol. 25, No. 5, pp. 655-664. doi: https://doi.org/10.5909/JBE.2020.25.5.655
  37. S. Kwak, J. Yun, J. -Y. Jeong, Y. Kim, I. Ihm, W. -S. Cheong, J. Seo. 2021. View synthesis with sparse light field for 6DoF immersive video. ETRI Journal, Vol. 44, No. 1, pp. 24-37. doi: https://doi.org/10.4218/etrij.2021-0205
  38. J. Jung, B. Kroon. 2021. Common Test Conditions for MPEG Immersive Video. 133th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00051.
  39. B. Salahieh, J. Jung, A. Dziembowski, C. Bachhuber. 2021. Test Model 8 for MPEG Immersive Video. 133th MPEG meeting of ISO/IEC JTC 1/SC 29/WG 4, MPEG/n00050.
  40. HM reference software, [Online]. Available at: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/.
  41. VTM reference software, [Online]. Available at: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftwareVTM/.
  42. Y. Sun, A. Lu, L. Yu. 2017. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE signal processing letters, Vol. 24, No. 9, pp. 1408-1412. doi: https://doi.org/10.1109/LSP.2017.2720693
  43. C. G. Bampis, A. C. Bovik, and Z. Li. 2018. A simple prediction fusion improves data-driven full-reference video quality assessment models. Picture Coding Symposium 2018 (PCS2018, pp. 298-302. doi: https://doi.org/10.1109/PCS.2018.8456293
  44. A. Dziembowski, D. Mieloch, J. Stankowski. A. Grzelka. 2022. IV-PSNR-the objective quality metric for immersive video applications. IEEE Transactions on Circuits and Systems for Video Technology. doi: https://doi.org/10.1109/TCSVT.2022.3179575
  45. J. -B. Jeong, S. Kim, S. Lee, I. Kim, E. -S. Ryu. 2021. Super Metric: Quality Assessment Methods for Immersive Video. Journal of Internet Computing and Services (JICS), Vol. 22, No. 2, pp. 51-58. doi: https://doi.org/10.7472/jksii.2021.22.2.51
  46. J. Jung, B. Kroon. 2021. Common Test Conditions for MPEG Immersive Video. 134th MPEG meeting of ISO/IEC JTC 1/SC 29/WG4, MPEG/n00085.
  47. Versatile video encoder (VVenC), [Online]. Available at: https://github.com/fraunhoferhhi/vvenc/.
  48. F. Bossen, K. Suhring, A. Wieckowski, S. Liu. 2021. VVC Complexity and Software Implementation Analysis. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, pp. 3765-3778. doi: https://doi.org/10.1109/TCSVT.2021.3072204