Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Implementing 3DoF+ 360 Video Compression System for Immersive Media

실감형 미디어를 위한 3DoF+ 360 비디오 압축 시스템 구현

  • Jeong, Jong-Beom (Sungkyunkwan University (SKKU), Department of Computer Education) ;
  • Lee, Soonbin (Gachon University, Department of Computer Engineering) ;
  • Jang, Dongmin (Sungkyunkwan University (SKKU), Department of Computer Education) ;
  • Lee, Sangsoon (Gachon University, Department of Computer Engineering) ;
  • Ryu, Eun-Seok (Sungkyunkwan University (SKKU), Department of Computer Education)
  • 정종범 (성균관대학교 컴퓨터교육과) ;
  • 이순빈 (가천대학교 컴퓨터공학과) ;
  • 장동민 (성균관대학교 컴퓨터교육과) ;
  • 이상순 (가천대학교 컴퓨터공학과) ;
  • 류은석 (성균관대학교 컴퓨터교육과)
  • Received : 2019.07.26
  • Accepted : 2019.08.29
  • Published : 2019.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

System for three degrees of freedom plus (3DoF+) and 6DoF requires multi-view high resolution 360 video transmission to provide user viewport adaptive 360 video streaming. In this paper, we implement 3DoF+ 360 video compression system which removes the redundancy between multi-view videos and merges the residual into one video to provide high quality 360 video corresponding to an user's head movement efficiently. Implementations about 3D warping based redundancy removal method between 3DoF+ 360 videos and residual extraction and merger are explained in this paper. With the proposed system, 20.14% of BD-rate reduction in maximum is shown compared to traditional high-efficiency video coding (HEVC) based system.

사용자 움직임 적응적 360 비디오 스트리밍을 지원하는 three degrees of freedom plus (3DoF+), 6DoF를 위한 시스템은 여러 위치에서 촬영된 다수의 고화질 360 비디오 전송을 요구한다. 이를 위해 본 논문에서는 사용자 시점에 대응하는 고화질 360 비디오 제공을 위해 3DoF+ 360 비디오 중복성 제거 기법을 적용하고 잔여 비디오를 하나의 영상으로 병합하여 압축 후 전송하는 시스템을 구현한다. 3DoF+ 360 비디오 간 중복성 제거를 위한 3D warping을 기반으로 하는 뷰 간 중복성 제거 기술과 비디오 복원에 필요한 타일들만 추출 및 병합해주는 잔여 뷰 병합 기술에 대한 구현 내용을 설명한다. 제안된 시스템을 기반으로 3DoF+ 360 비디오 전송을 수행하면, 기존 high-efficiency video coding (HEVC)을 사용하여 전송했을 때 대비 최대 20.14%의 BD-rate 감소가 가능함을 확인하였다.

Keywords

References

  1. Mary-Luc Champel, Thomas Stockhammer, Thierry Fautier, Emmanuel Thomas, Rob Koenen. 2016. Quality Requirements for VR. 116th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG 116/m39532.
  2. Hyun-Joon Roh, SungWon Han, and Eun-Seok Ryu. 2017. Prediction complexitybased HEVC parallel processing for asymmetric multicores. Multimedia Tools and Applications 76, 23 (2017), 25271-25284. https://doi.org/10.1007/s11042-017-4413-7
  3. Hyun-Joon Roh, Bok-Gi Lee, and Eun-Seok Ryu. 2018. Tile Partitioning and Allocation for HEVC Parallel Decoding on Asymmetric Multicores. The Journal of Korean Institute of Communications and Information Sciences (J-KICS), Vol.43, No.05, pp. 791-800 https://doi.org/10.7840/kics.2018.43.5.791
  4. Seehwan Yoo and Eun-Seok Ryu. 2017. Parallel HEVC decoding with asymmetric mobile multicores. Multimedia Tools and Applications 76, 16 (2017), 17337-17352. https://doi.org/10.1007/s11042-016-4269-2
  5. Dien Nguyen, Tuan Le, Sangsoon Lee, and Eun-Seok Ryu. 2018. SHVC Tile-Based 360-Degree Video Streaming for Mobile VR: PC Offloading Over mmWave. Sensors 18, 11 (2018), 3728. https://doi.org/10.3390/s18113728
  6. Tuan Thanh Le, Dien Van Nguyen, and Eun-Seok Ryu. 2018. Computing Offloading Over mmWave for Mobile VR: Make 360 Video Streaming Alive. IEEE Access (2018).
  7. Robert Skupin, Yago Sanchez, Karsten SAijhring, Thomas Schierl, Eun-Seok Ryu, and Jangwoo Son. 2018. Temporal MCTS Coding Constraints Implementation. 122th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG 122/m42423.
  8. Jang-Woo Son, Dongmin Jang, Eun-Seok Ryu. 2018. Implementing Motion-Constrained Tile and Viewport Extraction for VR Streaming. ACM Network and Operating System Support for Digital Audio and Video 2018 (NOSSDAV2018).
  9. Jang-Woo Son, Eun-Seok Ryu. 2018. Tile-Based 360-Degree Video Streaming for Mobile Virtual Reality in Cyber Physical System. Elsevier, Computers and Electrical Engineering.
  10. Xin Wang, Lulin Chen, Shuai Zhao, and Shawmin Lei. 2017. From OMAF for 3DoF VR to MPEG-I Media Format for 3DoF+, Windowed 6DoF and 6DoF VR. 119th MPEG meeting of ISO/IECJTC1/SC29/WG11, MPEG119/m44197.
  11. Jong-Beom Jeong, Dongmin Jang, Jangwoo Son, and Eun-Seok Ryu. 2018. 3DoF+ 360 Video Location-Based Asymmetric Down-Sampling for View Synthesis to Immersive VR Video Streaming. Sensors 18, 9(2018), 3148. https://doi.org/10.3390/s18093148
  12. ISO/IEC JTC1/SC29/WG11. 2019. Call for Proposals on 3DoF+ Visual. 125th MPEG meeting of ISO/IECJTC1/SC29/WG11, MPEG/n18145.
  13. ISO/IEC JTC1/SC29/WG11. 2018. 3DoF+ Software Platform description. 125th MPEG meeting of ISO/IECJTC1/SC29/WG11, MPEG2018/n18070.
  14. ISO/IEC JTC1/SC29/WG11. 2018. Reference View Synthesizer(RVS) manual. 124th MPEG meeting of ISO/IECJTC1/SC29/WG11, MPEG124/n18068.
  15. Patrick Boissonade, Joel Jung, and P.Nikitin. 2018. [MPEG-IVisual] View synthesis algorithm for windowed-6DoF. 124th MPEG meeting of ISO/IECJTC1/SC29/ WG11, MPEG2018/m44788.
  16. ISO/IECJTC1/SC29/WG11.2008. Overview of 3D Video Coding. 84th MPEG meeting of ISO/IECJTC1/SC29/WG11, MPEG2008/n9784.
  17. Gerhard Tech, Ying Chen, Karsten Muller, Jens-Rainer Ohm, Anthony Vetro, Ye-Kui Wang. 2016. Overview of the Multiview and 3D Extensions of High Efficiency Video Coding. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 26.1: 35-49. https://doi.org/10.1109/TCSVT.2015.2477935
  18. Ying Chen, Gerhard Tech, Krzysztof Wegner and Sehoon Yea. 2014. Test Model 10 of 3D-HEVC and MV-HEVC. 110th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2014/n14975.
  19. Joel Jung, P. Nikitin. 2017. [MPEG-I Visual] List of issues with MVand 3D-HEVC when coding 1D- and 2D-multi-view contents. 120th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2017/m41701.
  20. Renaud Dore, Franck Thudor. 2018. Outperforming 3DoF+ Anchors: first evidence. 123th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2018/m43504.
  21. Franck Thudor, Renaud Dore. RVS based 3DoF+ evidence results. 124th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2018/m44588.
  22. Bart Kroon, Bart Sonneveldt. 2019. Philips response to CfP on 3DoF+. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47179.
  23. Julien Fleureau, Franck Thudor, Renaud Doré, Max Dmitrichenko, Basel Salahieh, Jill Boyce. 2019. Description of Technicolor Intel response to MPEG-I 3DoF+ Call for Proposal. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47445.
  24. Marek Domański, Adrian Dziembowski, Dawid Mieloch, Olgierd Stankiewicz, Jakub Stankowski, Adam Grzelka, Gwangsoon Lee, Jun Young Jeong, Jeongil Seo. 2019. Technical description of proposal for Call for Proposals on 3DoF+ Visual prepared by Poznań University of Technology (PUT) and Electronics and Telecommunications Research Institute (ETRI). 126th MPEG meeting of ISO/IEC JTC1/SC29/WG11, MPEG2019/m47407.
  25. Vinod Kumar Malamal Vadakital, Kimmo Roimela, Lauri Ilola, Jaakko Keränen, Mika Pesonen, Sebastian Schwarz, Jani lainema, Miska Hannuksela. 2019. Description of Nokia's response to CFP for 3DOF+ visual. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47372.
  26. Bin Wang, Yule Sun, Wenyi Wang, Jiafeng Chen, Lu Yu, Huiming Yu. 2019. Description of Zhejiang University's response to 3DoF+ Visual CfP. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/m47684.
  27. Basel Salahieh, Bart Kroon, Joel Jung, Marek Domanski. 2019. Test Model for Immersive Video. 126th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2019/n18470.
  28. Ying Chen, Xin Zhao, Li Zhang, Je-Won Kang. 2016. Multiview and 3D Video Compression Using Neighboring Block Based Disparity Vectors. IEEE Transactions on Multimedia 18.4 (2016): 576-589. https://doi.org/10.1109/TMM.2016.2525010
  29. Je-Won Kang, Ying Chen, Li Zhang, Marta Karczewicz. 2014. Low Complexity Neighboring Block based Disparity Vector Derivation in 3D-HEVC. IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2014. p. 1921-1924.
  30. M Mary Synthuja Jain Preetha, L Padma Suresh, and M John Bosco. 2012. Image segmentation using seeded region growing. In 2012 International Conference on Computing, Electronics and Electrical Technologies (ICCEET). IEEE, 576-583.
  31. Bart Kroon. 2018. 3DoF+ test sequence ClassroomVideo. 122th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2018/m42415.
  32. Renaud Dore. 2018. Technicolor 3DoF+ test materials. 122th MPEG meeting of ISO/IEC JTC1/SC29/ WG11, MPEG2018/m42349.
  33. 2018. Open computer vision. [Online]. https://github.com/opencv/opencv/releases/tag/3.4.2