Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Toward 6 Degree-of-Freedom Video Coding Technique and Performance Analysis

6 자유도 전방위 몰입형 비디오의 압축 코덱 개발 및 성능 분석

  • Park, Hyeonsu (Department of Electronic and Electrical Engineering, Ewha W. University) ;
  • Park, Sang-hyo (Department of Electronic and Electrical Engineering, Ewha W. University) ;
  • Kang, Je-Won (Department of Electronic and Electrical Engineering, Ewha W. University)
  • 박현수 (이화여자대학교 엘텍공과대학 전자전기공학과) ;
  • 박상효 (이화여자대학교 엘텍공과대학 전자전기공학과) ;
  • 강제원 (이화여자대학교 엘텍공과대학 전자전기공학과)
  • Received : 2019.10.04
  • Accepted : 2019.11.08
  • Published : 2019.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, as the demand for immersive videos increases, efficient video processing techniques for omnidirectional immersive video is actively developed by MPEG-I. While the omnidirectional video provides a larger degree of freedom for a free viewpoint, the size of the video increases significantly. Furthermore, in order to compress 6 degree-of-freedom (6 DoF) videos that support motion parallax, it is required to develop a codec to yield better coding efficiency. In this paper, we develop a 6 DoF codec using Versatile Video Coding (VVC) as the next generation video coding standard. To the authors' best knowledge, this is the first VVC-based 6 DoF video codec toward the future ISO/IEC 23090 Part 7 (Metadata for Immersive Media (Video)) MPEG-I standardization. The experiments were conducted on the seven test video sequences specified in Common Test Condition (CTC) in two operation modes of TMIV (Test Model for Immersive Media) software. It is demonstrated that the proposed codec improves coding performance around 33.8% BD-rate reduction in the MIV (Metadata for Immersive Video) mode and 30.2% BD-rate reduction in the MIV view mode as compared to the state-of-the-art TMIV reference software. We also show the performance comparisons using Immersive Video PSNR (IV-PSNR) and Mean Structural Similarity (MSSIM).

최근 몰입형 비디오의 수요가 점차 늘어남에 따라 국제 표준 단체인 MPEG-I에서 전방위 몰입형 비디오의 처리 기술이 활발하게 개발 중이다. 전방위 몰입형 비디오는 사용자 시점의 자유도가 증가함에 따라 비디오 신호의 크기가 급격히 증가하여 효과적인 압축 기술이 필수적이다. 더욱이 사용자의 움직임에 따른 보다 자유로운 시점 변환을 지원하는 6 자유도 (6-Degree-of_Freedom, 6DoF) 비디오의 압축을 위해서는 보다 우수한 부호화 효율을 제공하는 코덱의 개발이 필요하다. 본 논문에서는 ISO/IEC 23090 Part 7 (Metadata for Immersive Media (Video))에서 진행 중인 몰입형 비디오의 압축 표준 프로젝트의 테스트 모델인 TMIV (Test Model for Immersive Video)에 기존 적용된 High Efficiency Video Coding (HEVC)를 최근 차세대 비디오 압축 표준 개발 중인 Versatile Video Coding (VVC)로 대체하여 성능 분석을 수행하고, VVC의 툴 분석으로부터 디블로킹 필터를 TMIV의 패치 아틀라스에 선택적으로 적용하는 것이 부호화 효율을 증대시킬 수 있음을 보인다. VVC 기반의 6 DoF 비디오 코덱의 성능 평가는 본 논문이 최초로 그에 따른 향후 6DoF지원 몰입형 비디오 표준 개발 방향을 제시한다. TMIV의 두 가지 작동 모드인 MIV (Metadata for Immersive Video) 모드와 MIV 시점 모드에서 공통 실험 조건에 명시된 일곱 가지 시퀀스에 대해 전체적으로 실험을 진행하였다. 기존 HEVC를 VVC로 대체함으로써 MIV 모드 방식에서 33.8%, MIV 시점 모드에서 30.2%의 Peak Signal-to-Noise Ratio (PSNR) 관점에서의 부호화 성능 향상을 제공하였다. 이외에도 3차원 비디오의 인지 화질 평가를 위하여 사용하는 평가 지표로 IV-PSNR (Immersive Video PSNR)와 MSSIM (Mean Structural Similarity)를 이용하여 성능을 평가하였다.

Keywords

References

  1. J. Lee, "Immersive Video Format Standardization Trends", Broadcast and Media, 23(4), pp.31-40, 2018.
  2. ISO/IEC JTC1/SC29/WG11/M41555, "Proposed architectures for supporting Windowed 6DoF, Omnidirectional 6DoF and 6DoF media", MPEG 120, Oct. 2017.
  3. https://www.theverge.com/2017/4/19/15345738/facebook-surround-360-video-cameras-f8-conference-2017
  4. Yu-Chuan su, Kristen Grauman, "Learning Compressible 360◦Video Isomers", In CVPR, 2018.
  5. ISO/IEC JTC1/SC29/WG11/M48278, "[MPEG-I Visual] Proposal of Depth Improvement for Depth Estimation Reference Software (DERS8.0)", MPEG 127, July. 2019.
  6. ISO/IEC JTC1/SC29/WG11/M45995, "[MPEG-I Visual] Results of Exploration Experiments for MPEG-I: Omnidirectional 6DoF - EEsynthesis", MPEG 125, Jan. 2019.
  7. ISO/IEC JTC1/SC29/WG11/M47705, "[MPEG-I Visual] 6DoF EE Proposal for ClassroomImage Pose e", MPEG 126, Mar. 2019.
  8. ISO/IEC JTC1/SC29/WG11/N18564, "Exploration Experiments for MPEG-I: 6DoF", MPEG 127, July. 2019.
  9. ISO/IEC JTC1/SC29/WG11/N18450, "Manual of Depth Estimation Reference Software (DERS 8.0)", MPEG 126, Mar. 2019.
  10. ISO/IEC JTC1/SC29/WG11/N18172, "Versatile View Synthesizer (VVS) 2.0 manual", MPEG 125, Jan. 2019.
  11. ISO/IEC JTC1/SC29/WG11/M47170, "[MPEG-I Visual] Proposal of Trimming and Color Matching of Multi-View Sequences", MPEG 126, Mar. 2019.
  12. ISO/IEC JTC1/SC29/WG11/N18145, "Call for Proposals on 3DoF+ Visual", MPEG 125, Jan 2019.
  13. ISO/IEC JTC1/SC29/WG11/N18577, "Test Model 2 for Immersive Video", MPEG 127, July 2019.
  14. ISO/IEC JTC1/SC29/WG11/M47544, "Description of Technicolor Intel response to MPEG-I 3DoF+ Call for Proposal", MPEG 126, Mar. 2019.
  15. ISO/IEC JTC1/SC29/WG11/M47407, "Technical description of proposal for Call for Proposals on 3DoF+ Visual prepared by Poznan University of Technology (PUT) and Electronics and Telecommunications Research Institute (ETRI)", MPEG 126, Mar. 2019.
  16. ISO/IEC JTC1/SC29/WG11/M47372, "Description of Nokia's response to CFP for 3DOF+ visual", MPEG 126, Mar. 2019.
  17. ISO/IEC JTC1/SC29/WG11/M47684, "Description of Zhejiang University's response to 3DoF+ Visual CfP", MPEG 126, Mar. 2019.
  18. ISO/IEC JTC1/SC29/WG11/M47179, "Philips response to CfP on 3DoF", MPEG 126, Mar. 2019.
  19. JVET of ITU-T and ISO/IEC, "Meeting Report of the 15th Meeting of the Joint Video Experts Team (JVET)", Document JVET-O2000, July 2019.
  20. ISO/IEC JTC1/SC29/WG11/N18563, "Common Test Conditions for Immersive Video", MPEG 127, July 2019.
  21. ISO/IEC JTC1/SC29/WG11/N18576, "Working Draft2 of Metadata for Immersive Video", MPEG 127, July 2019.
  22. JCT-VC of ITU-T and ISO/IEC, "Common test conditions", Document JCTVC-Z1100, January 2017
  23. JVET of ITU-T and ISO/IEC, "JVET common test conditions and software reference configurations for SDR video", Document JVETN1010, March 2019.
  24. JVET of ITU-T and ISO/IEC, "Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6)", Document JVET-O2002-v2, July 2019.
  25. ISO/IEC JTC1/SC29/WG11/N18068, "Reference View Synthesizer (RVS) manual", MPEG 124, Oct. 2018.
  26. ISO/IEC JTC1/SC29/WG11/M42415, "3DoF+ test sequence ClassroomVideo", MPEG 122, April. 2018.
  27. ISO/IEC JTC1/SC29/WG11/M43748, "Kermit test sequence for Windowed 6DoF Activities", MPEG 123, July. 2018.
  28. Y. Sun, A. Lu, L. Yu, "Weighted-to-spherically-uniform quality evaluation for omnidirectional video", IEEE Signal Process. Lett., 24(9), pp.1408-1412, 2017. https://doi.org/10.1109/LSP.2017.2720693
  29. G. Choi, G. Bang, "MPEG-I Video Standardization Trends", Broadcast and Media, 23(4), pp.41-57, 2018.
  30. ISO/IEC JTC1/SC29/WG11/N18709, "Software manual of IV-PSNR for Immersive Video", MPEG 127, July 2019.