Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

MPEG-I RVS Software Speed-up for Real-time Application

실시간 렌더링을 위한 MPEG-I RVS 가속화 기법

  • Ahn, Heejune (Dept. of Electrical and Information Engineering, Seoul National University of Science and Technology) ;
  • Lee, Myeong-jin (Dept. of Electronics and Information Engineering, Korea Aerospace University)
  • 안희준 (서울과학기술대학교 전기정보공학과) ;
  • 이명진 (한국항공대학교 항공전자정보공학과)
  • Received : 2020.07.06
  • Accepted : 2020.09.02
  • Published : 2020.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Free viewpoint image synthesis technology is one of the important technologies in the MPEG-I (Immersive) standard. RVS (Reference View Synthesizer) developed by MPEG-I and in use in MPEG group is a DIBR (Depth Information-Based Rendering) program that generates an image at a virtual (intermediate) viewpoint from multiple viewpoints' inputs. RVS uses the mesh surface method based on computer graphics, and outperforms the pixel-based ones by 2.5dB or more compared to the previous pixel method. Even though its OpenGL version provides 10 times speed up over the non OpenGL based one, it still shows a non-real-time processing speed, i.e., 0.75 fps on the two 2k resolution input images. In this paper, we analyze the internal of RVS implementation and modify its structure, achieving 34 times speed up, therefore, real-time performance (22-26 fps), through the 3 key improvements: 1) the reuse of OpenGL buffers and texture objects 2) the parallelization of file I/O and OpenGL execution 3) the parallelization of GPU shader program and buffer transfer.

자유시점 영상합성기술은 MPEG-I(Immersive) 표준에서 중요한 기술 중 하나이다. 현재 MPEG-I에서 개발하여 사용하는 RVS (Reference View Synthesizer) 프로그램은 다수의 시점의 컬러영상과 깊이영상을 바탕으로 임의시점의 영상을 생성하는 DIBR (Depth Information-Based Rendering) 프로그램이다. RVS는 기존의 DIBR이 갖는 깊이정보 전달의 문제를 컴퓨터 그래픽스의 메쉬 표면 방식으로 접근하여 이전 화소방식에 비하여 2.5dB 이상의 성능향상을 보이며 OpenGL을 사용하면 CPU에서 동작하는 코드보다 10배 이상의 속도를 보인다. 그러나 여전히 2개의 2k 해상도 입력 영상에서 0.75fps 정도의 비실시간 처리속도를 보인다. 본 논문에서는 현 RVS의 내부 구현을 분석하고 이를 바탕으로 1) OpenGL 버퍼와 텍스쳐 객체의 재사용 2) 파일 입출력과 OpenGL 실행의 병렬화 3) GPU 셰이더 프로그램과 버퍼 데이터 전송의 병렬화를 적용하였다. 그 결과 두 개의 2k 해상도 입력 영상의 처리속도를 34배 이상 가속하여 22-28fps의 실시간 성능을 확보하였다.

Keywords

References

  1. G. Lafruit, D. Bonatto, C. Tulvan, M. Preda and L. Yu, "Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications," in SMPTE Motion Imaging Journal, vol. 128, no. 10, pp. 33-39, Nov.-Dec. 2019.
  2. G. Park, “Trend of free-view point video and A case study of extensible view point selection,” Information & communications magazine, Vol. 36, No. 12, pp. 3-9, 2019.
  3. J. Yang, M. Song, G. Park, "Implementation of Integrated Player System based on Free-Viewpoint Video Service according to User Selection," J. of Broadcasting Engineering, Vol. 25, No. 2, 2020.
  4. C. Fehn, "Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV," Stereoscopic Displays and Virtual Reality Systems XI. Vol. 5291. International Society for Optics and Photonics, 2004.
  5. S. Rogge, D Bonatto, J. Sancho, R. Salvador, E. Juarez, A. Munteanu, and G. Lafruit, "MPEG-I Depth Estimation Reference Software," In IEEE 2019 International Conference on 3D Immersion (IC3D) pp. 1-6, December, 2019
  6. S. Fachada, D. Bonatto, A. Schenkel and G. Lafruit, "Depth image based view synthesis with multiple reference views for virtual reality," (3DTV-CON), Helsinki, 2018.
  7. B. Kroon, G. Lafruit, "Proposed update of the RVS manual," ISO/IEC JTC1/SC29/WG11, w18068, Oct. 2018.
  8. H.-H. Kim, J.-G. Kim, "Performance Analysis on View Synthesis of 360 Videos for Omnidirectional 6DoF in MPEG-I" J. of Broadcasting Engineering, Vol. 24, No. 2, 2019.
  9. T. Senoh, K. Yamamoto, N. Tetsutni, H. Yasuda, and K. Wenger, "View synthesis reference software (VSRS) 4.2 with improved in-painting and hole filling, " ISO/IEC JTC1/SC29/WG11, M40657, Apr. 2017.
  10. OpenGL Programming Guide, 9th Edition. ISBN 978-0-134-49549-1.
  11. Cozzi, P., & Riccio, C. (Eds.). (2012). OpenGL insights. CRC press.
  12. B. Kroon, "ClassroomImage: A frame of ClassroomVideo with less noise and more views," ISO/IEC JTC1/SC29/WG11, m44762, Oct. 2018.