Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Fast Content-preserving Seam Estimation for Real-time High-resolution Video Stitching

실시간 고해상도 동영상 스티칭을 위한 고속 콘텐츠 보존 시접선 추정 방법

  • 김태하 (서울과학기술대학교 전기정보공학과) ;
  • 양성엽 (서울과학기술대학교 전기정보공학과) ;
  • 강병근 (서울과학기술대학교 전자IT미디어공학과) ;
  • 이희경 (한국전자통신연구원 실감미디어연구실) ;
  • 서정일 (한국전자통신연구원 실감미디어연구실) ;
  • 이의진 (서울과학기술대학교 전기정보공학과)
  • Received : 2020.10.15
  • Accepted : 2020.11.16
  • Published : 2020.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

We present a novel content-preserving seam estimation algorithm for real-time high-resolution video stitching. Seam estimation is one of the fundamental steps in image/video stitching. It is to minimize visual artifacts in the transition areas between images. Typical seam estimation algorithms are based on optimization methods that demand intensive computations and large memory. The algorithms, however, often fail to avoid objects and results in cropped or duplicated objects. They also lack temporal consistency and induce flickering between frames. Hence, we propose an efficient and temporarily-consistent seam estimation algorithm that utilizes a straight line. The proposed method also uses convolutional neural network-based instance segmentation to locate seam at out-of-objects. Experimental results demonstrate that the proposed method produces visually plausible stitched videos with minimal visual artifacts in real-time.

본 논문은 실시간 고해상도 비디오 스티칭을 위한 새로운 콘텐츠 보존 시접선 추정 알고리즘을 제안한다. 시접선 추정은 영상 스티칭 후 중첩 영역에서의 시각적 왜곡을 최소화하기 위한 요소 기술 중 하나이다. 기존 시접선 추정 알고리즘들은 요구되는 연산량과 메모리 사용량이 높은 최적화 알고리즘에 기반을 두고 있음에도 불구하고, 추정된 시접선이 객체를 피하지 못해 객체를 자르거나 반복하는 현상을 유발한다. 또한, 프레임 간의 추정된 시접선의 시간적 일관성이 부족하여 불필요한 잦은 변동이 발생한다. 따라서, 본 논문에서는 직선의 시접선을 활용하여 효율적이고 시간적 일관성이 있으며, 심층신경망 기반 객체 세그먼테이션 알고리즘을 활용하여 객체를 피하여 시접선을 형성하는 시접선 추정 알고리즘을 제안하고자 한다. 고해상도 360° 다중 시점 동영상을 사용한 실험을 통해 제안하는 알고리즘이 기존 알고리즘보다 짧은 시간에 시각적으로 유사한 360VR 동영상을 생성하는 시접선을 추정함을 확인하였다.

Keywords

Acknowledgement

This work was supported by Electronics and Telecommunications Research Institute(ETRI) grant funded by ICT R&D program of MSIT/IITP[2018-0-00207, Immersive Media Research Laboratory].

References

  1. R. Szeliski, Computer Vision: Algorithms and Applications, Springer, 2010.
  2. M. Brown and D. Lowe., "Automatic Panoramic Image Stitching Using Invariant Features," International Journal of Computer Vision, Vol.74, No.1, pp. 59-73, Dec. 2007. https://doi.org/10.1007/s11263-006-0002-3
  3. J. Gao, S. J. Kim, and M. S. Brown. "Constructing Image Panoramas Using Dual-Homography Warping," Proceedings of Conference on Computer Vision and Pattern Recognition, CO, USA. pp. 49-56, 2011.
  4. J. Zaragoza, T. J. Chin, M. S. Brown, D. Suter, "As-Projective-As-Possible Image Stitching with Moving DLT," Proceedings of Conference on Computer Vision and Pattern Recognition, OR, USA, pp. 2339-2346, 2013.
  5. C. Herrmann, C. Wang, R.S. Bowen, E. Keyder, M. Krainin, C. Liu, R. Zabih, "Robust Image Stitching Using Multiple Registrations," Proceedings of the European Conference on Computer Vision, Munich, Germany, pp.53-69, 2018.
  6. V. Kwatra, A. Schodl, I. Essa, G. Turk, A. Bobick, "Graphcut Textures: Image and Video Synthesis Using Graph Cuts," ACM Transactions on Graphics, Vol.22, No.3, pp.277-286, Jul. 2003. https://doi.org/10.1145/882262.882264
  7. G. Zhang, Y. He, W. Chen, J. Jia, and H. Bao, "Multi-Viewpoint Panorama Construction with Wide-Baseline Images," IEEE Transactions on Image Processing, Vol.25, No.7, pp.3099-3111, Jul. 2016. https://doi.org/10.1109/TIP.2016.2535225
  8. P. Perez, M. Gangnet, A. Blake, "Poisson Image Editing," ACM Transactions on Graphics, Vol.22, No.3, pp. 313-318, Jul. 2003. https://doi.org/10.1145/882262.882269
  9. A. Levin, A. Zomet, S. Peleg, and Y. Weiss, "Seamless Image Stitching in the Gradient Domain," Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, pp.377-389, 2006.
  10. M. El-Saban, M. Izz, and A. Kaheel, "Fast Stitching of Videos Captured from Freely Moving Devices by Exploiting Temporal Redundancy," Proceedings of International Conference on Image Processing, Hong Kong, pp. 1193-1196, 2010.
  11. W. Jiang and J. Gu, "Video Stitching with Spatial-Temporal Content Preserving Warping," Proceedings of Conference on Computer Vision and Pattern Recognition Workshops, MA, USA, pp. 42-48, 2015.
  12. D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee, "YOLACT: Real-Time Instance Segmentation," Proceedings of International Conference on Computer Vision, Seoul, Korea, pp. 9157-9166, 2019.
  13. OpenCV SeamFinder, https://docs.opencv.org/3.4/d0/dd5/classcv_1_1detail_1_1_DpSeamFinder.html
  14. A. Efros, and T. William. "IImage Quilting for Texture Synthesis and Transfer," Proceedings of the 28th Anual Conference on Computer Graphics and InteractiveTechniques, CA, USA, pp. 341-346. 2001.
  15. A. Mittal, A. K. Moorthy and A. C. Bovik, "No-Reference Image Quality Assessment in the Spatial Domain," IEEE Transactions on Image Processing, Vol.21, No.12, pp. 4695-4708, Dec. 2012. https://doi.org/10.1109/TIP.2012.2214050