Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Image Stitching focused on Priority Object using Deep Learning based Object Detection

딥러닝 기반 사물 검출을 활용한 우선순위 사물 중심의 영상 스티칭

  • Rhee, Seongbae (Department of Electronic Engineering, Kyung Hee University) ;
  • Kang, Jeonho (Department of Electronic Engineering, Kyung Hee University) ;
  • Kim, Kyuheon (Department of Electronic Engineering, Kyung Hee University)
  • Received : 2020.09.14
  • Accepted : 2020.10.27
  • Published : 2020.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, the use of immersive media contents representing Panorama and 360° video is increasing. Since the viewing angle is limited to generate the content through a general camera, image stitching is mainly used to combine images taken with multiple cameras into one image having a wide field of view. However, if the parallax between the cameras is large, parallax distortion may occur in the stitched image, which disturbs the user's content immersion, thus an image stitching overcoming parallax distortion is required. The existing Seam Optimization based image stitching method to overcome parallax distortion uses energy function or object segment information to reflect the location information of objects, but the initial seam generation location, background information, performance of the object detector, and placement of objects may limit application. Therefore, in this paper, we propose an image stitching method that can overcome the limitations of the existing method by adding a weight value set differently according to the type of object to the energy value using object detection based on deep learning.

최근 Panorama와 360° 영상이 대표되는 몰입형 미디어 콘텐츠의 활용이 증가하고 있다. 일반적인 카메라 한 대를 통해서 해당 콘텐츠를 생성하기에는 시야각이 제한되기 때문에, 다수의 카메라로 촬영한 영상을 넓은 시야각을 갖는 하나의 영상으로 합성하는 영상 스티칭이 주로 사용되고 있다. 그러나 촬영하는 카메라 간의 시차(Parallax)가 크다면 스티칭 영상에서 시차 왜곡이 발생할 수 있고, 이는 사용자의 콘텐츠 몰입을 제한하기 때문에 시차 왜곡을 극복할 수 있는 영상 스티칭 기술이 필요하다. 시차 왜곡을 극복하기 위한 기존의 Seam Optimization 기반 영상 스티칭 방법은 사물의 위치 정보를 반영하기 위하여 에너지 함수나 객체 세그먼트 정보를 활용하고 있지만, 초기 Seam 생성 위치, 배경 정보, 사물 검출기의 성능 그리고 사물의 배치 등의 제한 사항으로 인해 기술의 적용이 제한될 수 있다. 이에 본 논문에서는 딥러닝 기반 사물 검출을 활용하여 사물의 종류에 따라 다르게 설정한 가중치 값을 시각적 인지 에너지 값에 더함으로써, 기존 기술의 제한 사항을 극복할 수 있는 영상 스티칭 방법을 제안하고자 한다.

Keywords

Acknowledgement

This research was supported by Korea Electric Power Corporation. (Grant number:R18XA02)

References

  1. Im, Jiheon, et al. "Images Grouping Technology based on Camera Sensors for Efficient Stitching of Multiple Images." Journal of Broadcast Engineering, Vol.22, No.6, pp.713-723, 2017. https://doi.org/10.5909/JBE.2017.22.6.713
  2. R. Szeliski, "Image Alignment and Stitching: A Tutorial." Foundations and Trends in Computer Graphics and Computer Vision, Vol. 2, No.1, 2006.
  3. Rhee, Seongbae, Jeonho Kang, and Kyuheon Kim. "Local Differential Pixel Assessment Method for Image Stitching." Journal of Broadcast Engineering, Vol.24, No.5, pp.775-784, 2019 https://doi.org/10.5909/JBE.2019.24.5.775
  4. Wei, L. Y. U., et al. "A survey on image and video stitching." Virtual Reality & Intelligent Hardware Vol. 1, No.1 pp.55-83, 2019. https://doi.org/10.3724/SP.J.2096-5796.2018.0008
  5. Zhang, Guofeng, et al. "Multi-viewpoint panorama construction with wide-baseline images." IEEE Transactions on Image Processing, Vol.25, No.7, pp.3099-3111, 2016. https://doi.org/10.1109/TIP.2016.2535225
  6. Lin, Kaimo, et al. "Seagull: Seam-guided local alignment for parallax-tolerant image stitching." European conference on computer vision. Springer, Cham, 2016.
  7. Avidan, Shai, and Ariel Shamir. "Seam carving for content-aware image resizing." ACM SIGGRAPH 2007 papers. pp.10-es, 2007.
  8. TANG, Yu; JIANG, Huiyan. "Highly efficient image stitching based on energy map". In: 2009 2nd International Congress on Image and Signal Processing IEEE, pp.1-5, 2009.
  9. TANG, Yu; SHIN, Jungpil. "Image stitching with efficient brightness fusion and automatic content awareness.". International Conference on Signal Processing and Multimedia Applications (SIGMAP), pp.60-66, 2014.
  10. Abdukholikov, Murodjon, and Taegkeun Whangbo. "Fast image stitching method for handling dynamic object problems in Panoramic Images." KSII Transactions on Internet & Information Systems, Vol.11, No.11, 2017.
  11. Kang, Jeonho, et al. "Minimum Error Seam-Based Efficient Panorama Video Stitching Method Robust to Parallax." IEEE Access 7, pp. 167127-167140, 2019. https://doi.org/10.1109/ACCESS.2019.2953705
  12. Rhee, Seongbae, Jeonho Kang, and Kyuheon Kim. "Parallax Distortion Detection and Correction Method for Video Stitching by using LDPM Image Assessment." Journal of Broadcast Engineering, Vol.25, No.5, pp.685-697, 2020. https://doi.org/10.5909/JBE.2020.25.5.685
  13. He, Kaiming, et al. "Mask r-cnn." Proceedings of the IEEE international conference on computer vision, pp.2961-2969, 2017.
  14. Bolya, Daniel, et al. "Yolact: Real-time instance segmentation." Proceedings of the IEEE international conference on computer vision., pp.9157-9166, 2019.
  15. Rublee, Ethan, et al. "ORB: An efficient alternative to SIFT or SURF." 2011 International conference on computer vision. IEEE, pp.2564-2571, 2011.