Broadcasting and Media Magazine (방송과미디어)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지

딥러닝 기반 비디오 프레임 보간 기술 연구 동향

  • Published : 2022.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

비디오 프레임 보간 기술은 연속되어 있는 두 개의 프레임 사이의 중간 프레임을 생성하는 기술로 비디오의 프레임율을 늘리거나 슬로우 모션 영상을 생성 시 사용된다. 최근 딥러닝 기술의 발전에 따라 다양한 알고리즘의 비디오 프레임 보간 기술이 연구되고 있다. 본 고에서는 이러한 기알고리즘들을 커널 기반 방식과 플로우 기반 방식으로 분류하고, 각 범주에 속하는 대표적인 알고리즘들의 특징 및 한계점에 대해 살펴본다.

Keywords

Acknowledgement

이 논문은 2021년도 정부(과학기술정보통신부)의 재원으로 정보통신기획평가원의 지원을 받아 수행된 연구임 (No. 2021-0-00087, SD/HD급 저화질 미디어의 고품질 변환 기술 개발)

References

  1. Liu, Ziwei, et al. "Video frame synthesis using deep voxel flow." Proceedings of the IEEE International Conference on Computer Vision. 2017.
  2. Niklaus, Simon, Long Mai, and Feng Liu. "Video frame interpolation via adaptive convolution." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
  3. Niklaus, Simon, Long Mai, and Feng Liu. "Video frame interpolation via adaptive separable convolution." Proceedings of the IEEE International Conference on Computer Vision. 2017.
  4. Niklaus, Simon, and Feng Liu. "Context-aware synthesis for video frame interpolation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  5. Jiang, Huaizu, et al. "Super slomo: High quality estimation of multiple intermediate frames for video interpolation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  6. Meyer, Simone, et al. "Phasenet for video frame interpolation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
  7. Bao, Wenbo, et al. "Depth-aware video frame interpolation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  8. Niklaus, Simon, and Feng Liu. "Softmax splatting for video frame interpolation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  9. Lee, Hyeongmin, et al. "Adacof: Adaptive collaboration of flows for video frame interpolation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  10. Choi, Myungsub, et al. "Channel attention is all you need for video frame interpolation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.
  11. Cheng, Xianhang, and Zhenzhong Chen. "Video frame interpolation via deformable separable convolution." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.
  12. Huang, Zhewei, et al. "Rife: Real-time intermediate flow estimation for video frame interpolation." arXiv preprint arXiv:2011.06294 (2020).
  13. Ding, Tianyu, et al. "Cdfi: Compression-driven network design for frame interpolation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
  14. BT.2020-2 (10/2015) : Parameter values for ultra-high definition television systems for production and international programme exchange
  15. Dosovitskiy, Alexey, et al. "Flownet: Learning optical flow with convolutional networks." Proceedings of the IEEE international conference on computer vision. 2015.
  16. https://www.svp-team.com/
  17. https://topten.ai/convert-videos-to-60-fps-with-ai-interpolation/
  18. Sun, Deqing, et al. "Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  19. Teed, Zachary, and Jia Deng. "Raft: Recurrent all-pairs field transforms for optical flow." European conference on computer vision. Springer, Cham, 2020.
  20. O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MIC-CAI, 2015.
  21. Bao, Wenbo, et al. "Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement." IEEE transactions on pattern analysis and machine intelligence 43.3 (2019): 933-948.