Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

A Review on Deep Learning-based Image Outpainting

딥러닝 기반 이미지 아웃페인팅 기술의 현황 및 최신 동향

  • Received : 2020.11.30
  • Accepted : 2021.01.07
  • Published : 2021.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

Image outpainting is a very interesting problem in that it can continuously fill the outside of a given image by considering the context of the image. There are two main challenges in this work. The first is to maintain the spatial consistency of the content of the generated area and the original input. The second is to generate high quality large image with a small amount of adjacent information. Existing image outpainting methods have difficulties such as generating inconsistent, blurry, and repetitive pixels. However, thanks to the recent development of deep learning technology, deep learning-based algorithms that show high performance compared to existing traditional techniques have been introduced. Deep learning-based image outpainting has been actively researched with various networks proposed until now. In this paper, we would like to introduce the latest technology and trends in the field of outpainting. This study compared recent techniques by analyzing representative networks among deep learning-based outpainting algorithms and showed experimental results through various data sets and comparison methods.

이미지 아웃페인팅은 이미지의 맥락을 고려하여 주어진 이미지의 외부를 지속적으로 채울 수 있다는 점에서 매우 흥미로운 문제이다. 이 작업에는 두 가지 주요 과제가 있다. 첫 번째는 생성된 영역의 내용과 원래 입력의 공간적 일관성을 유지하는 것이다. 두 번째는 적은 양의 인접 정보로 고품질의 큰 이미지를 생성하는 것이다. 기존의 이미지 아웃페인팅 방법은 일관되지 않고 흐릿하며 반복되는 픽셀을 생성하는 등 어려움을 겪고 있다. 하지만 최근 딥러닝 기술의 발달에 힘입어 기존의 전통적인 기법들에 비해 높은 성능을 보여주고 있는 알고리즘들이 소개되었다. 딥러닝 기반 아웃 페인팅은 현재까지도 다양한 네트워크가 제안되며 활발히 연구되고 있다. 본 논문에서는 아웃 페인팅 분야의 최신 기술 현황 및 동향을 소개하고자 한다. 딥러닝 기반의 아웃페인팅 알고리즘 중 대표적인 네트워크들을 분석하고 다양한 데이터 셋과 비교 방법을 통한 실험 결과를 보여줌으로써 최근 기법들을 비교하고자 한다.

Keywords

References

  1. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Verdera, J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing, 10(8), 1200-1211. https://doi.org/10.1109/83.935036
  2. Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417-424. ACM Press/AddisonWesley Publishing Co., 2000.
  3. Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 341-346. ACM, 2001
  4. Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1033-1038. IEEE, 1999
  5. Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
  6. Iizuka, Satoshi, Edgar Simo-Serra, and Hiroshi Ishikawa. "Globally and locally consistent image completion." ACM Transactions on Graphics (ToG) 36.4 (2017): 1-14.
  7. Liu, Guilin, et al. "Image inpainting for irregular holes using partial convolutions." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  8. Yu, Jiahui, et al. "Generative image inpainting with contextual attention." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
  9. Johannes Kopf, Wolf Kienzle, Steven Drucker, and Sing Bing Kang. Quality prediction for image completion. ACM Transactions on Graphics (TOG), 31(6):131, 2012.
  10. Josef Sivic, Biliana Kaneva, Antonio Torralba, Shai Avidan, and William T Freeman. Creating and exploring a large photorealistic virtual space. 2008.
  11. Miao Wang, Yu-Kun Lai, Yuan Liang, Ralph R. Martin, and Shi-Min Hu. Biggerpicture: Data-driven image extrapolation using graph matching. ACM Trans. Graph., 33(6):173:1-173:13, Nov. 2014.
  12. Yinda Zhang, Jianxiong Xiao, James Hays, and Ping Tan. Framebreak: Dramatic image extrapolation by guided shiftmaps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1171-1178, 2013.
  13. Yang, Zongxin, et al. "Very Long Natural Scenery Image Prediction by Outpainting." Proceedings of the IEEE International Conference on Computer Vision. 2019.
  14. Teterwak, Piotr and Sarna, Aaron and Krishnan, Dilip and Maschinot, Aaron and Belanger, David and Liu, Ce and Freeman, William T. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE International Conference on Computer Vision, pages 10521-10530, 2019.
  15. Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Wide context semantic image extrapolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1399-1408, 2019.
  16. Zhang, X., Chen, F., Wang, C., Tao, M., & Jiang, G. P. (2020). Sienet: Siamese expansion network for image extrapolation. IEEE Signal Processing Letters, 27, 1590-1594. https://doi.org/10.1109/LSP.2020.3019705
  17. Gatys, Leon A and Ecker, Alexander S and Bethge, Matthias. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
  18. Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li. Perceptual losses for real-time style transfer and superresolution. In European conference on computer vision, pages 694-711. Springer, 2016.
  19. Kim, K., Yun, Y., Kang, K. W., Kong, K., Lee, S., & Kang, S. J. (2021). Painting Outside As Inside: Edge Guided Image Outpainting via Bidirectional Rearrangement With Progressive Step Learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2122-2130).
  20. Xiao, Jianxiong and Hays, James and Ehinger, Krista A and Oliva, Aude and Torralba, Antonio. Sun database: Largescale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492. IEEE, 2010.
  21. Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing, 21(12):4695-4708, 2012. https://doi.org/10.1109/TIP.2012.2214050
  22. Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2242, 2016.
  23. Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.
  24. Mark Sabini and Gili Rusak. Painting outside the box: Image out-painting with gans. arXiv preprint arXiv:1808.08483, 2018.