Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Distortion-guided Module for Image Deblurring

왜곡 정보 모듈을 이용한 이미지 디블러 방법

  • Kim, Jeonghwan (Department of Artificial Intelligence, Konkuk University) ;
  • Kim, Wonjun (Department of Electrical and Electronics Engineering, Konkuk University)
  • 김정환 (건국대학교 인공지능학과) ;
  • 김원준 (건국대학교 전기전자공학부)
  • Received : 2022.03.21
  • Accepted : 2022.04.29
  • Published : 2022.05.30
Download PDF
⟨ Previous Next ⟩

Abstract

Image blurring is a phenomenon that occurs due to factors such as movement of a subject and shaking of a camera. Recently, the research for image deblurring has been actively conducted based on convolution neural networks. In particular, the method of guiding the restoration process via the difference between blur and sharp images has shown the promising performance. This paper proposes a novel method for improving the deblurring performance based on the distortion information. To this end, the transformer-based neural network module is designed to guide the restoration process. The proposed method efficiently reflects the distorted region, which is predicted through the global inference during the deblurring process. We demonstrate the efficiency and robustness of the proposed module based on experimental results with various deblurring architectures and benchmark datasets.

영상 흐려짐은 피사체의 움직임, 카메라의 흔들림 등의 요인으로 발생하는 현상이다. 최근 합성곱 심층신경망(Convolution Neural Network, CNN)을 활용하여 흐려짐 현상을 복원하는 연구가 활발하게 진행되었으며, 원본과 정답 영상의 차이를 이용하여 복원 과정을 가이드하는 방법이 뛰어난 성능을 보였다. 본 논문에서는 왜곡 정보를 기반으로 흐려진 영상 복원 성능을 개선하는 방법을 제안한다. 이를 위해 학습 시, 원본과 정답 영상 차이에 대한 이진화를 수행하여 복원 과정을 가이드 할 수 있도록 하는 트랜스포머(Transformer) 기반 신경망 모듈을 설계하였다. 제안하는 방법은 학습 과정에서 잠재 특징을 기반으로 전역적 추론을 통해 예측한 왜곡 위치 정보 분포를 흐려짐 복원 과정에 반영한다. 다양한 영상 흐려짐 복원 신경망에 제안하는 모듈을 적용하여 복원 성능을 효과적으로 향상시킬 수 있음을 확인하였다.

Keywords

Acknowledgement

이 논문은 정부(과학기술정보통신부)의 재원으로 한국연구재단의 지원을 받아 수행된 연구임 (No. 2020R1F1A1068080).

References

  1. R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis, and W.T. Freeman, "Removing camera shake from a single photograph," ACM Transactions on Graphics (TOG), vol. 25, no. 3, pp. 787-794, Jul. 2006. doi: https://doi.org/10.1145/1141911.1141956
  2. Q. Shan, J. Jia, A. Agarwala, "High-quality motion deblurring from a single image,"ACM Transactions on Graphics (TOG), vol. 27, no. 3, pp. 73:1-73:10, Aug. 2008. doi: https://doi.org/10.1145/1360612.1360672
  3. S. Cho and S. Lee, "Fast motion deblurring,"ACM Transactions on Graphics (TOG), vol. 28, no. 5, pp. 145:1-145:8, Dec. 2009. doi: https://doi.org/10.1145/1618452.1618491
  4. S. Nah, T. H. Kim, and K. M. Lee, "Deep multi-scale convolutional neural network for dynamic scene deblurring," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pp. 3883-3891, Jul. 2017. doi: https://doi.org/10.1109/CVPR.2017.35
  5. H. Zhang, Y. Dai, H. Li, and P. Koniusz, "Deep stacked hierarchical multi-patch network for image deblurring," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pp. 5978-5986, Jun. 2019. doi: https://doi.org/10.1109/CVPR.2019.00613
  6. S. Lim, J. Kim, and W. Kim, "Deep spectral-spatial network for single image deblurring," IEEE Signal Process. Lett., vol. 27, no. 1, pp. 835-839, Dec. 2020. doi: https://doi.org/10.1109/LSP.2020.2995106
  7. S. J. Cho, S. W. Ji, J. P. Hong, S. W. Jung, and S. J. Ko, "Rethinking coarse-to-fine approach in single image deblurring," in Proc. IEEE Int. Conf. Comput. Vis., pp. 4641-4650, Oct. 2021. doi: https://doi.org/10.1109/ICCV48922.2021.00460
  8. J. Zhang, J. Pan, J. Ren, Y. Song, L. Bao, R. Lau, and M. Yang, "Dynamic scene deblurring using spatially variant recurrent neural networks," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pp. 2521-2529, Jun. 2018. doi: https://doi.org/10.1109/CVPR.2018.00267
  9. O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, "DeblurGAN-v2: deblurring (orders-of-magnitude) faster and better," in Proc. IEEE Int. Conf. Comput. Vis., pp. 8878-8887, Oct. 2019. doi: https://doi.org/10.1109/ICCV.2019.00897
  10. K. Purohit, M. Suin, A. N. Rajagopalan, and V. N. Boddeti, "Spatially-adaptive image restoration using distortion-guided networks," in Proc. IEEE Int. Conf. Comput. Vis., pp. 2309-2319, Oct. 2021. doi: https://doi.org/10.1109/ICCV48922.2021.00231
  11. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Proc. Neural Inf. Process. Syst., pp. 5998-6008, Dec. 2017. doi: https://dl.acm.org/doi/10.5555/3295222.3295349
  12. Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao, "Human-aware motion deblurring," in Proc. IEEE Int. Conf. Comput. Vis., pp. 5572-5581, Oct. 2019. doi: https://doi.org/10.1109/ICCV.2019.00567
  13. J. Rim, H. Lee, J. Won, and S. Cho, "Real-world blur dataset for learning and benchmarking deblurring algorithms," in Proc. Eur. Conf. Comput. Vis., pp. 184-201, Aug. 2020. doi: https://doi.org/10.1007/978-3-030-58595-2_12
  14. R. Guo, D. Niu, L. Qu, and Z. Li, "SOTR: segmenting objects with transformers," in Proc. IEEE Int. Conf. Comput. Vis., pp. 7157-7166, Oct. 2021. doi: https://doi.org/10.1109/ICCV48922.2021.00707
  15. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., pp. 770-778, Jun. 2016. doi: https://doi.org/10.1109/CVPR.2016.90
  16. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Lin, A. Desmaison, L. Antiga, A. Lerer, "Automatic differentiation in pytorch". in Proc. Conference and Workshop on Neural Information Processing Systems (NIPS), pp. 1-4, 2017. doi: https://openreview.net/pdf?id=BJJsrmfCZ
  17. D. P. Kingma and J. Ba, "Adam: a method for stochastic optimization," in Proc. Int. Conf. Learn. Representation, pp. 1-15, May 2015. doi: https://doi.org/10.48550/arXiv.1412.6980