Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

A Deep Learning-based Real-time Deblurring Algorithm on HD Resolution

HD 해상도에서 실시간 구동이 가능한 딥러닝 기반 블러 제거 알고리즘

  • 심규진 (한국과학기술원 전기 및 전자공학부) ;
  • 고강욱 (한국과학기술원 전기 및 전자공학부) ;
  • 윤성준 (한국과학기술원 전기 및 전자공학부) ;
  • 하남구 (LIG 넥스원 전자광학연구소) ;
  • 이민석 (LIG 넥스원 전자광학연구소) ;
  • 장현성 (LIG 넥스원 전자광학연구소) ;
  • 권구용 (LIG 넥스원 전자광학연구소) ;
  • 김은준 (국방과학연구소) ;
  • 김창익 (한국과학기술원 전기 및 전자공학부)
  • Received : 2021.11.10
  • Accepted : 2021.12.21
  • Published : 2022.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

Image deblurring aims to remove image blur, which can be generated while shooting the pictures by the movement of objects, camera shake, blurring of focus, and so forth. With the rise in popularity of smartphones, it is common to carry portable digital cameras daily, so image deblurring techniques have become more significant recently. Originally, image deblurring techniques have been studied using traditional optimization techniques. Then with the recent attention on deep learning, deblurring methods based on convolutional neural networks have been actively proposed. However, most of them have been developed while focusing on better performance. Therefore, it is not easy to use in real situations due to the speed of their algorithms. To tackle this problem, we propose a novel deep learning-based deblurring algorithm that can be operated in real-time on HD resolution. In addition, we improved the training and inference process and could increase the performance of our model without any significant effect on the speed and the speed without any significant effect on the performance. As a result, our algorithm achieves real-time performance by processing 33.74 frames per second at 1280×720 resolution. Furthermore, it shows excellent performance compared to its speed with a PSNR of 29.78 and SSIM of 0.9287 with the GoPro dataset.

영상 블러 제거(deblurring)는 피사체의 움직임, 카메라의 흔들림, 초점의 흐림 등으로 인해 촬영 도중 발생한 영상 블러(blur)를 제거하는 것을 목표로 한다. 최근 스마트폰이 보급되며 휴대용 디지털카메라를 들고 다니는 것이 일상인 시대가 오면서 영상 블러 제거 기술은 그 필요성을 점점 더해가고 있다. 기존의 영상 블러 제거 기술들은 전통적인 최적화 기법을 활용하여 연구되어 오다가 최근에는 딥러닝이 주목받으며 합성곱 신경망 기반의 블러 제거 방법들이 활발하게 제안되고 있다. 하지만 많은 방법들이 성능에 먼저 초점을 맞추어 개발되어 알고리즘의 속도로 인하여 현실에서 실시간 활용이 어렵다는 문제점을 안고 있다. 이를 해결하고자 본 논문에서는여러 신경망 설계 기법을 활용하여 HD 영상에서도 30 FPS 이상의 실시간 구동이 가능한 딥러닝 기반 블러 제거 알고리즘을 설계하여 이를 제안한다. 또한 학습 및 추론 과정을 개선하여 속도에 별다른 영향 없이 신경망의 성능을 높이고 동시에 성능에 별다른 영향없이 신경망의 속도를 높였다. 이를 통해 최종적으로 1280×720 해상도에서 초당 33.74장의 프레임을 처리하며 실시간 동작이 가능함을 보여주었고 GoPro 데이터 세트를 기준으로 PSNR 29.79, SSIM 0.9287의 속도 대비 우수한 성능을 보여주었다.

Keywords

Acknowledgement

This work was supported by LIG Nex1.Co.,Ltd, originally funded by DAPA and ADD(UC190031FD).

References

  1. J. Pan, Z. Hu, Z. Su, and M. Yang, "Deblurring text images via L0-regularized intensity and gradient prior," IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAMI), vol. 39, no. 2, pp. 342-355, Feb 2017. https://doi.org/10.1109/TPAMI.2016.2551244
  2. T. Kim and K. Lee, "Segmentation-Free Dynamic Scene Deblurring," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Columbus, OH, USA, pp. 2766-2773, 2014
  3. J. Sun, W. Cao, Z. Xu, and J. Ponce, "Learning a convolutional neural network for non-uniform motion blur removal," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Boston, MA, USA, pp. 769-777, 2015
  4. M. Noroozi, P. Chandramouli, and P. Favaro, "Motion deblurring in the wild," Proceedings of the 39th German Conference on Pattern Recognition(GCPR), Basel, Switzerland, pp. 65-77, 2017.
  5. S. Nah, T. Kim, and K. Lee, "Deep multi-scale convolutional neural net work for dynamic scene deblurring,". Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Honolulu, Hawaii, USA, pp. 3883-3891, 2017.
  6. T. Kim, K. Lee, B. Scholkopf, and M. Hirsch, "Online Video Deblurring via Dynamic Temporal Blending Network," Proceedings of the IEEE International Conference on Computer Vision(ICCV), Venice, Italy, pp. 4058-4067, 2017
  7. X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, "Scale-recurrent network for deep image deblurring," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Salt Lake City, Utah, USA, pp. 8174-8182, 2018
  8. S. Nah, S. Son, and K. Lee, "Recurrent Neural Networks With Intra-Fra me Iterations for Video Deblurring," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, pp. 8094-8103, 2019
  9. Z. Zhong, Y. Gao, Y. Zheng, and B. Zheng. "Efficient spatio-temporal recurrent neural network for video deblurring," Proceedings of the 16th European Conference on Computer Vision(ECCV), pp. 191-207, 2020
  10. X. Hu, W. Ren, K. Yu, K. Zhang, X. Cao, W. Liu, and B. Menze, "Pyramid Architecture Search for Real-Time Image Deblurring," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4298-4307, 2021.
  11. H. Zhang, Y. Dai, H. Li, and P. Koniusz, "Deep stacked hierarchical multi-patch network for image deblurring," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, pp. 5978-5986, 2019.
  12. K. He, X. Zhang, S. Ren, and J. Sun. "Deep residual learning for image recognition," Proceedings of the IEEE conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, Nevada, USA, pp. 770-778, 2016.
  13. W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, "Real-time single image and video super-resoluti on using an efficient sub-pixel convolutional neural network," Proceed ings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, Nevada, USA, pp. 1874-1883, 2016.
  14. P. Goyal, P. Doll'ar, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, "Accurate, large minibatch SGD: training imagenet in 1 hour," CoRR, abs/1706.02677, 2017.
  15. I. Loshchilov and F. Hutter, "Sgdr: stochastic gradient descent with rest arts," CoRR, abs/1608.03983, 2016.
  16. T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li, "Bag of tricks for image classification with convolutional neural networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Long Beach, CA, USA, pp. 558-567, 2019.
  17. A. L. Maas, A. Y. Hannun, and A. Y. Ng, "Rectifier nonlinearities improve neural network acoustic models," Proceedings of the 30th International Conference on Machine Learning Workshop on Deep Learning for Audio, Speech and Language Processing, Atlanta, GA, USA, 2013
  18. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, "Beyond a gaussi an denoiser: residual learning of deep CNN for image denoising," IEEE Transactions on Image Processing(TIP), vol. 26, no. 7, pp. 3142-3155, July 2017. https://doi.org/10.1109/TIP.2017.2662206
  19. J. Kim, J. K. Lee, and K. Lee, "Accurate image super-resolution using very deep convolutional networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, Nevada, USA, pp. 1646-1654, 2016.
  20. H. Zhao, O. Gallo, I. Frosio, and J. Kautz, "Loss functions for neural networks for image processing," IEEE Transactions on Computational Imaging(TCI), vol. 3, no. 1, pp. 47-57, March 2016. https://doi.org/10.1109/TCI.2016.2644865