Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Screen-shot Image Demorieing Using Multiple Domain Learning

다중 도메인 학습을 이용한 화면 촬영 영상 내 모아레 무늬 제거 기법

  • Park, Hyunkook (Dongguk University, Department of Multimedia Engineering) ;
  • Vien, An Gia (Dongguk University, Department of Multimedia Engineering) ;
  • Lee, Chul (Dongguk University, Department of Multimedia Engineering)
  • 박현국 (동국대학교 멀티미디어공학과) ;
  • 비엔지아안 (동국대학교 멀티미디어공학과) ;
  • 이철 (동국대학교 멀티미디어공학과)
  • Received : 2020.11.23
  • Accepted : 2020.12.28
  • Published : 2021.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

We propose a moire artifacts removal algorithm for screen-shot images using multiple domain learning. First, we estimate clean preliminary images by exploiting complementary information of the moire artifacts in pixel value and frequency domains. Next, we estimate a clean edge map of the input moire image by developing a clean edge predictor. Then, we refine the pixel and frequency domain outputs to further improve the quality of the results using the estimated edge map as the guide information. Finally, the proposed algorithm obtains the final result by merging the two refined results. Experimental results on a public dataset demonstrate that the proposed algorithm outperforms conventional algorithms in quantitative and qualitative comparison.

본 논문은 다중 도메인 학습을 이용하여 화면 촬영 영상 내 모아레 무늬를 효과적으로 제거하는 기법을 제안한다. 제안하는 기법은 먼저 화소값 영역과 주파수 영역에서 입력 영상의 모아레 무늬를 각각 제거한다. 다음으로 모아레 영상에서 clean edge map을 추정하고, 추정된 clean edge map을 가이드 정보로 사용하여 화소값 영역과 주파수 영역에서 얻은 결과 영상의 품질을 향상시킨다. 마지막으로, 독립적으로 향상된 두 결과 영상을 적응적으로 결합하며 모아레 무늬가 제거된 최종 결과 영상을 생성한다. 컴퓨터 모의 실험결과를 통해 제안하는 기법이 기존의 알고리즘보다 모아레 무늬를 더욱 효과적으로 제거할 수 있음을 확인한다.

Keywords

Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. NRF-2019R1A2C4069806).

References

  1. M. Schoberl, W. Schnurrer, A. Oberdorster, S. Fossel, and A. Kaup, "Dimensioning of optical birefringent anti-alias filters for digital cameras," in Proceeding of IEEE International Conference on Image Processing, Sep. 2010.
  2. I. Pekkucuksen and Y. Altunbasak, "Multiscale gradients based color filter array interpolation," IEEE Transactions Image Processing, vol, 22, no. 1, pp. 157-165, Jan. 2013. https://doi.org/10.1109/TIP.2012.2210726
  3. Y. Sun, Y. Yu, and W. Wang, "Moire photo restoration using multiresolution convolutional neural networks," IEEE Transactions Image Processing, vol. 27, no. 8, pp. 4160-4172, Aug. 2018. https://doi.org/10.1109/TIP.2018.2834737
  4. X. Cheng, Z. Fu, and J. Yang, "Multi-scale dynamic feature encoding network for image demoireing," in Proceedings of IEEE/CVF International Conference on Computer Vision Workshops, Oct. 2019, pp. 3486-3493.
  5. B. He, C. Wang, B. Shi, and L. Duan, "Mop moire patterns using MopNet," in Proceedings of IEEE International Conference on Computer Vision, Oct./Nov. 2019, pp. 2424-2432.
  6. B. Zheng, S. Yuan, G. Slabaugh, and A. Leonardis, "Image demoireing with learnable bandpass filters," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp. 3636-3645.
  7. L. Cavigelli, P. Hager, and L. Benini, "CAS-CNN: A deep convolutional neural network for image compression artifact suppression," in Proceedings of International Joint Conference on Neural Networks, May 2017, pp. 752-759.
  8. A. G. Vien, H. Park, and C. Lee, "Dual-domain deep convolutional neural networks for image demoireing," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Jun. 2020, pp. 1934-1942.
  9. J. Yang, F. Liu, H. Yue, X. Fu, C. Hou, and F. Wu, "Textured image demoireing via signal decomposition and guided filtering," IEEE Transactions Image Processing, vol. 26, no. 7, pp. 3528-3541, Jul. 2017. https://doi.org/10.1109/TIP.2017.2698920
  10. B. He, C. Wang, B. Shi, and L.-Y. Duan, "FHDe2Net: Full high definition demoireing network." in Proceedings of European Conference on Computer Vision, Sep. 2020, pp. 713-729.
  11. B. Liu, X. Shu, and X. Wu, "Demoireing of camera-captured screen images using deep convolutional neural network," arXiv preprint arXiv:1804.03809, 2018.
  12. P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, "Multi-level wavelet CNN for image restoration," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Jun. 2018, pp. 886-895.
  13. B. Zheng, Y. Chen, X. Tian, F. Zhou, and X. Liu, "Implicit dual-domain convolutional network for robust color image compression artifact reduction," IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 11, pp. 3982-3994, Nov. 2020. https://doi.org/10.1109/TCSVT.2019.2931045
  14. X. Luo, J. Zhang, M. Hong, Y. Qu, Y. Xie, and C. Li, "Deep wavelet network with domain adaptation for single image demoireing," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Jun. 2020, pp. 1687-1694.
  15. L. Liu, J. Liu, S. Yuan, G. Slabaugh, A. Leonardis, W. Zhou, and Q. Tian, "Wavelet-based dual-branch network for image demoireing," in Proceedings of European Conference on Computer Vision, Aug. 2020, pp 86-102.
  16. X. Cheng, Z. Fu, and J. Yang, "Multi-scale dynamic feature encoding network for image demoireing," in Proceedings of IEEE/CVF International Conference on Computer Vision Workshops, Oct. 2019, pp. 3486-3493.
  17. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, "Residual dense net- work for image super-resolution," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp. 2472-2481.
  18. W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp. 1874-1883.
  19. F. Yu and V. Koltun, "Multi-scale context aggregation by dilated convolutions," in Proceedings of International Conference Learning Representations, May 2016.
  20. S. Yuan, R. Timofte, G. Slabaugh, and A. Leonardis, "AIM 2019 challenge on image demoireing: Dataset and study," in Proceedings of IEEE/CVF International Conference on Computer Vision Workshops, Oct. 2019, pp. 3526-3533.
  21. G. Huang, Z. Liu, L. V. D. Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp. 2261-2269.
  22. T. Tong, G. Li, X. Liu, and Q. Gao, "Image super-resolution using dense skip connections," in Proceedings of IEEE International Conference on Computer Vision, Jul. 2017, pp. 4809-4817.
  23. S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "CBAM: Convolutional block attention module," in Proceedings of European Conference on Computer Vision, Sep. 2018, pp. 3-19.
  24. O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation," in Proceedings of Medical Image Computing and Computer-Assisted Intervention, Nov. 2015, pp. 234.241.
  25. X. Wang, R. Girshick, A. Gupta, and K. He, "Non-local neural networks." in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2018, pp. 7794-7803.
  26. I. Loshchilov and F. Hutter, Decoupled weight decay regularization. in Proceedings in International Conference on Learning Representations, 2018.
  27. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004. https://doi.org/10.1109/TIP.2003.819861