Journal of Broadcast Engineering (방송공학회논문지)

The Korean Institute of Broadcast and Media Engineers (한국방송∙미디어공학회)
권호 표지
DOI QR코드

DOI QR Code

Raw Sensor Single Image Super Resolution Using Color Corrector-Attention Network

코렉터 어텐션 네트워크을 이용한 로우 센서 영상 초해상화 기법

  • Paul Shin (Dept. of Smart Energy System Engineering, Seoul National University of Science and Technology) ;
  • Teaha Kim (Dept. of Electrical and Information Engineering, Seoul National University of Science and Technology) ;
  • Yeejin Lee (Dept. of Electrical and Information Engineering, Dept. of Smart Energy System Engineering, Seoul National University of Science and Technology)
  • 신바울 (서울과학기술대학교 스마트에너지시스템학과) ;
  • 김태하 (서울과학기술대학교 전기정보공학과) ;
  • 이의진 (서울과학기술대학교 전기정보공학과, 스마트에너지시스템학과)
  • Received : 2022.02.07
  • Accepted : 2022.12.13
  • Published : 2023.01.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a super resolution network for raw sensor image which data size is lower comparatively to RGB image. But the actual capabilities of raw image super resolution depends on color correction because its absent of camera post processing that leads to unintended result having different white balance, saturation, etc. Thus, we introduce novel color corrector attention network by adopting the idea of precedent raw super resolution research, and tune to the our faced problem from data specification. The result is not superior to former researches but shows decent output on certain performance matrix. In the same time, we encounter new challenging problem of unexpected shadowing artifact around image objects that cause performance declination despite its good result overall. This problem remains a task to be solved in the future research.

본 연구에서는 전통적인 RGB 영상보다 데이터양이 적은 로우 센서 영상을 이용한 초해상화 네트워크를 제안하고 이에 대한 실험결과를 정리하였다. 로우 센서 영상의 초해상화는 일반적인 RGB 초해상화와 달리 카메라에서 일어나는 후처리 가공이 없는 무손실영상을 이용하기 때문에 결과물의 성능이 일반 RGB 초해상화 연구와 달리 색상 보정에 따라 많이 좌우된다. 따라서, 본 연구에서는색상 보정을 위한 모듈을 개발하여 기존 RGB 기반 네트워크에 삽입해 이를 이용해 성능 결과를 비교하였다. 연구 결과 색상 보정 모듈을 적용함으로 성능 지표의 향상이 있음을 확인하였다. 다만, 출력 영상의 의도하지 않은 아티팩트가 발생하는 현상을 확인하였고, 성능 지표 중 PSNR의 향상이 분명하나 SSIM의 성능이 일부 떨어지는 것으로 확인하였다.

Keywords

Acknowledgement

이 논문은 2022년도 정부(교육부)의 재원으로 한국연구재단 개인연구사업의 지원을 받아 수행된 연구임. (No. 2022R1F1A1062950)

References

  1. (2022) NITRE. [Online]. Available: https://data.vision.ee.ethz.ch/cvl/ntire22/
  2. Q. Yuan, L. Zhang, and H. Shen, "Regional Spatially Adaptive Total Variation Super-Resolution With Spatial Information Filtering and Clustering," IEEE Transactions on Image Processing, vol. 22, no. 6, pp.2327-2342, 2013. doi: https://doi.org/10.1109/TIP.2013.2251648
  3. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, "Non-local sparse models for image restoration," IEEE International Conference on Computer Vision (ICCV), pp.2272-2279, 2009. doi: https://doi.org/10.1109/ICCV.2009.5459452
  4. Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, "Deep Networks for Image Super-Resolution with Sparse Prior,"IEEE International Conference on Computer Vision (ICCV), pp.370-378, 2015. doi: https://doi.org/10.1109/ICCV.2015.50
  5. C. Dong, C. C. Loy, K. He, and X. Tang, "Learning a Deep Convolutional Network for Image Super-Resolution," European Conference on Computer Vision (ECCV), pp.184-199, 2014. doi: https://doi.org/10.1007/978-3-319-10593-2_13
  6. C. Dong, C. C. Loy, K. He, and X. Tang, "Image Super-Resolution Using Deep Convolutional Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp.295-307, 2016. doi: https://doi.org/10.48550/arXiv.1501.00092
  7. J. Kim, J. K. Lee, and K. M. Lee, "Accurate Image Super-Resolution Using Very Deep Convolutional Networks," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1646-1654, 2016. doi: https://doi.org/10.48550/arXiv.1511.04587
  8. K. He, X. Zhang, S. Ren, and J. Sun. "Deep residual learning for image recognition," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770-778, 2016. doi: https://doi.org/10.48550/arXiv.1512.03385
  9. J. Kim, J. K. Lee, and K. M. Lee, "Deeply-Recursive Convolutional Network for Image Super-Resolution," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1637-1645, 2016. doi: https://doi.org/10.48550/arXiv.1511.04491
  10. X.-J. Mao, C. Shen, and Y. Yang, "Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections," A journal extension of the version at arXiv:1603.09056, 2016. doi: https://doi.org/10.48550/arXiv.1606.08921
  11. C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.105-114, 2017. doi: https://doi.org/10.48550/arXiv.1609.04802
  12. G. Huang, Z. Liu, L. Maaten and K. Weinberger, "Densely Connected Convolutional Networks," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.4700-4708, 2018. doi: https://doi.org/10.48550/arXiv.1608.06993
  13. T. Tong, G. Li, X. Liu, and Q. Gao, "Image Super-Resolution Using Dense Skip Connections," IEEE International Conference on Computer Vision (ICCV), pp.4809-4817, 2017. doi: https://doi.org/10.1109/ICCV.2017.514
  14. S. Anwar, and N. Barnes, "Densely Residual Laplacian SuperResolution," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp.1192-1204, 2022. doi: https://doi.org/10.48550/arXiv.1906.12021
  15. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. "Attention is all you need," Advances in Neural Information Processing Systems(NeurIPS),
  16. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, "Image Super-Resolution Using Very Deep Residual Channel Attention Networks," European conference on computer vision (ECCV), pp.286-301, 2018. doi: https://doi.org/10.48550/arXiv.1807.02758
  17. T. Dai, J. Cai, Y. Zhang, S. Xia, and L. Zhang, "Second-Order Attention Network for Single Image Super-Resolution," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.11057-11066, 2019. doi: https://doi.org/10.1109/CVPR.2019.01132
  18. B. Niu, W. Wen, W. Ren, X. Zhang, L. Yang, S. Wang, K. Zhang, X. Cao, and H. Shen, "Single Image Super-Resolution via a Holistic Attention Network," European Conference on Computer Vision (ECCV), pp.191-207, 2020. doi: https://doi.org/10.48550/arXiv.2008.08767
  19. Y. Mei, Y. Fan, Y. Zhou, L. Huang, T. Huang, and H. Shi, "Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.5690-5699, 2020. doi: https://doi.org/10.48550/arXiv.2006.01424
  20. W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1874-1883, 2016. doi: https://doi.org/10.48550/arXiv.1609.05158
  21. C. Dong, C. C. Loy, and X. Tang, "Accelerating the Super-Resolution Convolutional Neural Network," European Conference on Computer Vision (ECCV) pp.391-407, 2016. doi: https://doi.org/10.48550/arXiv.1608.00367
  22. W. Lai, J. Huang, N. Ahuja, and M. Yang, "Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.624-632, 2017. doi: https://doi.org/10.48550/arXiv.1704.03915
  23. Y. Tai, J. Yang, X. Liu, and C. Xu, "MemNet: A Persistent Memory Network for Image Restoration," IEEE International Conference on Computer Vision (ICCV), pp.4549-4557, 2017. doi: https://doi.org/10.48550/arXiv.1708.02209
  24. W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T. S. Huang, "Image Super-Resolution via Dual-State Recurrent Networks," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1654-1663, 2018. doi: https://doi.org/10.48550/arXiv.1805.02704
  25. M. Haris, G. Shakhnarovich, and N. Ukita, "Deep Back-Projection Networks for Super-Resolution," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1654-1663, 2018. doi: https://doi.org/10.48550/arXiv.1803.02735
  26. Z. Li, J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wu, "Feedback Network for Image Super-Resolution," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.3862-3871, 2019. doi: https://doi.org/10.48550/arXiv.1903.09814
  27. X. C. Zhang, Q. Chen, R. Ng, and V. Koltun, "Zoom to Learn, Learn to Zoom," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.3757-3765, 2019. doi: https://doi.org/10.48550/arXiv.1905.05169
  28. X. Xu, Y. Ma, W. Sun, and M.-H. Yang, "Exploiting Raw Images for Real-Scene Super-Resolution," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp.1905-1921, 2022. doi: https://doi.org/10.48550/arXiv.2102.01579
  29. Y. Lee and K. Hirakawa, "Lossless White Balance for Improved Lossless CFA Image and Video Compression," IEEE Transactions on Image Processing, vol. 31, pp.3309-3321, 2022. doi: https://doi.org/10.48550/arXiv.2009.09137
  30. G. D. Evangelidis, and E. Z. Psarakis, "Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 10, pp. 1858-1865, 2008. doi: https://doi.org/10.1109/TPAMI.2008.113
  31. Xu, Xuan, and Xin Li. "Scan: Spatial color attention networks for real single image super-resolution." IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2019. doi: https://doi.org/10.1109/CVPRW.2019.00254