Journal of the Korean Society for Industrial and Applied Mathematics

Korean Society for Industrial and Applied Mathematics (한국산업응용수학회)
권호 표지
DOI QR코드

DOI QR Code

MULTI-APERTURE IMAGE PROCESSING USING DEEP LEARNING

  • GEONHO HWANG (DEPARTMENT OF MATHEMATICAL SCIENCES, SEOUL NATIONAL UNIVERSITY) ;
  • CHANG HOON SONG (DEPARTMENT OF MATHEMATICAL SCIENCES, SEOUL NATIONAL UNIVERSITY) ;
  • TAE KYUNG LEE (INTERDISCIPLINARY PROGRAM IN ARTIFICIAL INTELLIGENCE, SEOUL NATIONAL UNIVERSITY) ;
  • HOJUN NA (DEPARTMENT OF MATHEMATICAL SCIENCES, SEOUL NATIONAL UNIVERSITY) ;
  • MYUNGJOO KANG (DEPARTMENT OF MATHEMATICAL SCIENCES, SEOUL NATIONAL UNIVERSITY)
  • Received : 2022.12.30
  • Accepted : 2023.03.23
  • Published : 2023.03.25
Download PDF
⟨ Previous Next ⟩

Abstract

In order to obtain practical and high-quality satellite images containing high-frequency components, a large aperture optical system is required, which has a limitation in that it greatly increases the payload weight. As an attempt to overcome the problem, many multi-aperture optical systems have been proposed, but in many cases, these optical systems do not include high-frequency components in all directions, and making such an high-quality image is an ill-posed problem. In this paper, we use deep learning to overcome the limitation. A deep learning model receives low-quality images as input, estimates the Point Spread Function, PSF, and combines them to output a single high-quality image. We model images obtained from three rectangular apertures arranged in a regular polygon shape. We also propose the Modulation Transfer Function Loss, MTF Loss, which can capture the high-frequency components of the images. We present qualitative and quantitative results obtained through experiments.

Keywords

Acknowledgement

This work has been supported by the Challengeable Future Defense Technology Research and Development Program through ADD[No. 915020201], the NRF grant[2012R1A2C3010887] and the MSIT/IITP[No. 2021-0-01343].

References

  1. Campbell, J. & Wynne, R. Introduction to remote sensing, Guilford Press, 2011.
  2. Green, P., Sun, W., Matusik, W. & Durand, F. Multi-aperture photography, Association for Computing Machinery, Proceedings of ACM SIGGRAPH, San Diego, California, USA 2007.
  3. Lv, G., Xu, H., Feng, H., Xu, Z., Zhou, H., Li, Q. & Chen, Y. A Full-Aperture Image Synthesis Method for the Rotating Rectangular Aperture System Using Fourier Spectrum Restoration, Photonics, 8 (2021), 1-17. https://doi.org/10.3390/photonics8110522
  4. Rim, J., Lee, H., Won, J. & Cho, S. Real-world blur dataset for learning and benchmarking deblurring algorithms, Lecture Notes in Computer Science, Springer, Proceedings of ECCV, Glasgow, UK 2020.
  5. Zhang, K., Ren, W., Luo, W., Lai, W., Stenger, B., Yang, M. & Li, H. Deep image deblurring: A survey. International Journal Of Computer Vision. 130 (2022), 2103-2130. https://doi.org/10.1007/s11263-022-01633-5
  6. Bai, Y., Jia, H., Jiang, M., Liu, X., Xie, X. & Gao, W. Single-image blind deblurring using multi-scale latent structure prior, IEEE Transactions On Circuits And Systems For Video Technology, 30 (2019), 2033-2045.
  7. Zhang, K., Luo, W., Zhong, Y., Ma, L., Liu, W. & Li, H. Adversarial spatio-temporal learning for video deblurring, IEEE Transactions On Image Processing, 28 (2018), 291-301. https://doi.org/10.1109/TIP.2018.2867733
  8. Chen, L., Zhang, J., Lin, S., Fang, F. & Ren, J. Blind deblurring for saturated images, Proceedings Of Conference On Computer Vision And Pattern Recognition, 2021.
  9. Wang, P., Sun, J., Li, H., Chen, X., Zhu, Y. & Zhang, Y. Multi-image Deblurring Using Complement, Lecture Notes in Computer Science, Springer, Proceedings of International Conference On Intelligent Science And Big Data Engineering, Dalian, China, 2017.
  10. Zhang, J., Pan, J., Wang, D., Zhou, S., Wei, X., Zhao, F., Liu, J. & Ren, J. Deep Dynamic Scene Deblurring from Optical Flow, IEEE Transactions On Circuits And Systems For Video Technology, 32 (2021), 8250 -8260 https://doi.org/10.1109/TCSVT.2021.3084616
  11. Zhu, B., Lv, Q., Yang, Y., Sui, X., Zhang, Y., Tang, Y. & Tan, Z. Blind Deblurring of Remote-Sensing Single Images Based on Feature Alignment, Sensors, 22 (2022), 7894
  12. Zhang, Z., Zheng, L., Piao, Y., Tao, S., Xu, W., Gao, T. & Wu, X. Blind Remote Sensing Image Deblurring Using Local Binary Pattern Prior, Remote Sensing, 14 (2022), 1276
  13. Kim, H., Seo, D., Jung, J., Cha, D. & Lee, D. Blind motion deblurring for satellite image using convolutional neural network, IEEE, Proceedings of Digital Image Computing: Techniques And Applications(DICTA), Perth, Australia 2019.
  14. Born, M. & Wolf, E. Principles of optics: electromagnetic theory of propagation, interference and diffraction of light, Elsevier, 2013
  15. Williams, C. & Becklund, O. Introduction to the optical transfer function, SPIE Press, 2002.
  16. Condon, J. & Ransom, S. Essential radio astronomy, Princeton University Press, 2016.
  17. Viallefont-Robinet, F., Helder, D., Fraisse, R., Newbury, A., Bergh, F., Lee, D. & Saunier, S. Comparison of MTF measurements using edge method: towards reference data set, Optics Express, 26 (2018), 33625-33648 https://doi.org/10.1364/OE.26.033625
  18. Liang, J., Sun, G., Zhang, K., Van Gool, L. & Timofte, R. Mutual affine network for spatially variant kernel estimation in blind image super-resolution, Proceedings Of The IEEE/CVF International Conference On Computer Vision, Montreal, QC, Canada 2021.
  19. Wang, X., Yu, K., Dong, C. & Loy, C. Recovering realistic texture in image super-resolution by deep spatial feature transform, Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition, Salt Lake City, UT 2018.
  20. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y. & Change Loy, C. Esrgan: Enhanced super-resolution generative adversarial networks, Proceedings Of The European Conference On Computer Vision (ECCV) Workshops, Munich, Germany 2018.
  21. Ji, S., Wei, S. & Lu, M. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set, IEEE Transactions On Geoscience And Remote Sensing, 57 (2018), 574-586. https://doi.org/10.1109/TGRS.2018.2858817
  22. Goldberg, H., Brown, M. & Wang, S. A benchmark for building footprint classification using orthorectified rgb imagery and digital surface models from commercial satellites, IEEE, Proceedings Of IEEE Applied Imagery Pattern Recognition Workshop(AIPR), Washington, DC, USA 2017.