Current Optics and Photonics

Optical Society of Korea (한국광학회)
권호 표지
DOI QR코드

DOI QR Code

Restoring Turbulent Images Based on an Adaptive Feature-fusion Multi-input-Multi-output Dense U-shaped Network

  • Haiqiang Qian (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology) ;
  • Leihong Zhang (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology) ;
  • Dawei Zhang (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology) ;
  • Kaimin Wang (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology)
  • Received : 2023.11.03
  • Accepted : 2024.02.23
  • Published : 2024.06.25
Download PDF
⟨ Previous Next ⟩

Abstract

In medium- and long-range optical imaging systems, atmospheric turbulence causes blurring and distortion of images, resulting in loss of image information. An image-restoration method based on an adaptive feature-fusion multi-input-multi-output (MIMO) dense U-shaped network (Unet) is proposed, to restore a single image degraded by atmospheric turbulence. The network's model is based on the MIMO-Unet framework and incorporates patch-embedding shallow-convolution modules. These modules help in extracting shallow features of images and facilitate the processing of the multi-input dense encoding modules that follow. The combination of these modules improves the model's ability to analyze and extract features effectively. An asymmetric feature-fusion module is utilized to combine encoded features at varying scales, facilitating the feature reconstruction of the subsequent multi-output decoding modules for restoration of turbulence-degraded images. Based on experimental results, the adaptive feature-fusion MIMO dense U-shaped network outperforms traditional restoration methods, CMFNet network models, and standard MIMO-Unet network models, in terms of image-quality restoration. It effectively minimizes geometric deformation and blurring of images.

Keywords

Acknowledgement

National Nature Science Foundation of China (Grant No. 61805144, 61875125, 61775140 and 61405115); Natural Science Foundation of Shanghai (Grant No. 18ZR1425800).

References

  1. B. L. Ellerbroek, "First-order performance evaluation of adaptive-optics systems for atmospheric-turbulence compensation in extended-field-of-view astronomical telescopes," J. Opt. Soc. Am. A 11, 783-805 (1994).
  2. J. Zhang and X. Zhou, "Research on feature recognition algorithm for space target," Proc. SPIE 6786, 678616 (2007).
  3. P.-A. Moreau, E. Toninelli, T. Gregory, and M. J. Padgett, "Ghost imaging using optical correlations," Laser Photonics Rev. 12, 1700143 (2018).
  4. C. P. Lau, Y. H. Lai, and L. M. Lui, "Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps," Inverse Probl. 35, 074002 (2019).
  5. X. Zhu and P. Milanfar, "Removing atmospheric turbulence via space-invariant deconvolution," IEEE Trans. Pattern Anal. Mach. Intell. 35, 157-170 (2013).
  6. C. P. Lau, Y. H. Lai, and L. M. Lui, "Variational models for joint subsampling and reconstruction of turbulence-degraded images," J. Sci. Comput. 78, 1488-1525 (2019).
  7. C. P. Lau, C. D. Castillo, and R. Chellappa, "ATFaceGAN: Single face semantic aware image restoration and recognition from atmospheric turbulence," IEEE Trans. Biom. Behav. Identity Sci. 3, 240-251 (2021).
  8. D. Jin, Y. Chen, Y. Lu, J. Chen, P. Wang, Z. Liu, S. Guo, and X. Bai, "Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning," Nat. Mach. Intell. 3, 876-884 (2021).
  9. S. N. Rai and C. V. Jawahar, "Removing atmospheric turbulence via deep adversarial learning," IEEE Trans. Image Process. 31, 2633-2646 (2022).
  10. Y. Xie, W. Zhang, D. Tao, W. Hu, Y. Qu, and H. Wang, "Removing turbulence effect via hybrid total variation and deformation-guided kernel regression," IEEE Trans. Image Process. 25, 4943-4958 (2016).
  11. N. Chimitt and S. H. Chan, "Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated Zernike coefficients," Opt. Eng. 59, 083101 (2020).
  12. S. Basu, J. E. McCrae, and S. T. Fiorino, "Estimation of the path-averaged atmospheric refractive index structure constant from time-lapse imagery," Proc. SPIE 9465, 94650T (2015).
  13. G. A. Chanan, "Calculation of wave-front tilt correlations associated with atmospheric turbulence," J. Opt. Soc. Am. A 9, 298-301 (1992).
  14. B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, "Places: A 10 million image database for scene recognition," IEEE Trans. Pattern Anal. Mach. Intell. 40, 1452-1464 (2018).
  15. N. Anantrasirichar and D. Bull, "BVI-CLEAR," (University of Bristol, Published date: Apr 11, 2022), https://doi.org/10.5523/bris.1yh1e51t7tg2g2q9cwv96sdfc2 (Accessed date: May 11, 2023).
  16. S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko, "Rethinking coarse-to-fine approach in single image deblurring," in Proc. IEEE/CVF International Conference on Computer Vision (Montreal, QC, Canada, Oct. 10-17, 2021), pp. 4621-4630.
  17. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, "Residual dense network for image super-resolution," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (Salt Lake City, UT, USA, Jun. 18-23, 2018), pp. 2472-2481.
  18. J. Cao, Y. Li, M. Sun, Y. Chen, D. Lischinski, D. Cohen-Or, B. Chen, and C. Tu, "DO-conv: Depthwise over-parameterized convolutional layer," IEEE Trans. Image Process. 31, 3726-3736 (2022).
  19. S. Liu, D. Huang, and Y. Wang, "Learning Spatial fusion for single-shot object detection," arXiv:1911.09516 (2019).
  20. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proc. IEEE/CVF International Conference on Computer Vision (Montreal, QC, Canada, Oct. 10-17, 2021), pp. 9992-10002.
  21. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, "Multi-Stage Progressive Image Restoration," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (Nashville, TN, USA, Jun. 20-25, 2021), pp. 14816-14826.
  22. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv:1608.03983 (2014).
  23. I. Loshchilov and F. Hutter, "SGDR: Stochastic gradient descent with warm restarts," arXiv:1608.03983 (2016).
  24. H. R. Sheikh, A. C. Bovik, and G. de Veciana, "An information fidelity criterion for image quality assessment using natural scene statistics," IEEE Trans. Image Process. 14, 2117-2128 (2005).
  25. L. Zhang, L. Zhang, X. Mou, and D. Zhang, "FSIM: A feature similarity index for image quality assessment," IEEE Trans. Image Process. 20, 2378-2386 (2011).
  26. Z. Mao, N. Chimitt, and S. H. Chan, "Image reconstruction of static and dynamic scenes through anisoplanatic turbulence," IEEE Trans. Comput. Imaging 6, 1415-1428 (2020).
  27. C.-M. Fan, T.-J. Liu, and K.-H. Liu, "Compound multi-branch feature fusion for real image restoration," arXiv:2206.02748 (2022).