Current Optics and Photonics

  • 제8권3호
  • /
  • Pages.215-224
  • /
  • 2024
  • /
  • 2508-7266(pISSN)
  • /
  • 2508-7274(eISSN)
한국광학회 (Optical Society of Korea)
권호 표지
DOI QR코드

DOI QR Code

Restoring Turbulent Images Based on an Adaptive Feature-fusion Multi-input-Multi-output Dense U-shaped Network

  • Haiqiang Qian (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology) ;
  • Leihong Zhang (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology) ;
  • Dawei Zhang (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology) ;
  • Kaimin Wang (Engineering Research Center of Optical Instrument and System, Ministry of Education and Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology)
  • 투고 : 2023.11.03
  • 심사 : 2024.02.23
  • 발행 : 2024.06.25
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

In medium- and long-range optical imaging systems, atmospheric turbulence causes blurring and distortion of images, resulting in loss of image information. An image-restoration method based on an adaptive feature-fusion multi-input-multi-output (MIMO) dense U-shaped network (Unet) is proposed, to restore a single image degraded by atmospheric turbulence. The network's model is based on the MIMO-Unet framework and incorporates patch-embedding shallow-convolution modules. These modules help in extracting shallow features of images and facilitate the processing of the multi-input dense encoding modules that follow. The combination of these modules improves the model's ability to analyze and extract features effectively. An asymmetric feature-fusion module is utilized to combine encoded features at varying scales, facilitating the feature reconstruction of the subsequent multi-output decoding modules for restoration of turbulence-degraded images. Based on experimental results, the adaptive feature-fusion MIMO dense U-shaped network outperforms traditional restoration methods, CMFNet network models, and standard MIMO-Unet network models, in terms of image-quality restoration. It effectively minimizes geometric deformation and blurring of images.

키워드

과제정보

National Nature Science Foundation of China (Grant No. 61805144, 61875125, 61775140 and 61405115); Natural Science Foundation of Shanghai (Grant No. 18ZR1425800).

참고문헌

  1. B. L. Ellerbroek, "First-order performance evaluation of adaptive-optics systems for atmospheric-turbulence compensation in extended-field-of-view astronomical telescopes," J. Opt. Soc. Am. A 11, 783-805 (1994). https://doi.org/10.1364/JOSAA.11.000783
  2. J. Zhang and X. Zhou, "Research on feature recognition algorithm for space target," Proc. SPIE 6786, 678616 (2007).
  3. P.-A. Moreau, E. Toninelli, T. Gregory, and M. J. Padgett, "Ghost imaging using optical correlations," Laser Photonics Rev. 12, 1700143 (2018).
  4. C. P. Lau, Y. H. Lai, and L. M. Lui, "Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps," Inverse Probl. 35, 074002 (2019).
  5. X. Zhu and P. Milanfar, "Removing atmospheric turbulence via space-invariant deconvolution," IEEE Trans. Pattern Anal. Mach. Intell. 35, 157-170 (2013). https://doi.org/10.1109/TPAMI.2012.82
  6. C. P. Lau, Y. H. Lai, and L. M. Lui, "Variational models for joint subsampling and reconstruction of turbulence-degraded images," J. Sci. Comput. 78, 1488-1525 (2019). https://doi.org/10.1007/s10915-018-0833-4
  7. C. P. Lau, C. D. Castillo, and R. Chellappa, "ATFaceGAN: Single face semantic aware image restoration and recognition from atmospheric turbulence," IEEE Trans. Biom. Behav. Identity Sci. 3, 240-251 (2021). https://doi.org/10.1109/TBIOM.2021.3058316
  8. D. Jin, Y. Chen, Y. Lu, J. Chen, P. Wang, Z. Liu, S. Guo, and X. Bai, "Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning," Nat. Mach. Intell. 3, 876-884 (2021). https://doi.org/10.1038/s42256-021-00392-1
  9. S. N. Rai and C. V. Jawahar, "Removing atmospheric turbulence via deep adversarial learning," IEEE Trans. Image Process. 31, 2633-2646 (2022). https://doi.org/10.1109/TIP.2022.3158547
  10. Y. Xie, W. Zhang, D. Tao, W. Hu, Y. Qu, and H. Wang, "Removing turbulence effect via hybrid total variation and deformation-guided kernel regression," IEEE Trans. Image Process. 25, 4943-4958 (2016). https://doi.org/10.1109/TIP.2016.2598638
  11. N. Chimitt and S. H. Chan, "Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated Zernike coefficients," Opt. Eng. 59, 083101 (2020).
  12. S. Basu, J. E. McCrae, and S. T. Fiorino, "Estimation of the path-averaged atmospheric refractive index structure constant from time-lapse imagery," Proc. SPIE 9465, 94650T (2015).
  13. G. A. Chanan, "Calculation of wave-front tilt correlations associated with atmospheric turbulence," J. Opt. Soc. Am. A 9, 298-301 (1992). https://doi.org/10.1364/JOSAA.9.000298
  14. B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, "Places: A 10 million image database for scene recognition," IEEE Trans. Pattern Anal. Mach. Intell. 40, 1452-1464 (2018). https://doi.org/10.1109/TPAMI.2017.2723009
  15. N. Anantrasirichar and D. Bull, "BVI-CLEAR," (University of Bristol, Published date: Apr 11, 2022), https://doi.org/10.5523/bris.1yh1e51t7tg2g2q9cwv96sdfc2 (Accessed date: May 11, 2023).
  16. S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko, "Rethinking coarse-to-fine approach in single image deblurring," in Proc. IEEE/CVF International Conference on Computer Vision (Montreal, QC, Canada, Oct. 10-17, 2021), pp. 4621-4630.
  17. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, "Residual dense network for image super-resolution," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (Salt Lake City, UT, USA, Jun. 18-23, 2018), pp. 2472-2481.
  18. J. Cao, Y. Li, M. Sun, Y. Chen, D. Lischinski, D. Cohen-Or, B. Chen, and C. Tu, "DO-conv: Depthwise over-parameterized convolutional layer," IEEE Trans. Image Process. 31, 3726-3736 (2022). https://doi.org/10.1109/TIP.2022.3175432
  19. S. Liu, D. Huang, and Y. Wang, "Learning Spatial fusion for single-shot object detection," arXiv:1911.09516 (2019).
  20. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proc. IEEE/CVF International Conference on Computer Vision (Montreal, QC, Canada, Oct. 10-17, 2021), pp. 9992-10002.
  21. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, "Multi-Stage Progressive Image Restoration," in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (Nashville, TN, USA, Jun. 20-25, 2021), pp. 14816-14826.
  22. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv:1608.03983 (2014).
  23. I. Loshchilov and F. Hutter, "SGDR: Stochastic gradient descent with warm restarts," arXiv:1608.03983 (2016).
  24. H. R. Sheikh, A. C. Bovik, and G. de Veciana, "An information fidelity criterion for image quality assessment using natural scene statistics," IEEE Trans. Image Process. 14, 2117-2128 (2005). https://doi.org/10.1109/TIP.2005.859389
  25. L. Zhang, L. Zhang, X. Mou, and D. Zhang, "FSIM: A feature similarity index for image quality assessment," IEEE Trans. Image Process. 20, 2378-2386 (2011). https://doi.org/10.1109/TIP.2011.2109730
  26. Z. Mao, N. Chimitt, and S. H. Chan, "Image reconstruction of static and dynamic scenes through anisoplanatic turbulence," IEEE Trans. Comput. Imaging 6, 1415-1428 (2020). https://doi.org/10.1109/TCI.2020.3029401
  27. C.-M. Fan, T.-J. Liu, and K.-H. Liu, "Compound multi-branch feature fusion for real image restoration," arXiv:2206.02748 (2022).