Journal of the Korea Institute of Military Science and Technology (한국군사과학기술학회지)

The Korea Institute of Military Science and Technology (한국군사과학기술학회)
권호 표지
DOI QR코드

DOI QR Code

Synthetic Infra-Red Image Dataset Generation by CycleGAN based on SSIM Loss Function

SSIM 목적 함수와 CycleGAN을 이용한 적외선 이미지 데이터셋 생성 기법 연구

  • Lee, Sky (Department of Aerospace Engineering, Chosun University) ;
  • Leeghim, Henzeh (Department of Aerospace Engineering, Chosun University)
  • 이하늘 (조선대학교 항공우주공학과) ;
  • 이현재 (조선대학교 항공우주공학과)
  • Received : 2022.07.05
  • Accepted : 2022.09.30
  • Published : 2022.10.05
Download PDF
⟨ Previous Next ⟩

Abstract

Synthetic dynamic infrared image generation from the given virtual environment is being the primary goal to simulate the output of the infra-red(IR) camera installed on a vehicle to evaluate the control algorithm for various search & reconnaissance missions. Due to the difficulty to obtain actual IR data in complex environments, Artificial intelligence(AI) has been used recently in the field of image data generation. In this paper, CycleGAN technique is applied to obtain a more realistic synthetic IR image. We added the Structural Similarity Index Measure(SSIM) loss function to the L1 loss function to generate a more realistic synthetic IR image when the CycleGAN image is generated. From the simulation, it is applicable to the guided-missile flight simulation tests by using the synthetic infrared image generated by the proposed technique.

Keywords

Acknowledgement

이 연구는 인공지능 비행제어 특화연구실 프로그램의 일환으로 국방과학연구소와 방위사업청의 지원으로 수행되었음.

References

  1. Zhang, Ruiheng and Mu, Chengpo and Xu, Min and Xu, Lixin and Shi, Qiaolin and Wang, Junbo, "Synthetic IR Image Refinement Using Adversarial Learning with Bidirectional Mappings," IEEE Access, Vol. 7, pp. 153734-153750, 2019. https://doi.org/10.1109/ACCESS.2019.2947657
  2. Kniaz, VV and Gorbatsevich, VS and Mizginov, VA, "Thermalnet: a Deep Convolutional Network for Synthetic Thermal Image Generation," The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 42, p. 41, 2017,
  3. Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza and Bing Xu, "Generative Adversarial Networks," Neural Information Processing Systems(NIPS), Motreal, 2014.
  4. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, "Image-to-Image Translation with Conditional Adversarial Networks," CVPR, 2017.
  5. Jun-Yan Zhu, Taesung Park, and Alexei A. Efros, "Unpaired Image-to-Image Translation using CycleConsistent Adversarial Networks," Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
  6. Karras, Tero and Aila, Timo and Laine, Samuli and Lehtinen, Jaakko, "Progressive Growing of Gans for Improved Quality, Stability, and Variation," arXiv preprint arXiv:1710.10196, 2017.
  7. Karras, Tero and Laine, Samuli and Aila, Timo, "A Style-based Generator Architecture for Generative Adversarial Networks," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
  8. Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas, "U-Net: Convolutional Networks for Biomedical Image Segmentation," International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234-241, 2015.
  9. Kaiming He, Xiangyu Zhang, and Jian Sun, "Deep Residual Learning for Image Recognition," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016
  10. Li, Wenda and Wang, Jian, "Residual Learning of Cycle-GAN for Seismic Data Denoising," IEEE Access, Vol. 9, pp. 11585-11597, 2021. https://doi.org/10.1109/ACCESS.2021.3049479
  11. Maniyath, Shima Ramesh and Vijayakumar, K and Singh, Laxman and Sharma, Sudhir Kumar and Olabiyisi, Tunde, "Learningbased Approach to Underwater Image Dehazing Using CycleGAN," IEEE Access, Vol. 14, No. 18, pp. 1-11, 2021.
  12. Teng, Long and Fu, Zhongliang and Yao, Yu, "Interactive Translation in Echocardiography Training System with Enhanced Cycle-GAN," IEEE Access, Vol. 8, pp. 106147-106156, 2020. https://doi.org/10.1109/ACCESS.2020.3000666
  13. Hang Zhao, Orazio Gallo, and Jan Kautz, "Loss Functions for Image Restoration With Neural Networks," IEEE Transactions on Computational Imaging, pp. 47-57, 2017.
  14. Hwang, Jieon and Yu, Chushi and Shin, Yoan, "SAR-to-Optical Image Translation Using SSIM and Perceptual Loss based Cycleconsistent GAN," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 191-194. 2020.
  15. Tao, Li and Zhu, Chuang and Xiang, Guoqing and Li, Yuan and Jia, Huizhu and Xie, Xiaodong, "LLCNN: A Convolutional Neural Network for Low-Light Image Enhancement," 2017 IEEE Visual Communications and Image Processing(VCIP), pp. 1-4, 2017.
  16. Shi, Haoyue and Wang, Le and Zheng, Nanning and Hua, Gang and Tang, Wei, "Loss Functions for Pose Guided Person Image Generation," Pattern Recognition, Vol. 122, p. 108351, 2022. https://doi.org/10.1016/j.patcog.2021.108351
  17. G. E. Hinton and S. T. Roweis, "Stochastic Neighbor Embedding," Proceedings of Advances in Neural Information Processing Systems(NIPS), pp. 833-840, 2002.
  18. L. J. P. van der Maaten and G. E. Hinton, "Visualizing High-Dimensional Data Using t-SNE," Journal of Machine Learning Research, Vol. 9, pp. 2579-2695, 2008.
  19. A. van der SCHAAF, J. H. van HATEREN "Modelling the Power Spectra of Natural Images: Statistics and Information," Vision Research, Volume 36, Issue 17, pp. 2759-2770, 1996. https://doi.org/10.1016/0042-6989(96)00002-8
  20. Koch, Michael and Denzler, Joachim and Redies, Christoph, "1/f2 Characteristics and Isotropy in the Fourier Power Spectra of Visual Art, Cartoons, Comics, Mangas, and Different Categories of Photographs," PLoS One, Vol. 5, No. 8, p. e12268, 2010. https://doi.org/10.1371/journal.pone.0012268
  21. Venkataramanan, Abhinau K and Wu, Chengyang and Bovik, Alan C and Katsavounidis "A Hitchhiker's Guide to Structural Similarity," IEEE Access, Vol. 9, pp. 28872-28896, 2021. https://doi.org/10.1109/ACCESS.2021.3056504
  22. Moorthy, Anush K and Bovik, Alan C, "Perceptually Significant Spatial Pooling Techniques for Image Quality Assessment," Human Vision and Electronic Imaging XIV, Vol. 7240, pp. 339-349, 2009.
  23. Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp, "Gans Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium," Advances in Neural Information Processing Systems, Vol. 30, 2017.
  24. Obukhov, Artem and Krasnyanskiy, Mikhail, "Quality Assessment Method for GAN based on Modified Metrics Inception Score and Frechet Inception Distance," Proceedings of the Computational Methods in Systems and Software, pp. 102-114, 2020.
  25. FLIR Systems, 2019. FREE FLIR Thermal Dataset for Algorithm Training. https://flir.com/oem/adas/adas-dataset-form