KSCE Journal of Civil and Environmental Engineering Research (대한토목학회논문집)

Korean Society of Civil Engeneers (대한토목학회)
권호 표지
DOI QR코드

DOI QR Code

Learning-Based Daytime Image Simulation Improvement of Thermal Images Using Unsupervised Segmentation

무감독 세그멘테이션을 이용한 열영상의 학습기반 주간영상 모의 개선

  • 원태연 (건국대학교 대학원 기술융합공학과) ;
  • 조수민 (건국대학교 대학원 기술융합공학과) ;
  • 정지헌 (한국건설기술연구원 미래스마트건설연구본부) ;
  • 장명도 (건국대학교 대학원 기술융합공학과) ;
  • 김용민 (국방과학연구소 국방인공지능기술연구원)
  • Received : 2025.05.07
  • Accepted : 2025.05.12
  • Published : 2025.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

Nightyime image serves as critical visual information in a wide range of applications, including disaster response and surveillance operations. To improve the clarity and visibility of such imagery, extensive research has been conducted on imaging devices, sensor technologies, and image processing techniques. This study proposes a method to simulate daytime images from nighttime image acquired by various sensors by learning object-specific feature regions within the image. Considering that temperature differences between daytime and nighttime images vary depending on the object, a CNN-based unsupervised segmentation technique, and the UNSB (Unpaired Neural Schrödinger Bridge) model were applied. The experimental setting for the unsupervised segmentation technique was designed to satisfy three constraints: feature similarity, spatial continuity, and a limited number of clusters. Training data were composed by compositing thermal infrared images, LiDAR intensity images, LiDAR range images, and segmentation images, which were then learned in conjunction with corresponding optical images using the UNSB model to simulate daytime optical image. The experimental results showed SSIM 0.593, PSNR 14.34, and R2 0.218 based on 120 epochs of training. For each object detail, the colors were similar to the original image, proving that the conversion result is excellent when the object features are clear.

야간영상은 재난상황과 감시 작전 등 다양한 분야에서 중요한 영상정보로 활용되었고, 야간영상의 선명도와 가시성 향상을 위해 카메라, 센서와 영상처리 연구가 활발히 이루어지고 있다. 본 연구에서는 다양한 특성을 가진 센서로 촬영한 야간영상에 대하여 영상 내 객체별 특징영역을 구분하여 학습시켜 주간영상으로 모의하는 방법을 제안하였다. 주간과 야간영상 간의 온도 차이가 피복별로 다른 점을 고려해 학습방법을 제안한 것으로써, CNN 기반 무감독 세그멘테이션 기법과 UNSB 모델을 적용하였다. 무감독 세그멘테이션 기법을 적용한 실험환경은, 특성 유사성, 공간 연속성, 클러스터 수 제한과 같은 세 가지 제약 조건을 만족하도록 하였다. 딥러닝 학습 자료로 열적외 영상, 라이다 강도 영상, 라이다 거리 영상 그리고 피복별로 구별될 수 있는 세그멘테이션 영상을 정합하였으며, 해당 영상과 광학영상을 UNSB 모델 기반으로 학습하여 주간 광학영상을 모의하였다. 실험결과, 120 에포크 학습 기준으로 SSIM 0.593, PSNR 14.34, R2 0.218을 나타내었다. 시각적으로 유사한 형상이 생성되고, 객체별 세부 요소들에 대하여 원영상과 유사한 색상을 구현하여, 객체의 특징이 명확한 경우 변환 결과가 우수함을 입증하였다.

Keywords

Acknowledgement

This work was supported by KOITA grant funded by MSIT(R&DCENTER Capability Enhancement Project, 1711199726).

References

  1. Bengio, Y., Courville, A. and Vincent, P. (2014). "Representation learning: A review and new perspectives." IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 8, pp. 1798-1828, https://doi.org/10.1109/TPAMI.2013.50.
  2. Ho, J., Jain, A. and Abbeel, P. (2020). "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems, Vol. 33, pp. 6840-6851, https://doi.org/10.48550/arXiv.2006.11239.
  3. Hore, A. and Ziou, D. (2010). "Image quality metrics: PSNR vs. SSIM." 2010 20th International Conference on Pattern Recognition (ICPR), Istanbul, Turkey, 23-26 August 2010, pp. 2366-2369, https://doi.org/10.1109/ICPR.2010.579.
  4. Isola, P., Zhu, J. Y., Zhou, T. and Efros, A. A. (2017). "Image-to-image translation with conditional adversarial networks." IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967-5976, https://doi.org/10.1109/CVPR.2017.632
  5. Jensen, J. R. (2007). Remote sensing of the environment: An earth resource perspective. Pearson Prentice Hall.
  6. Jiang, Q., Zhou, T., He, Y., Ma, W., Hou, J., Abdul Ghani, A. S., Miao, S. and Jin, X. (2025). "CMMF-Net: A generative network based on CLIP-guided multi-modal feature fusion for thermal infrared image colorization." Intelligence & Robotics, Vol. 5, No. 1, pp. 34-49, https://doi.org/10.20517/ir.2025.03.
  7. Jung, J. H., Park, J., Hwang, C., Lee, J. and Eo, Y. D. (2025). "Colorization based on LiDAR and thermal image." Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 43, No. 1, pp. 91-99, https://doi.org/10.7848/ksgpc.2025.43.1.91 (in Korean).
  8. Kim, B., Kwon, G., Kim, K. and Ye, J. C. (2024). "Unpaired image-to-image translation via neural schrödinger bridge." The 12th International Conference on Learning Representations (ICLR 2024), Vienna, Austria, https://doi.org/10.48550/arXiv.2305.15086.
  9. Kanezaki, A. (2018). "Unsupervised image segmentation by backpropagation." IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 15-20 April 2018, pp. 1543-1547, https://doi.org/10.1109/ICASSP.2018.8462533.
  10. Lee, S., Kim, T. and Choi, Y. (2023). "Pseudo-RGB-based place recognition through thermal-to-RGB image translation." Journal of Korea Robotics Society, Vol. 18, No. 1, pp. 48-52, https://doi.org/10.7746/jkros.2023.18.1.048 (in Korean).
  11. Luo, F., Li, Y., Zeng, G., Peng, P., Wang, G. and Li, Y. (2022). "Thermal infrared image colorization for nighttime driving scenes with top-down guided attention." IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 9, pp. 15808-15823. https://doi.org/10.1109/TITS.2022.3145476
  12. Saharia, C., Chan, W., Chang, H., Lee, C. A., Ho, J., Fleet, D. J. and Norouzi, M. (2022). "Palette: Image-to-image diffusion models." IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18661-18671, https://doi.org/10.48550/arXiv.2111.05826.
  13. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S. and Poole, B. (2021). "Score-based generative modeling through stochastic differential equations." International Conference on Learning Representations (ICLR), https://doi.org/10.48550/arXiv.2011.13456.
  14. Wang, D. (2001). "Unsupervised learning: Foundations of neural computation." Neural Networks, Vol. 22, No. 2, pp. 101-102, https://doi.org/10.1609/aimag.v22i2.1565.
  15. Wang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. (2004). "Image quality assessment: From error visibility to structural similarity." IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600-612, https://doi.org/10.1109/TIP.2003.819861.
  16. Zhu, J. Y., Park, T., Isola, P. and Efros, A. A. (2017). "Unpaired image-to-image translation using cycle-consistent adversarial networks." IEEE International Conference on Computer Vision (ICCV), pp. 2223-2232, https://doi.org/10.1109/ICCV.2017.244.