Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography (한국측량학회지)

Korean Society of Surveying, Geodesy, Photogrammetry and Cartography (한국측량학회)
권호 표지
DOI QR코드

DOI QR Code

True Orthoimage Generation from LiDAR Intensity Using Deep Learning

딥러닝에 의한 라이다 반사강도로부터 엄밀정사영상 생성

  • Shin, Young Ha (Dept. of Environment, Energy & Geoinformatics, Sejong University) ;
  • Hyung, Sung Woong (Dept. of Environment, Energy & Geoinformatics, Sejong University) ;
  • Lee, Dong-Cheon (Dept. of Environment, Energy & Geoinformatics, Sejong University)
  • Received : 2020.07.22
  • Accepted : 2020.08.26
  • Published : 2020.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

During last decades numerous studies generating orthoimage have been carried out. Traditional methods require exterior orientation parameters of aerial images and precise 3D object modeling data and DTM (Digital Terrain Model) to detect and recover occlusion areas. Furthermore, it is challenging task to automate the complicated process. In this paper, we proposed a new concept of true orthoimage generation using DL (Deep Learning). DL is rapidly used in wide range of fields. In particular, GAN (Generative Adversarial Network) is one of the DL models for various tasks in imaging processing and computer vision. The generator tries to produce results similar to the real images, while discriminator judges fake and real images until the results are satisfied. Such mutually adversarial mechanism improves quality of the results. Experiments were performed using GAN-based Pix2Pix model by utilizing IR (Infrared) orthoimages, intensity from LiDAR data provided by the German Society for Photogrammetry, Remote Sensing and Geoinformation (DGPF) through the ISPRS (International Society for Photogrammetry and Remote Sensing). Two approaches were implemented: (1) One-step training with intensity data and high resolution orthoimages, (2) Recursive training with intensity data and color-coded low resolution intensity images for progressive enhancement of the results. Two methods provided similar quality based on FID (Fréchet Inception Distance) measures. However, if quality of the input data is close to the target image, better results could be obtained by increasing epoch. This paper is an early experimental study for feasibility of DL-based true orthoimage generation and further improvement would be necessary.

정사영상 생성을 위한 많은 연구들이 진행되어 왔다. 기존의 방법은 정사영상을 제작할 경우, 폐색지역을 탐지하고 복원하기 위해 항공영상의 외부표정요소와 정밀 3D 객체 모델링 데이터가 필요하며, 일련의 복잡한 과정을 자동화하는 것은 어렵다. 본 논문에서는 기존의 방법에서 탈피하여 딥러닝(DL)을 이용하여 엄밀정사영상을 제작하는 새로운 방법을 제안하였다. 딥러닝은 여러 분야에서 더욱 급속하게 활용되고 있으며, 최근 생성적 적대 신경망(GAN)은 영상처리 및 컴퓨터비전 분야에서 많은 관심의 대상이다. GAN을 구성하는 생성망은 실제 영상과 유사한 결과가 생성되도록 학습을 수행하고, 판별망은 생성망의 결과가 실제 영상으로 판단될 때까지 반복적으로 수행한다. 본 논문에서 독일 사진측량, 원격탐사 및 공간정보학회(DGPF)가 구축하고 국제 사진측량 및 원격탐사학회(ISPRS)가 제공하는 데이터 셋 중에서 라이다 반사강도 데이터와 적외선 정사영상을 GAN기반의 Pix2Pix 모델 학습에 사용하여 엄밀정사영상을 생성하는 두 가지 방법을 제안하였다. 첫 번째 방법은 라이다 반사강도영상을 입력하고 고해상도의 정사영상을 목적영상으로 사용하여 학습하는 방식이고, 두 번째 방법에서도 입력영상은 첫 번째 방법과 같이 라이다 반사강도영상이지만 목적영상은 라이다 점군집 데이터에 칼라를 지정한 저해상도의 영상을 이용하여 재귀적으로 학습하여 점진적으로 화질을 개선하는 방법이다. 두 가지 방법으로 생성된 정사영상을 FID(Fréchet Inception Distance)를 이용하여 정량적 수치로 비교하면 큰 차이는 없었지만, 입력영상과 목적영상의 품질이 유사할수록, 학습 수행 시 epoch를 증가시키면 우수한 결과를 얻을 수 있었다. 본 논문은 딥러닝으로 엄밀정사영상 생성 가능성을 확인하기 위한 초기단계의 실험적 연구로서 향후 보완 및 개선할 사항을 파악할 수 있었다.

Keywords

References

  1. Amhar, F., Jansa, J., and Ries, C. (1998), The generation of true orthophotos using a 3d building model in conjunction with a conventional DTM, International Archives of Photogrammetry and Remote Sensing, Vol. 32, pp. 16-22.
  2. Arjovsky, M., Chintala, S., and Bottou, L. (2017), Wasserstein generative adversarial networks, International Conference on Machine Learning-2017, 6-11 August, Sydney, Australia, pp. 214-223.
  3. Bashmal, L., Bazi, Y., AlHichri, H., AlRahhal, M., Ammour, N., and Alajlan, N. (2018), Siamese-GAN: Learning invariant representations for aerial vehicle image categorization, Remote Sensing, Vol. 10, No. 2, pp. 351. https://doi.org/10.3390/rs10020351
  4. Behnke, S. (2003), Hierarchical Neural Networks for Image Interpretation, Springer-Verlag, Berlin, Germany.
  5. Cramer, M. (2010), The DGPF test on digital aerial camera evaluation - Overview and test design. Photogrammetrie, Fernerkundung, Geoinformation, Vol. 2, pp. 73-82. https://doi.org/10.1127/1432-8364/2010/0041
  6. Demir, U. and Unal, G. (2018), Patch-Based image inpainting with generative adversarial networks, arXiv:1803.07422v1.
  7. Demiray, B., Sit, M., and Demir, I. (2020), D-SRGAN: DEM super-resolution with generative adversarial networks, arXiv:2004.04788v2.
  8. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014), Generative adversarial nets, Neural Information Processing Systems, pp. 2672-2680.
  9. Gui, J., Sun, Z., Wen, Y., Tao, D., and Ye, J. (2020), A review on generative adversarial networks: algorithms, theory, and applications, arXiv:2001.06937v1.
  10. Habib, A., Kim, E.M., and Kim, C.J. (2007), New methodologies for true orthophoto generation, Photogrammetric Engineering & Remote Sensing, Vol. 73, No. 1, pp. 25-36. https://doi.org/10.14358/PERS.73.1.25
  11. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., and Hochreiter, S. (2017), GANs trained by a two time-scale update rule converge to a Nash equilibrium, arXiv:1706.08500v6.
  12. Hyung, S., Shin, Y., and Lee, D.C. (2020), Artificial neural network for IR image to true color image conversion, Proceedings of Korean Society of Surveying, Geodesy, Photogrammetry & Cartography-2020, 2-3 July, Seoul, Korea, pp. 212-217. (in Korean with English abstract)
  13. Isola, P., Zhu, J.Y., Zhou, T., and Efros, A. (2017), Image-toimage translation with conditional adversarial networks, IEEE Conference on Computer Vision and Pattern Recognition-2017, 21-26 July, Honolulu, HI, USA, pp. 1125-1134.
  14. Kim, E., Choi, H., and Park, J. (2017), Analysis of applicability of orthophoto using 3d mesh on aerial image with large file size, Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 35, No. 3, pp. 155-166. https://doi.org/10.7848/ksgpc.2017.35.3.155
  15. Kim, J. and Um, D. (2015), High quality ortho-image production using the high resolution DMC aerial image, Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 33, No. 1, pp. 11-21. https://doi.org/10.7848/ksgpc.2015.33.1.11
  16. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., and Jackel, L.D. (1989), Backpropagation applied to handwritten zip code recognition, Neural Computation, Vol. 4, No. 1, pp. 541-551.
  17. Lee, H. (2008), Producing true orthophoto using multidimensional spatial information, Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 26, No. 3, pp. 241-253.
  18. Li, C. and Wand, M. (2016), Precomputed real-time texture synthesis with markovian generative adversarial networks, European Conference on Computer Vision-2016, 11-14 October, Amsterdam, The Netherlands, pp. 702-716.
  19. Mirza, M. and Osindero, S. (2014), Conditional generative adversarial nets, arXiv:1411.1784.
  20. Panagiotou, E., Chochlakis, G., Grammatikopoulos, L., and Charou, E. (2020), Generating elevation surface from a single RGB remotely sensed image using deep learning, Remote Sensing, Vol. 12, No. 12.
  21. Radford, A., Metz, L., and Chintala, S. (2015), Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv:1511.06434.
  22. Ronneberger, O., Fischer, P., and Brox, T. (2015), U-Net: Convolutional networks for biomedical image segmentation, Medical Image Computing and Computer-Assisted Intervention, pp. 234-241.
  23. Rottensteiner, F., Sohn, G., Gerke, M., and Wegner, J. (2013), ISPRS test project on urban classification and 3D building reconstruction, http://www2.isprs.org/tl_files/isprs/wg34/docs/ComplexScenes_revision_v4.pdf (last date accessed: 6 April 2018).
  24. Sajjadi, M., Scholkopf, B., and Hirsch, M. (2017), Enhancenet: Single image super-resolution through automated texture synthesis, The IEEE International Conference on Computer Vision-2017, 22-29 October, Venice, Italy, pp. 4501-4510.
  25. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016), Improved Techniques for Training GANs, arXiv:1606.03498v1.
  26. Shelhamer, E., Long, J., and Darrell, T. (2017), Fully convolutional networks for semantic segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 4, pp. 640-651. https://doi.org/10.1109/TPAMI.2016.2572683
  27. Simard, P., Steinkraus, D., and Platt, J. (2003), Best practices for convolutional neural networks applied to visual document analysis, International Conference on Document Analysis and Recognition-2003, 3-6 August, Edinburgh, Scotland, UK, pp. 958-963.
  28. Vaananen, P. (2019), Removing 3D Point Cloud Occlusion Artifacts with Generative Adversarial Networks, PhD thesis, Department of Computer Science, University of Helsinki, Helsinki, Finland, 56p.
  29. Wang, A., Li, Y., Jiang, K., Zhao, L., and Iwahori, Y. (2019), Lidar data classification algorithm based on generative adversarial network, IEEE International Geoscience and Remote Sensing Symposium, pp. 2487-2490.
  30. Yarlagadda, S., Guera, D., Bestagini, P., Zhu, F., Tubaro, S., and Delp, E. (2018), Satellite image forgery detection and localization using GAN and one-class classifier, Electronic Imaging, Vol. 2018, No. 7, pp. 214-1-214-9.
  31. Yoo, E. and Lee D.C. (2016), True orthoimage generation by mutual recovery of occlusion areas, GIScience & Remote Sensing, Vol. 53, No. 2, pp. 227-246. https://doi.org/10.1080/15481603.2015.1128629