Browse > Article
http://dx.doi.org/10.7848/ksgpc.2020.38.4.363

True Orthoimage Generation from LiDAR Intensity Using Deep Learning  

Shin, Young Ha (Dept. of Environment, Energy & Geoinformatics, Sejong University)
Hyung, Sung Woong (Dept. of Environment, Energy & Geoinformatics, Sejong University)
Lee, Dong-Cheon (Dept. of Environment, Energy & Geoinformatics, Sejong University)
Publication Information
Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography / v.38, no.4, 2020 , pp. 363-373 More about this Journal
Abstract
During last decades numerous studies generating orthoimage have been carried out. Traditional methods require exterior orientation parameters of aerial images and precise 3D object modeling data and DTM (Digital Terrain Model) to detect and recover occlusion areas. Furthermore, it is challenging task to automate the complicated process. In this paper, we proposed a new concept of true orthoimage generation using DL (Deep Learning). DL is rapidly used in wide range of fields. In particular, GAN (Generative Adversarial Network) is one of the DL models for various tasks in imaging processing and computer vision. The generator tries to produce results similar to the real images, while discriminator judges fake and real images until the results are satisfied. Such mutually adversarial mechanism improves quality of the results. Experiments were performed using GAN-based Pix2Pix model by utilizing IR (Infrared) orthoimages, intensity from LiDAR data provided by the German Society for Photogrammetry, Remote Sensing and Geoinformation (DGPF) through the ISPRS (International Society for Photogrammetry and Remote Sensing). Two approaches were implemented: (1) One-step training with intensity data and high resolution orthoimages, (2) Recursive training with intensity data and color-coded low resolution intensity images for progressive enhancement of the results. Two methods provided similar quality based on FID (Fréchet Inception Distance) measures. However, if quality of the input data is close to the target image, better results could be obtained by increasing epoch. This paper is an early experimental study for feasibility of DL-based true orthoimage generation and further improvement would be necessary.
Keywords
LiDAR Intensity; True Orthoimage; Deep Learning; GAN;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 Kim, J. and Um, D. (2015), High quality ortho-image production using the high resolution DMC aerial image, Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 33, No. 1, pp. 11-21.   DOI
2 LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., and Jackel, L.D. (1989), Backpropagation applied to handwritten zip code recognition, Neural Computation, Vol. 4, No. 1, pp. 541-551.
3 Lee, H. (2008), Producing true orthophoto using multidimensional spatial information, Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 26, No. 3, pp. 241-253.
4 Li, C. and Wand, M. (2016), Precomputed real-time texture synthesis with markovian generative adversarial networks, European Conference on Computer Vision-2016, 11-14 October, Amsterdam, The Netherlands, pp. 702-716.
5 Mirza, M. and Osindero, S. (2014), Conditional generative adversarial nets, arXiv:1411.1784.
6 Panagiotou, E., Chochlakis, G., Grammatikopoulos, L., and Charou, E. (2020), Generating elevation surface from a single RGB remotely sensed image using deep learning, Remote Sensing, Vol. 12, No. 12.
7 Radford, A., Metz, L., and Chintala, S. (2015), Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv:1511.06434.
8 Ronneberger, O., Fischer, P., and Brox, T. (2015), U-Net: Convolutional networks for biomedical image segmentation, Medical Image Computing and Computer-Assisted Intervention, pp. 234-241.
9 Rottensteiner, F., Sohn, G., Gerke, M., and Wegner, J. (2013), ISPRS test project on urban classification and 3D building reconstruction, http://www2.isprs.org/tl_files/isprs/wg34/docs/ComplexScenes_revision_v4.pdf (last date accessed: 6 April 2018).
10 Sajjadi, M., Scholkopf, B., and Hirsch, M. (2017), Enhancenet: Single image super-resolution through automated texture synthesis, The IEEE International Conference on Computer Vision-2017, 22-29 October, Venice, Italy, pp. 4501-4510.
11 Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016), Improved Techniques for Training GANs, arXiv:1606.03498v1.
12 Shelhamer, E., Long, J., and Darrell, T. (2017), Fully convolutional networks for semantic segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 4, pp. 640-651.   DOI
13 Simard, P., Steinkraus, D., and Platt, J. (2003), Best practices for convolutional neural networks applied to visual document analysis, International Conference on Document Analysis and Recognition-2003, 3-6 August, Edinburgh, Scotland, UK, pp. 958-963.
14 Yoo, E. and Lee D.C. (2016), True orthoimage generation by mutual recovery of occlusion areas, GIScience & Remote Sensing, Vol. 53, No. 2, pp. 227-246.   DOI
15 Vaananen, P. (2019), Removing 3D Point Cloud Occlusion Artifacts with Generative Adversarial Networks, PhD thesis, Department of Computer Science, University of Helsinki, Helsinki, Finland, 56p.
16 Wang, A., Li, Y., Jiang, K., Zhao, L., and Iwahori, Y. (2019), Lidar data classification algorithm based on generative adversarial network, IEEE International Geoscience and Remote Sensing Symposium, pp. 2487-2490.
17 Yarlagadda, S., Guera, D., Bestagini, P., Zhu, F., Tubaro, S., and Delp, E. (2018), Satellite image forgery detection and localization using GAN and one-class classifier, Electronic Imaging, Vol. 2018, No. 7, pp. 214-1-214-9.
18 Bashmal, L., Bazi, Y., AlHichri, H., AlRahhal, M., Ammour, N., and Alajlan, N. (2018), Siamese-GAN: Learning invariant representations for aerial vehicle image categorization, Remote Sensing, Vol. 10, No. 2, pp. 351.   DOI
19 Amhar, F., Jansa, J., and Ries, C. (1998), The generation of true orthophotos using a 3d building model in conjunction with a conventional DTM, International Archives of Photogrammetry and Remote Sensing, Vol. 32, pp. 16-22.
20 Arjovsky, M., Chintala, S., and Bottou, L. (2017), Wasserstein generative adversarial networks, International Conference on Machine Learning-2017, 6-11 August, Sydney, Australia, pp. 214-223.
21 Behnke, S. (2003), Hierarchical Neural Networks for Image Interpretation, Springer-Verlag, Berlin, Germany.
22 Habib, A., Kim, E.M., and Kim, C.J. (2007), New methodologies for true orthophoto generation, Photogrammetric Engineering & Remote Sensing, Vol. 73, No. 1, pp. 25-36.   DOI
23 Cramer, M. (2010), The DGPF test on digital aerial camera evaluation - Overview and test design. Photogrammetrie, Fernerkundung, Geoinformation, Vol. 2, pp. 73-82.   DOI
24 Demir, U. and Unal, G. (2018), Patch-Based image inpainting with generative adversarial networks, arXiv:1803.07422v1.
25 Demiray, B., Sit, M., and Demir, I. (2020), D-SRGAN: DEM super-resolution with generative adversarial networks, arXiv:2004.04788v2.
26 Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014), Generative adversarial nets, Neural Information Processing Systems, pp. 2672-2680.
27 Gui, J., Sun, Z., Wen, Y., Tao, D., and Ye, J. (2020), A review on generative adversarial networks: algorithms, theory, and applications, arXiv:2001.06937v1.
28 Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., and Hochreiter, S. (2017), GANs trained by a two time-scale update rule converge to a Nash equilibrium, arXiv:1706.08500v6.
29 Hyung, S., Shin, Y., and Lee, D.C. (2020), Artificial neural network for IR image to true color image conversion, Proceedings of Korean Society of Surveying, Geodesy, Photogrammetry & Cartography-2020, 2-3 July, Seoul, Korea, pp. 212-217. (in Korean with English abstract)
30 Isola, P., Zhu, J.Y., Zhou, T., and Efros, A. (2017), Image-toimage translation with conditional adversarial networks, IEEE Conference on Computer Vision and Pattern Recognition-2017, 21-26 July, Honolulu, HI, USA, pp. 1125-1134.
31 Kim, E., Choi, H., and Park, J. (2017), Analysis of applicability of orthophoto using 3d mesh on aerial image with large file size, Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography, Vol. 35, No. 3, pp. 155-166.   DOI