Browse > Article
http://dx.doi.org/10.7848/ksgpc.2022.40.3.177

The Effect of Training Patch Size and ConvNeXt application on the Accuracy of CycleGAN-based Satellite Image Simulation  

Won, Taeyeon (Dept. of Advanced Technology Fusion, Konkuk University, Realtimevisual Inc.)
Jo, Su Min (Dept. of Technology Fusion Engineering, Konkuk University)
Eo, Yang Dam (Dept. of Civil and Environmental Engineering, Konkuk University)
Publication Information
Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography / v.40, no.3, 2022 , pp. 177-185 More about this Journal
Abstract
A method of restoring the occluded area was proposed by referring to images taken with the same types of sensors on high-resolution optical satellite images through deep learning. For the natural continuity of the simulated image with the occlusion region and the surrounding image while maintaining the pixel distribution of the original image as much as possible in the patch segmentation image, CycleGAN (Cycle Generative Adversarial Network) method with ConvNeXt block applied was used to analyze three experimental regions. In addition, We compared the experimental results of a training patch size of 512*512 pixels and a 1024*1024 pixel size that was doubled. As a result of experimenting with three regions with different characteristics,the ConvNeXt CycleGAN methodology showed an improved R2 value compared to the existing CycleGAN-applied image and histogram matching image. For the experiment by patch size used for training, an R2 value of about 0.98 was generated for a patch of 1024*1024 pixels. Furthermore, As a result of comparing the pixel distribution for each image band, the simulation result trained with a large patch size showed a more similar histogram distribution to the original image. Therefore, by using ConvNeXt CycleGAN, which is more advanced than the image applied with the existing CycleGAN method and the histogram-matching image, it is possible to derive simulation results similar to the original image and perform a successful simulation.
Keywords
Occlusion; Image Restoration; ConvNeXt; CycleGAN; Training Patch;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014), Generative adversarial nets, Neural Information Processing Systems-2014, 8-13 December, Montreal, Canada, pp. 2672-2680.
2 Isola, P., Zhu, J.Y., Zhou, T., and Efros, and A.A. (2017), Image-to-image translation with conditional adversarial networks, IEEE Conference on computer vision and pattern recognition-2017, 21-26 July, Honolulu, USA, pp. 1125-1134.
3 Kim, D., Wang, K., Sclaroff, S., and Saenko, K. (2022), A Broad Study of Pre-training for Domain Generalization and Adaptation, arXiv preprint arXiv:2203.11819. https://doi.org/10.48550/arXiv.2203.11819   DOI
4 Lee, M.H., Lee, S.B., Eo, Y.D., Kim, S.W., Woo, J.H., and Han, S.H. (2017), A comparative study on generating simulated Landsat NDVI images using data fusion and regression method-the case of the Korean Peninsula, Environmental Monitoring and Assessment, Vol. 189, No. 7, pp. 1-13. https://doi.org/10.1007/s10661-017-6034-z   DOI
5 LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., and Jackel, L.D. (1989), Backpropagation applied to handwritten zip code recognition, Neural computation, Vol. 1, No. 4, pp. 541-551.https://doi.org/10.1162/neco.1989.1.4.541   DOI
6 Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. 2021), Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, arXiv preprint arXiv:2103.14020. https://doi.org/10.48550/arXiv.2103.14030   DOI
7 Liu, Y., Yano, T., Nishiyama, S., and Kimura, R. (2017), Radiometric correction for linear change-detection techniques: analysis in bi-temporal space, International Journal of Remote Sensing, Vol. 28, No. 22, pp. 5143-5157. https://doi.org/10.1080/01431160701268954   DOI
8 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017), Attention is all you need, Neural Information Processing Systems-2017, 4-9 December, Long Beach, USA, pp. 5998-6008.
9 Yoo, E.J. and Lee, D.C. (2010), Patch-based processing and occlusion area recovery for true orthoimage generation, Journal of the Korean Society of Surveying,Vol. 28, No. 1, pp. 83-92. (in Korean with English abstract)
10 Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017), Unpaired image-to-image translation using cycle-consistent adversarial networks, International Conference on Computer Vision, pp. 2223-2232. https://doi.org/10.48550/arXiv.1703.10593   DOI
11 Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021), An image is worth 16x16 words: transformers for image recognition at scale, International Conference on Learning Representations 2021, 3-7 May. https://doi.org/10.48550/arXiv.2010.11929   DOI
12 Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T.S. (2019), Free-form image inpainting with gated convolution, International Conference on Computer Vision, pp. 4471-4480. https://doi.org/10.48550/arXiv.1806.03589   DOI
13 Zhang, Q., Yuan, Q., Zeng, C., Li, X., and Wei, Y. (2018), Missing data reconstruction in remote sensing image with a unified spatial-temporal-spectral deep convolutional neural network, IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, No. 8, pp. 4274-4288. https://doi.org/10.1109/TGRS.2018.2810208   DOI
14 Su, N., Zhang, Y., Tian, S., Yan, Y., and Miao, X. (2016), Shadow detection and removal for occluded object information recovery in urban high-resolution panchromatic satellite images, IEEE Applied Earth Observations and Remote Sensing, Vol. 9, No. 6, pp. 2568-2582. https://doi.org/10.1109/JSTARS.2016.2570234   DOI
15 Won.T. and Eo, Y.D. (2022), An experiment on image restoration Applying the Cycle Generative adversarial network to partial occlusion Kompsat-3A image, Korean Journal of Remote Sensing, Vol. 38, No. 1, pp.33-43. https://doi.org/10.7780/kjrs.2022.38.1.3   DOI
16 Guo, Q., He, M., and Li, A. (2018), High-resolution remote-sensing image registration based on angle matching of edge point features, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 11, No. 8, pp. 2881-2895. https://doi.org/10.1109/JSTARS.2018.2844295   DOI
17 Lee, S.B., Park, W.Y., Eo, Y.D., Pyeon, M.W., Han, S., Yeon, S.H., and Lee, B.K. (2017), Analysis on the applicability of simulated image from SPOT 4 HRVIR image, Korean Society of Civil Engineers, Vol. 21, No. 4, pp. 1434-1442. https://doi.org/10.1007/s12205-016-0522-5   DOI
18 Liu, M., Ma, J., Zheng, Q., Liu, Y., and Shi, G. (2022), 3D Object Detection Based on Attention and Multi-Scale Feature Fusion, Sensor 2022, Vol. 22, No. 10. https://doi.org/10.3390/s22103935   DOI
19 Liu Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S., (2022), A ConvNet for the 2020s, arXiv preprint arXiv:2201.03545. https://doi.org/10.48550/arXiv.2201.03545   DOI
20 Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. (2018), Mixed precision training, International Conference on Learning Representations-2018, 30-3 April-May, Vancouver, Canada.