Browse > Article
http://dx.doi.org/10.7780/kjrs.2019.35.5.1.5

Combining 2D CNN and Bidirectional LSTM to Consider Spatio-Temporal Features in Crop Classification  

Kwak, Geun-Ho (Department of Geoinformatic Engineering, Inha University)
Park, Min-Gyu (Department of Geoinformatic Engineering, Inha University)
Park, Chan-Won (National Institute of Agriculture Sciences, Rural Development Administration)
Lee, Kyung-Do (National Institute of Agriculture Sciences, Rural Development Administration)
Na, Sang-Il (National Institute of Agriculture Sciences, Rural Development Administration)
Ahn, Ho-Yong (National Institute of Agriculture Sciences, Rural Development Administration)
Park, No-Wook (Department of Geoinformatic Engineering, Inha University)
Publication Information
Korean Journal of Remote Sensing / v.35, no.5_1, 2019 , pp. 681-692 More about this Journal
Abstract
In this paper, a hybrid deep learning model, called 2D convolution with bidirectional long short-term memory (2DCBLSTM), is presented that can effectively combine both spatial and temporal features for crop classification. In the proposed model, 2D convolution operators are first applied to extract spatial features of crops and the extracted spatial features are then used as inputs for a bidirectional LSTM model that can effectively process temporal features. To evaluate the classification performance of the proposed model, a case study of crop classification was carried out using multi-temporal unmanned aerial vehicle images acquired in Anbandegi, Korea. For comparison purposes, we applied conventional deep learning models including two-dimensional convolutional neural network (CNN) using spatial features, LSTM using temporal features, and three-dimensional CNN using spatio-temporal features. Through the impact analysis of hyper-parameters on the classification performance, the use of both spatial and temporal features greatly reduced misclassification patterns of crops and the proposed hybrid model showed the best classification accuracy, compared to the conventional deep learning models that considered either spatial features or temporal features. Therefore, it is expected that the proposed model can be effectively applied to crop classification owing to its ability to consider spatio-temporal features of crops.
Keywords
Crop classification; Convolutional neural network; Long short-term memory; Spatio-temporal features;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 Hua, Y., L. Mou, and X. X. Zhu, 2019. Recurrently exploring class-wise attention in a hybrid convolu - tional and bidirectional LSTM network for multilabel aerial image classification, ISPRS Journal of Photogrammetry and Remote Sensing, 149: 188-199.   DOI
2 Ji, S., C. Zhang, A. Xu, Y. Shi, and Y. Duan, 2018. 3D convolutional neural networks for crop classification with multi-temporal remote sensing images, Remote Sensing, 10(1): 75.   DOI
3 Kim, Y., N.-W. Park, and K.-D. Lee, 2017. Selflearning based land-cover classification using sequential class patterns from past land-cover maps, Remote Sensing, 9(9): 921.   DOI
4 Kim, Y., G.-H. Kwak, K.-D. Lee, S.-I. Na, C.-W. Park, and N.-W. Park, 2018. Performance evaluation of machine learning and deep learning algorithms in crop classification: Impact of hyper-parameters and training sample size, Korean Journal of Remote Sensing, 34(5): 811-827 (in Korean with English abstract).   DOI
5 Kussul, N., G. Lemoine, F. J. Gallego, S. V. Skakun, M. Lavreniuk, and A. Y. Shelestov, 2016. Parcelbased crop classification in Ukraine using Landsat-8 data and Sentinel-1A data, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(6): 2500-2508.   DOI
6 Kwak, G.-H. and N.-W. Park, 2019. Impact of texture information on crop classification with machine learning and UAV images, Applied Sciences, 9(4): 643.   DOI
7 Lee, S. and J. Kim, 2019. Land cover classification using sematic image segmentation with deep learning, Korean Journal of Remote Sensing, 35(2): 279-288 (in Korean with English abstract).   DOI
8 Liu, Q., F. Zhou, R. Hang, and X. Yuan, 2017. Bidirectional-convolutional LSTM based spectralspatial feature learning for hyperspectral image classification, Remote Sensing, 9(12): 1330.   DOI
9 Mou, L., L. Bruzzone, and X. X. Zhu, 2018. Learning spectral-spatial-temporal features via a recurrent convolutional neural network for change detection in multispectral imagery, IEEE Transactions on Geoscience and Remote Sensing, 57(2): 924-935.   DOI
10 Ma, A., A. M. Filippi, Z. Wang, and Z. Yin, 2019. Hyperspectral image classification using similarity measurements-based deep recurrent neural networks, Remote Sensing, 11(2): 194.   DOI
11 Russwurm, M. and M. Korner, 2017. Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images, Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2017, Honolulu, HI, Jul. 21-26, pp. 11-19.
12 Russwurm, M. and M. Korner, 2018. Multi-temporal land cover classification with sequential recurrent encoders, ISPRS International Journal of Geo-Information, 7(4): 129.   DOI
13 Schuster, M. and K. K. Paliwal, 1997. Bidirectional recurrent neural networks, IEEE Transactions on Signal Processing, 45(11): 2673-2681.   DOI
14 Seydgar, M., A. Alizadeh Naeini, M. Zhang, W. Li, and M. Satari, 2019. 3-D convolution-recurrent networks for spectral-spatial classification of hyperspectral images, Remote Sensing, 11(7): 883.   DOI
15 Siachalou, S., G. Mallinis, and M. Tsakiri-Strati, 2015. A hidden Markov models approach for crop classification: Linking crop phenology to time series of multi-sensor remote sensing data, Remote Sensing, 7(4): 3633-3650.   DOI
16 Song, A. and Y. Kim, 2017. Deep learning-based hyperspectral image classification with application to environmental geographic information systems, Korean Journal of Remote Sensing, 33(6-2): 1061-1073 (in Korean with English abstract).   DOI
17 Xie, B., H. K. Zhang, and J. Xue, 2019. Deep convolutional neural network for mapping smallholder agriculture using high spatial resolution satellite image, Sensors, 19(10): 2398.   DOI
18 Tatsumi, K., Y. Yamashiki, M. A. C. Torres, and C. L. R. Taipe, 2015. Crop classification of upland fields using Random forest of time-series Landsat 7 ETM+ data, Computers and Electronics in Agriculture, 115: 171-179.   DOI
19 Ullah, A., J. Ahmad, K. Muhammad, M. Sajjad, and S. W. Baik, 2017. Action recognition in video sequences using deep bi-directional LSTM with CNN features, IEEE Access, 6: 1155-1166.   DOI
20 Wei, S., H. Zhang, C. Wang, Y. Wang, and L. Xu, 2019. Multi-temporal SAR data large-scale crop mapping based on U-Net model, Remote Sensing, 11(1): 68.   DOI
21 Zhang, C., X. Pan, H. Li, A. Gardiner, I. Sargent, J. Hare, and P. M. Atkinson, 2018. A hybrid MLPCNN classifier for very fine resolution remotely sensed image classification, ISPRS Journal of Photogrammetry and Remote Sensing, 140: 133-144.   DOI
22 Zhong, L., L. Hu, and H. Zhou, 2019. Deep learning based multi-temporal crop classification, Remote Sensing of Environment, 221: 430-443.   DOI
23 Zhou, F., R. Hang, Q. Liu, and X. Yuan, 2019. Hyperspectral image classification using spectralspatial LSTMs, Neurocomputing, 328: 39-47.   DOI
24 Feng, Q., D. Zhu, J. Yang, and B. Li, 2019. Multisource hyperspectral and LiDAR data fusion for urban land-use mapping based on a modified two-branch convolutional neural network, ISPRS International Journal of Geo-Information, 8(1): 28.   DOI
25 Chiu, J. P. C. and E. Nichols, 2016. Named entity recognition with bidirectional LSTM-CNNs, Transactions of the Association for Computational Linguistics, 4: 357-370.   DOI