Journal of Korea Water Resources Association (한국수자원학회논문집)

Korea Water Resources Association (한국수자원학회)
권호 표지
DOI QR코드

DOI QR Code

Prediction of multipurpose dam inflow utilizing catchment attributes with LSTM and transformer models

유역정보 기반 Transformer및 LSTM을 활용한 다목적댐 일 단위 유입량 예측

  • Kim, Hyung Ju (Department Civil Engineering, Seoul National University of Science and Technology) ;
  • Song, Young Hoon (Institute of Construction Future Talent, Seoul National University of Science and Technology) ;
  • Chung, Eun Sung (Department Civil Engineering, Seoul National University of Science and Technology)
  • 김형주 (서울과학기술대학교 건설시스템공학과) ;
  • 송영훈 (서울과학기술대학교 건설미래인재연구소) ;
  • 정은성 (서울과학기술대학교 건설시스템공학과)
  • Received : 2024.04.17
  • Accepted : 2024.06.12
  • Published : 2024.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

Rainfall-runoff prediction studies using deep learning while considering catchment attributes have been gaining attention. In this study, we selected two models: the Transformer model, which is suitable for large-scale data training through the self-attention mechanism, and the LSTM-based multi-state-vector sequence-to-sequence (LSTM-MSV-S2S) model with an encoder-decoder structure. These models were constructed to incorporate catchment attributes and predict the inflow of 10 multi-purpose dam watersheds in South Korea. The experimental design consisted of three training methods: Single-basin Training (ST), Pretraining (PT), and Pretraining-Finetuning (PT-FT). The input data for the models included 10 selected watershed attributes along with meteorological data. The inflow prediction performance was compared based on the training methods. The results showed that the Transformer model outperformed the LSTM-MSV-S2S model when using the PT and PT-FT methods, with the PT-FT method yielding the highest performance. The LSTM-MSV-S2S model showed better performance than the Transformer when using the ST method; however, it showed lower performance when using the PT and PT-FT methods. Additionally, the embedding layer activation vectors and raw catchment attributes were used to cluster watersheds and analyze whether the models learned the similarities between them. The Transformer model demonstrated improved performance among watersheds with similar activation vectors, proving that utilizing information from other pre-trained watersheds enhances the prediction performance. This study compared the suitable models and training methods for each multi-purpose dam and highlighted the necessity of constructing deep learning models using PT and PT-FT methods for domestic watersheds. Furthermore, the results confirmed that the Transformer model outperforms the LSTM-MSV-S2S model when applying PT and PT-FT methods.

딥러닝을 활용하여 유역 특성을 반영한 유량 예측 및 비교 연구가 주목받고 있다. 본 연구는 셀프 어텐션 메커니즘을 통해 대용량 데이터 훈련에 적합한 Transformer와 인코더-디코더(Encoder-Decoder) 구조를 가지는 LSTM-based multi-state-vector sequence-to-sequence (LSTM-MSV-S2S) 모형을 선정하여 유역정보(catchment attributes)를 고려할 수 있는 모형을 구축하였고 이를 토대로 국내 10개 다목적댐 유역의 유입량을 예측하였다. 본 연구에서 설계한 실험 구성은 단일유역-단일훈련(Single-basin Training, ST), 다수유역-단일훈련(Pretraining, PT), 사전학습-파인튜닝(Pretraining-Finetuning, PT-FT)의 세 가지 훈련 방법을 사용하였다. 모형의 입력 자료는 선정된 10가지 유역정보와 함께 기상 자료를 사용하였으며, 훈련 방법에 따른 유입량 예측 성능을 비교하였다. 그 결과, Transformer 모형은 PT와 PT-FT 방법에서 LSTM-MSV-S2S보다 우수한 성능을 보였으며, 특히 PT-FT 기법 적용 시 가장 높은 성능을 나타냈다. LSTM-MSV-S2S는 ST 방법에서는 Transformer보다 높은 성능을 보였으나, PT 및 PT-FT 방법에서는 낮은 성능을 보였다. 또한, 임베딩 레이어 활성화 값과 원본 유역정보를 군집화하여 모형의 유역 간 유사성 학습 여부를 분석하였다. Transformer는 활성화 벡터가 유사한 유역들에서 성능이 향상되었으며, 이는 사전에 학습된 다른 유역의 정보를 활용해 성능이 개선됨을 입증하였다. 본 연구는 다목적댐별 적합한 모형 및 훈련 방법을 비교하고, 국내 유역에 PT 및 PT-FT 방법을 적용한 딥러닝 모형 구축의 필요성을 제시하였다. 또한, PT 및 PT-FT 방법 적용 시 Transformer가 LSTM-MSV-S2S보다 성능이 더 우수하였다.

Keywords

Acknowledgement

본 연구는 한국연구재단(RS-2023-00246767_2)의 지원을 받아 수행되었습니다. 이에 감사드립니다.

References

  1. Addor, N., Newman, A.J., Mizukami, N., and Clark, M.P. (2017). "The CAMELS data set: Catchment attributes and meteorology for large-sample studies." Hydrology and Earth System Sciences, Vol. 21, No. 10, pp. 5293-5313. https://doi.org/10.5194/hess-21-5293-2017
  2. Beck, H.E., Vergopolan, N., Pan, M., Levizzani, V., Van Dijk, A.I., Weedon, G.P., Brocca, L., Pappenberger, F., Huffman, G.J., and Wood, E.F. (2017). "Global-scale evaluation of 22 precipitation datasets using gauge observations and hydrological modeling." Hydrology and Earth System Sciences, Vol. 21, No. 12, pp. 6201-6217. https://doi.org/10.5194/hess-21-6201-2017
  3. Bloschl, G., and Sivapalan, M. (1995). "Scale issues in hydrological modelling: A review." Hydrological Processes, Vol. 9, No. 3-4, pp. 251-290. https://doi.org/10.1002/hyp.3360090305
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). "Language models are few-shot learners." Advances in Neural Information Processing Systems, Vol. 33, pp. 1877-1901.
  5. Brunner, M.I., Slater, L., Tallaksen, L.M., and Clark, M. (2021). "Challenges in modeling and predicting floods and droughts: A review" Wiley Interdisciplinary Reviews: Water, Vol. 8, No. 3, e1520.
  6. Chen, C., Hui, Q., Xie, W., Wan, S., Zhou, Y., and Pei, Q. (2021). "Convolutional Neural Networks for forecasting flood process in Internet-of-Things enabled smart city." Computer Networks, Vol. 186, 107744.
  7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint, arXiv:1810.04805.
  8. Ding, Y., Zhu, Y., Feng, J., Zhang, P., and Cheng, Z. (2020). "Interpretable spatio-temporal attention LSTM model for flood forecasting." Neurocomputing, Vol. 403, pp. 348-359. https://doi.org/10.1016/j.neucom.2020.04.110
  9. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). "An image is worth 16x16 words: Transformers for image recognition at scale" arXiv preprint, arXiv: 2010.11929.
  10. Fang, K., Kifer, D., Lawson, K., Feng, D., and Shen, C. (2022). "The data synergy effects of time-series deep learning models in hydrology." Water Resources Research, Vol. 58, No. 4, e2021 WR029583.
  11. Gao, S., Zhang, S., Huang, Y., Han, J., Luo, H., Zhang, Y., and Wang, G. (2022). "A new seq2seq architecture for hourly runoff prediction using historical rainfall and runoff as input." Journal of Hydrology, Vol. 612, 128099.
  12. Gupta, H., Perrin, C., Bloschl, G., Montanari, A., Kumar, R., Clark, M., and Andreassian, V. (2014). "Large-sample hydrology: A need to balance depth with breadth." Hydrology and Earth System Sciences, Vol. 18, No. 2, pp. 463-477. https://doi.org/10.5194/hess-18-463-2014
  13. Gupta, H.V., Kling, H., Yilmaz, K.K., and Martinez, G.F. (2009). "Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling." Journal of Hydrology, Vol. 377, No. 1-2, pp. 80-91. https://doi.org/10.1016/j.jhydrol.2009.08.003
  14. Hochreiter, S., and Schmidhuber, J. (1997). "Long short-term memory." Neural Computation, Vol. 9, No. 8, pp. 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
  15. Jeong, J., and Park, E. (2019). "Comparative applications of datadriven models representing water table fluctuations." Journal of Hydrology, Vol. 572, pp. 261-273. https://doi.org/10.1016/j.jhydrol.2019.02.051
  16. Jun, H., and Lee, J. (2013). "A methodology for flood forecasting and warning based on the characteristic of observed water levels between upstream and downstream." Journal of the Korean Society of Hazard Mitigation, Vol. 13, No. 6, pp. 367-374. https://doi.org/10.9798/KOSHAM.2013.13.6.367
  17. Jung, J., Mo, H., Lee, J., Yoo, Y., and Kim, H.S. (2021). "Flood stage forecasting at the Gurye-Gyo station in Sumjin River Using LSTM-based deep learning models." Journal of the Korean Society of Hazard Mitigation, Vol. 21, No. 3, pp. 193-201. https://doi.org/10.9798/KOSHAM.2021.21.3.193
  18. Jung, S., Lee, D., and Lee, K. (2018). "Prediction of river water level using deep-learning open library." Journal of the Korean Society of Hazard Mitigation, Vol. 18, No. 1, pp. 1-11.
  19. Kao, I.-F., Zhou, Y., Chang, L.-C., and Chang, F.-J. (2020). "Exploring a Long Short-Term Memory based Encoder-Decoder framework for multi-step-ahead flood forecasting." Journal of Hydrology, Vol. 583, 124631.
  20. Kratzert, F., Klotz, D., Brenner, C., Schulz, K., and Herrnegger, M. (2018). "Rainfall-runoff modelling using Long Short-Term Memory (LSTM) networks." Hydrology and Earth System Sciences, Vol. 22, No. 11, pp. 6005-6022. https://doi.org/10.5194/hess-22-6005-2018
  21. Kratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., and Nearing, G. (2019). "Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets." Hydrology and Earth System Sciences, Vol. 23, No. 12, pp. 5089-5110. https://doi.org/10.5194/hess-23-5089-2019
  22. Kratzert, F., Nearing, G., Addor, N., Erickson, T., Gauch, M., Gilon, O., Gudmundsson, L., Hassidim, A., Klotz, D., and Nevo, S. (2023). "Caravan-A global community dataset for large-sample hydrology." Scientific Data, Vol. 10, No. 1, 61.
  23. LeCun, Y., Bengio, Y., and Hinton, G. (2015). "Deep learning." Nature, Vol. 521, No. 7553, pp. 436-444. https://doi.org/10.1038/nature14539
  24. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021). "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF International Conference on Computer Vision, Microsoft Research Asia, pp. 10012-10022.
  25. Mok, J.-Y., Choi, J.-H., and Moon, Y.-I. (2020). "Prediction of multipurpose dam inflow using deep learning." Journal of Korea Water Resources Association, Vol. 53, No. 2, pp. 97-105. https://doi.org/10.3741/JKWRA.2020.53.2.97
  26. Nash, J.E., and Sutcliffe, J.V. (1970). "River flow forecasting through conceptual models part I - A discussion of principles." Journal of Hydrology, Vol. 10, No. 3, pp. 282-290. https://doi.org/10.1016/0022-1694(70)90255-6
  27. Nearing, G.S., Kratzert, F., Sampson, A.K., Pelissier, C.S., Klotz, D., Frame, J.M., Prieto, C., and Gupta, H.V. (2021). "What role does hydrological science play in the age of machine learning?" Water Resources Research, Vol. 57, No. 3, e2020WR028091.
  28. Newman, A.J., Clark, M.P., Sampson, K., Wood, A., Hay, L.E., Bock, A., Viger, R.J., Blodgett, D., Brekke, L., and Arnold, J. (2015). "Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: Data set characteristics and assessment of regional variability in hydrologic model performance" Hydrology and Earth System Sciences, Vol. 19, No. 1, pp. 209-223. https://doi.org/10.5194/hess-19-209-2015
  29. Oudin, L., Andreassian, V., Perrin, C., Michel, C., and Le Moine, N. (2008). "Spatial proximity, physical similarity, regression and ungaged catchments: A comparison of regionalization approaches based on 913 French catchments." Water Resources Research, Vol. 44, No. 3, W03413.
  30. Razavi, T., and Coulibaly, P. (2013). "Streamflow prediction in ungauged basins: review of regionalization methods" Journal of Hydrologic Engineering, Vol. 18, No. 8, pp. 958-975. https://doi.org/10.1061/(ASCE)HE.1943-5584.0000690
  31. Schmidhuber, J. (2015). "Deep learning in neural networks: An overview" Neural Networks, Vol. 61, pp. 85-117. https://doi.org/10.1016/j.neunet.2014.09.003
  32. Shen, C., Laloy, E., Elshorbagy, A., Albert, A., Bales, J., Chang, F.-J., Ganguly, S., Hsu, K.-L., Kifer, D., and Fang, Z. (2018). "HESS Opinions: Incubating deep-learning-powered hydrologic science advances as a community." Hydrology and Earth System Sciences, Vol. 22, No. 11, pp. 5639-5656. https://doi.org/10.5194/hess-22-5639-2018
  33. Sivapalan, M., Takeuchi, K., Franks, S.W., Gupta, V.K., Karambiri, H., Lakshmi, V., Liang, X., McDonnell, J.J., Mendiondo, E.M., O'Connell, P.E., Oki, T., Pomeroy, J.W., Schertzer, D., Uhlenbrook, S., and Zehe, E. (2003). "IAHS Decade on Predictions in Ungauged Basins (PUB), 2003-2012: Shaping an exciting future for the hydrological sciences." Hydrological Sciences Journal, Vol. 48, No. 6, pp. 857-880. https://doi.org/10.1623/hysj.48.6.857.51421
  34. Tuli, S., Casale, G., and Jennings, N.R. (2022). "Tranad: Deep transformer networks for anomaly detection in multivariate time series data." arXiv preprint, arXiv:2201.07284.
  35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). "Attention is all you need." Advances in Neural Information Processing Systems, Vol. 30, Long Beach, CA, U.S.
  36. Wen, Q., He, K., Sun, L., Zhang, Y., Ke, M., and Xu, H. (2021). "RobustPeriod: Robust time-frequency mining for multiple periodicity detection." Proceedings of the 2021 International Conference on Management of Data, China, pp. 2328-2337.
  37. Wu, H., Xu, J., Wang, J., and Long, M. (2021). "Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting." Advances in Neural Information Processing Systems, Vol. 34, pp. 22419-22430.
  38. Xu, J., Wu, H., Wang, J., and Long, M. (2021). "Anomaly transformer: Time series anomaly detection with association discrepancy" arXiv preprint, arXiv:2110.02642.
  39. Xu, Y., Lin, K., Hu, C., Wang, S., Wu, Q., Zhang, L., and Ran, G. (2023). "Deep transfer learning based on transformer for flood forecasting in data-sparse basins." Journal of Hydrology, Vol. 625, 129956.
  40. Yang, C.-H.H., Tsai, Y.-Y., and Chen, P.-Y. (2021). "Voice2series: Reprogramming acoustic models for time series classification." International Conference on Machine Learning, PMLR, pp. 11808-11819.
  41. Yin, H., Guo, Z., Zhang, X., Chen, J., and Zhang, Y. (2022). "RRFormer: Rainfall-runoff modeling based on Transformer." Journal of Hydrology, Vol. 609, 127781.
  42. Yin, H., Zhang, X., Wang, F., Zhang, Y., Xia, R., and Jin, J. (2021). "Rainfall-runoff modeling using LSTM-based multi-state-vector sequence-to-sequence model." Journal of Hydrology, Vol. 598, 126378.
  43. Yin, H., Zhu, W., Zhang, X., Xing, Y., Xia, R., Liu, J., and Zhang, Y. (2023). "Runoff predictions in new-gauged basins using two transformer-based models" Journal of Hydrology, Vol. 622, 129684.
  44. Zhang, Y., Chiew, F. H., Li, M., and Post, D. (2018). "Predicting runoff signatures using regression and hydrological modeling approaches." Water Resources Research, Vol. 54, No. 10, pp. 7859-7878. https://doi.org/10.1029/2018WR023325
  45. Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, Canada, Vol. 35, No. 12, pp. 11106-11115.