Journal of Korea Water Resources Association (한국수자원학회논문집)

Korea Water Resources Association (한국수자원학회)
권호 표지
DOI QR코드

DOI QR Code

Comparative analysis of activation functions of artificial neural network for prediction of optimal groundwater level in the middle mountainous area of Pyoseon watershed in Jeju Island

제주도 표선유역 중산간지역의 최적 지하수위 예측을 위한 인공신경망의 활성화함수 비교분석

  • Shin, Mun-Ju (Water Resources Research Team, Jeju Province Development Corporation) ;
  • Kim, Jin-Woo (Water Resources Research Team, Jeju Province Development Corporation) ;
  • Moon, Duk-Chul (Water Resources Research Team, Jeju Province Development Corporation) ;
  • Lee, Jeong-Han (Water Resources Research Team, Jeju Province Development Corporation) ;
  • Kang, Kyung Goo (R&D Innovation Center, Jeju Province Development Corporation)
  • 신문주 (제주특별자치도개발공사 수자원연구팀) ;
  • 김진우 (제주특별자치도개발공사 수자원연구팀) ;
  • 문덕철 (제주특별자치도개발공사 수자원연구팀) ;
  • 이정한 (제주특별자치도개발공사 수자원연구팀) ;
  • 강경구 (제주특별자치도개발공사 R&D 혁신센터)
  • Received : 2021.09.27
  • Accepted : 2021.11.18
  • Published : 2021.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

The selection of activation function has a great influence on the groundwater level prediction performance of artificial neural network (ANN) model. In this study, five activation functions were applied to ANN model for two groundwater level observation wells in the middle mountainous area of the Pyoseon watershed in Jeju Island. The results of the prediction of the groundwater level were compared and analyzed, and the optimal activation function was derived. In addition, the results of LSTM model, which is a widely used recurrent neural network model, were compared and analyzed with the results of the ANN models with each activation function. As a result, ELU and Leaky ReLU functions were derived as the optimal activation functions for the prediction of the groundwater level for observation well with relatively large fluctuations in groundwater level and for observation well with relatively small fluctuations, respectively. On the other hand, sigmoid function had the lowest predictive performance among the five activation functions for training period, and produced inappropriate results in peak and lowest groundwater level prediction. The ANN-ELU and ANN-Leaky ReLU models showed groundwater level prediction performance comparable to that of the LSTM model, and thus had sufficient potential for application. The methods and results of this study can be usefully used in other studies.

활성화함수의 선택은 인공신경망(Artificial Neural Network, ANN) 모델의 지하수위 예측성능에 큰 영향을 미친다. 특히 제주도의 중산간 지역과 같이 지하수위의 변동폭이 크고 변동양상이 복잡한 경우 적절한 지하수위 예측을 위해서는 다양한 활성화함수의 비교분석을 통한 최적의 활성화함수 선택이 반드시 필요하다. 본 연구에서는 지하수위의 변동폭이 크고 변동양상이 복잡한 제주도 표선유역 중산간지역 2개 지하수위 관측정을 대상으로 5개의 활성화함수(sigmoid, hyperbolic tangent (tanh), Rectified Linear Unit (ReLU), Leaky Rectified Linear Unit (Leaky ReLU), Exponential Linear Unit (ELU))를 ANN 모델에 적용하여 지하수위 예측결과를 비교 및 분석하고 최적 활성화함수를 도출하였다. 그리고 최근 널리 사용되고 있는 순환신경망 모델인 Long Short-Term Memory (LSTM) 모델의 결과와 비교분석하였다. 분석결과 지하수위 변동폭이 상대적으로 큰 관측정과 상대적으로 작은 관측정에 대한 지하수위 예측에 대해서는 각각 ELU와 Leaky ReLU 함수가 최적의 활성화함수로 도출되었다. 반면 sigmoid 함수는 학습기간에 대해 5개 활성화함수 중 예측성능이 가장 낮았으며 첨두 및 최저 지하수위 예측에서 적절하지 못한 결과를 도출하였다. 따라서 ANN-sigmoid 모델은 가뭄기간의 지하수위 예측을 통한 지하수자원 관리목적으로 사용할 경우 주의가 필요하다. ANN-ELU와 ANN-Leaky ReLU 모델은 LSTM 모델과 대등한 지하수위 예측성능을 보여 활용가능성이 충분히 있으며 LSTM 모델은 ANN 모델들 보다 예측성능이 높아 인공지능 모델의 예측성능 비교분석 시 참고 모델로 활용될 수 있다. 마지막으로 학습기간의 정보량에 따라 학습기간의 지하수위 예측성능이 검증 및 테스트 기간의 예측성능보다 낮을 수 있다는 것을 확인하였으며, 관측지하수위의 변동폭이 크고 변동양상이 복잡할수록 인공지능 모델별 지하수위 예측능력의 차이는 커졌다. 본 연구에서 제시한 5개의 활성화함수를 적용한 연구방법 및 비교분석 결과는 지하수위 예측뿐만 아니라 일단위 하천유출량 및 시간단위 홍수량 등 지표수 예측을 포함한 다양한 연구에 유용하게 사용될 수 있다.

Keywords

References

  1. Afzaal, H., Farooque, A.A., Abbas, F., Acharya, B., and Esau, T. (2020). "Groundwater estimation from major physical hydrology components using artificial neural networks and deep learning." Water, Vol. 12, No. 1, p. 5. https://doi.org/10.3390/w12010005
  2. Barthel, R., and Banzhaf, S. (2016). "Groundwater and surface water interaction at the regional-scale - a review with focus on regional integrated models." Water Resources Management, Vol. 30, No. 1, pp. 1-32. https://doi.org/10.1007/s11269-015-1163-z
  3. Bengio, Y., Simard, P., and Frasconi, P. (1994). "Learning long-term dependencies with gradient descent is difficult." IEEE Transactions on Neural Networks, Vol. 5, No. 2, pp. 157-166. https://doi.org/10.1109/72.279181
  4. Chang, J., Wang, G., and Mao, T. (2015). "Simulation and prediction of suprapermafrost groundwater level variation in response to climate change using a neural network model." Journal of Hydrology, Vol. 529, pp. 1211-1220. https://doi.org/10.1016/j.jhydrol.2015.09.038
  5. Chollet, F. Allaire, J.J. (2018). Deep learning with R, Manning Publications, Shelter Island, NY, U.S., p. 360.
  6. Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2016). "Fast and accurate deep network learning by exponential linear units (ELUs)." arXiv preprint arXiv:1511.07289.
  7. Coulibaly, P., Anctil, F., Aravena, R., and Bobee, B. (2001). "Artificial neural network modeling of water table depth fluctuations." Water Resources Research, Vol. 37, No. 4, pp. 885-896. https://doi.org/10.1029/2000wr900368
  8. Emamgholizadeh, S., Moslemi, K., and Karami, G. (2014). "Prediction the groundwater level of bastam plain (Iran) by artificial neural network (ANN) and adaptive neuro-fuzzy inference system (ANFIS)." Water Resources Management, Vol. 28, No. 15, pp. 5433-5446. https://doi.org/10.1007/s11269-014-0810-0
  9. Falbel, D., Allaire, J.J., Chollet, F., Tang, Y., Van Der Bijl, W., Studer, M., Keydana, S. (2019). R interface to 'Keras'. R package version 2.2.4.1, accessed on 5 April 2019, .
  10. Hahnloser, R.H., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., and Seung, H.S. (2000). "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit." Nature, Vol. 405, No. 6789, pp. 947-951. https://doi.org/10.1038/35016072
  11. Haykin, S. (2009). Neural networks and learning machines, Pearson Prentice Hall, Upper Saddle River, NJ, U.S.
  12. Hochreiter, S., and Schmidhuber, J. (1997). "Long short-term memory." Neural Computation, Vol. 9, No. 8, pp. 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
  13. Hosseini, Z., Gharechelou, S., Nakhaei, M., and Gharechelou, S. (2016). "Optimal design of BP algorithm by ACOR model for groundwater-level forecasting: A case study on Shabestar plain, Iran." Arabian Journal of Geosciences, Vol. 9, No. 6, p. 436. https://doi.org/10.1007/s12517-016-2454-2
  14. Jeju Special Self-Governing Province (JSSGP) (2018). Comprehensive water resources management plan in Jeju Island. pp. 1-328.
  15. Jeong, J., and Park, E. (2019). "Comparative applications of data-driven models representing water table fluctuations." Journal of Hydrology, Vol. 572, pp. 261-273. https://doi.org/10.1016/j.jhydrol.2019.02.051
  16. Jha, M.K., and Sahoo, S. (2014). "Efficacy of neural network and genetic algorithm techniques in simulating spatio-temporal fluctuations of groundwater." Hydrological Processes, Vol. 29, No. 5, pp. 671-691. https://doi.org/10.1002/hyp.10166
  17. Kim, G.B., and Oh, D.H. (2018). "Determination of the groundwater yield of horizontal wells using an artificial neural network model incorporating riverside groundwater level data." The Journal of Engineering Geology, Vol. 28, No. 4, pp. 583-592. https://doi.org/10.9720/KSEG.2018.4.583
  18. Kim, J., Jun, S.M., Hwang, S., Kim, H.K., Heo, J., and Kang, M.S. (2021a). "Impact of activation functions on flood forecasting model based on artificial neural networks." Journal of The Korean Society of Agricultural Engineers, Vol. 63, No. 1, pp. 11-25. https://doi.org/10.5389/KSAE.2021.63.1.011
  19. Kim, M., Choi, J.Y., Bang, J., Yoon, P., and Kim, K. (2021b). "Comparison of artificial neural network model capability for runoff estimation about activation functions." Journal of The Korean Society of Agricultural Engineers, Vol. 63, No. 1, pp. 103-116. https://doi.org/10.5389/KSAE.2021.63.1.103
  20. Kim, T.W., and Valdes, J.B. (2003). "Nonlinear model for drought forecasting based on a conjunction of wavelet transforms and neural networks." Journal of Hydrologic Engineering, Vol. 8, No. 6, pp. 319-328. https://doi.org/10.1061/(ASCE)1084-0699(2003)8:6(319)
  21. Kingma, D.P., and Ba, J. (2014). "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980.
  22. Klemes, V. (1986). "Operational testing of hydrological simulation models." Hydrological Sciences Journal, Vol. 31, No. 1, pp. 13-24. https://doi.org/10.1080/02626668609491024
  23. Krishna, B., Satyaji Rao, Y.R., and Vijaya, T. (2008). "Modelling groundwater levels in an urban coastal aquifer using artificial neural networks." Hydrological Processes, Vol. 22, No. 8, pp. 1180-1188. https://doi.org/10.1002/hyp.6686
  24. Le, X.H., Ho, H.V., Lee, G., and Jung, S. (2019). "Application of long short-term memory (LSTM) neural network for flood forecasting." Water, Vol. 11, No. 7, p. 1387. https://doi.org/10.3390/w11071387
  25. Lee, S., Lee, K.K., and Yoon, H. (2019). "Using artificial neural network models for groundwater level forecasting and assessment of the relative impacts of influencing factors." Hydrogeology Journal, Vol. 27, No. 2, pp. 567-579. https://doi.org/10.1007/s10040-018-1866-3
  26. Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013). "Rectifier nonlinearities improve neural network acoustic models." Proceedings of the 30 th International Conference on Machine Learning, Atlanta, GA, U.S., Vol. 30, No. 1, p. 3.
  27. Maier, H.R., and Dandy, G.C. (2000). "Neural networks for the prediction and forecasting of water resources variables: A review of modelling issues and applications." Environmental Modelling & Software, Vol. 15, No. 1, pp. 101-124. https://doi.org/10.1016/S1364-8152(99)00007-9
  28. Maxwell, R.M., Condon, L.E., and Kollet, S.J. (2015). "A high-resolution simulation of groundwater and surface water over most of the continental US with the integrated hydrologic model ParFlow v3." Geoscientific Model Development, Vol. 8, No. 3, pp. 923-937. https://doi.org/10.5194/gmd-8-923-2015
  29. McDonald, M.G., and Harbaugh, A.W. (1988). A modular three-dimensional finite-difference ground-water flow model. Vol. 6. US Geological Survey, Reston, VA, U.S.
  30. Mohanty, S., Jha, M.K., Kumar, A., and Panda, D.K. (2013). "Comparative evaluation of numerical model and artificial neural network for simulating groundwater flow in Kathajodi - Surua Inter-basin of Odisha, India." Journal of Hydrology, Vol. 495, pp. 38-51. https://doi.org/10.1016/j.jhydrol.2013.04.041
  31. Moriasi, D.N., Arnold, J.G., Van Liew, M.W., Bingner, R.L., Harmel, R.D., and Veith, T.L. (2007). "Model evaluation guidelines for systematic quantification of accuracy in watershed simulations." Transactions of the ASABE, Vol. 50, No. 3, pp. 885-900. https://doi.org/10.13031/2013.23153
  32. Muller, J., Park, J., Sahu, R., Varadharajan, C., Arora, B., Faybishenko, B., and Agarwal, D. (2021). "Surrogate optimization of deep neural networks for groundwater predictions." Journal of Global Optimization, Vol. 81, No. 1, pp. 203-231. https://doi.org/10.1007/s10898-020-00912-0
  33. Nash, J.E., and Sutcliffe, J.V. (1970). "River flow forecasting through conceptual models part I - A discussion of principles." Journal of Hydrology, Vol. 10, No. 3, pp. 282-290. https://doi.org/10.1016/0022-1694(70)90255-6
  34. Nayak, P.C., Rao, Y.S., and Sudheer, K.P. (2006). "Groundwater level forecasting in a shallow aquifer using artificial neural network approach." Water Resources Management, Vol. 20, No. 1, pp. 77-90. https://doi.org/10.1007/s11269-006-4007-z
  35. Prechelt, L. (2012) "Early stopping - But when?." Neural networks: Tricks of the trade, Edited by Montavon G., Orr G.B., and Muller KR., Springer, Berlin Heidelberg, pp. 53-67.
  36. Rajaee, T., Ebrahimi, H., and Nourani, V. (2019). "A review of the artificial intelligence methods in groundwater level modeling." Journal of Hydrology, Vol. 572, pp. 336-351. https://doi.org/10.1016/j.jhydrol.2018.12.037
  37. Rakhshandehroo, G.R., Vaghefi, M., and Aghbolaghi, M.A. (2012). "Forecasting groundwater level in Shiraz plain using artificial neural networks." Arabian Journal for Science and Engineering, Vol. 37, No. 7, pp. 1871-1883. https://doi.org/10.1007/s13369-012-0291-5
  38. Sahoo, S., and Jha, M.K. (2013). "Groundwater-level prediction using multiple linear regression and artificial neural network techniques: a comparative assessment." Hydrogeology Journal, Vol. 21, No. 8, pp. 1865-1887. https://doi.org/10.1007/s10040-013-1029-5
  39. Sahoo, S., Russo, T.A., Elliott, J., and Foster, I. (2017). "Machine learning algorithms for modeling groundwater level changes in agricultural regions of the US." Water Resources Research, Vol. 53, No. 5, pp. 3878-3895. https://doi.org/10.1002/2016WR019933
  40. Shin, M.J., Moon, S.H., Kang, K.G., Moon, D.C., and Koh, H.J. (2020). "Analysis of groundwater level variations caused by the changes in groundwater withdrawals using long short-term memory network." Hydrology, Vol. 7, No. 3, p. 64. https://doi.org/10.3390/hydrology7030064
  41. Sit, M., Demiray, B.Z., Xiang, Z., Ewing, G.J., Sermet, Y., and Demir, I. (2020). "A comprehensive review of deep learning applications in hydrology and water resources." Water Science and Technology, Vol. 82, No. 12, pp. 2635-2670. https://doi.org/10.2166/wst.2020.369
  42. Sun, Y., Wendi, D., Kim, D.E., and Liong, S.Y. (2016). "Application of artificial neural networks in groundwater table forecasting-a case study in a Singapore swamp forest." Hydrology and Earth System Sciences, Vol. 20, No. 4. pp. 1405-1412. https://doi.org/10.5194/hess-20-1405-2016
  43. Taormina, R., Chau, K.W., and Sethi, R. (2012). "Artificial neural network simulation of hourly groundwater levels in a coastal aquifer system of the Venice lagoon." Engineering Applications of Artificial Intelligence, Vol. 25, No. 8, pp. 1670-1676. https://doi.org/10.1016/j.engappai.2012.02.009
  44. Todd, D.K. and Larry, W.M. (2004). Groundwater hydrology, Third edition. John Wiley & Sons Inc., Hoboken, NJ, U.S., pp. 1-656.
  45. Ukkonen, P., and Makela, A. (2019). "Evaluation of machine learning classifiers for predicting deep convection." Journal of Advances in Modeling Earth Systems, Vol. 11, No. 6, pp. 1784-1802. https://doi.org/10.1029/2018MS001561
  46. Vu, M.T., Jardani, A., Massei, N., and Fournier, M. (2021). "Reconstruction of missing groundwater level data by using Long Short-Term Memory (LSTM) deep neural network." Journal of Hydrology, Vol. 597, p. 125776. https://doi.org/10.1016/j.jhydrol.2020.125776
  47. Wen, X., Feng, Q., Deo, R.C., Wu, M., and Si, J. (2017). "Wavelet analysis - artificial neural network conjunction models for multi-scale monthly groundwater level predicting in an arid inland river basin, northwestern China." Hydrology Research, Vol. 48, No. 6, pp. 1710-1729. https://doi.org/10.2166/nh.2016.396
  48. White, J.T., Doherty, J.E., and Hughes, J.D. (2014). "Quantifying the predictive consequences of model error with linear subspace analysis." Water Resources Research, Vol. 50, No. 2, pp. 1152-1173. https://doi.org/10.1002/2013WR014767
  49. White, J.T., Knowling, M.J., and Moore, C.R. (2020). "Consequences of groundwater-Model vertical discretization in risk-Based decision-Making." Groundwater, Vol. 58, No. 5, pp. 695-709.
  50. Worland, S.C., Steinschneider, S., Asquith, W., Knight, R., and Wieczorek, M. (2019). "Prediction and inference of flow duration curves using multioutput neural networks." Water Resources Research, Vol. 55, No. 8, pp. 6850-6868. https://doi.org/10.1029/2018wr024463
  51. Xu, B., Wang, N., Chen, T., and Li, M. (2015). "Empirical evaluation of rectified activations in convolutional network." arXiv preprint arXiv:1505.00853.
  52. Yoon, H., Hyun, Y., Ha, K., Lee, K.K., and Kim, G.B. (2016). "A method to improve the stability and accuracy of ANN- and SVM-based time series models for long-term groundwater level predictions." Computers & Geosciences, Vol. 90, pp. 144-155.
  53. Yoon, H., Jun, S.C., Hyun, Y., Bae, G.O., and Lee, K.K. (2011). "A comparative study of artificial neural networks and support vector machines for predicting groundwater levels in a coastal aquifer." Journal of Hydrology, Vol. 396, No. 1-2, pp. 128-138. https://doi.org/10.1016/j.jhydrol.2010.11.002
  54. Yu, H., Wen, X., Feng, Q., Deo, R.C., Si, J., and Wu, M. (2018). "Comparative study of hybrid-wavelet artificial intelligence models for monthly groundwater depth forecasting in extreme arid regions, Northwest China." Water Resources Management, Vol. 32, No. 1, pp. 301-323. https://doi.org/10.1007/s11269-017-1811-6