Browse > Article
http://dx.doi.org/10.3741/JKWRA.2021.54.S-1.1143

Comparative analysis of activation functions of artificial neural network for prediction of optimal groundwater level in the middle mountainous area of Pyoseon watershed in Jeju Island  

Shin, Mun-Ju (Water Resources Research Team, Jeju Province Development Corporation)
Kim, Jin-Woo (Water Resources Research Team, Jeju Province Development Corporation)
Moon, Duk-Chul (Water Resources Research Team, Jeju Province Development Corporation)
Lee, Jeong-Han (Water Resources Research Team, Jeju Province Development Corporation)
Kang, Kyung Goo (R&D Innovation Center, Jeju Province Development Corporation)
Publication Information
Journal of Korea Water Resources Association / v.54, no.spc1, 2021 , pp. 1143-1154 More about this Journal
Abstract
The selection of activation function has a great influence on the groundwater level prediction performance of artificial neural network (ANN) model. In this study, five activation functions were applied to ANN model for two groundwater level observation wells in the middle mountainous area of the Pyoseon watershed in Jeju Island. The results of the prediction of the groundwater level were compared and analyzed, and the optimal activation function was derived. In addition, the results of LSTM model, which is a widely used recurrent neural network model, were compared and analyzed with the results of the ANN models with each activation function. As a result, ELU and Leaky ReLU functions were derived as the optimal activation functions for the prediction of the groundwater level for observation well with relatively large fluctuations in groundwater level and for observation well with relatively small fluctuations, respectively. On the other hand, sigmoid function had the lowest predictive performance among the five activation functions for training period, and produced inappropriate results in peak and lowest groundwater level prediction. The ANN-ELU and ANN-Leaky ReLU models showed groundwater level prediction performance comparable to that of the LSTM model, and thus had sufficient potential for application. The methods and results of this study can be usefully used in other studies.
Keywords
Activation function; ANN; LSTM; Groundwater level prediction; Pyoseon watershed in Jeju Island;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Yoon, H., Hyun, Y., Ha, K., Lee, K.K., and Kim, G.B. (2016). "A method to improve the stability and accuracy of ANN- and SVM-based time series models for long-term groundwater level predictions." Computers & Geosciences, Vol. 90, pp. 144-155.
2 Kim, G.B., and Oh, D.H. (2018). "Determination of the groundwater yield of horizontal wells using an artificial neural network model incorporating riverside groundwater level data." The Journal of Engineering Geology, Vol. 28, No. 4, pp. 583-592.   DOI
3 Yoon, H., Jun, S.C., Hyun, Y., Bae, G.O., and Lee, K.K. (2011). "A comparative study of artificial neural networks and support vector machines for predicting groundwater levels in a coastal aquifer." Journal of Hydrology, Vol. 396, No. 1-2, pp. 128-138.   DOI
4 Yu, H., Wen, X., Feng, Q., Deo, R.C., Si, J., and Wu, M. (2018). "Comparative study of hybrid-wavelet artificial intelligence models for monthly groundwater depth forecasting in extreme arid regions, Northwest China." Water Resources Management, Vol. 32, No. 1, pp. 301-323.   DOI
5 Kim, J., Jun, S.M., Hwang, S., Kim, H.K., Heo, J., and Kang, M.S. (2021a). "Impact of activation functions on flood forecasting model based on artificial neural networks." Journal of The Korean Society of Agricultural Engineers, Vol. 63, No. 1, pp. 11-25.   DOI
6 Kim, T.W., and Valdes, J.B. (2003). "Nonlinear model for drought forecasting based on a conjunction of wavelet transforms and neural networks." Journal of Hydrologic Engineering, Vol. 8, No. 6, pp. 319-328.   DOI
7 Kingma, D.P., and Ba, J. (2014). "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980.
8 Le, X.H., Ho, H.V., Lee, G., and Jung, S. (2019). "Application of long short-term memory (LSTM) neural network for flood forecasting." Water, Vol. 11, No. 7, p. 1387.   DOI
9 Maier, H.R., and Dandy, G.C. (2000). "Neural networks for the prediction and forecasting of water resources variables: A review of modelling issues and applications." Environmental Modelling & Software, Vol. 15, No. 1, pp. 101-124.   DOI
10 Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013). "Rectifier nonlinearities improve neural network acoustic models." Proceedings of the 30 th International Conference on Machine Learning, Atlanta, GA, U.S., Vol. 30, No. 1, p. 3.
11 McDonald, M.G., and Harbaugh, A.W. (1988). A modular three-dimensional finite-difference ground-water flow model. Vol. 6. US Geological Survey, Reston, VA, U.S.
12 Muller, J., Park, J., Sahu, R., Varadharajan, C., Arora, B., Faybishenko, B., and Agarwal, D. (2021). "Surrogate optimization of deep neural networks for groundwater predictions." Journal of Global Optimization, Vol. 81, No. 1, pp. 203-231.   DOI
13 Jeong, J., and Park, E. (2019). "Comparative applications of data-driven models representing water table fluctuations." Journal of Hydrology, Vol. 572, pp. 261-273.   DOI
14 Mohanty, S., Jha, M.K., Kumar, A., and Panda, D.K. (2013). "Comparative evaluation of numerical model and artificial neural network for simulating groundwater flow in Kathajodi - Surua Inter-basin of Odisha, India." Journal of Hydrology, Vol. 495, pp. 38-51.   DOI
15 Krishna, B., Satyaji Rao, Y.R., and Vijaya, T. (2008). "Modelling groundwater levels in an urban coastal aquifer using artificial neural networks." Hydrological Processes, Vol. 22, No. 8, pp. 1180-1188.   DOI
16 Shin, M.J., Moon, S.H., Kang, K.G., Moon, D.C., and Koh, H.J. (2020). "Analysis of groundwater level variations caused by the changes in groundwater withdrawals using long short-term memory network." Hydrology, Vol. 7, No. 3, p. 64.   DOI
17 Prechelt, L. (2012) "Early stopping - But when?." Neural networks: Tricks of the trade, Edited by Montavon G., Orr G.B., and Muller KR., Springer, Berlin Heidelberg, pp. 53-67.
18 Rajaee, T., Ebrahimi, H., and Nourani, V. (2019). "A review of the artificial intelligence methods in groundwater level modeling." Journal of Hydrology, Vol. 572, pp. 336-351.   DOI
19 Sahoo, S., and Jha, M.K. (2013). "Groundwater-level prediction using multiple linear regression and artificial neural network techniques: a comparative assessment." Hydrogeology Journal, Vol. 21, No. 8, pp. 1865-1887.   DOI
20 Nash, J.E., and Sutcliffe, J.V. (1970). "River flow forecasting through conceptual models part I - A discussion of principles." Journal of Hydrology, Vol. 10, No. 3, pp. 282-290.   DOI
21 Jha, M.K., and Sahoo, S. (2014). "Efficacy of neural network and genetic algorithm techniques in simulating spatio-temporal fluctuations of groundwater." Hydrological Processes, Vol. 29, No. 5, pp. 671-691.   DOI
22 Barthel, R., and Banzhaf, S. (2016). "Groundwater and surface water interaction at the regional-scale - a review with focus on regional integrated models." Water Resources Management, Vol. 30, No. 1, pp. 1-32.   DOI
23 Bengio, Y., Simard, P., and Frasconi, P. (1994). "Learning long-term dependencies with gradient descent is difficult." IEEE Transactions on Neural Networks, Vol. 5, No. 2, pp. 157-166.   DOI
24 Sahoo, S., Russo, T.A., Elliott, J., and Foster, I. (2017). "Machine learning algorithms for modeling groundwater level changes in agricultural regions of the US." Water Resources Research, Vol. 53, No. 5, pp. 3878-3895.   DOI
25 Chang, J., Wang, G., and Mao, T. (2015). "Simulation and prediction of suprapermafrost groundwater level variation in response to climate change using a neural network model." Journal of Hydrology, Vol. 529, pp. 1211-1220.   DOI
26 Coulibaly, P., Anctil, F., Aravena, R., and Bobee, B. (2001). "Artificial neural network modeling of water table depth fluctuations." Water Resources Research, Vol. 37, No. 4, pp. 885-896.   DOI
27 Hahnloser, R.H., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., and Seung, H.S. (2000). "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit." Nature, Vol. 405, No. 6789, pp. 947-951.   DOI
28 Hosseini, Z., Gharechelou, S., Nakhaei, M., and Gharechelou, S. (2016). "Optimal design of BP algorithm by ACOR model for groundwater-level forecasting: A case study on Shabestar plain, Iran." Arabian Journal of Geosciences, Vol. 9, No. 6, p. 436.   DOI
29 Klemes, V. (1986). "Operational testing of hydrological simulation models." Hydrological Sciences Journal, Vol. 31, No. 1, pp. 13-24.   DOI
30 Sit, M., Demiray, B.Z., Xiang, Z., Ewing, G.J., Sermet, Y., and Demir, I. (2020). "A comprehensive review of deep learning applications in hydrology and water resources." Water Science and Technology, Vol. 82, No. 12, pp. 2635-2670.   DOI
31 Sun, Y., Wendi, D., Kim, D.E., and Liong, S.Y. (2016). "Application of artificial neural networks in groundwater table forecasting-a case study in a Singapore swamp forest." Hydrology and Earth System Sciences, Vol. 20, No. 4. pp. 1405-1412.   DOI
32 Wen, X., Feng, Q., Deo, R.C., Wu, M., and Si, J. (2017). "Wavelet analysis - artificial neural network conjunction models for multi-scale monthly groundwater level predicting in an arid inland river basin, northwestern China." Hydrology Research, Vol. 48, No. 6, pp. 1710-1729.   DOI
33 Worland, S.C., Steinschneider, S., Asquith, W., Knight, R., and Wieczorek, M. (2019). "Prediction and inference of flow duration curves using multioutput neural networks." Water Resources Research, Vol. 55, No. 8, pp. 6850-6868.   DOI
34 Xu, B., Wang, N., Chen, T., and Li, M. (2015). "Empirical evaluation of rectified activations in convolutional network." arXiv preprint arXiv:1505.00853.
35 Haykin, S. (2009). Neural networks and learning machines, Pearson Prentice Hall, Upper Saddle River, NJ, U.S.
36 Ukkonen, P., and Makela, A. (2019). "Evaluation of machine learning classifiers for predicting deep convection." Journal of Advances in Modeling Earth Systems, Vol. 11, No. 6, pp. 1784-1802.   DOI
37 Emamgholizadeh, S., Moslemi, K., and Karami, G. (2014). "Prediction the groundwater level of bastam plain (Iran) by artificial neural network (ANN) and adaptive neuro-fuzzy inference system (ANFIS)." Water Resources Management, Vol. 28, No. 15, pp. 5433-5446.   DOI
38 Falbel, D., Allaire, J.J., Chollet, F., Tang, Y., Van Der Bijl, W., Studer, M., Keydana, S. (2019). R interface to 'Keras'. R package version 2.2.4.1, accessed on 5 April 2019, .
39 Hochreiter, S., and Schmidhuber, J. (1997). "Long short-term memory." Neural Computation, Vol. 9, No. 8, pp. 1735-1780.   DOI
40 Jeju Special Self-Governing Province (JSSGP) (2018). Comprehensive water resources management plan in Jeju Island. pp. 1-328.
41 Afzaal, H., Farooque, A.A., Abbas, F., Acharya, B., and Esau, T. (2020). "Groundwater estimation from major physical hydrology components using artificial neural networks and deep learning." Water, Vol. 12, No. 1, p. 5.   DOI
42 Rakhshandehroo, G.R., Vaghefi, M., and Aghbolaghi, M.A. (2012). "Forecasting groundwater level in Shiraz plain using artificial neural networks." Arabian Journal for Science and Engineering, Vol. 37, No. 7, pp. 1871-1883.   DOI
43 Chollet, F. Allaire, J.J. (2018). Deep learning with R, Manning Publications, Shelter Island, NY, U.S., p. 360.
44 Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2016). "Fast and accurate deep network learning by exponential linear units (ELUs)." arXiv preprint arXiv:1511.07289.
45 Lee, S., Lee, K.K., and Yoon, H. (2019). "Using artificial neural network models for groundwater level forecasting and assessment of the relative impacts of influencing factors." Hydrogeology Journal, Vol. 27, No. 2, pp. 567-579.   DOI
46 Maxwell, R.M., Condon, L.E., and Kollet, S.J. (2015). "A high-resolution simulation of groundwater and surface water over most of the continental US with the integrated hydrologic model ParFlow v3." Geoscientific Model Development, Vol. 8, No. 3, pp. 923-937.   DOI
47 Moriasi, D.N., Arnold, J.G., Van Liew, M.W., Bingner, R.L., Harmel, R.D., and Veith, T.L. (2007). "Model evaluation guidelines for systematic quantification of accuracy in watershed simulations." Transactions of the ASABE, Vol. 50, No. 3, pp. 885-900.   DOI
48 Nayak, P.C., Rao, Y.S., and Sudheer, K.P. (2006). "Groundwater level forecasting in a shallow aquifer using artificial neural network approach." Water Resources Management, Vol. 20, No. 1, pp. 77-90.   DOI
49 Kim, M., Choi, J.Y., Bang, J., Yoon, P., and Kim, K. (2021b). "Comparison of artificial neural network model capability for runoff estimation about activation functions." Journal of The Korean Society of Agricultural Engineers, Vol. 63, No. 1, pp. 103-116.   DOI
50 Taormina, R., Chau, K.W., and Sethi, R. (2012). "Artificial neural network simulation of hourly groundwater levels in a coastal aquifer system of the Venice lagoon." Engineering Applications of Artificial Intelligence, Vol. 25, No. 8, pp. 1670-1676.   DOI
51 Vu, M.T., Jardani, A., Massei, N., and Fournier, M. (2021). "Reconstruction of missing groundwater level data by using Long Short-Term Memory (LSTM) deep neural network." Journal of Hydrology, Vol. 597, p. 125776.   DOI
52 White, J.T., Knowling, M.J., and Moore, C.R. (2020). "Consequences of groundwater-Model vertical discretization in risk-Based decision-Making." Groundwater, Vol. 58, No. 5, pp. 695-709.
53 White, J.T., Doherty, J.E., and Hughes, J.D. (2014). "Quantifying the predictive consequences of model error with linear subspace analysis." Water Resources Research, Vol. 50, No. 2, pp. 1152-1173.   DOI
54 Todd, D.K. and Larry, W.M. (2004). Groundwater hydrology, Third edition. John Wiley & Sons Inc., Hoboken, NJ, U.S., pp. 1-656.