Browse > Article
http://dx.doi.org/10.5302/J.ICROS.2004.10.12.1295

A Study on the Bayesian Recurrent Neural Network for Time Series Prediction  

Hong Chan-Young (삼성전자)
Park Jung-Hoon (연세대학교 전기전자공학과)
Yoon Tae-Sung (창원대학교 전기공학과)
Park Jin-Bae (연세대학교 전기전자공학과)
Publication Information
Journal of Institute of Control, Robotics and Systems / v.10, no.12, 2004 , pp. 1295-1304 More about this Journal
Abstract
In this paper, the Bayesian recurrent neural network is proposed to predict time series data. A neural network predictor requests proper learning strategy to adjust the network weights, and one needs to prepare for non-linear and non-stationary evolution of network weights. The Bayesian neural network in this paper estimates not the single set of weights but the probability distributions of weights. In other words, the weights vector is set as a state vector of state space method, and its probability distributions are estimated in accordance with the particle filtering process. This approach makes it possible to obtain more exact estimation of the weights. In the aspect of network architecture, it is known that the recurrent feedback structure is superior to the feedforward structure for the problem of time series prediction. Therefore, the recurrent neural network with Bayesian inference, what we call Bayesian recurrent neural network (BRNN), is expected to show higher performance than the normal neural network. To verify the proposed method, the time series data are numerically generated and various kinds of neural network predictor are applied on it in order to be compared. As a result, feedback structure and Bayesian learning are better than feedforward structure and backpropagation learning, respectively. Consequently, it is verified that the Bayesian reccurent neural network shows better a prediction result than the common Bayesian neural network.
Keywords
time series prediction; Bayesian inference; recurrent neural network; Bayesian neural network; Bayesian recurrent neural network;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 김해경, 김태수, 시계열 분석과 예측 이론, 경문사, 2003
2 이덕기, 예측 방법의 이해, 고려정보산업, 1999
3 C. Chatfield, Time-series forecasting, Chapman & Hall/ CRC, 2000
4 P. J. Brockwell and R. A. Davis, Introduction to time series and forecasting, Springer, New York, 2002
5 A. Vehtari and J. Lampinen, 'Bayesian neural networks for industrial applications', Proceedings of the 1999 IEEE Midnight-Sun Workshop on Soft Computing Methods in Industrial Applications, pp. 63-68, 1999   DOI
6 M. Crucianu, R. Bone and J.-P. A. De Beauville, 'Bayesian learning for time series prediction with exogenous variables', International Joint Conference on Neural Networks, vol. 4, pp. 2594-2599, 1999   DOI
7 D. J. C. Mackay, 'A practical bayesian framework for backpropagation networks', Neural Computation, vol. 4, no. 3, pp. 448-472, 1992   DOI
8 J. F. D. Freitas, M. Niranjan and A. H. Gee, 'Hybrid sequential Monte carlo/kalman methods to train neural networks in non-stationary environments', IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, pp. 1057-1060, 1999   DOI
9 T. Zhang and A. Fukushige, 'Forecasting time series by bayesian neural networks', Proceedings of the 2002 International Joint Conference on Neural Networks, vol. 1, pp. 382-387, 2002   DOI
10 E. H. Tito, G. Zaverucha, M. Vellasco and M. Pacheco, 'Bayesian neural networks for electric load forecasting', 6th International Conference on Neural Information Processing, vol. 1, pp. 407 - 411, 1999   DOI
11 R. M. Neal, Bayesian learning for neural network, Lecture Notes in Statistics No. 118, Springer-Verlag, New York, 1996
12 N. Bergman, Recursive Bayesian estimation : navigation and tracking applications, Linkoping University, 1999
13 V. Kadirkamanathan and M. Niranjan, 'A function estimation approach to sequential learning with neural networks', Neural Computation, vol. 5, pp. 954-975, 1993   DOI   ScienceOn
14 J. Durbin and S. J. Koopman, Time Series Analysis by State Space Methods, Oxford University Press, 2001
15 N. J. Gordon, D. J. Salmond and A. F. M. Smith, 'Novel approach to nonlinear/non-Gaussian bayesian state estimation', Radar and Signal Processing, vol. 140, Issue 2, pp. 107-113, 1993   DOI
16 M. Crucianu, 'Bayesian learning for recurrent neural networks', Neurocomputing, 36, pp. 235-242, 2001   DOI   ScienceOn
17 T. L. Song, 'Filtering theory', Journal of Control, Automation, and Systems Engineering, vol. 9, no. 6, pp. 413-419, 2003   DOI   ScienceOn
18 D. Pena, G. C. Tiao and R. S. Tsay, A course in time series analysis, John Wiley & Sons, 2001
19 J. Zhang, K. S. Tang and K. F. Man, 'Recurrent NN model for chaotic time series prediction', 23rd International Conference on Industrial Electronics, Control and Instrumentation, vol. 3, pp. 1108-1112, 1997   DOI
20 C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995
21 G. C. Franco and R. C. Souza, 'A comparison of methods for bootstrapping in the local level model', Journal of Forecasting, 21, pp. 27-38, 2002   DOI   ScienceOn
22 G. Kitagawa, 'Non-gaussian state-space modeling of nonstationary time series', Journal of the American Statistical Association, vol. 82, no. 400, pp. 1032-1036, 1987   DOI
23 O. A. Alsayegh, 'Annual energy consumption prediction using particle filters', Seventh International Symposium on Signal Processing and Its Applications, vol. 2, pp. 571-574, 2003   DOI
24 A. C. Tsakoumis, S. S. Vladov and V. M. Mladenov, 'Electric load forecasting with multilayer perceptron and elman neural network', 2002 6th Seminar on Neural Network Applications in Electrical Engineering, pp. 87-90, 2002
25 R. G. Brown and P. Y. C. Hwang, Introduction to random signals and applied Kalman filter, John Wiley & Sons, 1997