Browse > Article

Enhanced Backpropagation Algorithm by Auto-Tuning Method of Learning Rate using Fuzzy Control System  

김광백 (신라대학교 컴퓨터공학과)
박충식 (영동대학교 컴퓨터공학과)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.8, no.2, 2004 , pp. 464-470 More about this Journal
Abstract
We propose an enhanced backpropagation algorithm by auto-tuning of learning rate using fuzzy control system for performance improvement of backpropagation algorithm. We propose two methods, which improve local minima and loaming times problem. First, if absolute value of difference between target and actual output value is smaller than $\varepsilon$ or the same, we define it as correctness. And if bigger than $\varepsilon$, we define it as incorrectness. Second, instead of choosing a fixed learning rate, the proposed method is used to dynamically adjust learning rate using fuzzy control system. The inputs of fuzzy control system are number of correctness and incorrectness, and the output is the Loaming rate. For the evaluation of performance of the proposed method, we applied the XOR problem and numeral patterns classification The experimentation results showed that the proposed method has improved the performance compared to the conventional backpropagatiot the backpropagation with momentum, and the Jacob's delta-bar-delta method.
Keywords
역전파 알고리즘;퍼지 제어 시스템;학습률;모멘텀;delta-bar-delta 방식;
Citations & Related Records
연도 인용수 순위
  • Reference
1 Peiman G. Maghami and Dean W. Sparks, 'Design of Neural Networks for Fast Convergence and Accuracy: Dynamics and Control', IEEE Transactions on Neural Networks, Vol.11, No.11, pp.113-123, 2000   DOI   ScienceOn
2 M. Jamshidi, N. Vadiee and T. J. Ross, Fuzzy Logic and Control, Prentice-Hall, 1993
3 C. Charalambous, 'Conjugate gradient algorithm for efficient training of artificial neural networks,' IEEE Proceedings of Neural Networks, Vol.139, No.3, pp.301-310, 1992
4 M. Hagiwaea, 'Theoretical Derivation of Momentum Term in Backprogation,' Proceedings of IJCNN, Vol.I, pp.682-686, 1992
5 R. Hecht-Nielsen, 'Theory of backpropagation Neural Networks,' Proceedings of IJCNN, Vol.1, pp.593-605, 1989
6 Y. Hirose, K. Yamashita and S. Hijiya, 'Backpropagation Algorithm Which Varies the Number of Hidden Units', Neural Networks, Vol.4, pp.61-66, 1991   DOI   ScienceOn
7 R. A. Jacobs, 'Increased rates of convergence through learning rate adaptation,' IEEE Transactions on Neural Networks, Vol.1, No.4, pp.295-308, 1988   DOI   ScienceOn
8 J. W. Kim, K. K. Jung and K. H Eom, 'Auto-Tuning Method of Learning Rate for Performance Improvement of Back propagation Algorithm,' Journal of Korea Institute of Electronics Engineers, Vol.39, No.4, pp.19-27, 2002
9 Cheung, et al, 'Relative Effectiveness of Training Set Patterns for Back- propagation,' Proceedings of IJCNN, Vol.1, pp. 673-678, 1990
10 M. T. Hagan and M. Menhaj, 'Training Feedforward Networks with the Marquardt Algorithm,' IEEE Transaction on Neural Networks, Vol.5, No.6, 1994