Journal of KIISE:Software and Applications (한국정보과학회논문지:소프트웨어및응용)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지

A Separate Learning Algorithm of Two-Layered Networks with Target Values of Hidden Nodes

은닉노드 목표 값을 가진 2개 층 신경망의 분리학습 알고리즘

  • 최범기 (인하대학교 컴퓨터정보공학과) ;
  • 이주홍 (인하대학교 컴퓨터정보공학과) ;
  • 박태수 (인하대학교 컴퓨터정보공학과)
  • Published : 2006.12.15
Download PDF
⟨ Previous Next ⟩

Abstract

The Backpropagation learning algorithm is known to have slow and false convergence aroused from plateau and local minima. Many substitutes for backpropagation announced so far appear to pay some trade-off for convergence speed and stability of convergence according to parameters. Here, a new algorithm is proposed, which avoids some of those problems associated with the conventional backpropagation problems, especially with local minima, and gives relatively stable and fast convergence with low storage requirement. This is the separate learning algorithm in which the upper connections, hidden-to-output, and the lower connections, input-to-hidden, separately trained. This algorithm requires less computational work than the conventional backpropagation and other improved algorithms. It is shown in various classification problems to be relatively reliable on the overall performance.

역전파 학습 방법은 속도가 느리고, 지역 최소점이나 고원에 빠져 수렴에 실패하는 경우가 많다고 알려져 있다. 이제까지 알려진 역전파의 대체 방법들은 수렴 속도와 변수에 따른 수렴의 안정성 사이에서 불균형이라는 대가를 치루고 있다. 기존의 전통적인 역전파에서 발생하는 위와 같은 문제점 중, 특히 지역 최소점을 탈피하는 기능을 추가하여 적은 저장 공간으로 안정성이 보장되면서도 빠른 수렴속도를 유지하는 알고리즘을 제안한다. 이 방법은 전체 신경망을 은닉층-출력층(hidden to output)을 의미하는 상위 연결(upper connections)과 입력층-은닉층(input to hidden)을 의미하는 하위 연결(lower connections) 2개로 분리하여 번갈아 훈련을 시키는 분리 학습방법을 적용한다. 본 논문에서 제안하는 알고리즘은 다양한 classification 문제에 적용한 실험 결과에서 보듯이 전통적인 역전파 및 기타 개선된 알고리즘에 비해 계산량이 적고, 성능이 매우 좋으며 높은 신뢰성을 보장한다.

Keywords

References

  1. McCulloch, W.S., Pitts, W., 'A logical Calculus of Ideas Immanent in Nervous Activity,' Bulletin of Mathematical Biophsics 5, 115-133, 1962 https://doi.org/10.1007/BF02478259
  2. Rosenblatt, F., 'Principle of Neurodynarnics,' New York: Spartan, 1962
  3. Minsky, M.L and Papert, S.A., 'Perceptrons,' Cambridge: MIT Press, 1969
  4. Rumelhart, D.E., G.E. Hinton, and Williams, R.J., 'Learning Internal Representations by Error propagation,' In Parallel Distributed Processing, vol. 1, chap8, 1986
  5. Parker, D.B., 'Learning Logic,' Technical Report TR-47, Center for Computational Research in Economics and Management Science, Massachusetts Institute of Technology, Cambridge, MA, 1985
  6. Kolen, J. F. and Pollack, J. B., 'Back Propagation is Sensitive to Initial Conditions,' Complex System 4, 269-280, 1990
  7. Jacobs, R. A., 'Increased Rates of Convergence Through Learning Rate Adaptation,' Neural Networks 1, 293-280, 1988 https://doi.org/10.1016/0893-6080(88)90003-2
  8. Vogl, T. P., J.X. Magis, A.K. Rigler, W.T. Zink, and D.L. Alkon., 'Accelerating the Convergence of the Back-Propagation Method,' Biological Cybernetics 59, 257-263, 1988 https://doi.org/10.1007/BF00332914
  9. Allred, L. G., Kelly, G. E., 'Supervised learning techniques for backpropagation networks,' In Proc. of IJCNN, vol. 1, 702-709, 1990 https://doi.org/10.1109/IJCNN.1990.137654
  10. Fahlman, S. E., 'Fast learning variations on backpropagation: An empirical study,' in Proc. Connectionist Models Summer School, 1989
  11. Riedmiller, M. and Braun, H., 'A direct adaptive method for faster backpropagation learning: The RPROP algorithm,' in Pro. Int. Conf, Neural Networks, vol. 1, 586-591, 1993 https://doi.org/10.1109/ICNN.1993.298623
  12. Montana D. J., Davis L., 'Training feedforward neural networks using genetic algorithms,' in Proc. Int. Joint Conf. Artificial Intelligence, Detroit, 762-767, 1989
  13. Nicholas K. Treadgold and Tamas D. Gedeon., 'Simulaed Annealing and Weight Decay in Adaptive Learning: The SARPRO Algorithm,' IEEE Trans. On Neural Networks, vol. 9, pp. 662-668, 1998 https://doi.org/10.1109/72.701179
  14. S. C. Ng and S. H. Leung, 'On Solving the Local Minima Probem of Adaptive Learning by Detrministic Weight Evolutionary Algorithm,' Proc. of Comgress in Evolutionary Computation(CEC2001), Seoul, Korea, May 27-20, 2001, vol. 1, 251-255, 2001 https://doi.org/10.1109/CEC.2001.934397
  15. Watrous, R. L., 'Learning algorithms for connectionist network: applied gradient methods
  16. Touretzky, D. S., 'San Mateo,' Morgan Kaufmann, 1989
  17. Grossman, T., 'The CHAIR Algorithm for Feed Forward Networks with Binary Weights,' In Advances Neural Information Processing Systems II, 1989
  18. Krogh, A., G.I. Thorbergerson, and J.A. Hertz., 'A Cost Function for Internal Representations,' In Advances in Neural Information Processing Systems II, 1989
  19. Saad, D. and E. Marom., 'Learning by Choice of Internal Representations-An Energy Minimization Approach,' Complex Systems 4, 107-118, 1990
  20. Saad, D. and E. Marom., 'Training Feedforward Nets with Binary Weighted via a Modified CHIR Algorithm,' ComplexSystems 4, 573-586, 1990 of nonlinear optimization,' in Proc. 1st. Int . Conf, Neural Networks, vol. II, 619-628, 1987