Journal of KIISE:Software and Applications (한국정보과학회논문지:소프트웨어및응용)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지

Modified Error Back Propagation Algorithm using the Approximating of the Hidden Nodes in Multi-Layer Perceptron

다층퍼셉트론의 은닉노드 근사화를 이용한 개선된 오류역전파 학습

  • Kwak, Young-Tae (Dept. of Computer Engineering, Chungnam National University) ;
  • Lee, young-Gik (Electronics and Telecommunications Research Institute) ;
  • Kwon, Oh-Seok (Dept. of Computer Engineering, Chungnam National University)
  • Published : 2001.09.01
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a novel fast layer-by-layer algorithm that has better generalization capability. In the proposed algorithm, the weights of the hidden layer are updated by the target vector of the hidden layer obtained by least squares method. The proposed algorithm improves the learning speed that can occur due to the small magnitude of the gradient vector in the hidden layer. This algorithm was tested in a handwritten digits recognition problem. The learning speed of the proposed algorithm was faster than those of error back propagation algorithm and modified error function algorithm, and similar to those of Ooyen's method and layer-by-layer algorithm. Moreover, the simulation results showed that the proposed algorithm had the best generalization capability among them regardless of the number of hidden nodes. The proposed algorithm has the advantages of the learning speed of layer-by-layer algorithm and the generalization capability of error back propagation algorithm and modified error function algorithm.

본 논문은 학습 속도가 계층별 학습처럼 빠르며, 일반화 성능이 우수한 학습 방법을 제안한다. 제안한 방법은 최소 제곡법을 통해 구한 은닉층의 목표값을 이용하여 은닉층의 가중치를 조정하는 방법으로, 은닉층 경사 벡터의 크기가 작아 학습이 지연되는 것을 막을 수 있다. 필기체 숫자인식 문제를 대상으로 실험한 결과, 제안한 방법의 학습 속도는 오류역전파 학습과 수정된 오차 함수의 학습보다 빠르고, Ooyen의 방법과 계층별 학습과는 비슷했다. 또한, 일반화 성능은 은닉노드의 수에 관련없이 가장 좋은 결과를 얻었다. 결국, 제안한 방법은 계층별 학습의 학습 속도와 오류역전파 학습과 수정된 오차 함수의 일반화 성능을 장점으로 가지고 있음을 확인하였다.

Keywords

References

  1. D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing, MIT Press, Cambridge, MA, pp.318-362, 1986
  2. R. P. Lippmann, 'An Introduction to Computing with Neural Nets,' IEEE ASSP Magazine, pp. 4-22, April, 1987
  3. Dan Hammerstrom, 'Working with Neural Networks,' IEEE Spectrum, pp.46-53, July, 1993 https://doi.org/10.1109/6.222230
  4. A. Van Ooyen and B. Nienhuis, 'Improving the convergence of the back-propagation algorithm,' Neural Networks, vol. 78, pp.465-471, 1992 https://doi.org/10.1016/0893-6080(92)90008-7
  5. J. R. Chen and P. Mars, 'Stepsize variation methods for accelerating the backpropation algorithm,' Proc. IJCNN Jan. 15-19, 1990, Washington, DC, USA, vol. I, pp. 601-604
  6. Sang-Hoon Oh and Youngjik Lee, 'A modified error function to improve the Error Back-Propagation algorithm of Multi-Layer Perceptrons,' ETRI Journal, vol. 17, pp.11-22, April, 1995
  7. C. M. Bishop, Neural Networks for Pattern Recognition, Clarendon Press, Oxford, 1997
  8. G.-J. Wang and C.-C. Chen, 'A Fast Multilayer Neural-Network Training Algorithm Based on the Layer-By-Layer Optimizing Procedures,' IEEE Trans. Neural Networks, vol. 7, pp.768-775, May, 1996 https://doi.org/10.1109/72.501734
  9. Jim. Y. F. Yam and Tommy W. S. Chow, 'Extended Least Squares Based Algorithm for Training Feedforward Networks,' IEEE Trans. Neural Networks, vol. 8, pp.806-810, May, 1997 https://doi.org/10.1109/72.572119
  10. R. S. Scalero and N. Tepedelenlioglu, 'A Fast New Algorithm for Training Feedforward Neural Networks,' IEEE Trans. Signal Processing, vol. 40, pp.202-210, Jan. 1992 https://doi.org/10.1109/78.157194
  11. S. Ergezinger and E. Thomsen, 'An Accelerated Learning Algorithm for Multilayer Perceptrons: Optimization Layer by Layer,' IEEE Trans. Neural Networks, vol. 6, pp.31-42, Jan. 1995 https://doi.org/10.1109/72.363452
  12. David J. Winter, Matrix Algebra, Macmillan Publishing Company, 1992
  13. James M. Ortega, Matrix Theory, Plenum Press, 1987
  14. J. J. Hull, A database for handwritten text recognition research, IEEE Trans. Pattern and Machine Intell., vol. 16, pp.550-554, 1994 https://doi.org/10.1109/34.291440
  15. 오상훈, 이영직, 김명원, '역전파 학습시 초기 가중치가 학습의 조기 포화에 미치는 영향', 전자공학회 논문지, 제28권 4호, pp.90-97, 1991