ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지

An Adaptive Learning Rate with Limited Error Signals for Training of Multilayer Perceptrons

  • Oh, Sang-Hoon (Department of Electrical Engineering, Brain Science Research Center, KAIST) ;
  • Lee, Soo-Young (Department of Electrical Engineering, Brain Science Research Center, KAIST)
  • Received : 1999.10.08
  • Published : 2000.09.30
Download PDF
⟨ Previous Next ⟩

Abstract

Although an n-th order cross-entropy (nCE) error function resolves the incorrect saturation problem of conventional error backpropagation (EBP) algorithm, performance of multilayer perceptrons (MLPs) trained using the nCE function depends heavily on the order of nCE. In this paper, we propose an adaptive learning rate to markedly reduce the sensitivity of MLP performance to the order of nCE. Additionally, we propose to limit error signal values at out-put nodes for stable learning with the adaptive learning rate. Through simulations of handwritten digit recognition and isolated-word recognition tasks, it was verified that the proposed method successfully reduced the performance dependency of MLPs on the nCE order while maintaining advantages of the nCE function.

Keywords

References

  1. IEEE Trans. Neural Networks v.4 A Programmable Analog VLSI Neural Network Processor for Communication Receivers Choi, J.;Bang, S.H.;Sheu, B.J.
  2. IEEE Trans. Communications v.40 Neural Networks for Multiuser Detection in Code-division Multiple-access Communications Aazhang, B.;Paris, B.P.;Orsak, G.C.
  3. IEEE Communications Magazine v.33 Neural Networks for Adaptive Traffic Control in ATM Networks Nordstrom, E.;Gallmo, O.;Asplund, L.
  4. IEEE Communications Magazine v.33 Intelligent Traffic Control for ATM Broadband Networks Habib, I.W.;Tarraf, A.A.;Saadawi, T.N.
  5. IEEE Communications Magazine no.Nov. Pattern Classification Using Neural Networks Lippmann, R.P.
  6. IEEE Trans. Neural Networks v.1 no.June A Novel Objective Function for Improved Phoneme Recognition Using Time-delay Neural Networks Hampshire, J.B. II;Waibel, A.H.
  7. Time Series Prediction: Forecasting the future and understanding the past Weigend, A.S.;Gershenfeld, N.A.
  8. IEEE Trans. Neural Networks v.1 Identification and Control of Dynamic System Using Neural Networks Narendra, K.S.;Parthasarathy, K.
  9. Neural Networks v.2 Multilayer Feedforward Networks are Universal Approximators Hornik, K.;Stincombe, M.;White, H.
  10. Neural Computation v.1 Learning in Artificial Neural Networks: a Statistical Perspective White, H.
  11. Neural Computation v.3 Neural Network Classifier Estimate Bayesian a Posteriori Probabilities Richard, M.D.;Lippmann, R.P.
  12. ETRI Journal v.16 no.April Input Noise Immunity of Multilayer Perceptrons Lee, Y.;Oh, S.H.
  13. Parallel Distributed Processing Rumelhart, D.E.;McClelland, J.L.
  14. Proc. IJCNN v.I Stepsize Variation Methods for Accelerating the Backpropagation Algorithm Chen, J.R.;Mars, P.
  15. Proc. IJCNN v.I The Effect of the Slope of the Activation Function on the Back Propagation Algorithm Rezgui, A.;Tepedelenlioglu, N.
  16. Neural Networks v.6 An Analysis of Premature Saturation in Back-propagation Learning Lee, Y.;Oh, S.H.;Kim, M.W.
  17. Neural Networks v.5 Improving the Convergence of the Back-propagation Algorithm van Ooyen, A.;Nienhuis, B.
  18. IEEE Trans. Neural Networks v.8 no.3 Improving the Error Backpropagation Algorithm with a Modified Error Function Oh, S.H.
  19. Biol. Cybern v.59 Accelerating the Convergence of the Back-propagation Method Vogl, T.P.;Mangis, J.K.;Rigler, A.K.;Zink, W.T.;Alkon, D.L.
  20. Complex Systems v.3 Accelerated Backpropagation Learning: Two Optimization Methods Battiti, R.
  21. Neural Networks for Pattern Recognition Bishop, C.M.
  22. IEEE Trans. Neural Networks v.10 A New Error Function at Hidden Layers for Fast Training of Multilayer Perecptrons Oh, S.H.;Lee, Soo-Young
  23. Neural Networks v.7 Energy Functions for Minimizing Misclassification Error with Minimum-complexity Networks Telfer, B.A.;Szu, H.H.
  24. IEEE Trans. Pat. Ana. Mach. Int. v.16 no.5 A Database for Handwritten Text Recognition Research Hull, J.J.
  25. IEEE Trans. Speech and Audio Processing v.7 Auditory Processing of Speech Signals for Robust Speech Recogtion in Real-world Noisy Environments Kim, D.S.;Lee, S.Y.;Kil, R.M.
  26. IEEE Trans. Neural Networks v.5 Effect of Nonlinear Transformations on Correlation between Weighted Sums in Multilayer Perceptrons Oh, S.H.;Lee, Y.