ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지

A Modified Error Function to Improve the Error Back-Propagation Algorithm for Multi-Layer Perceptrons

  • Published : 1995.04.20
Download PDF
⟨ Previous Next ⟩

Abstract

This paper proposes a modified error function to improve the error back-propagation (EBP) algorithm for multi-Layer perceptrons (MLPs) which suffers from slow learning speed. It can also suppress over-specialization for training patterns that occurs in an algorithm based on a cross-entropy cost function which markedly reduces learning time. In the similar way as the cross-entropy function, our new function accelerates the learning speed of the EBP algorithm by allowing the output node of the MLP to generate a strong error signal when the output node is far from the desired value. Moreover, it prevents the overspecialization of learning for training patterns by letting the output node, whose value is close to the desired value, generate a weak error signal. In a simulation study to classify handwritten digits in the CEDAR [1] database, the proposed method attained 100% correct classification for the training patterns after only 50 sweeps of learning, while the original EBP attained only 98.8% after 500 sweeps. Also, our method shows mean-squared error of 0.627 for the test patterns, which is superior to the error 0.667 in the cross-entropy method. These results demonstrate that our new method excels others in learning speed as well as in generalization.

Keywords

References

  1. IEEE Trans. Pat. Ana. March. Int. A database for handwritten text recognition research Hull, J.J.
  2. Parallel Distributed Processing Rumelhart, D.E.;McClelland, J.L.
  3. Proc. IJCNN v.I Relative effectiveness of training set patterns for back propagation Cheung, R.K.;Lustig, I.;Kornhauser, A.L.
  4. Neural Networks v.6 An analysis of premature saturation in backpropagation learning Lee, Y.;Oh, S.H.;Kim, M.W.
  5. Proc. IJCNN v.I Stepsize variation methods for accelerating the backpropagation algorithm Chen, J.R.;Mars, P.
  6. Proc. IJCNN v.I The effect of the slope of the activation function on the back propagation algorithm Rezgui, A.;Tepedelenlioglu, N.
  7. Proc. IJCNN v.III Classification of large set of handwritten characters using modified back propagation model Krzyzak, A.;Dai, W.;Suen, C.Y.
  8. Neural Networks v.5 Improving the convergence of the back-propagation algorithm van Ooyen, A.;Nienhuis, B.
  9. Neural Computation v.3 Neural network classifier estimate Bayesian a posteriori probabilities Richard, M.D.;Lippmann, R.P.
  10. Neural Networks v.7 Energy functions for minimizing misclassification error with minimum-complexity networks Telfer, B.A.;Szu, H.H.
  11. Proc. WCNN v.III A double-well potentional energy function resolving bistable ambiguity figures by neural network Szu, H.H.;Lu, F.
  12. Proc. ICONIP v.I Maximum entropy principle: Improving generalization performance by maximizing the number of internal representations Kamimura, R.;Nakanishi, S.
  13. IEEE Trans. Neural Networks v.5 Weight smoothing to improve network generalization Jean, J.S.N.;Wang, J.
  14. Proc. ICONIP v.I Hybrid Hebbian/backpropagation learning rule for improved generalization of multilayer feed-forward neural networks Lee, S.Y.;Jeong, D.G.
  15. Proc. ICONIP v.II Improving the error back-propagation algorithm Lee, Y.;Oh, S.H.