Journal of KIISE:Computing Practices and Letters (한국정보과학회논문지:컴퓨팅의 실제 및 레터)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지

A Fast-Loaming Algorithm for MLP in Pattern Recognition

패턴인식의 MLP 고속학습 알고리즘

  • 이태승 (한국항공대학교 항공전자공학과) ;
  • 최호진 (한국정보통산대학교 공학부)
  • Published : 2002.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

Having a variety of good characteristics against other pattern recognition techniques, Multilayer Perceptron (MLP) has been used in wide applications. But, it is known that Error Backpropagation (EBP) algorithm which MLP uses in learning has a defect that requires relatively long leaning time. Because learning data in pattern recognition contain abundant redundancies, in order to increase learning speed it is very effective to use online-based teaming methods, which update parameters of MLP pattern by pattern. Typical online EBP algorithm applies fixed learning rate for each update of parameters. Though a large amount of speedup with online EBP can be obtained by choosing an appropriate fixed rate, fixing the rate leads to the problem that the algorithm cannot respond effectively to different leaning phases as the phases change and the learning pattern areas vary. To solve this problem, this paper defines learning as three phases and proposes a Instant Learning by Varying Rate and Skipping (ILVRS) method to reflect only necessary patterns when learning phases change. The basic concept of ILVRS is as follows. To discriminate and use necessary patterns which change as learning proceeds, (1) ILVRS uses a variable learning rate which is an error calculated from each pattern and is suppressed within a proper range, and (2) ILVRS bypasses unnecessary patterns in loaming phases. In this paper, an experimentation is conducted for speaker verification as an application of pattern recognition, and the results are presented to verify the performance of ILVRS.

MLP(multilayer perceptron)는 다른 패턴인식 방법에 비해 여러 가지 훌륭한 특성을 가지고 있어 패턴인식에서 폭넓게 사용되고 있다. 그러나 MLP의 학습에 일반적으로 사용되는 EBP(error backpropagation) 알고리즘은 학습시간이 비교적 오래 걸린다는 단점이 있다. 패턴인식에 사용되는 학습 데이타는 풍부한 중복특성을 내포하고 있으므로 패턴마다 MLP의 내부변수를 갱신하는 온라인 계열의 학습방식이 속도의 향상에 상당한 효과가 있다. 일반적인 온라인 EBP 알고리즘에서는 내부변수 갱신시 고정된 학습률을 적용한다. 고정 학습률을 적절히 선택함으로써 패턴인식 웅용에서 상당한 속도개선을 얻을 수 있지만, 학습률이 고정되고 학습이 진행됨에 따라 학습패턴 영역이 달라지는 학습과정의 각 단계에 효과적으로 대웅하지 못하는 문제가 있다. 이 문제에 대해 본 논문에서는 학습과정을 세 단계로 정의하고, 각 단계별로 필요한 패턴만을 학습에 반영하는 패턴별 가변학습속도 및 학습생략(ILVRS) 방법을 제안한다. ILVRS의 기본개념은 다음과 같다. 학습단계마다 학습에 필요한 패턴의 부분이 달라지므로 이를 구별 하여 학습에 적용할 수 있도록 (1)패턴마다 발생하는 오류치를 적절한 범위 이내로 제한하여 가변 학습률로 사용하고, (2)학습이 진행됨에 따라 불필요한 부분의 패턴을 학습에서 생략한다. 제안한 ILVRS의 성능을 입증하기 위해 본 논문에서는 패턴인식 응용의 한 갈래인 화자증명을 실험하고 그 결과를 제시한다.

Keywords

References

  1. R. O. Duba et. al., Pattern Classification, Wiley-Interscience, 2001
  2. H. Bourlard and N Morgan, 'Hybrid Connectionist Models for Continuous Speech Recorgnition,' Automatic Speech and Speaker Recognition, pp. 259-283, Kluwer Academic Publishers, 1996
  3. S. Haykin, Neural Networks, Prentice Hall, 1999
  4. H. Demuth and M. Beale, Neural Network Toolbox, The MathWorks, 2001
  5. M. Riedmiller and H. Braun, 'A Direct Adaptive Method for Faster Backpropagation Learning: The PROP Algorithm,' IEEE International Conference on Neural Networks, pp. 586-591, Vol. 1, Mar 1993
  6. R. Fletcher, Practical Methods of Optimization, Wiley, 1987
  7. M. Moller, 'Supervised Learning on Large Redundant Training Sets,' Proceedings of the 1992 IEEE-SP Workshop Neural Networks for Signal Processing, pp. 79-89, Aug 1992
  8. S. Becker and Y. LeCun, 'Improving the Convergence of Back-Propagation Learning with Second-Order Methods,' in Proceedings of the 1988 Connectionist Models Summer School, pp. 29-37, 1988
  9. Y. Bengio, Neural Networks for Speech and Sequence Recognition, International Thomson Computer Press, 1995
  10. Y. LeCun et. al., 'Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors,' Advances in Neural Information Processing Systems, Vol. 5, pp. 156-163, 1993
  11. D. R. Wilson and T. R. Martinez, 'The Need for Small Learning Rates on Large Problems,' International Joint Conference on Neural Networks, Vol. 1, pp. 115-119, 2001 https://doi.org/10.1109/IJCNN.2001.939002
  12. J. M. Naik, 'Speaker Verification: A Tutorial,' IEEE Communications Magazine, Vol. 1, pp. 42-48, Jan 1990 https://doi.org/10.1109/35.46670
  13. M. Savic and J. Sorensen, 'Phoneme Based Speaker Verification,' 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 2, pp. 165-168, 1992
  14. C. Becchetti and L. P. Ricotti, Speech Recognition, John Wiley & Sons, 1999
  15. P. Cristea and Z. Valsan, 'New Cepstrum Frequency Scale for Neural Network Speaker Verification,' IEEE International Conference on Electronics, Circuits and Systems, Vol. 3, pp. 1573-1576, 1999 https://doi.org/10.1109/ICECS.1999.814472