Journal of the Institute of Electronics Engineers of Korea CI (전자공학회논문지CI)

The Institute of Electronics and Information Engineers (대한전자공학회)
권호 표지

A Pruning Algorithm of Neural Networks Using Impact Factors

임팩트 팩터를 이용한 신경 회로망의 연결 소거 알고리즘

  • 이하준 (한국과학기술원 전자전산학과) ;
  • 정승범 (삼성전자(주) 기술총괄 소프트웨어센) ;
  • 박철훈 (한국과학기술원 전자전산학과)
  • Published : 2004.03.01
Download PDF
⟨ Previous Next ⟩

Abstract

In general, small-sized neural networks, even though they show good generalization performance, tend to fail to team the training data within a given error bound, whereas large-sized ones learn the training data easily but yield poor generalization. Therefore, a way of achieving good generalization is to find the smallest network that can learn the data, called the optimal-sized neural network. This paper proposes a new scheme for network pruning with ‘impact factor’ which is defined as a multiplication of the variance of a neuron output and the square of its outgoing weight. Simulation results of function approximation problems show that the proposed method is effective in regression.

일반적으로 작은 구조의 신경 회로망은 좋은 일반화 성능을 나타내지만 원하는 학습 목표까지 학습하기가 어려운 경향이 있다. 반면에 큰 구조의 신경 회로망은 학습 데이터는 쉽게 배우지만 일반화 성능이 좋지 않은 경향이 있다. 따라서 좋은 일반화 성능을 얻기 위한 일반적인 방법은 학습이 되는 한도 내에서 최소 구조의 신경 회로망 즉 최적 구조 신경 회로망을 찾는 것이다. 본 논문에서는 가중치의 제곱과 뉴런 출력의 분산의 곱으로 정의되는 임팩트 팩터(ImF: Impact Factor)를 이용한 새로운 연결 소거 알고리즘을 제안한다. 그리고 함수 근사화 문제에 적용하여 제안된 방법이 효율적임을 보인다.

Keywords

References

  1. S. Haykin, Neural Networks: A Comprehensive Foundation, NJ Prentice Hall, 1999
  2. R. Read, 'Pruning algorithm - a survey,' IEEE Transactions on Neural Networks, vol. 4, no. 5, pp. 740-747, Sep, 1993 https://doi.org/10.1109/72.248452
  3. T. Y. Kwok, and D. Y. Yeung, 'Constructive algorithm for structure learning in feedfor ward neural networks for regression problems,' IEEE Transactions on Neural Net works, vol. 8, no. 3, pp. 630-645, May 1997 https://doi.org/10.1109/72.572102
  4. A. P. Engelbrecht, 'A new pruning heuristic based on variance analysis of sensitivity in formation,' IEEE Transactions on Neural Networks, vol. 12, no. 6, pp. 1386-1399, Nov. 2001 https://doi.org/10.1109/72.963775
  5. H. Lee, and C. H. Park, 'A pruning algorithm of neural networks using impact factor regularization,' Proceedings of the 9th Inter national Conference of Neural Information Processing, vol. 5, pp. 2605-2609, Singapore, Nov. 2002 https://doi.org/10.1109/ICONIP.2002.1201967
  6. J. -S. Lee, and C. H. Park, 'Self-organizing neural networks using adaptive neurons,' Proceedings of the 9th International Conference of Neural Information Processing, vol. 2, pp. 935-939, Singapore, Nov. 2002 https://doi.org/10.1109/ICONIP.2002.1198198
  7. C. M. Bishop, Neural Networks for Pattern Recognition, Oxford, U.K: Oxford Univ, Press, 1995
  8. Y. LeCun, J. S. Denker, and S. A. Solla, 'Optimal brain damage,' Advances in Neural Information Processing Systems, vol. 2, pp. 598-605, San Mateo, CA:Morgan Kaufmann, 1990
  9. B. Hassabi, and D. G. Stork, 'Second order derivative for network pruning: optimal brain surgeon,' Advances in Neural Information Processing Systems, vol. 5, pp. 164-171, 1993
  10. L. Breiman, 'The pi method for estimating multivariate functions from noisy data,' Technometrics, vol. 3, no. 2, pp. 125-160, 1991 https://doi.org/10.2307/1269038
  11. M. Maechler, D. Martin, J. Schimert, M. Csoppensky, and J. N. Hwang, 'Project pursuit learning networks for regression,' Proceedings of International Conference on Too Is for AI, Washington D.C., pp, 350-358, Nov. 1990 https://doi.org/10.1109/TAI.1990.130362
  12. V. Cherkassky, and H. Lari-Naiafi, 'Cons trained topological mapping for nonparametric regression analysis,' Neural Networks, vol. 4, pp, 27-40, 1991 https://doi.org/10.1016/0893-6080(91)90028-4
  13. M. T. Hagan, and M. B. Menhaj, 'Training feedforward networks with the Marquardt algorithm,' IEEE Transactions on Neural Networks, vol. 5, no. 6, pp. 989-993, Nov. 1994 https://doi.org/10.1109/72.329697