The Transactions of the Korean Institute of Electrical Engineers D (대한전기학회논문지:시스템및제어부문D)

The Korean Institute of Electrical Engineers (대한전기학회)
권호 표지

Performance Improvement of Ensemble Speciated Neural Networks using Kullback-Leibler Entropy

Kullback-Leibler 엔트로피를 이용한 종분화 신경망 결합의 성능향상

  • 김경중 (연세대학교 컴퓨터과학과) ;
  • 조성배 (연세대학교 컴퓨터과학과)
  • Published : 2002.04.01
Download PDF
⟨ Previous Next ⟩

Abstract

Fitness sharing that shares fitness if calculated distance between individuals is smaller than sharing radius is one of the representative speciation methods and can complement evolutionary algorithm which converges one solution. Recently, there are many researches on designing neural network architecture using evolutionary algorithm but most of them use only the fittest solution in the last generation. In this paper, we elaborate generating diverse neural networks using fitness sharing and combing them to compute outputs then, propose calculating distance between individuals using modified Kullback-Leibler entropy for improvement of fitness sharing performance. In the experiment of Australian credit card assessment, breast cancer, and diabetes in UCI database, proposed method performs better than not only simple average output or Pearson Correlation but also previous published methods.

Keywords

References

  1. D. Montana and L. Davis, 'Training feedforward neural networks using genetic algorithms,' Proc. of Eleventh Int'l Joint Conf. on Artificial Intelligence, pp. 762-767, San Mateo, CA, 1989
  2. D. B. Fogel, L. J. Fogel, and V. W. Porto, 'Evolving neural networks,' Biological Cybernetics, Vol. 63, pp. 487-493, 1990 https://doi.org/10.1007/BF00199581
  3. X. Yao, 'Evolving artificial neural networks,' Proceedings of the IEEE, vol. 87, no. 9, pp. 1423-1447, September 1999 https://doi.org/10.1109/5.784219
  4. X. Yao and Y. Liu, 'A new evolutionary system for evolving artificial neural networks,' IEEE Transactions on Neural Networks, vol. 8, pp. 694-713, May 1998 https://doi.org/10.1109/72.572107
  5. T. Back, D. B. Fogel, and Z. Michalewicz, Handbook of Evolutionary Computation, IOP Publishing and Oxford University Press, New York and Bristol (UK), 1997
  6. D. E. Goldberg, Genetic Algorithms in Serarch, Optimization, and Machine Learning, Addison-Wesley, Reading Massachusetts, 1989
  7. K. A. De Jong, 'An analysis of the behavior of a class of genetic adaptive systems,' Doctorial dissertation, University of Michigan, 1975
  8. J.-H. Ahn and S.-B. Cho, 'Speciated neural networks evolved with fitness sharing technique,' Proceedings of Congress on Evolutionary Computation, Vol. 1, pp. 390-396, May 2001 https://doi.org/10.1109/CEC.2001.934417
  9. S. Kullback and R. A. Leibler, 'On information and sufficiency,' Ann. Math. Stat., 22, pp. 79-86, 1951 https://doi.org/10.1214/aoms/1177729694
  10. S. A. Harp, T. Samad, and A. Guha, 'Toward the genetic synthesis of neural networks,' in Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, J. D. Schaffer, Ed. San Mateo, CA: Morgan Kaufmann, pp. 379-384, 1989
  11. S.-W. Lee, 'Off-line recognition of totally unconstrained hand-written numerals using multilayer cluster neural network,' IEEE Trans. Pattern Anal. Machine Intell., Vol. 18, pp. 648-652, 1996 https://doi.org/10.1109/34.506416
  12. P. A. Castillo, V. Rivas, J.J. Merelo, J. Gonzalez, A. Prieto and G. Romero, 'G-Prop-II: Global optimization of multilayer perceptrons using GAs,' Proceedings of the 1999 Congress on Evolutionary Computation, Vol. 3, pp. 2022-2027, May 1999 https://doi.org/10.1109/CEC.1999.785523
  13. A. J. C. Sharkey, 'On combining artificial neural nets,' Connection Science, Vol. 8, pp. 299-313, 1996 https://doi.org/10.1080/095400996116785
  14. L. Xu, A. Krzyzak and C. Y. Suen, 'Methods of combining multiple classifiers and their applications to handwriting recognition,' IEEE Trans. on Systems, Man and Cybernetics, vol. SMC-22, no. 3, pp. 418-435, 1992 https://doi.org/10.1109/21.155943
  15. 백종현, 다중 인식기의 다단계 결합을 통한 무제약 필기숫자 인식, 연세대학교 대학원 박사학위 논문, 1996
  16. J. C. Bioch, O. V. D. Meer, and R. Potharst, 'Classification using bayesian neural nets,' IEEE International Conference on Neural Networks, vol. 3, pp. 1488-1493, 1996 https://doi.org/10.1109/ICNN.1996.549120
  17. A. Khotanzad, and C. Chung, 'Hand written digit recognition using BKS combination of neural network classifiers,' Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation, pp. 94-99, 1994 https://doi.org/10.1109/IAI.1998.666880
  18. M. Perrone and L. N. Cooper, 'When networks disagree: Ensemble methods for hybrid neural networks,' Neural Networks for Speech and Image Processing, Chapman Hall, 1993
  19. A. D. Gordon, Classification: Methods for the Exploratory Analysis of Multivariate Data, Chapman and Hall, 1981
  20. K. Viele and C. Srinivasan, 'Parsimonious estimation of multiplicative interaction in analysis of variance using Kullback-Leibler information,' Journal of Statistical Planning and Inference, Vol. 84, pp. 201-219, 2000 https://doi.org/10.1016/S0378-3758(99)00151-2
  21. J. A. Garcia, J. Fdez-Valdivia, X. R. Fdez-Vidal, and R. Rodriguez-Sanchez, 'Information theoretic measure for visual target distinctness,' IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, pp. 362-383, April 2001 https://doi.org/10.1109/34.917572
  22. M. N. Do and M. Vetterli, 'Texture similarity measurement using Kullback-Leibler distance on wavelet subbands,' Proc. IEEE Int. Conf. on Image Proc. (ICIP), Vancouver, Canada, Sep. 2000 https://doi.org/10.1109/ICIP.2000.899558
  23. UCI Machine Learning Repository, http://www1.ics.uci.edu/~mlearn/MLRepository.html
  24. R. Setino and L. C. K. Hui, 'Use of a quasi-newton method in a feedforward neural network construction algorithm,' IEEE Trans. on Neural Networks, Vol. 6, no. 1, pp. 273-277, 1995 https://doi.org/10.1109/72.363426
  25. L. Prechelt, 'Probenl-a set of neural network benchmark probelms and benchmarking rules,' Tech. Rep. 21/94, Fakultat fur Informatik, Universitat Karlsruhe, 76128, Germany, 1994