Browse > Article

Performance Improvement of Ensemble Speciated Neural Networks using Kullback-Leibler Entropy  

Kim, Kyung-Joong (연세대학교 컴퓨터과학과)
Cho, Sung-Bae (연세대학교 컴퓨터과학과)
Publication Information
The Transactions of the Korean Institute of Electrical Engineers D / v.51, no.4, 2002 , pp. 152-159 More about this Journal
Abstract
Fitness sharing that shares fitness if calculated distance between individuals is smaller than sharing radius is one of the representative speciation methods and can complement evolutionary algorithm which converges one solution. Recently, there are many researches on designing neural network architecture using evolutionary algorithm but most of them use only the fittest solution in the last generation. In this paper, we elaborate generating diverse neural networks using fitness sharing and combing them to compute outputs then, propose calculating distance between individuals using modified Kullback-Leibler entropy for improvement of fitness sharing performance. In the experiment of Australian credit card assessment, breast cancer, and diabetes in UCI database, proposed method performs better than not only simple average output or Pearson Correlation but also previous published methods.
Keywords
진화 신경망;Kulback-Leibler 엔트로피;적합도 공유;다중 신경망;
Citations & Related Records
연도 인용수 순위
  • Reference
1 A. J. C. Sharkey, 'On combining artificial neural nets,' Connection Science, Vol. 8, pp. 299-313, 1996   DOI
2 L. Xu, A. Krzyzak and C. Y. Suen, 'Methods of combining multiple classifiers and their applications to handwriting recognition,' IEEE Trans. on Systems, Man and Cybernetics, vol. SMC-22, no. 3, pp. 418-435, 1992   DOI   ScienceOn
3 백종현, 다중 인식기의 다단계 결합을 통한 무제약 필기숫자 인식, 연세대학교 대학원 박사학위 논문, 1996
4 J. C. Bioch, O. V. D. Meer, and R. Potharst, 'Classification using bayesian neural nets,' IEEE International Conference on Neural Networks, vol. 3, pp. 1488-1493, 1996   DOI
5 A. Khotanzad, and C. Chung, 'Hand written digit recognition using BKS combination of neural network classifiers,' Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation, pp. 94-99, 1994   DOI
6 M. N. Do and M. Vetterli, 'Texture similarity measurement using Kullback-Leibler distance on wavelet subbands,' Proc. IEEE Int. Conf. on Image Proc. (ICIP), Vancouver, Canada, Sep. 2000   DOI
7 M. Perrone and L. N. Cooper, 'When networks disagree: Ensemble methods for hybrid neural networks,' Neural Networks for Speech and Image Processing, Chapman Hall, 1993
8 A. D. Gordon, Classification: Methods for the Exploratory Analysis of Multivariate Data, Chapman and Hall, 1981
9 K. Viele and C. Srinivasan, 'Parsimonious estimation of multiplicative interaction in analysis of variance using Kullback-Leibler information,' Journal of Statistical Planning and Inference, Vol. 84, pp. 201-219, 2000   DOI   ScienceOn
10 J. A. Garcia, J. Fdez-Valdivia, X. R. Fdez-Vidal, and R. Rodriguez-Sanchez, 'Information theoretic measure for visual target distinctness,' IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, pp. 362-383, April 2001   DOI   ScienceOn
11 R. Setino and L. C. K. Hui, 'Use of a quasi-newton method in a feedforward neural network construction algorithm,' IEEE Trans. on Neural Networks, Vol. 6, no. 1, pp. 273-277, 1995   DOI   ScienceOn
12 X. Yao, 'Evolving artificial neural networks,' Proceedings of the IEEE, vol. 87, no. 9, pp. 1423-1447, September 1999   DOI   ScienceOn
13 L. Prechelt, 'Probenl-a set of neural network benchmark probelms and benchmarking rules,' Tech. Rep. 21/94, Fakultat fur Informatik, Universitat Karlsruhe, 76128, Germany, 1994
14 UCI Machine Learning Repository, http://www1.ics.uci.edu/~mlearn/MLRepository.html
15 D. Montana and L. Davis, 'Training feedforward neural networks using genetic algorithms,' Proc. of Eleventh Int'l Joint Conf. on Artificial Intelligence, pp. 762-767, San Mateo, CA, 1989
16 X. Yao and Y. Liu, 'A new evolutionary system for evolving artificial neural networks,' IEEE Transactions on Neural Networks, vol. 8, pp. 694-713, May 1998   DOI   ScienceOn
17 T. Back, D. B. Fogel, and Z. Michalewicz, Handbook of Evolutionary Computation, IOP Publishing and Oxford University Press, New York and Bristol (UK), 1997
18 D. E. Goldberg, Genetic Algorithms in Serarch, Optimization, and Machine Learning, Addison-Wesley, Reading Massachusetts, 1989
19 K. A. De Jong, 'An analysis of the behavior of a class of genetic adaptive systems,' Doctorial dissertation, University of Michigan, 1975
20 D. B. Fogel, L. J. Fogel, and V. W. Porto, 'Evolving neural networks,' Biological Cybernetics, Vol. 63, pp. 487-493, 1990   DOI
21 S. A. Harp, T. Samad, and A. Guha, 'Toward the genetic synthesis of neural networks,' in Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, J. D. Schaffer, Ed. San Mateo, CA: Morgan Kaufmann, pp. 379-384, 1989
22 S.-W. Lee, 'Off-line recognition of totally unconstrained hand-written numerals using multilayer cluster neural network,' IEEE Trans. Pattern Anal. Machine Intell., Vol. 18, pp. 648-652, 1996   DOI   ScienceOn
23 P. A. Castillo, V. Rivas, J.J. Merelo, J. Gonzalez, A. Prieto and G. Romero, 'G-Prop-II: Global optimization of multilayer perceptrons using GAs,' Proceedings of the 1999 Congress on Evolutionary Computation, Vol. 3, pp. 2022-2027, May 1999   DOI
24 J.-H. Ahn and S.-B. Cho, 'Speciated neural networks evolved with fitness sharing technique,' Proceedings of Congress on Evolutionary Computation, Vol. 1, pp. 390-396, May 2001   DOI
25 S. Kullback and R. A. Leibler, 'On information and sufficiency,' Ann. Math. Stat., 22, pp. 79-86, 1951   DOI   ScienceOn