Browse > Article
http://dx.doi.org/10.5370/KIEE.2018.67.6.767

Analysis of Evolutionary Optimization Methods for CNN Structures  

Seo, Kisung (Dept. of Electronics Engineering, Seokyeong University)
Publication Information
The Transactions of The Korean Institute of Electrical Engineers / v.67, no.6, 2018 , pp. 767-772 More about this Journal
Abstract
Recently, some meta-heuristic algorithms, such as GA(Genetic Algorithm) and GP(Genetic Programming), have been used to optimize CNN(Convolutional Neural Network). The CNN, which is one of the deep learning models, has seen much success in a variety of computer vision tasks. However, designing CNN architectures still requires expert knowledge and a lot of trial and error. In this paper, the recent attempts to automatically construct CNN architectures are investigated and analyzed. First, two GA based methods are summarized. One is the optimization of CNN structures with the number and size of filters, connection between consecutive layers, and activation functions of each layer. The other is an new encoding method to represent complex convolutional layers in a fixed-length binary string, Second, CGP(Cartesian Genetic Programming) based method is surveyed for CNN structure optimization with highly functional modules, such as convolutional blocks and tensor concatenation, as the node functions in CGP. The comparison for three approaches is analysed and the outlook for the potential next steps is suggested.
Keywords
Convolutional neural network; Optimization; Genetic algorithm; Cartesian genetic programming;
Citations & Related Records
연도 인용수 순위
  • Reference
1 A. Rikhtegar, M. Pooyan, M. Manzuri-Shalmani, "Genetic algorithm-optimised structure of convolutional neural network for face recognition applications," IET Computer Vision, Vol. 10, Iss. 6, pp. 559-566, 2016   DOI
2 L. Xie, A. Yuille, "Genetic CNN," CVPR 2017
3 M. Suganuma, s, Shirakawa, T. Nagao, "A Genetic Programming Approach to Designing Convolutional Neural Network Architectures," Proceedings of GECCO 2017, pp. 497-504, 2017
4 LeCun, Yann, et al. "Gradient based learning applied to document recognition," Proceedings of the IEEE, pp. 2278-2324, 1998
5 J. R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, The MIT Press, 1992.
6 J. Miller, P. Thomson, "Cartesian Genetic Programming," EuroGP 2000. LNCS, vol. 1802, pp. 121-132. Springer, 2000
7 K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," International Conference on Learning Representations, 2014.
8 C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going Deeper with Convolutions," Computer Vision and Pattern Recognition, 2015
9 K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Computer Vision and Pattern Recognition, 2016.
10 S. Zagoruyko and N. Komodakis, "Wide Residual Networks," arXiv: 1605.07146, 2016.
11 L. Xie, J. Wang, W. Lin, B. Zhang, and Q. Tian, "Towards Reversal-Invariant Image Representation," International Journal on Computer Vision, 2016.
12 G. Huang, Z. Liu, and K. Weinberger, "Densely Connected Convolutional Networks," arXiv: 1608.06993, 2016.
13 L. Xie, J. Wang, W. Lin, B. Zhang, and Q. Tian, "Towards Reversal-Invariant Image Representation", International Journal on Computer Vision, 2016.
14 K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," International Conference on Learning Representations, 2014.
15 K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition,"Computer Vision and Pattern Recognition, 2016.
16 K. O. Stanley, D. B. D'Ambrosio, and J. Gauci, "A hypercube-based indirect encoding for evolving large scale neural networks," Artificial Life, vol. 15, 2009
17 J. Schmidhuber, "Deep Learning in Neural Networks: An Overview," Neural Networks, Vol. 61, pp. 85-117, 2015.   DOI
18 Y. LeCun, Y. Bengio, G. Hinton, "Deep learning," Nature Vol. 521, pp. 436-444, 2015.   DOI
19 J. D. Goldberg, Genetic Algorithms in Search, Optimition and Machine Learning, Addison-Wesley, Reading, MA, 1989.
20 K. O. Stanley and R. Miikkulainen, "Competitive coevolution through evolutionary complexification," Journal of Artificial Intelligence Research, vol. 21, pp. 63-100, 2004.   DOI
21 K. O. Stanley, "Compositional pattern producing networks: A novel abstraction of development," Genetic Programming and Evolvable Machines Special Issue on Dev. Sys., vol. 8, no. 2, pp. 131-162, 2007.   DOI
22 C. Fernando et al. "Convolution by Evolution: Differentiable Pattern Producing Networks," In Proceedings of the 2016 Genetic and Evolutionary Computation Conference, Denver, CO, USA, pp. 109-116. 2016.