The Transactions of The Korean Institute of Electrical Engineers (전기학회논문지)

The Korean Institute of Electrical Engineers (대한전기학회)
권호 표지
DOI QR코드

DOI QR Code

Analysis of Evolutionary Optimization Methods for CNN Structures

CNN 구조의 진화 최적화 방식 분석

  • Seo, Kisung (Dept. of Electronics Engineering, Seokyeong University)
  • Received : 2018.05.02
  • Accepted : 2018.05.29
  • Published : 2018.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, some meta-heuristic algorithms, such as GA(Genetic Algorithm) and GP(Genetic Programming), have been used to optimize CNN(Convolutional Neural Network). The CNN, which is one of the deep learning models, has seen much success in a variety of computer vision tasks. However, designing CNN architectures still requires expert knowledge and a lot of trial and error. In this paper, the recent attempts to automatically construct CNN architectures are investigated and analyzed. First, two GA based methods are summarized. One is the optimization of CNN structures with the number and size of filters, connection between consecutive layers, and activation functions of each layer. The other is an new encoding method to represent complex convolutional layers in a fixed-length binary string, Second, CGP(Cartesian Genetic Programming) based method is surveyed for CNN structure optimization with highly functional modules, such as convolutional blocks and tensor concatenation, as the node functions in CGP. The comparison for three approaches is analysed and the outlook for the potential next steps is suggested.

Keywords

References

  1. J. Schmidhuber, "Deep Learning in Neural Networks: An Overview," Neural Networks, Vol. 61, pp. 85-117, 2015. https://doi.org/10.1016/j.neunet.2014.09.003
  2. Y. LeCun, Y. Bengio, G. Hinton, "Deep learning," Nature Vol. 521, pp. 436-444, 2015. https://doi.org/10.1038/nature14539
  3. J. D. Goldberg, Genetic Algorithms in Search, Optimition and Machine Learning, Addison-Wesley, Reading, MA, 1989.
  4. K. O. Stanley and R. Miikkulainen, "Competitive coevolution through evolutionary complexification," Journal of Artificial Intelligence Research, vol. 21, pp. 63-100, 2004. https://doi.org/10.1613/jair.1338
  5. K. O. Stanley, D. B. D'Ambrosio, and J. Gauci, "A hypercube-based indirect encoding for evolving large scale neural networks," Artificial Life, vol. 15, 2009
  6. K. O. Stanley, "Compositional pattern producing networks: A novel abstraction of development," Genetic Programming and Evolvable Machines Special Issue on Dev. Sys., vol. 8, no. 2, pp. 131-162, 2007. https://doi.org/10.1007/s10710-007-9028-8
  7. C. Fernando et al. "Convolution by Evolution: Differentiable Pattern Producing Networks," In Proceedings of the 2016 Genetic and Evolutionary Computation Conference, Denver, CO, USA, pp. 109-116. 2016.
  8. A. Rikhtegar, M. Pooyan, M. Manzuri-Shalmani, "Genetic algorithm-optimised structure of convolutional neural network for face recognition applications," IET Computer Vision, Vol. 10, Iss. 6, pp. 559-566, 2016 https://doi.org/10.1049/iet-cvi.2015.0037
  9. L. Xie, A. Yuille, "Genetic CNN," CVPR 2017
  10. M. Suganuma, s, Shirakawa, T. Nagao, "A Genetic Programming Approach to Designing Convolutional Neural Network Architectures," Proceedings of GECCO 2017, pp. 497-504, 2017
  11. LeCun, Yann, et al. "Gradient based learning applied to document recognition," Proceedings of the IEEE, pp. 2278-2324, 1998
  12. J. R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, The MIT Press, 1992.
  13. J. Miller, P. Thomson, "Cartesian Genetic Programming," EuroGP 2000. LNCS, vol. 1802, pp. 121-132. Springer, 2000
  14. K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," International Conference on Learning Representations, 2014.
  15. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going Deeper with Convolutions," Computer Vision and Pattern Recognition, 2015
  16. K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Computer Vision and Pattern Recognition, 2016.
  17. S. Zagoruyko and N. Komodakis, "Wide Residual Networks," arXiv: 1605.07146, 2016.
  18. L. Xie, J. Wang, W. Lin, B. Zhang, and Q. Tian, "Towards Reversal-Invariant Image Representation," International Journal on Computer Vision, 2016.
  19. L. Xie, J. Wang, W. Lin, B. Zhang, and Q. Tian, "Towards Reversal-Invariant Image Representation", International Journal on Computer Vision, 2016.
  20. K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," International Conference on Learning Representations, 2014.
  21. K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition,"Computer Vision and Pattern Recognition, 2016.
  22. G. Huang, Z. Liu, and K. Weinberger, "Densely Connected Convolutional Networks," arXiv: 1608.06993, 2016.