Browse > Article

Hyperparameter Optimization for Image Classification in Convolutional Neural Network  

Lee, Jae-Eun (Dept. of Convergence & Applications Engineering, Pukyong National University)
Kim, Young-Bong (Dept. of Convergence & Applications Engineering, Pukyong National University)
Kim, Jong-Nam (Dept. of Convergence & Applications Engineering, Pukyong National University)
Publication Information
Journal of the Institute of Convergence Signal Processing / v.21, no.3, 2020 , pp. 148-153 More about this Journal
Abstract
In order to obtain high accuracy with an convolutional neural network(CNN), it is necessary to set the optimal hyperparameters. However, the exact value of the hyperparameter that can make high performance is not known, and the optimal hyperparameter value is different based on the type of the dataset, therefore, it is necessary to find it through various experiments. In addition, since the range of hyperparameter values is wide and the number of combinations is large, it is necessary to find the optimal values of the hyperparameters after the experimental design in order to save time and computational costs. In this paper, we suggest an algorithm that use the design of experiments and grid search algorithm to determine the optimal hyperparameters for a classification problem. This algorithm determines the optima values of the hyperparameters that yields high performance using the factorial design of experiments. It is shown that the amount of computational time can be efficiently reduced and the accuracy can be improved by performing a grid search after reducing the search range of each hyperparameter through the experimental design. Moreover, Based on the experimental results, it was shown that the learning rate is the only hyperparameter that has the greatest effect on the performance of the model.
Keywords
Grid search; Design of experiment; Hyperparameter optimization; Convolutional neural network(CNN);
Citations & Related Records
연도 인용수 순위
  • Reference
1 A. Klein, S. Falkner, S. Bartels, P. Hennig, F. Hutter, "Fast bayesian optimization of machine learning hyperparameters on large datasets," Proceeding of International Conference on Artificial Intelligence and Statistics, pp. 528-536, 2017.
2 S. Albelwi, A. Mahmood, "A framework for designing the architectures of deep convolutional neural networks," Entropy, vol. 19, no. 6, 2017.
3 G. A. Lujan-Moreno, P. R. Howard, O. G. Rojas, D. C. Montgomery, "Design of experiments and response surface methodology to tune machine learning hyperparameters, with a random forest case-study," Expert Systems with Applications, vol. 109, pp. 195-205, 2018.   DOI
4 J. Bergstra, Y. Bengio, "Random search for hyper-parameter optimization," The Journal of Machine Learning Research, vol. 13, no. 1, pp. 281-305, 2012.
5 J. Snoek, H. Larochelle, R. P. Adams, "Practical bayesian optimization of machine learning algorithms," Proceeding of Neural Information Processing Systems, pp. 2951-2959, 2012.
6 J. Bergstra, R. Bardenet, Y. Bengio, B. Kegl, "Algorithms for hyper-parameter optimization," Proceeding of Advances in Neural Information Processing Systems, vol. 63, no. 3, pp. 2546-2554, 2011.
7 S. Falkner, A. Klein, F. Hutter, "BOHB: Robust and efficient hyperparameter optimization at scale," Proceeding of International Conference on Machine Learning, pp. 1437-1446, 2018.
8 T. Hinz, N. Navarro-Guerrero, S. Magg, S. Wermter, "Speeding up the hyperparameter optimization of deep convolutional neural networks," International Journal of Computational Intelligence and Applications, vol. 17, no. 2, 2018.
9 S. S. Talathi, "Hyper-parameter optimization of deep convolutional networks for object recognition," Proceeding of IEEE International Conference on Image Processing, pp. 3982-3986, 2015.
10 R. Andonie, A. C. Florea, "Weighted random search for CNN hyperparameter optimization," International Journal of Computers Communications & Control, vol. 15, no. 2, pp. 432-445, 2020.