Browse > Article
http://dx.doi.org/10.3745/KTSDE.2022.11.1.1

Performance Improvement Method of Fully Connected Neural Network Using Combined Parametric Activation Functions  

Ko, Young Min (전주대학교 인공지능학과)
Li, Peng Hang (전주대학교 인공지능학과)
Ko, Sun Woo (전주대학교 인공지능학과)
Publication Information
KIPS Transactions on Software and Data Engineering / v.11, no.1, 2022 , pp. 1-10 More about this Journal
Abstract
Deep neural networks are widely used to solve various problems. In a fully connected neural network, the nonlinear activation function is a function that nonlinearly transforms the input value and outputs it. The nonlinear activation function plays an important role in solving the nonlinear problem, and various nonlinear activation functions have been studied. In this study, we propose a combined parametric activation function that can improve the performance of a fully connected neural network. Combined parametric activation functions can be created by simply adding parametric activation functions. The parametric activation function is a function that can be optimized in the direction of minimizing the loss function by applying a parameter that converts the scale and location of the activation function according to the input data. By combining the parametric activation functions, more diverse nonlinear intervals can be created, and the parameters of the parametric activation functions can be optimized in the direction of minimizing the loss function. The performance of the combined parametric activation function was tested through the MNIST classification problem and the Fashion MNIST classification problem, and as a result, it was confirmed that it has better performance than the existing nonlinear activation function and parametric activation function.
Keywords
Fully Connected Neural Network; Nonlinear Activation Function; Combined Parametric Activation Function; Learning;
Citations & Related Records
Times Cited By KSCI : 3  (Citation Analysis)
연도 인용수 순위
1 Y. Bengio, I. Goodfellow, and A. Courville, "Deep learning," MIT Press, 2017.
2 Y. M. Ko and S. W. Ko, "Alleviation of vanishing gradient problem using parametric activation functions," KIPS Transactions on Software and Data Engineering, Vol.10, No.10, pp.407-420, 2021.   DOI
3 Y. Qin, X. Wang, and J. Zou, "The optimized deep belief networkswith improved logistic Sigmoid units and their application in faultdiagnosis for planetary gearboxes of wind turbines," IEEE Transactions on Industrial Electronics, Vol.66, No.5, pp.3814-3824, 2018.   DOI
4 X. Wang, Y. Qin, Y. Wang, S. Xiang, and H. Chen, "ReLTanh: An activation function with vanishing gradient resistance for SAE-based DNNs and its application to rotating machinery fault diagnosis," Neurocomputing, Vol.363, pp.88-98, 2019.   DOI
5 R. Pascanu, T. Mikolov, and Y. Bengio, "Understanding the exploding gradient problem," arXiv:1211.5063, 2012.
6 N. Y. Kong, Y. M. Ko, and S. W. Ko, "Performance improvement method of convolutional neural network using agile activation function," KIPS Transactions on Software and Data Engineering, Vol.9, No.7, pp.213-220, 2020.   DOI
7 V. Nair and G. Hinton, "Rectified linear units improve restricted boltzmann machines," ICML, pp.807-814, 2010.
8 K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Networks, Vol.2, Iss.5, pp.359-366, 1989.   DOI
9 N. Y. Kong and S. W. Ko, "Performance improvement method of deep neural network using parametric activation functions," Journal of the Korea Contents Association, Vol.21, No.3, pp.616-625, 2021.   DOI
10 M. Roodschild, J. Gotay Sardinas, and A. Will, "A new approach for the vanishing gradient problem on sigmoid activation," Springer Nature, Vol.20, Iss.4, pp.351-360, 2020.
11 S. Kong and M. Takatsuka, "Hexpo: A vanishing-proof activation function," International Joint Conference on Neural Networks, pp.2562-2567, 2017.
12 B. Xu, N. Wang, T. Chen, and M. Li, "Empirical evaluation of rectified activations in convolution network," arXiv:1505. 00853, 2015.
13 R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks," arXiv:1211.5063, 2013.
14 K. He, X. Zhang, S. Ren, and J. Sun, "Delving deep into rectifiers: Surpassing human-level performance on imagenet classification," International Conference on Computer Vision, arXiv:1502.01852, 2015.
15 D. Clevert, T. Unterthiner, and S. Hochreiter, "Fast and accurate deep network learning by exponential linear units(ELUs)," arXiv:1511.07289, 2016.