다중계층 퍼셉트론 내 Sigmoid 활성함수의 구간 선형 근사와 양자화 근사와의 비교

A piecewise affine approximation of sigmoid activation functions in multi-layered perceptrons and a comparison with a quantization scheme

Multi-layered perceptrons that are a nonlinear neural network model, have been widely used for various applications mainly thanks to good function approximation capability for nonlinear fuctions. However, for digital hardware implementation of the multi-layere perceptrons, the quantization scheme using "look-up tables (LUTs)" is commonly employed to handle nonlinear signmoid activation functions in the neworks, and thus requires large amount of storage to prevent unacceptable quantization errors. This paper is concerned with a new effective methodology for digital hardware implementation of multi-layered perceptrons, and proposes a "piecewise affine approximation" method in which input domain is divided into (small number of) sub-intervals and nonlinear sigmoid function is linearly approximated within each sub-interval. Using the proposed method, we develop an expression and an error backpropagation type learning algorithm for a multi-layered perceptron, and compare the performance with the quantization method through Monte Carlo simulations on XOR problems. Simulation results show that, in terms of learning convergece, the proposed method with a small number of sub-intervals significantly outperforms the quantization method with a very large storage requirement. We expect from these results that the proposed method can be utilized in digital system implementation to significantly reduce the storage requirement, quantization error, and learning time of the quantization method.quantization method.