DOI QR코드

DOI QR Code

A Study on Compression of Connections in Deep Artificial Neural Networks

인공신경망의 연결압축에 대한 연구

  • 안희준 (서울과학기술대학교 전기정보공학과)
  • Received : 2017.08.31
  • Accepted : 2017.10.29
  • Published : 2017.10.31

Abstract

Recently Deep-learning, Technologies using Large or Deep Artificial Neural Networks, have Shown Remarkable Performance, and the Increasing Size of the Network Contributes to its Performance Improvement. However, the Increase in the Size of the Neural Network Leads to an Increase in the Calculation Amount, which Causes Problems Such as Circuit Complexity, Price, Heat Generation, and Real-time Restriction. In This Paper, We Propose and Test a Method to Reduce the Number of Network Connections by Effectively Pruning the Redundancy in the Connection and Showing the Difference between the Performance and the Desired Range of the Original Neural Network. In Particular, we Proposed a Simple Method to Improve the Performance by Re-learning and to Guarantee the Desired Performance by Allocating the Error Rate per Layer in Order to Consider the Difference of each Layer. Experiments have been Performed on a Typical Neural Network Structure such as FCN (full connection network) and CNN (convolution neural network) Structure and Confirmed that the Performance Similar to that of the Original Neural Network can be Obtained by Only about 1/10 Connection.

최근 딥러닝, 즉 거대 또는 깊은 인공신경망을 사용한 기술이 놀라운 성능을 보이고 있고, 점차로 그 네트워크의 규모가 커지고 있다. 하지만, 신경망 크기의 증가는 계산양의 증가로 이어져서 회로의 복잡성, 가격, 발열, 실시간성 제약 등의 문제를 야기한다. 또한, 신경망 연결에는 많은 중복성이 존재한다, 본 연구에서는 이 중복성을 효과적으로 제거하여 이용하여 원 신경망의 성능과 원하는 범위안의 차이를 보이면서, 네트워크 연결의 수를 줄이는 방법을 제안하고 실험하였다. 특히, 재학습에 의하여 성능을 향상시키고, 각 계층별 차이를 고려하기 위하여 계층별 오류율을 할당하여 원하는 성능을 보장할 수 있는 간단한 방법을 제안하였다. 대표적인 영상인식 신경망구조인 FCN (전연결) 구조와 CNN (컨벌루션 신경망) 구조에서 대하여 실험한 결과 약 1/10 정도의 연결만으로도 원 신경망과 유사한 성능을 보일 수 있음을 확인하였다.

Keywords

References

  1. LeCuns, Y., Bengio, Y., and Hinton, G., "Deep learning," Nature, Vol. 521, No. 7553, pp. 436-444, 2015. https://doi.org/10.1038/nature14539
  2. Lee, K., H, lee, S.-Y., and Chung, N., Koo, C., "The Effects of IT Usage of Exhibition Onsite and Overall Effectiveness Toward Attendee's Satisfaction", The Journal of Internet Electronic Commerce Research, Vol. 16, No. 6, pp. 77-94, 2016.
  3. Hong, T. H. and Kim, J. W., "Intelligent Intrusion Detection Systems Using the Asymmetric costs of Errors in Data Mining," The Journal of Information Systems, Vol. 15, No. 4, pp. 211-224, 2016
  4. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P., "Gradient-based Learning Applied to Document Recognition," Proceedings of the IEEE, Vol. 86, No. 11, pp.2278-2324, 1998. https://doi.org/10.1109/5.726791
  5. Han, S., Pool, J., Tran, J., and Dally, W., "Learning both Weights and Connections for Efficient Neural Network." In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015.
  6. Naumov, M., Chien, L. S., Vandermersch, P., and Kapasi, U. "Cusparse Library," In GPU Technology Conference, Spetmber, 2010.
  7. Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., and Temam, O. "Diannao: A Small-Footprint High-throughput Accelerator for Ubiquitous Machine-learning. In ACM Sigplan Notices, Vol. 49, No. 4, pp. 269-284. 2014.
  8. Zhang, S., Du, Z., Zhang, L., Lan, H., Liu, S., Li, L., and Chen, Y., "Cambricon-X: An Accelerator for Sparse Neural Networks," 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1-12, 2016.