Browse > Article
http://dx.doi.org/10.6109/jkiice.2021.25.1.27

Improvement of Catastrophic Forgetting using variable Lambda value in EWC  

Park, Seong-Hyeon (Department of Embedded Systems Engineering, Incheon National University)
Kang, Seok-Hoon (Department of Embedded Systems Engineering, Incheon National University)
Publication Information
Journal of the Korea Institute of Information and Communication Engineering / v.25, no.1, 2021 , pp. 27-35 More about this Journal
Abstract
This paper proposes a method to mitigate the Catastrophic Forgetting phenomenon in which artificial neural networks forget information on previous data. This method adjusts the Regularization strength by measuring the relationship between previous data and present data. MNIST and EMNIST data were used for performance evaluation and experimented in three scenarios. The experiment results showed a 0.1~3% improvement in the accuracy of the previous task for the same domain data and a 10~13% improvement in the accuracy of the previous task for different domain data. When continuously learning data with various domains, the accuracy of all previous tasks achieved more than 50% and the average accuracy improved by about 7%. This result shows that neural network learning can be properly performed in a CL environment in which data of different domains are successively entered by the method of this paper.
Keywords
Artificial neural network; Continuous learning; Catastrophic forgetting; Regularization; EWC;
Citations & Related Records
연도 인용수 순위
  • Reference
1 R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan, "Measuring catastrophic forgetting in neural networks," In: Thirty-second AAAI conference on artificial intelligence, 2018.
2 S. J. Park, S. M. Choi, H. J. Lee, and J. B. Kim, "Spatial analysis using R based Deep Learning," In Asia-pacific Journal of Multimedia Services Convergent with Art, Humanities, and Sociology, ISSN:2383-5281, vol. 6, no. 4, pp. 1-8, 2016.   DOI
3 R. M. French, "Catastrophic forgetting in connectionist networks," Trends in cognitive sciences, vol. 3, no. 4, pp. 128-135, Apr. 1999.   DOI
4 Y. Hsu, Y. Liu, A. Ramasamy, and Z. Kira, "Re-evaluating continual learning scenarios: A categorization and case for strong baselines," arXiv preprint, arXiv:1810.12488, 2018.
5 I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, "An empirical investigation of catastrophic forgetting in gradient-based neural networks," arXiv preprint arXiv: 1312.6211, 2013.
6 G. I. Parisi, R. Kemker, J. L.Part, C. Kanan, and S. Wermter, "Continual lifelong learning with neural networks: A review," Neural Networks, 2019.
7 J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, "Overcoming catastrophic forgetting in neural networks," Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521-3526, 2017.   DOI
8 Z. Li and D. Hoiem, "Learning without forgetting," IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 12, pp. 2935-2947, 2017.   DOI
9 F. Zenke, B. Poole, and S. Ganguli, "Continual learning through synaptic intelligence," In: Proceedings of the 34th International Conference on Machine Learning, Volume 70, pp. 3987-3995, 2017.
10 D. Maltoni and V. Lomonaco, "Continuous learning in single-incremental-task scenarios," Neural Networks, vol. 116, pp. 56-73, 2019.   DOI
11 V. Lomonaco and D. Maltoni, "Core50: a new dataset and benchmark for continuous object recognition," arXiv preprint arXiv:1705.03550, 2017.
12 Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.   DOI
13 G. Cohen, S. Afshar, J. Tapson, and A. Schaik, "EMNIST: Extending MNIST to handwritten letters," In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 2921-2926, 2017.