KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

A Survey on Neural Networks Using Memory Component

메모리 요소를 활용한 신경망 연구 동향

  • 이지환 (연세대학교 컴퓨터과학과) ;
  • 박진욱 (연세대학교 컴퓨터과학과) ;
  • 김재형 (연세대학교 컴퓨터과학과) ;
  • 김재인 (연세대학교 컴퓨터과학과) ;
  • 노홍찬 (SK Telecom ICK 종합원) ;
  • 박상현 (연세대학교 컴퓨터과학과)
  • Received : 2018.02.08
  • Accepted : 2018.05.03
  • Published : 2018.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, recurrent neural networks have been attracting attention in solving prediction problem of sequential data through structure considering time dependency. However, as the time step of sequential data increases, the problem of the gradient vanishing is occurred. Long short-term memory models have been proposed to solve this problem, but there is a limit to storing a lot of data and preserving it for a long time. Therefore, research on memory-augmented neural network (MANN), which is a learning model using recurrent neural networks and memory elements, has been actively conducted. In this paper, we describe the structure and characteristics of MANN models that emerged as a hot topic in deep learning field and present the latest techniques and future research that utilize MANN.

최근 순환 신경 망(Recurrent Neural Networks)은 시간에 대한 의존성을 고려한 구조를 통해 순차 데이터(Sequential data)의 예측 문제 해결에서 각광받고 있다. 하지만 순차 데이터의 시간 스텝이 늘어남에 따라 발생하는 그라디언트 소실(Gradients vanishing)이 문제로 대두되었다. 이를 해결하기 위해 장단기 기억 모델(Long Short-Term Memory)이 제안되었지만, 많은 데이터를 저장하고 장기간 보존하는 데에 한계가 있다. 따라서 순환 신경망과 메모리 요소(Memory component)를 활용한 학습 모델인 메모리-증대 신경망(Memory-Augmented Neural Networks)에 대한 연구가 최근 활발히 진행되고 있다. 본 논문에서는 딥 러닝(Deep Learning) 분야의 화두로 떠오른 메모리-증대 신경망 주요 모델들의 구조와 특징을 열거하고, 이를 활용한 최신 기법들과 향후 연구 방향을 제시한다.

Keywords

References

  1. S. Y. Hea and E. G. Kim, "Design and implementation of the differential contents organization system based on each learner's level," The KIPS Transactions: Part A, Vol.18, No.6, pp.19-31, 2011.
  2. S. Russell and P. Norvig, "Artificial Intelligence: A Modern Approach," 3th ed., New York: Prentice Hall, 2009.
  3. J. L. Hennessy and D. A. Patterson, "Instruction-level parallelism and its exploitation," in Computer Architecture: A Quantitative Approach, 4th ed., San Francisco, CA: Morgan Kaufmann Pub., ch. 2, pp.66-153, 2007.
  4. D. B. Lenat, "Programming articitical interlligence," in Understanding Artificial Intelligence, Scientific American, Ed., New York: Warner Books Inc., pp.23-29, 2002.
  5. A. Stoffel, D. Spretke, H. Kinnemann, and D. A. Keim, "Enhancing document structure analysis using visual analytics," in Proceedings of the ACM Symposium on Applied Computing, Sierre, pp.8-12, 2010.
  6. J. Y. Seo, "Text driven construction of discourse structures for understanding descriptive texts," Ph.D. Dissertation, University of Texas at Austin, TX, USA, 1990.
  7. Thomas Claburn, Google Chrome 18 Brings Faster Graphics [Internet], http://www.techweb.com/news/232800057/google-chrome-18-brings-faster-graphics.html.
  8. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S.Khudanpur, "Recurrent neural network based language model," In Interspeech, pp.1045-1048, 2010.
  9. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, Vol.9, No.8, pp.1735-1780, 1997. https://doi.org/10.1162/neco.1997.9.8.1735
  10. J. Weston, S. Chopra, and A. Bordes, "Memory networks," arXiv preprint arXiv:1410.3916, 2014.
  11. A. Graves, G. Wayne, and I. Danihelka, "Neural turing machines," arXiv preprint arXiv:1410.5401, 2014.
  12. A. Joulin and T. Mikolov, "Inferring algorithmic patterns with stack-augmented recurrent nets," In Advances in Neural Information Processing Systems, 2015.
  13. E. Grefenstette, K. M. Hermann, M. Suleyman, and P. Blunsom, "Learning to Transduce with Unbounded Memory," In Advances in Neural Information Processing Systems, 2015.
  14. A. Fader, L. Zettlemoyer, and O. Etzioni, "Paraphrase-driven learning for open question answering," In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pp.1608-1618, 2013.
  15. A. Bordes, N. Usunier, R. Collobert, and J. Weston, "Towards understanding situated natural language," In Proceedings of Machine Learning Research, pp.65-72, 2010.
  16. S. Das, C. L. Giles, and G. Sun, "Learning context-free grammars: Capabilities and limitations of a recurrent neural network with an external stack memory," In Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society, 1992.
  17. G. Z. Sun, C. L. Giles, H. H. Chen, and Y. C. Lee, "The neural network pushdown automaton: Model, stack and learning simulations," UNIVERSITY OF MARYLAND TR NOs. UMIACS-TR-93-77 & CS-TR-3118, 1993.
  18. K. Cho, B. Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase representations using rnn encoder-decoder for statistical machine translation," In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp.1724-1734, 2014.
  19. S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus, "Endto-end memory networks," In Advances in Neural Information Processing Systems, pp.2431-2439, 2015.
  20. J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, A. H. Miller, A. Szlam, and J. Weston, "Evaluating prerequisite qualities for learning end-to-end dialog systems," arXiv preprint arXiv:1511.06931, 2015.
  21. J. Weston, "Dialog-based language learning," In Advances in Neural Information Processing Systems, 2016.
  22. A. Bordes, Y. Boureau, and J. Weston, "Learning End-To-End Goal-Oriented Dialog," arXiv preprint arXiv:1605.07683, 2016.
  23. A. Bordes, N. Usunier, S. Chopra, and J. Weston, "Largescale simple question answering with memory networks," arXiv preprint arXiv:1506.02075, 2015.
  24. J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. Merrienboer, A. Joulin, and T. Mikolov, In The 4th International Conference on Learning Representations, 2015.
  25. F. Hill, A. Bordes, S. Chopra, and J. Weston, "The goldilocks principle: Reading children's books with explicit memory representations," In The 4th International Conference on Learning Representations, 2015.
  26. A. H. Miller, A. Fisch, J. Dodge, A. Karimi, A. Bordes, and J. Weston, "Key-Value Memory Networks for Directly Reading Documents," arXiv preprint arXiv:1606.03126, 2016.
  27. J. Berant, A. Chou, R. Frostig, and P. Liang, "Semantic parsing on Freebase from question-answer pairs," In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
  28. A. Kumar, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, V. Zhong, R. Paulus, and R. Socher, "Ask Me Anything: Dynamic memory networks for Natural Language Processing," In Proceedings of the 33rd International Conference on Machine Learning, 2016.
  29. C. Xiong, S. Merity, and R. Socher, "Dynamic memory networks for visual and textual question answering," In Proceedings of the 33rd International Conference on Machine Learning, 2016.
  30. M, Henaff, J. Weston, A. Szlam, A. Bordes, and Y. LeCun, "Tracking the World State with Recurrent Entity Networks," In The 5th International Conference on Learning Representations, 2016.
  31. S. Chandar, S. Ahn, H. Larochelle, P. Vincent, G. Tesauro, and Y. Bengio, "Hierarchical memory networks," arXiv preprint arXiv:1605.07427, 2016.
  32. M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, "Building a large annotated corpus of english: The Penn Treebank," Computational Linguistics, Vol.19, No.2, pp.313-330, Jun. 1993.
  33. T. Mikolov, A. Joulin, S. Chopra, M. Mathieu, and M. Ranzato, "Learning longer memory in recurrent neural networks," In The 4th International Conference on Learning Representations, 2015.
  34. K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, "Teaching Machines to Read and Comprehend," In Advances in Neural Information Processing Systems, 2015.
  35. M. Henderson, B. Thomson, and J. Williams, "The second dialog state tracking challenge," In 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp.263-272, 2014.
  36. Y. Yang, W. Yih, and C. Meek, "Wikiqa: A challenge dataset for open-domain question answering," In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.2013-2018, 2015.
  37. R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, "Recursive deep models for semantic compositionality over a sentiment treebank," In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp.1631-1642, 2013.
  38. M. Malinowski and M. Fritz, "A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input," In Advances in Neural Information Processing Systems, 2014.
  39. A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, "Meta-learning with memory-augmented neural networks," In Proceedings of the 33rd International Conference on Machine Learning, 2016.
  40. C. Gulcehre, S. Chandar, K. Cho, and Y. Bengio, "Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes," arXiv preprint arXiv:1607.00036, 2017.
  41. A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwinska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis, "Hybrid computing using a neural network with dynamic external memory," Nature, Vol.538, pp.471-476, 2016. https://doi.org/10.1038/nature20101
  42. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, "Curriculum learning," In Proceedings of the 26th Annual International Conference on Machine Learning, pp.41-48, 2009.
  43. W. Zaremba, and I. Sutskever, "Reinforcement learning neural turing machines," arXiv preprint arXiv:1505.00521, 2015.
  44. R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning," Machine Learning, Vol.8, No.3-4, pp.229-256, 1992. https://doi.org/10.1007/BF00992696
  45. J. W. Rae, J. J. Hunt, T. Harley, I. Danihelka, A. Senior, G. Wayne, A. Graves, and T. P. Lilicrap, "Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes," In Advances in Neural Information Processing Systems, 2016.
  46. S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, "A large annotated corpus for learning natural language inference," arXiv preprint arXiv:1508.05326, 2015.