Korean Journal of Cognitive Science (인지과학)

The Korean Society for Cognitive Science (한국인지과학회)
권호 표지
DOI QR코드

DOI QR Code

RBM-based distributed representation of language

RBM을 이용한 언어의 분산 표상화

  • You, Heejo (Department of Psychology, Korea University) ;
  • Nam, Kichun (Department of Psychology, Korea University) ;
  • Nam, Hosung (Department of English language and literature, Korea University)
  • Received : 2017.06.05
  • Accepted : 2017.06.21
  • Published : 2017.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

The connectionist model is one approach to studying language processing from a computational perspective. And building a representation in the connectionist model study is just as important as making the structure of the model in that it determines the level of learning and performance of the model. The connectionist model has been constructed in two different ways: localist representation and distributed representation. However, the localist representation used in the previous studies had limitations in that the unit of the output layer having a rare target activation value is inactivated, and the past distributed representation has the limitation of difficulty in confirming the result by the opacity of the displayed information. This has been a limitation of the overall connection model study. In this paper, we present a new method to induce distributed representation with local representation using abstraction of information, which is a feature of restricted Boltzmann machine, with respect to the limitation of such representation of the past. As a result, our proposed method effectively solves the problem of conventional representation by using the method of information compression and inverse transformation of distributed representation into local representation.

연결주의 모델은 계산주의적 관점에서 언어 처리를 연구하는 한 가지 접근법이다. 그리고 연결주의 모델 연구를 진행하는데 있어서 표상(representation)을 구축하는 것은, 모델의 학습 수준 및 수행 능력을 결정한다는 점에서 모델의 구조를 만드는 것만큼이나 중요한 일이다. 연결주의 모델은 크게 지역 표상(localist representation)과 분산 표상(distributed representation)이라는 두 가지 서로 다른 방식으로 표상을 구축해 왔다. 하지만 종래 연구들에서 사용된 지역 표상은 드문 목표 활성화 값을 갖고 있는 출력층의 유닛이 불활성화 하는 제한점을, 그리고 과거의 분산 표상은 표상된 정보의 불투명성에 의한 결과 확인의 어려움이라는 제한점을 갖고 있었으며 이는 연결주의 모델 연구 전반의 제한점이 되어 왔다. 본 연구는 이와 같은 과거의 표상 구축의 제한점에 대하여, 제한된 볼츠만 머신(restricted Boltzmann machine)이 갖고 있는 특징인 정보의 추상화를 활용하여 지역 표상을 가지고 분산 표상을 유도하는 새로운 방안을 제시하였다. 결과적으로 본 연구가 제안한 방법은 정보의 압축과 분산 표상을 지역 표상으로 역변환하는 방안을 활용하여 종래의 표상 구축 방법이 갖고 있는 문제를 효과적으로 해결함을 보였다.

Keywords

Acknowledgement

Supported by : 한국연구재단

References

  1. 유희조, 남기춘, 남호성. (2015). 속성적 관점에 기반한 단어 재인시의 의미 처리 가능성 연구: 연결주의 모델링, 한국심리학회지: 인지 및 생물, 27, 613-638.
  2. Cree, G. S., McNorgan, C., & McRae, K. (2006). Distinctive features hold a privileged status in the computation of word meaning: Implications for theories of semantic memory. Journal of Experimental Psychology. Learning, Memory, and Cognition, 32, 643-658. https://doi.org/10.1037/0278-7393.32.4.643
  3. Dilkina, K., McClelland, J. L., & Plaut, D. C. (2008). A single-system account of semantic and lexical deficits in five semantic dementia patients. Cognitive Neuropsychology, 25, 136-164. https://doi.org/10.1080/02643290701723948
  4. Dilkina, K., McClelland, J. L., & Plaut, D. C. (2010). Are There Mental Lexicons? The Role of Semantics in Lexical Decision. Brain Research, 1365, 66-81. https://doi.org/10.1016/j.biotechadv.2011.08.021.Secreted
  5. Harm, M. W., & Seidenberg, M. S. (1999). Phonology, reading acquisition, and dyslexia: insights from connectionist models. Psychological Review, 106, 491-528. https://doi.org/10.1037/0033-295X.106.3.491
  6. Hinton, G. (2010). A Practical Guide to Training Restricted Boltzmann Machines A Practical Guide to Training Restricted Boltzmann Machines. Computer, 9, 1. https://doi.org/10.1007/978-3-642-35289-8_32
  7. Hinton, G., & Shallice, T. (1991). Lesioning an attractor network: investigations of acquired dyslexia. Psychological Review, 98, 74-95. https://doi.org/10.1037/0033-295X.98.1.74
  8. Kawamoto, A. (1993). Nonlinear dynamics in the resolution of lexical ambiguity: A parallel distributed processing account. Journal of Memory and Language, 32, 474-516. Retrieved from http://psycnet.apa.org/psycinfo/1994-07885-001 https://doi.org/10.1006/jmla.1993.1026
  9. Laszlo, S., & Plaut, D. C. (2012). A neurally plausible parallel distributed processing model of event-related potential word reading data. Brain and Language, 120, 271-281. https://doi.org/10.1016/j.bandl.2011.09.001.A
  10. McClelland, J., & Rumelhart, D. (1981). An interactive activation model of context effects in letter perception: I. An account of basic findings. Psychological Review, (September). Retrieved from http://psycnet.apa.org/journals/rev/88/5/375/
  11. Mikolov, T., Corrado, G., Chen, K., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. In Proceedings of the International Conference on Learning Representations (pp. 1-12). https://doi.org/10.1162/153244303322533223
  12. Plaut, D. C. (1997). Structure and Function in the Lexical System: Insights from Distributed Models of Word Reading and Lexical Decision. Language and Cognitive Processes, 12, 765-806. https://doi.org/10.1080/016909697386682
  13. Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996). Understanding normal and impaired word reading: computational principles in quasi-regular domains. Psychological Review, 103, 56-115. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/8650300 https://doi.org/10.1037/0033-295X.103.1.56
  14. Plaut, D. C., & Shallice, T. (1993). Deep dyslexia: A case study of connectionist neuropsychology. Cognitive Neuropsychology, 10, 377-500. https://doi.org/10.1080/02643299308253469
  15. Rogers, T. T., Lambon Ralph, M. a, Garrard, P., Bozeat, S., McClelland, J. L., Hodges, J. R., & Patterson, K. (2004). Structure and deterioration of semantic memory: a neuropsychological and computational investigation. Psychological Review, 111, 205-235. https://doi.org/10.1037/0033-295X.111.1.205
  16. Seidenberg, M. S., & McClelland, J. L. (1989). A distributed, developmental model of word recognition and naming. Psychological Review, 96, 523-568. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2798649 https://doi.org/10.1037/0033-295X.96.4.523
  17. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. In Parallel Distributed Processing Explorations in the Microstructure of Cognition (Vol. 1, pp. 194-281). Retrieved from http://portal.acm.org/citation.cfm?id=104279.104290