Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Inference of Context-Free Grammars using Binary Third-order Recurrent Neural Networks with Genetic Algorithm

이진 삼차 재귀 신경망과 유전자 알고리즘을 이용한 문맥-자유 문법의 추론

  • Jung, Soon-Ho (Dept. of Computer Engineering, Pukyong National University)
  • 정순호 (부경대학교 컴퓨터공학과)
  • Received : 2011.12.14
  • Accepted : 2012.02.21
  • Published : 2012.03.30
Download PDF
⟨ Previous Next ⟩

Abstract

We present the method to infer Context-Free Grammars by applying genetic algorithm to the Binary Third-order Recurrent Neural Networks(BTRNN). BTRNN is a multiple-layered architecture of recurrent neural networks, each of which is corresponding to an input symbol, and is combined with external stack. All parameters of BTRNN are represented as binary numbers and each state transition is performed with any stack operation simultaneously. We apply Genetic Algorithm to BTRNN chromosomes and obtain the optimal BTRNN inferring context-free grammar of positive and negative input patterns. This proposed method infers BTRNN, which includes the number of its states equal to or less than those of existing methods of Discrete Recurrent Neural Networks, with less examples and less learning trials. Also BTRNN is superior to the recent method of chromosomes representing grammars at recognition time complexity because of performing deterministic state transitions and stack operations at parsing process. If the number of non-terminals is p, the number of terminals q, the length of an input string k, and the max number of BTRNN states m, the parallel processing time is O(k) and the sequential processing time is O(km).

이 논문은 이진 삼차 재귀 신경망(Binary Third-order Recurrent Neural Networks: BTRNN)에 유전자 알고리즘을 적용하여 문맥-자유 문법을 추론하는 방법을 제안한다. BTRNN은 각 입력심볼에 대응되는 재귀 신경망들의 다층적 구조이고 외부의 스택과 결합된다. BTRNN의 매개변수들은 모두 이진수로 표현되며 상태 전이와 동시에 스택의 한 동작이 실행된다. 염색체로 표현된 BTRNN들에 유전자 알고리즘을 적용하여 긍정과 부정의 입력 패턴들의 문맥-자유 문법을 추론하는 최적의 BTRNN를 얻는다. 이 방법은 기존의 신경망 이용방법보다 적은 학습량과 적은 학습회수로 작거나 같은 상태 수를 갖는 BTRNN을 추론한다. 또한 문법 표현의 염색체 이용방법보다 parsing과정에서 결정적인 상태전이와 스택동작이 실행되므로 입력 패턴에 대한 인식처리 시간복잡도가 우수하다. 문맥-자유 문법의 비단말 심볼의 개수 p, 단말 심볼의 개수 q, 그리고 길이가 k인 문자열이 입력이 될 때, BTRNN의 최대 상태수가 m이라고 하면, BTRNN의 인식처리 병렬처리 시간은 O(k)이고 순차처리 시간은 O(km)이다.

Keywords

References

  1. J. L. Elman, "Finding structure in time," Cognitive Science, vol.14, pp.179-211, Apr./Jun. 1990 https://doi.org/10.1207/s15516709cog1402_1
  2. S. Porat and J. A. Feldman, "Learning automata from ordered examples," Machine Learning, vol.7, no. 2-3, pp.109-138, Sep. 1991. https://doi.org/10.1007/BF00114841
  3. K. Fu and T. L. Booth, "Grammatical inference: Introduction and survey," IEEE Trans. Pattern Anal. & Machine Intell., vol.8, no.3, pp.343-375, May 1986. https://doi.org/10.1109/TPAMI.1986.4767796
  4. A. Cleeremans, D. Servan-Schreiber, and J. L. McClelland, "Finite state automata and simple recurrent networks," Neural Comp., vol.1, no.3, pp.327-381, Mar. 1989.
  5. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Second-order recurrent neural networks," Neural Comp., vol.4, no.3, pp.393-405, Mar. 1972.
  6. C. L. Giles, and C. B. Miller, "The effect of higher order in recurrent neural networks: Experiments," Artificial Neural Networks for Speech and Vision, Chapman & Hall, pp.54-78, 1993.
  7. C. L. Giles, C. B. Miller, D. Chen, H. H. Chen, G. Z. Sun, and Y. C. Lee, "Learning and Extracting finite state automata with second-order recurrent neural networks," Comm. ACM, vol.4, no.3, pp.393-405, March 1992.
  8. Z. Zeng, R. M. Goodman, and P. Smyth, "Discrete recurrent neural networks for grammatical inference," IEEE Trans. Neural Networks, vol.5, no.2, pp.320-330, June 1994. https://doi.org/10.1109/72.279194
  9. M. W. Goudreau, C. L. Giles, S. T. Chakradhar, and D. Chen, "First-order versus second-order single-layer recurrent neural networks," IEEE Trans. Neural Networks, vol.5, no.3, pp.511-513, May 1994. https://doi.org/10.1109/72.286928
  10. S. Jung and H. Yoon, "Binary Second-Order Recurrent Neural Networks for Inferring Regular Grammars," IEICE Trans. on Info. and Sys., vol.E83-D, no.11, pp.1996-2007, Nov. 2000.
  11. I. Tsoulos, D. Gavrilis, and E. Glavas, "Neural network construction and training using grammatical evolution," Neurocomputing, pp.1-9, Dec. 2008.
  12. P. Wyard, "Context Free Grammar Induction Using Genetic Algorithm," 1994, Proceedings, Sixth Int'l Conference on Tools with Artificial Intelligence, pp.828-831, April 1994.
  13. N. S. Choubey, and M. V. Kharat, "PDA Simulator for CFG Induction Using Genetic Algorithm," 12th Int'l Conference on Computer Modeling and Simulation, pp.92-97, March 2010.
  14. H. M. Pandey, "Context Free Grammar Induction Library Using Genetic Algorithms," Int'l Conference on Computer & Communication Technology(ICCCT), pp.752-758, Sept. 2010.
  15. S. Lawrence, L. Giles, and S. Fong, "Natural Language Grammatical Inference with RNN," IEEE Trans. Knowledge and Data Engineering, vo.12, No.1, JAN/FEB, 2000.
  16. E. Rodrigues and H. S. Lopes, "Genetic Programming for Induciton of Context-free Grammars," 7th International Conference on Intelligent Systems Design and Applications, pp.297-302, Oct. 2007.
  17. J. E. Hopcroft and J. D. Ullman, "Introduction Automata Theory, Languages, and Computation," Addison-Wesley, pp.217-232, 1979.
  18. U. Harigopal, and H. C. Chen, "Grammatical Inference using Higher Order Recurrent Neural Networks," Proceedings SSST '93., 25th Southeastern Symposium on System Theory, pp.338-342, March 1993.
  19. S, Jung, "Generalized Binary Second-order Recurrent Neural Networks Equivalent to Regular Grammars," Journal Of Korea Intelligent Information System Society(KIISS), vol. 12, no. 1, pp.107-123, MAR, 2006.
  20. N. S. Chouney and M. U. Kharat, "Stochastic Mutation Approach for Grammar Induction Using Genetic Algorithm," International Conference on Electronic Computer Technology (ICECT), pp.143-146, May 2010.