KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

CRNN-Based Korean Phoneme Recognition Model with CTC Algorithm

CTC를 적용한 CRNN 기반 한국어 음소인식 모델 연구

  • 홍윤석 (서울대학교 융합과학부 인지컴퓨팅연구실) ;
  • 기경서 (서울대학교 융합과학부 인지컴퓨팅연구실) ;
  • 권가진 (서울대학교 융합과학부)
  • Received : 2018.07.06
  • Accepted : 2018.10.21
  • Published : 2019.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

For Korean phoneme recognition, Hidden Markov-Gaussian Mixture model(HMM-GMM) or hybrid models which combine artificial neural network with HMM have been mainly used. However, current approach has limitations in that such models require force-aligned corpus training data that is manually annotated by experts. Recently, researchers used neural network based phoneme recognition model which combines recurrent neural network(RNN)-based structure with connectionist temporal classification(CTC) algorithm to overcome the problem of obtaining manually annotated training data. Yet, in terms of implementation, these RNN-based models have another difficulty in that the amount of data gets larger as the structure gets more sophisticated. This problem of large data size is particularly problematic in the Korean language, which lacks refined corpora. In this study, we introduce CTC algorithm that does not require force-alignment to create a Korean phoneme recognition model. Specifically, the phoneme recognition model is based on convolutional neural network(CNN) which requires relatively small amount of data and can be trained faster when compared to RNN based models. We present the results from two different experiments and a resulting best performing phoneme recognition model which distinguishes 49 Korean phonemes. The best performing phoneme recognition model combines CNN with 3hop Bidirectional LSTM with the final Phoneme Error Rate(PER) at 3.26. The PER is a considerable improvement compared to existing Korean phoneme recognition models that report PER ranging from 10 to 12.

지금까지의 한국어 음소 인식에는 은닉 마르코프-가우시안 믹스쳐 모델(HMM-GMM)이나 인공신경망-HMM을 결합한 하이브리드 시스템이 주로 사용되어 왔다. 하지만 이 방법은 성능 개선 여지가 적으며, 전문가에 의해 제작된 강제정렬(force-alignment) 코퍼스 없이는 학습이 불가능하다는 단점이 있다. 이 모델의 문제로 인해 타 언어를 대상으로 한 음소 인식 연구에서는 이 단점을 보완하기 위해 순환 신경망(RNN) 계열 구조와 Connectionist Temporal Classification(CTC) 알고리즘을 결합한 신경망 기반 음소 인식 모델이 연구된 바 있다. 그러나 RNN 계열 모델을 학습시키기 위해 많은 음성 말뭉치가 필요하고 구조가 복잡해질 경우 학습이 까다로워, 정제된 말뭉치가 부족하고 기반 연구가 비교적 부족한 한국어의 경우 사용에 제약이 있었다. 이에 본 연구는 강제정렬이 불필요한 CTC 알고리즘을 도입하되, RNN에 비해 더 학습 속도가 빠르고 더 적은 말뭉치로도 학습이 가능한 합성곱 신경망(CNN)을 기반으로 한국어 음소 인식 모델을 구축하여 보고자 시도하였다. 총 2가지의 비교 실험을 통해 본 연구에서는 한국어에 존재하는 49가지의 음소를 판별하는 음소 인식기 모델을 제작하였으며, 실험 결과 최종적으로 선정된 음소 인식 모델은 CNN과 3층의 Bidirectional LSTM을 결합한 구조로, 이 모델의 최종 PER(Phoneme Error Rate)은 3.26으로 나타났다. 이는 한국어 음소 인식 분야에서 보고된 기존 선행 연구들의 PER인 10~12와 비교하면 상당한 성능 향상이라고 할 수 있다.

Keywords

JBCRJM_2019_v8n3_115_f0001.png 이미지

Fig. 1. Phoneme Classifier Overview

JBCRJM_2019_v8n3_115_f0002.png 이미지

Fig. 2. VGG Networks based DeepCNN, DeepCNN+BiLSTM, DeepCNN+BiGRU Models

JBCRJM_2019_v8n3_115_f0003.png 이미지

Fig. 3. DeepCNN+BiGRU(1hop~5hop), DeepCNN+BiLSTM(1hop~5hop) Models with Improved DeepCNN Models

JBCRJM_2019_v8n3_115_f0004.png 이미지

Fig. 4. Temporal Labels for Phoneme Recognition Models. One Phoneme Label per 30ms is Displayed in Each Cell

Table 1. Results for Phoneme Recognition

JBCRJM_2019_v8n3_115_t0001.png 이미지

Table 2. Results for Phoneme Recognition

JBCRJM_2019_v8n3_115_t0002.png 이미지

References

  1. Gales, Mark JF. "Maximum likelihood linear transformations for HMM-based speech recognition," Computer Speech & Language, Vol.12, No.2, pp.75-98, 1998. https://doi.org/10.1006/csla.1998.0043
  2. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradientbased learning applied to document recognition," Proceedings of the IEEE, Vol.86, No.11, pp.2278-2324, 1998. https://doi.org/10.1109/5.726791
  3. Glass, James R. "A probabilistic framework for segmentbased speech recognition," Computer Speech & Language , Vol.17, No.2-3, pp.137-152, 2003. https://doi.org/10.1016/S0885-2308(03)00006-8
  4. Schwarz, Petr, Pavel Matejka, and Jan Cernocky. "Towards lower error rates in phoneme recognition," International Conference on Text, Speech and Dialogue. Springer, Berlin, Heidelberg, 2004.
  5. Waibel, Alexander, et al., "Phoneme recognition using timedelay neural networks," Readings in Speech Recognition, 1990. 393-404.
  6. Bengio, Yoshua. "A connectionist approach to speech recognition," Advances in Pattern Recognition Systems Using Neural Network Technologies, pp.3-23. 1993.
  7. Mohamed, Abdel-rahman, George E. Dahl, and Geoffrey Hinton. "Acoustic modeling using deep belief networks," IEEE Transactions on Audio, Speech, and Language Processing, Vol.20, No.1, pp.14-22, 2012. https://doi.org/10.1109/TASL.2011.2109382
  8. Ardussi Mines, M., Hanson, B. F., & Shoup, J. E. "Frequency of Occurrence of Phonemes in Conversational English," Language and Speech, Vol.21, No.3, pp.221-241, 1978. https://doi.org/10.1177/002383097802100302
  9. Ji-Young Shin. "Phoneme and Syllable Frequencies of Korean Based on the Analysis of Spontaneous Speech Data," Communication Sciences and Disorders, Vol.13, No.2, pp.193-215, 2008.
  10. Seltzer, Michael L., and Jasha Droppo. "Multi-task learning in deep neural networks for improved phoneme recognition," Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013.
  11. Graves, Alex, Navdeep Jaitly, and Abdel-rahman Mohamed. "Hybrid speech recognition with deep bidirectional LSTM," Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013.
  12. Graves, Alex, Abdel-rahman Mohamed, and Geoffrey Hinton. "Speech recognition with deep recurrent neural networks," Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013.
  13. Minsoo Na and Minhwa Chung, "Assistive Program for Automatic Speech Transcription based on G2P Conversion and Speech Recognition," Proc. Conference on Korean Society of Speech Sciences, pp.131-132, 2016.
  14. Palaz, Dimitri, Ronan Collobert, and Mathew Magimai Doss. "End-to-end phoneme sequence recognition using convolutional neural networks," arXiv preprint arXiv: 1312.2137 (2013).
  15. Palaz, Dimitri, Mathew Magimai Doss, and Ronan Collobert. "Convolutional neural networks-based continuous speech recognition using raw speech signal," Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015.
  16. Heck, Michael, et al., "Ensembles of Multi-scale VGG Acoustic Models," Proc. Interspeech 2017 (2017): 1616-1620.
  17. Zhang, Ying, et al., "Towards end-to-end speech recognition with deep convolutional neural networks," arXiv preprint arXiv:1701.02720 (2017).
  18. Graves, Alex, et al., "Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks," Proceedings of the 23rd international conference on Machine learning. ACM, 2006.
  19. Hori, Takaaki, et al., "Advances in joint CTC-attention based end-to-end speech recognition with a deep CNN encoder and RNN-LM," arXiv preprint arXiv:1706.02737 (2017).
  20. National Institute of the Korean Language (NIKL), Seoul Reading Speech Corpus("서울말 낭독체 발화 말뭉치"), 2003. URL: https://ithub.korean.go.kr
  21. Yejin Cho, Korean Grapheme-to-Phoneme Analyzer (KoG2P), 2017. GitHub repository : https://github.com/scarletcho/KoG2P
  22. Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556 (2014).
  23. Amodei, Dario, et al., "Deep speech 2: End-to-end speech recognition in english and mandarin," International Conference on Machine Learning. 2016.
  24. Xiong, Wayne, et al., "The Microsoft 2016 conversational speech recognition system," Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 2017.
  25. Sainath, Tara N., et al., "Convolutional, long short-term memory, fully connected deep neural networks," Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015.
  26. Chung, Junyoung, et al., "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint arXiv:1412.3555 (2014).
  27. Xiong, Wayne, et al., "The Microsoft 2016 conversational speech recognition system," Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 2017.