Journal of the Korea Institute of Information and Communication Engineering (한국정보통신학회논문지)

The Korea Institute of Information and Commucation Engineering (한국정보통신학회)
권호 표지
DOI QR코드

DOI QR Code

Speech Recognition based Message Transmission System for the Hearing Impaired Persons

청각장애인을 위한 음성인식 기반 메시지 전송 시스템

  • Kim, Sung-jin (Department of Electrical, Electronics & Communication Engineering, Korea University of Technology and Education(KOREATECH)) ;
  • Cho, Kyoung-woo (Department of Electrical, Electronics & Communication Engineering, Korea University of Technology and Education(KOREATECH)) ;
  • Oh, Chang-heon (Department of Electrical, Electronics & Communication Engineering, Korea University of Technology and Education(KOREATECH))
  • Received : 2018.08.24
  • Accepted : 2018.09.24
  • Published : 2018.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

The speech recognition service is used as an ancillary means of communication by converting and visualizing the speaker's voice into text to the hearing impaired persons. However, in open environments such as classrooms and conference rooms it is difficult to provide speech recognition service to many hearing impaired persons. For this, a method is needed to efficiently provide it according to the surrounding environment. In this paper, we propose a system that recognizes the speaker's voice and transmits the converted text to many hearing impaired persons as messages. The proposed system uses the MQTT protocol to deliver messages to many users at the same time. The end-to-end delay was measured to confirm the service delay of the proposed system according to the QoS level setting of the MQTT protocol. As a result of the measurement, the delay between the most reliable Qos level 2 and 0 is 111ms, confirming that it does not have a great influence on conversation recognition.

음성인식 서비스는 청각장애인에게 화자의 음성을 텍스트로 변환하여 시각화함으로써 의사소통의 보조적인 수단으로 사용되고 있다. 하지만 강의실 및 회의실과 같은 개방된 환경에서는 다수의 청각장애인에게 음성인식 서비스를 제공하기 힘들다. 이를 위해 주변 환경에 따라 음성 인식 서비스를 효율적으로 제공하기 위한 방법이 필요하다. 본 논문에서는 화자의 음성을 인식하여 변환된 텍스트를 다수의 청각장애인에게 메시지로 전달하는 시스템을 제안한다. 제안하는 시스템은 다수의 사용자에게 동시에 메시지를 전달하기 위해 MQTT 프로토콜을 사용한다. MQTT 프로토콜의 QoS level 설정에 따른 제안 시스템의 서비스 지연을 확인하기 위해 종단 간 지연을 측정하였다. 측정 결과 가장 신뢰성이 높은 QoS level 2와 0간의 지연이 111ms로 대화 인식에 큰 영향을 끼치지 않음을 확인하였다.

Keywords

HOJBC0_2018_v22n12_1604_f0001.png 이미지

Fig. 1 Publish/Subscribe Model

HOJBC0_2018_v22n12_1604_f0002.png 이미지

Fig. 2 Speech Recognition based Message Transmission Scenario for the Hearing Impaired Persons

HOJBC0_2018_v22n12_1604_f0003.png 이미지

Fig. 3 System Configuration

HOJBC0_2018_v22n12_1604_f0004.png 이미지

Fig. 4 Process Flow Chart of Speech Recognition System

HOJBC0_2018_v22n12_1604_f0005.png 이미지

Fig. 5 Android Applications

HOJBC0_2018_v22n12_1604_f0006.png 이미지

Fig. 6 Experiment Environment

HOJBC0_2018_v22n12_1604_f0007.png 이미지

Fig. 7 Screen of Each Smartphone

HOJBC0_2018_v22n12_1604_f0008.png 이미지

Fig. 8 End-to-End Delay According to QoS Level of Each Smartphone

References

  1. T. Aujeszky, and M. Eid, "A gesture recogintion architecture for arabic sign language communication system," Multimedia Tools and Applications, vol. 75, no. 14, pp. 8493-8511, Jul. 2016. https://doi.org/10.1007/s11042-015-2767-2
  2. E. W. Healy, and S. E. Yoho, "Difficulty understanding speech in noise by the hearing impaired: underlying causes and technological solutions," in Proceedings of the 38th Annual International Conference of The IEEE in Medicine and Biology Society, pp. 89-92, Orlando: Florida, 2016.
  3. R. Akmeliawati, D. Bailey, S. Bilal, S. Demidenko, N. Gamage, S. Khan, Y. C. Kuang, M. Ooi, and G. S. Gupta, "Assistive technology for relieving communication lumber between hearing/speech impaired and hearing people," The Journal of Engineering, vol. 2014, no. 6, pp. 312-323, Jun. 2014. https://doi.org/10.1049/joe.2014.0039
  4. D. Watanabe, Y. Takeuchim, T. Matsumoto, H. Kudo, and N. Ohnishi, "Communication support system of smart glasses for the hearing impaired," in Proceedings of the 16th International Conference on Computers Helping People with Special Needs, Linz: Austria, pp. 225-232, Jul. 2018.
  5. S. E. Han, S. A. Kim, and G. H. Hwang, "E-book to sign-language translation program based on morpheme analysis," Journal of Korea Institute of Information and Communication Engineering, vol. 21, no. 2, pp. 461-467, 2017. https://doi.org/10.6109/jkiice.2017.21.2.461
  6. A. Chern, Y. H. Lai, Y. P. Chang, Y. Tsao, R. Y. Chang, and H. W. Chang, "A smartphone-based multi-functional hearing assistive system to facilitate speech recognition in the classroom," IEEE Access, vol. 5, pp. 10339-10351, Jun. 2017. https://doi.org/10.1109/ACCESS.2017.2711489
  7. P. Patil, and J. Prajapat, "Implementation of a real time communication system for deaf people using internet of things," in Proceedings of the International Conference on Trends in Electronics and Informatics, Tirunelveli: India, pp. 313-316, May 2017.
  8. J. Ming, and D. Crookes, "Speech enhancement based on full-sentence correlation and clean speech recognition," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 3, pp. 531-543, Mar. 2017. https://doi.org/10.1109/TASLP.2017.2651406
  9. C. K. A. Reddy, N. Shankar, G. S. Bhat, R. Charan, and I. Panahi, "An individualized super-gaussian single microphone speech enhancement for hearing aid users with smartphone as an assistive device," IEEE Signal Processing Letters, vol. 24, no. 11, pp. 1601-1605, Nov. 2017. https://doi.org/10.1109/LSP.2017.2750979
  10. OASIS Standard, MQTT version 3.1.1[Internet], Available: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html.
  11. Y. T. Lee, W. H. Hsiao, C. M. Huang, and S. C. T. Chou, "An integrated cloud-based smart home management system with community hierarchy," IEEE Transactions on Consumer Electronics, vol. 62, no. 1, pp. 1-9, Feb. 2016. https://doi.org/10.1109/TCE.2016.7448556
  12. S. H. Kim, D. H. Kim, H. S. Oh, H. S. Jeon, and H. J. Park, "The data collection solution based on MQTT for stable IoT platforms," Journal of Korea Institute of Information and Communication Engineering, vol. 20, no. 4, pp. 728-738, Apr. 2016. https://doi.org/10.6109/JKIICE.2016.20.4.728
  13. Google Cloud, Cloud Speech-to-Text[Internet], Available: https://cloud.google.com/speech-to-text/.
  14. Mosquitto, Mosquitto[Internet], Available: http://mosquitto-.org/.
  15. eclipse paho, paho[Internet], Available: http://eclipse.org-/paho/.