Telemedicine robot system for visual inspection and auscultation using WebRTC

WebRTC를 이용한 육안 검사 및 청진용 원격진료 로봇 시스템

Abstract

When a doctor examines a patient in a hospital, the doctor directly checks the patient's condition and conducts a face-to-face diagnosis through dialogue with the patient. However, it is often difficult for doctors to directly treat patients. Recently, several types of telemedicine systems have been developed. However, the systems have lack of capabilities to observe heart disease, neck condition, skin condition, inside ear condition, etc. To solve this problem, in this paper, an interactive telemedicine robot system with autonomous driving in a room capable of visual examination and auscultation of patients is developed. The developed robot can be controlled remotely through the WebRTC platform to move toward the patient and check a patient's condition under the doctor's observation using the multi-joint robot arm. The video information, audio information, patient's heart sound, and other data obtained remotely from patients can be transmitted to a doctor through the web RTC platform. The developed system can be applied to the various places where doctors are not possible to attend.

의사가 병원에서 환자를 진찰할 때 의사는 환자의 상태를 직접 확인하고 환자와의 대화를 통해 대면 진단을 한다. 그러나 의사가 환자를 직접 진료하기 어려운 경우가 많다. 최근에는 여러 유형의 원격 의료 시스템이 개발되었다. 그러나 현존하는 많은 시스템이 심장질환, 목상태, 피부상태, 귀의 내부상태 등을 관찰할 수 있는 능력이 부족하다. 이러한 문제를 해결하기 위하여 본 논문에서는 환자의 육안 검사와 청진이 가능하도록 실내에서 자율주행이 가능한 대화형 원격진료 로봇 시스템을 개발한다. 개발된 로봇은 WebRTC 플랫폼을 통해 원격 제어가 가능하도록 다관절 로봇팔을 이용해 의사의 관찰 하에 환자에게 다가가 환자의 상태를 확인할 수 있다. 환자로부터 원격으로 얻은 영상 정보, 음성 정보, 환자의 심음 및 기타 데이터를 WebRTC 플랫폼을 통해 의사에게 전송할 수 있다. 개발된 시스템은 의사가 참석할 수 없는 다양한 장소에 적용이 가능하다.

Ⅰ. Introduction

Due to the development of IT(Information Technology) and 5G(5th Generation) communication technology, the interface between humans and remote computers could be possible. With this, the foundation for providing remote medical services such as telemedicine services through video communication are being laid. [1],[2].

The telemedicine is a service that allows you to receive medical services anywhere outside the hospital, measure biometric information, and consult with a doctor without going to the hospital. As life expectancy increases, the elderly population increases, and the demand for remote management and examination of patients without going to a hospital is increasing due to inconvenient movements. The system also allows doctors to perform remote treatment without seeing patients in person staying outside of metropolitan area or far from hospital or minimize infection(such as COVID-19 [3] by avoiding direct contact to the patients. The Fig.1 shows a basics of telemedicine system.

그림 1. 원격의료 서비스 분류

Fig. 1. Telemedicine service classification

Since the telemedicine system is used in a limited space, effective measurement or diagnosis is difficult. A minimum examination and a stethoscope are required for a doctor to accurately judge a patient's condition. Observing the patient's trauma is possible through a remote imaging robot. However, it is difficult to consult with patients with heart disease because they cannot auscultate the heart sound remotely. It should be possible to measure heart sounds during consultations between doctors and patients and transmit them in real time. The effective remote medical treatment could be possible if the system that can measure the actual sound and video images are transmitted in real time.

In this paper, a video robot system that can be operated remotely by a doctor to check the patient's condition is proposed. The doctor can conveniently control the remote examination robot using a smartphone ore a PC(Personal computer), and it is configured to move freely in a designated ward as it is capable of autonomous driving [4] The system is devised, designed, and implemented for a telemedicine consultation robot with a built-in micro-camera that transmits heart sounds through the Internet and can check the patient's mouth or attribution. For video communication, the WebRTC(Web Real-Time Communication) [5] platform, which is more effective than the existing P2P(Peer to Peer) communication, is used to separate and transmit image information, audio and data information [6],[7].

The contents are organized as follows. In section 2, structure of the remote medical consultation robot system is given. In section 3, the performance experiment of the implemented system is shown. The last section draws a conclusion.

Ⅱ. Structure of the telemedicine robot system

2-1 Telemedicine system configuration

The developed system for remote medical consultation consists of a robot arm, a body with wheel base and display part. The robot arm helps examination and auscultation. Fig. 2 shows the developed system configuration.

그림 2. 개발된 시스템의 구성

Fig. 2. Configuration of a developed system

The robot arm can be moved simply by X and Y coordinates. With the X axis, the robot arm can be expanded and contracted horizontally up to a certain distance in a linear way.

The built-in functions of the robot arm are a non-contact auscultation function and a built-in endoscope to enlarge the patient's mouth, ear, and skin condition. Fig. 3 shows the structure of the robot arm and the possible structure for auscultation, examination, and body temperature. Fig. 4 shows the arm structure of the autonomous driving rounding robot for telemedicine.

그림 3. 로봇 팔(청진, 검사, 온도계)의 구조 및 측정 기능

Fig. 3. Structure and measurement function of robot arm (auscultation, examination, thermometer).

그림 4. 원격진료용 로봇팔 구조

Fig. 4. Robot arm structure for telemedicine

The designed robot has two wheels and a built in 12-inch tablet PC monitor with a height of 120 cm. The screen is designed to be moved by Fan/Tilt. Fig. 5 shows the structure and function of the fabricated remote video robot. The attached robot arm makes possible to examination and auscultation remotely.

그림 5. 개발된 원격진료용 로봇팔 외형

Fig. 5. Appearance of developed telemedicine robot

The base part of the robot is consisting of 2-axis DC(Direct Current) motors and ball casters. The two motors are controlled by the feedback control algorithm. Fig. 6 shows the robot base for the developed telemedicine robot. With this structure, the robot can be moved forward, backward and rotation.

그림 6. 원격진료용 로봇의 이동부

Fig. 6. Moving parts of the telemedicine robot

The sensor unit of the Mobile Robot is equipped with an IR(Infrared) sensor, an ultrasonic sensor, and a LiDAR(Light Detection and Ranging) to enable the robot to avoid collisions in real-time during autonomous driving.

그림 7. 라이다맵에 의한 자율주행 구조

Fig. 7. Self-driving structure by LiDAR map

A monitor unit for video communication is designed to allow the fan/tilt moves by using a servomotor connected to the monitor. This structure makes possible a doctor can remotely operate the monitor to check the patient's condition.

그림 8. 화상 통신용 모니터 유닛

Fig. 8. Monitor unit for video communication

The robot body is designed according to the world-class standards for safe movement of the robot. The robot motion was confirmed through experiments. Table 1 show the final specifications reached in this development.

표 1. 개발된 시스템의 정량적 수준

Table 1. The quantitative level of developed system

2-2 Electronic stethoscope system

Auscultation is a diagnostic process that is performed to find out defects in the digestive system, abnormal heart function. This is confirmed by listening to heart sounds or lung sounds using a stethoscope. A doctor places the head of a stethoscope at the heart, lungs, or intestines, which are the source of the sound, and listens the sound and examines it. Fig. 9 shows the structure of the stethoscope. However, the conventional stethoscope is hard to capture the sounds and transmit them through the communication line as the sound source captured by the diaphragm goes up the tube.

그림 9. 청진기의 구조

Fig. 9. Structure of a stethoscope

Recently, electronic stethoscopes have been distributed. Using the electronic stethoscopes, more clear sound can be captured, stored, and transmitted in real-time after filtering and amplification. Fig. 10 shows the structure in which an electronic stethoscope is connected to a tablet. This system is mounted on the developed robot to transmit the sound signal of the electronic stethoscope to the doctor.

그림 10. 전자 청진기의 구조 및 제어 순서

Fig. 10. Structure and control sequence of the electronic stethoscope

2-3 Miniature endoscope camera system

A miniature endoscope camera is mounted on the developed telemedicine robot arm. A doctor can remotely operate the robot arm to check the patient's skin, mouth, and ears while performing examinations. The camera image/video data is transferred to a doctor in real-time. Fig. 11 shows the process of confirming the attribution with an endoscope.

그림 11. 소형 내시경 카메라의 측정

Fig. 11. Measurement of the miniature endoscope camera

2-4 Software for telemedicine robot system

It is important to obtain accurate patient care information in real-time for treatment between a patient and a doctor with a remote video robot. Hardware configuration is important for accurate medical information, and software configuration is also important. To control the robot all the control signals need to be sent in real-time. Also, the obtained sound, image and video information about the patient need to be sent to a doctor in real-time. In this study, the development is performed by securing secure communication using the WebRTC platform. Fig.12 shows the structure of WebRTC [8].

그림 12. WebRTC의 아키텍처

Fig. 12. Architecture of WebRTC

Browsers and mobile applications use WebRTC through a simple API(Application Programming Interface) to consume audio and video RTC(Real-Time Communication). The WebRTC component has been optimized to best serve this purpose. WebRTC-based web applications provide rich real-time multimedia features on the web (think video chat) without plug-ins, downloads or installations, and help build a powerful WebRTC platform that works across multiple web browsers and across multiple platforms.

Fig. 13 shows the process of sending video and audio signals through WebRTC Peer to Peer Communication. WebRTC is in-built in Firefox browser. It has improved video and audio streaming and VP8 video codec and OPUS audio codec provides much less data transmission without packet loss. WebRTC based Peer to Peer communication can be run from Firefox browser without any plugin or software installation. Using WebRTC, audio and video streaming can be done via local networks.

그림 13. WebRTC 피어 투 피어 통신의 작업 프로세스 (a) 지원되는 브라우저 (b) WebRTC 구조

Fig. 13. Working process of WebRTC peer to peer communication (a) Supported browsers (b) WebRTC structure

For remote video control through a smart phone, a remote video experiment is performed between a smart phone and a robot. Fig. 14 show the experimental process of remote video control between the robot and the smartphone.

그림 14. 로봇에 장착 후 이미지 전송 실험

Fig. 14. Image transmission test after mounting on a robot

Ⅲ. Implementation and test of the telemedicine robot system

In this section, the performance experiment of the implemented system is shown. The experiment mainly consisted of the test for the robot control unit, the auscultation, and for remote medical examination. The test result of the control experiments for auscultation, video communication control is also performed. Then, the experimental process and results for the evaluation items of the developed technology are discussed.

3-1 Control board and image transmission test

To test the robot control unit, a program written in C language using ARM GCC was downloaded to the Cortex M4 board. Then, after Bluetooth pairing, commands are set up to 0x31, 0x32, 0x34 ... 0x38, and a program is executed to drive the servo motor using data such as 0x40, 0x41, 0x42, 0x43. Then, a motion control test is performed using a remote-control application. To evaluate the developed system, tests are conducted under the experimental environmental conditions: temperature: (20 ± 2) ℃, humidity: (56 ± 5) % R.H. The test results are summarized in Table 2. Fig. 14 shows the test scene.

표 2. 시험항목 및 시험방법

Table 2. Test items and test methods

Tests are carried out according to the evaluation method for each item, and among the evaluation items, the sound and speed of the robot and the frequency measurement of the sound with a stethoscope are summarized in Table 2 and Table 3. The speaker's maximum output noise is measured during a video call with a tablet PC. The noise measurement room (width x length x height) is composed of 4.98 x 3.1 x 3,42 m, and the distance to the robot is measured at a distance of 1 m from the front. The noise meter is performed with NA-27 (RION), and there are 10 measurement functions, so it is possible to measure conveniently. The experiment is performed three times and the average value is obtained.

표 3. 테스트 결과

Table 3. Test results

In order to facilitate the interaction between the patient and the doctor through the robot, the robot's sensor reaction speed must be fast for the doctor to interact and the remote robot's environmental recognition function. A stopwatch was used as the measuring device, and the HS-6 (CASIO) device was used. The test result of stopping time after reaction of object detection sensor 0.17 ~ 0.22 seconds.

3-2 Measurement with endoscope camera and stethoscope sound storage history

Because the distance to the patient for examination should be measured while maintaining a distance of 10 cm through the ultrasonic sensor built into the robot arm of the remote examination robot, the change in the distance during measurement was measured and confirmed. The distance measurement was measured three times using a Vernier caliper to obtain an average value. The result of diametrical distance measurement of robot arm is 102.6 mm.

A function to receive the auscultation sound transmitted from the telemedicine robot and to store and reproduce the auscultation sound on the tablet PC was added. Tests for storing and playing auscultation sounds on the tablet screen are performed. Since the auscultation sound must be transmitted and stored, it is encoded in 8000 Hz 8bit mono type and is configured to provide playback and recording functions of the auscultation sound received by the doctor. The auscultation sound transmitted from the patient is stored on the doctor's monitor and can be played again, so that the history of the image can be managed. Through the test, it is confirmed that the remote imaging robot can hear auscultation sound and make a diagnosis. In addition, since images are transmitted using WebRTC, images of 20 frames or more were acquired and telemedicine are performed.

Ⅳ. Conclusions

In this study, a telemedicine robot that can perform examination and auscultation remotely is developed. The robot is configured to measure the patient's condition and perform examination even if the doctor and the patient are far apart. For the configuration of the video system, real-time video transmission is implemented using the H263 codec, which is an existing video communication. Unlike the method that provides a login function as a server agent using a TCP socket, when using the webRTC platform, the video signal, audio signal, and control signal are transmitted separately, so the video is transmitted stably at a transmission rate of more than 20 frames per second without interruption. A miniature endoscope camera for examination is installed to inspect the condition of the patient's mouth or skin. An electronic stethoscope system is installed so that the doctor can check the heart sound in real-time while consulting with the patient and receive the diagnosis.

Using the system implemented in this study, it is thought that monitoring and telemedicine counseling will be possible for the elderly at home or for patients discharged from hospital after heart disease surgery. As the sound quality of the minimum auscultation sound required for accurate diagnosis by medical staff must be guaranteed, in a future research project, the sound quality before and after transmission of the auscultation sound is compared to improve the performance by conducting continuous experiments.

For the accurate control of a robot arm mounted on a remote medical robot, additional research such as interaction that can safely control a remote robot using various sensors and multi-touch technology mounted on a smartphone or tablet PC, and environmental recognition technology of a remote robot may be required.