IoT-based Architecture and Implementation for Automatic Shock Treatment

Abstract

The Internet of Things (IoT) is being used in a wide variety of fields due to the recent 4th industrial revolution. In particular, research is being conducted that combines IoT with the medical field such as telemedicine. Among them, the field of shock detection is a big issue in the medical field because the causes of shock are diverse, treatments are very complex, and require a high level of medical knowledge and experience. The transmission of infectious diseases is common when treating critically ill patients, especially patients with shock. Thus, to effectively care for shock patients, we propose an architecture that continuously monitors the patient's condition, and automatically recommends a drug injection treatment according to the patient's shock condition. The patient's hemodynamic information is continuously monitored, and the patient's shock generation information is recorded periodically. With the recorded patient information, the patient's condition is determined and automatically injected with necessary medication. The medical team can find out whether the patient's condition has improved by checking the recorded information through web applications. The study can help relieve the shortage of medical personnel and help prevent transmission of infectious disease in medical staff. We look forward to playing a role in helping medical staff by making recommendations for the diagnosis and treatment of complex and difficult shocks.

1. Introduction

We live in a generation where rapid changes are taking place. Internet of Things (IoT) is being used and researched in various fields such as smart factories and autonomous vehicles. Among them, research is being conducted in fields such as CPS (Cyber-Physical Systems), which combine IoT and medical services, and telemedicine [1-4]. Among them, Telemedicine is a technology that makes it possible to receive medical services without the barrier of distance by using telecommunications and information technology to provide clinical health care from a distance [5]. Telemedicine is mainly used in environments where it is difficult to continuously receive medical services. Telemedicine can be used in a variety of medical fields, e.g., Parkinson’s disease, [6], surgery using robots, digital monitoring devices, and physical therapy through application combinations [7,8].

The process of diagnosing a patient's shock in a hospital is very complex. In order to treat the patient's shock, the cause of the patient's shock must be identified. Healthcare providers should set and follow target indicators such as blood pressure, pulse rate, urine output, base deficiency, and consciousness. The goal of the treatment of patients in shock is to maintain adequate perfusion to provide an oxygen equivalent to the tissue's oxygen demand. This requires a lot of medical staff and time because it has the characteristic of continuously monitoring the patient's condition.

The monitoring system is very important in solving the problem of lack of medical resources in developing countries or remote areas that lack experience in treating critically ill patients. It is necessary to monitor medical data values to determine the cause of shock and provide appropriate treatment. In addition, the patient's shock state changes every moment, and there is a possibility of drug change or re-diagnosis, so continuous monitoring is necessary. The hemodynamic monitoring equipment supports the medical professionals to cope with various diseases by outputting the aforementioned medical data values in real time, and it is possible to check the occurrence of shock and the type of shock in real time [9]. As such, hemodynamic monitoring can solve the problem of shortage of medical staff and provide rapid treatment of shock patients. In addition, in the process of treating a patient, contact between the patient and the medical staff frequently occurs. Recently, the spread of COVID-19 (COVID-19) and super bacteria is often caused by contact with patients by medical staff. If the hemodynamic monitoring and automatic shock treatment system are applied to the hospital, the previous problem can be solved by minimizing the contact between the patient and the medical staff.

In this paper, the treatment algorithm according to the change in the value of the medical data variable that can be checked through the hemodynamic monitoring system is applied to the IoT, and based on this, the overall architecture is conceived to automatically manage the treatment method according to the change of the patient’s condition through the infusion pump. First, a treatment algorithm suitable for each patient’s situation is developed, and each situation is judged through the hemodynamic monitoring system and an appropriate command is transmitted to the infusion pump. The software has been developed so that the received command can be properly executed by the infusion pump, and any changes of infusion pump rate is shared with the medical staff using IoT. software architecture continuously monitors hemodynamics and repeats the above process until the patient stabilizes. We envision that our architecture can help patients with shock and solve the shortage of medical staff even in environments where continuous medical care is difficult.

The remainder of this paper is organized as follows. In Section 2, related research and brief explanations are provided. Section 3 describes the architecture and software algorithm for the proposed model. Section 4 describes how to implement the proposed algorithm. Finally, Section 5 is the conclusion.

2. Related Work

2.1 Shock Treatment

Shock is a condition in which blood flow is insufficient in body tissues due to circulatory problems [10]. Shock can occur in various ways. The treatment of shock is important at what levels of abnormalities the patient shows [11]. Shock can be classified using hemodynamic information. For example, if blood pressure decreases in hemodynamic information, cardiac output per minute is normal, but the systemic vascular index decreases, a vasopressor is administered. Conversely, when blood pressure decreases, the cardiac output per minute and systemic vascular index are normal, but the intravascular volume decreases, the rate of fluid administration increases. Hemodynamics monitoring watches the patient’s various parameters including blood pressure, pulse rate, cardiac output, vascular resistance and intravascular volume. Hemodynamics monitoring helps identify the patient’s shock status [9]. Equipment for hemodynamic monitoring has been developed in a variety of ways. Monitoring equipment includes equipment such as the Medis’ BioZ [12] and the Philips’ SureSigns VM6 [13]. Above all, since shock is a disease that is directly related to life, time is very important in shock treatment. The importance of constant monitoring and detection of abnormal conditions is much higher than that of other diseases, because the patient's emergency can be overcome only when appropriate treatment is given at the right time. Various studies are currently underway to quickly detect such a shock situation. Two studies [14, 15] predicted hemodynamic shock by analyzing and learning image data. Even today, there are various methods for various shocks. However, there is a trade-off between accuracy and processing speed, and these factors often lead to poor accuracy or unclear results. On the other hand, the method presented in this paper recognizes and uses bio-signals directly input from the monitoring device, so the accuracy is high and the process is simpler than other studies. Therefore, it is possible to detect shock as soon as possible and give prompt and accurate notification to the medical staff.

2.2 Internet of Things (IoT)

IoT refers to the junction of exchanging data by connecting various objects such as smart light bulbs, smart locks, IP cameras, and alarm clocks to the Internet [16]. IoT function is currently being used in many fields. There is also a paper making a system for storing big data in cloud computing using the IoT system [17]. In addition, Lenovo has also developed an IoT system that helps healthcare by using exercise data by inserting a chip into a smart shoe [18].

Integrating image processing, computer vision, and networking frameworks [19] will help develop new and challenging scientific research areas at the intersection of video, infrared, microphone and network technologies. and according to [20-22], already proposed many IoT services such as smart parking, precision agriculture, and water usage management. In addition, extensive research has been conducted into the use of IoT system such as traffic congestion minimization [23], structure health monitoring [24], crash-avoiding cars [25], and smart grids [26].

IoT techniques have been applied to various medical fields. As IoT is applied to the medical field, treatment can be received in places other than hospitals. In several studies, IoT healthcare systems have been developed for specific purposes, such as rehabilitation and diabetes management. In [27], a system that creates a recurrence plan tailored to individuals based on symptoms was developed, and to perform this, the patient’s condition was compared with the data of the previous patient’s symptoms. This system is inconvenient for a doctor to manually enter symptoms and approve recommended treatment. [28] describes a system for monitoring patients suffering from Parkinson’s disease. The study says that wearable sensors for observing gait patterns, tremors, and general activity levels can be used in conjunction with vision-based technology (i.e., cameras) around the house to monitor the progression of Parkinson’s disease. In [29], a system for monitoring blood sugar levels in diabetic patients was proposed. This system is cumbersome, requiring the patient to manually measure blood sugar levels at set intervals. This system takes into account two types of blood sugar abnormalities: an abnormal blood sugar level and a missing blood sugar level. The system analyzes the severity of the anomaly and decides who to notify. The system can notify caregivers, family members, or emergency care providers such as doctors. Although this system is feasible, there is an inconvenience of having to periodically check the blood sugar level by the patient. [30] has developed a system for detecting heart attacks. This system was built using an antenna. The ECG sensor detects cardiac activity that is processed by a microcontroller. This information is displayed on the user’s smartphone via Bluetooth. However, this development is a system that only detects heart attacks and cannot treat them. These developments only serve as an assistant to medical staff, and they cannot sufficiently perform shock treatment or transmission prevention by medical staff in place of medical staff. We devised the architecture in this paper to improve these points.

2.3 Optical Character Recognition (OCR)

Patient monitoring equipment cannot directly import data because it uses equipment that has already been developed. Therefore, we used a method of taking a screen display on the monitoring equipment and extracting patient information from the monitoring equipment using optical character recognition (OCR). Computer vision refers to a method of collecting, processing, analyzing, and understanding digital images and extracting high-dimensional data from the real world. Image processing can be implemented in a variety of ways, but the most widely used method is the one using AI [31]. OCR refers to a method of recognizing text from an image and converting it into digital data. OCR can be used for cognitive computing, machine translation, text-to-speech, and text mining by electronically editing and storing text and displaying it online. OCR is also a field of pattern recognition and artificial intelligence research. OCR is used in various fields. Representatively, it is Traffic Sign Recognition (TSR) [32]. TSR is a technology that allows vehicles to recognize traffic signs placed on the road, such as speed limits or children or moving forward.

We use hemodynamic monitoring equipment to monitor the patient’s condition in real time. When the screen of the device is photographed with a camera and transmitted to a computer, OCR is performed based on the photographed picture. It collects the printed information. Based on the collected information, an IoT system was conceived to perform shock treatments necessary for patients.

3. Proposed Architecture

The architecture of the proposed model can be seen in Fig. 1 which can be largely divided into three areas: image recognition, decision of shock treatment, and control of infusion pump. The patient’s condition is checked through hemodynamic monitoring. The identified information is transferred over the network to the server and stored in the database. Based on the stored information, the server diagnoses the necessary drugs according to the patient’s condition. The determined information is passed to the infusion pump, and the infusion pump injects drugs into the patient. Details are described below for each part.

Fig. 1. System architecture of the proposed work

3.1 OCR for hemodynamic Values

In this section, we describe recognition of the hemodynamic values using Optical Character Recognition (OCR). Fig. 2. describes the process of character detection and content recognition using OCR. First, the chart page of the device is input in the form of an image from the hemodynamic monitoring device through the VGA capture device. The input image is converted to text through OCR, and EasyOCR was used for this process [33]. EasyOCR is an OCR algorithm based on Deep Learning. First, NAVER-Clova AI’s Character Region Awareness for Text (CRAFT) detection model was used in the text detection part. Similar to the Generative Adversarial Network (GAN), CRAFT is a model that learns the shape of a character image by using real and created images [34]. For text detection, a total of about 10,000 datasets such as ICDAR2013, ICDAR2015, ICDAR2017, MSRA-TD500, TotalText, and CTW-1500 were used. After text detection, a hybrid convolutional recurrent neural network (CRNN) model, in which a convolution layer and a recurrent layer are combined, was used to recognize the contents of the detected text box. CRNN is a hybrid deep learning model consisting of a convolutional layer for feature extraction and learning from text in an image form, and a recurrent layer for labeling sequential text [35]. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. Datasets such as IC03, IC13, IIIT5k, and SVT were used, and images of about 5,000 words were used as the data. Through the above EasyOCR Process, the screen of the actual monitoring device is received as an image, and the coordinates where the text is located on the image, the contents of the text, and the probability can be known. The first study uses EasyOCR for character recognition of high-resolution images restored with super resolution[36]. Second, the mail contents were detected using EasyOCR to detect spam[37]. Finally, the third study used EasyOCR for vehicle license plate recognition. Through the above EasyOCR Process, the screen of the actual monitoring device is received as an image, and the coordinates where the text is located on the image, the contents of the text, and the probability can be known.

Fig. 2. OCR Process for Hemodynamic Values Detection

3.2 Decision of Shock Treatment

In this part, based on the monitoring data, it is determined which drugs to inject according to the patient’s condition. The hemodynamic monitoring device collects parameters: HR, BP, MAP, SpO2, Resp, TFC, TFCI, SVV, SV, SI, CO, CI, EF, EDV, EDI, SI, CI, ACI, LCWI, LCW, LSWI, SVRI., SVR, SSVRI, PP, SVRI. The parameters for diagnosing shock are shown in Table 1.

Table 1. Cardioscreen Series Parameters

In order to diagnose shock, hemodynamic information stored in the database is first retrieved and the server diagnoses the shock in the following order:

● Step 1: By analyzing the hemodynamic information, only the information necessary for the occurrence of shock is extracted.

● Step 2: Using the extracted information, it is determined whether the hemodynamic information is in the normal range.

● Step 3: Determine which of three drugs the patient needs according to the identified content.

● Step 4: The determined command is transmitted to the microcontroller connected to the infusion pump through the wireless Internet.

For the shock detection SI, ACI, SVRI, and SVV parameters are needed. There are three types of drugs: general fluid, dobutamine, and norepinephrine. The algorithm for injecting drugs can be seen in Fig. 3.

Fig. 3. Flow chart of the drug infusion algorithm. [38]

Check if the SI value is within the normal range. If it is in the normal range, check whether the SVRI value is within the normal range. If it is in the normal range, the algorithm is terminated, and if it is decreased, norepinephrine is injected. If the SI value is decreased, the normal range of the LCWI value is determined. If the LCWI value is in the normal range, the normal range of the SVV is determined. If it is increased, fluid is injected until normal blood pressure is achieved. If the LCWI value is decreased and SVV value is within the normal range, dobutamine is injected.

After determining which drug to be administered, how much of the drug to be injected must be determined. The amount of drug is determined according to the value of each parameter.

3.3 Control of Infusion Pumps

In our architecture, three infusion pumps, as shown in Fig. 4. (a), are used for an automated shock treatment. We use only the main parts and control the infusion pump by adding an embedded board, i.e., Arduino. Arduino plays a role in controlling the infusion pump by connecting the infusion pump’s main parts. All three infusion pumps have the same configuration.

Fig. 4. Infusion Pump Hardware Prototype.

Fig. 4. (b) shows a state flow of the infusion pump. The operation of the infusion pump consists of three states: Idle, Infusion, and Empty Alarm. The state of the infusion pump changes according to internal commands (start and stop), physical switches (empty reservoir and low reservoir), and a physical button (bolus request). For example, an infusion pump starts in an Idle state waiting for a trigger input such as a start command or a bolus request. When the trigger input is received, the infusion pump activates the drug infusion until it receives an interrupt input, such as a stop command. The active state of the infusion pump indicates the Infusion state. The Empty Alarm state refers to a condition in which the drug needs to be changed due to low drug volume. A buzzer alarm is activated when a reservoir switch turns on until an empty reservoir switch is triggered or a stop command is received. For automated shock treatment, a central server selects the appropriate infusion pump and controls the infusion pump’s operations through internal commands.

4. Implementation of the Architecture

Experiments are divided into three main categories. The first is to recognize and store the patient’s data in hemodynamic monitoring. The second part is knowing the patient’s condition using stored data and choosing the appropriate medication. The third is the part when the selected drug is delivered to the infusion pump control to inject the drug into the patient. The detailed experimental process is described below.

4.1 Hemodynamic Monitor Recognition

Algorithm. 1 Parsing Process for Recognition of Hemodynamic Value using EasyOCR

Require: Image data from the VGA Capture Device

Use EasyOCR for text detection (recognition) from the image

img_RGB = Image form data from the capture device

csv_Hemo = Csv data includes information of hemodynamic values to detect

items_Hemo = Size of the data detected from the monitoring device

while img_RGB is not NULL do

img_Gray = Convert img_RGB to grayscale

img_Gray = EasyOCR.readText(img_Gray)

for idx = 0, 1, ..., items_Hemo do

if textResult is in csvHemo then

bufsend.append(textresult)

end if

end for

Send buf_send to server

end while

Algorithm 1 describes the process of extracting hemodynamic values using EasyOCR from the image data through the VGA capture device. First, the image array input through the hemodynamic device and capture device is converted to grayscale. At this time, OpenCV 3.4.0 was used for the grayscale conversion. The transformed array uses EasyOCR to detect and recognize text in the image. When performing recognition, only necessary parameters are recognized by parsing the information from a csv file in which only information of parameters necessary for recognition are specified. Recognized parameters and values are concatenated to a buffer and transmitted to the server for UI and data storage.

Fig. 5 shows the patient’s hemodynamic monitoring information sent to the server. On this screen, the SI, LCWI, SVRI, SVV values are extracted as required for shock diagnosis. Hemodynamics monitors used Medis’s products [13]. Fig. 6 shows that the patient’s data is extracted from the monitor once a second using OCR.

Fig. 5. Hemodynamic monitoring

Fig. 6. Real-time OCR recognition

The received information is stored directly in the database. Each patient’s information is stored individually in the database in the patient’s name, patient’s ID, hemodynamic information, time, etc.

Patient information stored in the database can also be viewed directly on the web page as seen in Fig. 7. Medical staff can see the patient’s name and information are stored for each patient. Fig. 7 shows a patient’s hemodynamics information data.

Fig. 7. Patient's data in webpage.

4.2 Drug Determination

Fig. 8 shows that hemodynamic data stored in the database was retrieved from the main server. Based on this data, the patient’s shock is determined. If a shock occurs, the algorithm determines drug selection and injection volume. The determined drug and infusion amount are then stored in the database. The infusion data record time, drugs injected, injection volume and injection velocity.

Fig. 8. Drug Selection

The determined drug information uses a wireless Internet network to communicate commands to the microcontroller of the infusion pump. Upon receipt of the command, the microcontroller adjusts the cylinder according to the command and injects drugs into the patient. Data changed after drug injection is stored in the database as result data. Monitor how the hemodynamic data of the patient changes after drug injection. Result data is stored as result data whether drug injection was successful or failed and if hemodynamic information was normalized after injection.

4.3 Implementation Results

In Fig. 9, the red graph is the interval in which the Infusion Station receives the data, and the green graph shows the interval in which the EasyOCR processes the data.

Fig. 9. EasyOCR processing time and infusion station receive time variation trend.

Table 2. Implementation Environment

The average interval is 600ms to 750ms, and the average time it takes to send and receive data to and from the server is 500ms, and it takes 1.1sec to 1.25sec for hemodynamic data recognized by the learning model to be loaded into the database.

The experiment was conducted in two environments. In experiment 1, when the program was implemented on a GTX1050 and i7 environment, it took 2 seconds for the entire program to be executed. In the second experiment, when the program was implemented on a Titan and Ryzen 7, it was shortened to 1.15 seconds before the program was executed. As a result, we implemented a system that can check the patient’s condition and enable treatment in a short time. However, it was confirmed that there is a difference in speed depending on the computer hardware.

5. Discussion and Conclusion

We proposed an architecture that can detect shock and treat it according to the type of shock by combining the shock detection field and IoT to prevent a shortage of human resources and transmission of disease to and from medical staff in the medical field. A thorough and systematic overview of the techniques involved in each component of the proposed model was presented. Hemodynamic data was monitored, patients’ shock status was checked, and treatment methods were presented and analyzed. Then, we compared the execution times of the architecture according to the environment of the hardware in terms of suitability for medical applications. We confirmed that the execution time of the system was different depending on the computer hardware. This technology further discussed the direction of research so that the type of shock can be accurately diagnosed using artificial intelligence and the drug can be used accordingly. However, some parameters, especially SVV are less accurate in patients with spontaneous breathing. This parameter is accurate and reliable only in patients who have applied a positive pressure ventilator with a constant and sufficient tidal volume and no arrhythmia, requiring careful interpretation. Finally, we concluded that existing IoT-based healthcare systems remain in the early stages of development. Currently, the data derived from the hemodynamic monitoring equipment has about 50 parameter values and includes information on various biosignals of the patient. The number of parameters covered in this paper is about 4 to 5, and monitoring is focused on specific diseases such as shock. In the future, it is necessary to consider the methodology of using all 50 parameters provided by the hemodynamic monitoring equipment to check and learn whether more various types of diseases occur and at the same time proceed with treatment.