Analysis of Abnormal Event Detection Research using Intelligent IoT Devices for Human Health Cares

Abstract

With the outbreak of COVID-19, non-face-to-face activities such as remote learning and telecommuting have increased rapidly. As a result, the number of people staying at home and the number of hours spent inside the house have also increased since the pandemic. Our team had previously worked on methods for detecting abnormal conditions in a person's health in various circumstances within the house by converging single sensor-based algorithms. In our previous research, we installed IoT sensors indoors to detect people emergency situations requiring aids, the scope of detection was limited to indoor space due to the limitation in sensors. In this study, we have come up with a system that integrates our previous study with a new method for detecting abnormal conditions in outdoor environments using outdoor security cameras and wearable devices. The proposed system enables users to be notified of emergency situations in both indoor and outdoor areas and respond to them as quickly as possible.

1. Introduction

Approximately 93.6% of the people responded that the time they spent staying home has increased since the outbreak of COVID-19 [1]. The populations previously involved in outdoor production activities are currently spending more time indoors as non-contact activities such as remote working or remote learning have been triggered due to COVID-19. According to the report published by the Bank of Korea [2], the number of individuals who will continue teleworking is expected to increase even after the COVID-19 crisis ends. As the time spent indoors increases, the risk of being exposed to dangers that may occur indoors also increases for residents. According to the 2020 Consumer Safety Report republished by the Korea Consumer Agency [3], the portion of safety accidents occurred indoors among all accidents occurring annually was 47.6% in 2017 and increased to 53.0% in 2018 and 55.5% in 2019. In particular, the highest percentage of indoor accidents occur among children and the elderly. In the case of emergency situations requiring assistance, IoT objects at home can detect abnormalities of humans and send alarms to external entities. 100 elderly were promptly rescued from an emergency through artificial intelligence (AI) speakers [4].

We conducted a study on determining abnormalities occurring indoors by integrating various devices such as AI speakers and indoor security cameras [5-13]. However, abnormalities occurring in a variety of outdoor places could not be detected in previous studies since the detection range of abnormalities occurring indoors was limited.

This study, therefore, proposes an algorithm and a system for detecting abnormalities of users in various places in both indoor and outdoor using wearable devices or outdoor security cameras in addition to detecting abnormalities of users in indoor spaces. In Chapter 2, our previous studies related to abnormality detection are explained as well as diverse algorithms for abnormality detection and the products for detecting emergency situations requiring help in outdoor environments. Chapter 3 provides an explanation of the solution for expanding the abnormality detection range that is limited for indoor spaces, and lastly, Chapter 4 concludes this paper.

2. Algorithm and System Research

2.1 Our Previous Indoor Healthcare Research

We conducted studies on detecting abnormalities in various situations by integrating different algorithms for detecting the movements of a person using a single sensor [5-13]. Indoor sensors including video, voice, and movement sensors collect data, and a pattern detection algorithm is applied to the collected data for detecting a person’s movements. Abnormalities are defined as when movements of a person are not detected for a certain period of time or abnormal movements different from usual patterns are detected.

We studied single sensor-based algorithms for detecting the movements of a person using various sensors. Vision sensor-based algorithms collect video data using vision sensors such as indoor security cameras, and object detection algorithms such as Faster R-CNN [14], YOLO [15, 16] are used for detecting persons or objects. Then, a person is determined to have moved if the bounding box coordinate values of a person detected by the object detection algorithm are changed. It is determined as abnormalities if there is no movement of a person for a certain period of time or if there is a pattern different from normal behaviors such as suddenly falling on the ground. Table 1 shows the average experiment performance of the vision sensor-based algorithm of our previous study.

(Table 1) Performance of the vision pattern detection algorithm

Activity sensor-based algorithms determine the movements of a person using an acceleration sensor equipped in smartphones. The acceleration sensor collects the movement of a smartphone as tri-axial data. Subsequently, the average value of tri-axial data collected by the pattern detection algorithm is calculated and saved in a window having the size of 5. When saving the values in the window array as the acceleration sensor moves, the maximum and minimum values of the middle value of the window array are defined as high peak and low peak. A person’s movement is determined based on the impact value calculated by finding the difference between high peak and low peak; it is determined as abnormalities when the impact value is measured to be below the threshold value for a certain period of time. Table 2 shows the average experiment performance of the activity sensor-based algorithm.

(Table 2) Performance of the activity pattern detection algorithm

LiDAR sensor-based algorithms use a 2D LiDAR sensor. The 2D LiDAR sensor measures the distance based on the time for a pulse signal emitted by a laser to reach the receiver. The measured distance data are applied to the point cloud-based algorithm for determining the location of an object, thus distinguishing walls from a person. If the central coordinate values of a point cloud set of an object determined as a person are the change, the respective person can be determined to have moved. Table 3 shows the experiment performance of the 2D LiDAR sensor-based algorithm in terms of accuracy of distinguishing wall and person.

(Table 3) Performance of the LiDAR pattern detection algorithm

A single sensor-based pattern detection algorithm can detect abnormalities only in certain circumstances depending on sensors. The vision sensor-based algorithm cannot detect situations outside the sensor range, the activity algorithm is significantly affected by the person’s possession of a smartphone, and the LiDAR algorithm cannot precisely measure distances in space with a large number of obstacles. In order to overcome such drawbacks, we proposed an algorithm and an intelligent system for detecting abnormalities in various situations by converging different sensor-based algorithms.

The convergence-based algorithm can detect abnormalities in more diverse situations than using single sensor-based algorithms. For example, when a resident lies down on a bed and use a smartphone, a video sensor-based algorithm determines that there is no movement and thus wrongly detect as an abnormality, while an activity sensor-based algorithm may decide a person is in a normal state based on the movement of a smartphone. However, various sensors and sensor-based algorithms in our previous studies have a limited range of detecting abnormalities within an indoor environment. Therefore, we studied the measures for detecting users’ abnormalities in indoor and outdoor environments and proposed an algorithm and a system for detecting abnormalities in all environments by converging previously studied methods.

2.2 Related Healthcare Research

The latest research has been conducted that expanded the findings of our previous studies on detecting abnormalities of a person [17-23]. One of the studies analyzed the patterns of a person’s daily life to determine abnormalities such as accidents [17]. Abnormal patterns of a person are analyzed using the sensor data collected from a variety of sensors in a smart home environment.

Other studies analyzed a person’s behavior based on the detection of an object or a person in vision-based video data [18, 19]. One of the studies proposed a system for monitoring and recognizing abnormal activities of the elderly based on a dynamic Bayesian network (DBN) [20]. The video data are processed using Otsu’s thresholding and ROI to extract features, and then the elderly’s abnormal behaviors such as falling or feeling pain are analyzed using a DBN model. Another study estimated a person’s pose based on the depth information in the data collected by an RGB-D camera and analyzed behaviors [21]. Humans are detected through depth information based object detection in the data collected by an RGB-D camera, and a human body model is represented with 14 key points using convolutional pose machines (CPM) to analyze human behaviors of a smart monitoring system.

Another research was conducted on recognizing a person’s activity based on the acceleration sensor data of a smartphone [22, 23]. Different behaviors of a person such as walking, running, sitting, and using stairs can be analyzed by learning the data of an acceleration sensor in a space reconfigured by a Gaussian mixture model.

One of the studies classified human and non-human classes from the data scanned by 2D LiDAR and followed the traces of the human class [24]. The Kalman filter was used for following the traces of an object detected as a person, and 15 activities that a resident can do within a house can be analyzed based on the collected trace data.

Other studies researched the detection of abnormal data using vision sensors such as outdoor security cameras [25-27]. One of the studies extracted features using a CNN in a video frame, and classified abnormal activity classes such as fighting and fainting and normal activity classes such as walking and standing while waiting.

2.3 The Existing Recent Outdoor Healthcare Research

In this chapter, various commercial products that detect unusual events of a person and provide help through healthcare are examined. There are a variety of devices available that can identify a person’s state and detect abnormal situations. Such devices can be categorized into portable or stationary depending on how the device is used. Portable devices that can be worn or carried by a person include wearable devices, while stationary devices include outdoor security cameras fixed at a certain place for detecting indoor and outdoor situations.

2.3.1 Wearable Devices for Healthcare

Wearable devices can be “worn” by a person and are integrated with daily items such as watches or eye glasses to provide a computing environment for users anywhere and anytime. In particular, wearable devices with various functions in the form of a watch have emerged as the smartwatch market grows rapidly. Table 4 shows the comparison of various wearable devices having healthcare functions.

(Figure 1) Sensors equipped in wearable devices

(Table 4) Performance comparison of wearable products

Ava [26] is a device worn on a wrist and collects the data of women’s infertility and pregnancy. Ovulation and menstruation cycles are estimated and the stress level is measured based on sleep state and heart rate to be delivered to a user through an application. Motiv [27] is a ring-type device in which a user’s heart rate is monitored while walking or sleeping, and the data are saved in the application and delivered to the user. Apple Watch [28] and FitBit [29] which are worn on a wrist are connected with a smartphone through Bluetooth and various information can be reported to users in real time. Furthermore, users can identify time, weather, and collected data through a simple display attached to wearable devices. TempTraq [30] is a patch-type, disposable thermometer which is typically used by attaching to the body of children aged less than five. This device not only measures the temperature, but also detects a user’s sleep state based on the measured temperature and monitors a heart rate to be reported through a mobile application. Owlet [31] can be worn as socks for monitoring the health state of children under the age of five and reporting to parents in real time through an application.

2.3.2 Vision Research for Healthcare

Cameras are used both indoors and outdoors for a variety of purposes in addition to security. Security cameras installed inside houses can detect emergency situations requiring help, while those installed at entrances can prevent crimes including trespassing. Table 5 shows the comparison and analysis of different types of security cameras.

(Table 5) Performance comparison of the latest outdoor security camera products

The latest security cameras monitor situations inside and outside houses by analyzing a person’s behavior or voice. Outdoor security camera and indoor security camera slightly differ in charging method or monitoring subject range, and thus are often used together. If a security camera detects a person and determines the person’s movement as abnormal, a user can check the situation in real time through a mobile application connected to the security camera and take necessary actions. Arlo Pro3 [32] and Wyze Cam Outdoor [33] are security cameras installed at entrances of houses where an alarm is sent to a user through a mobile application when a person is detected. Also, the microphone and speaker equipped in the products enable users to communicate with visitors. Google Nest Cam [34] discerns whether a visitor had visited before based on a face detection technology and even detects animals and vehicles in addition to humans. Logitech Circle View [35] and Maximus Camera Floodlight [36] can detect objects having the size of up to 70 feet using a floodlight attached to the devices, and automatically record and stream the video when a user is not monitoring the video.

3. Intelligence Solution for Healthcare

Our previous studies enabled ways to detect abnormal conditions in various situations by converging single sensor-based detection algorithms. However, the range of detection was limited to indoor space. We addressed this issue by integrating with recent studies that provided ways to detect emergency situations in outdoor areas using sensors such as outdoor security cameras and wearable devices. Table 6 shows whether the sensors were able to successfully detect abnormal conditions in various environments.

(Table 6) Performance of the emergency detection system in various environments

We divided possible emergency environments into three categories: indoors, semi-outdoors, and outdoors. For the purpose of our studies, we defined semi-outdoors as outdoor areas limited to the vicinity of the house such as the yard or parking lot. In the indoors, our system uses convergence algorithms and indoor security cameras, Activity, and LiDAR sensors to detect emergencies. In the semi-outdoors, outdoor security cameras are used to detect abnormal conditions in the user near the house, while outdoor security cameras are used to prevent emergency situations caused by outsiders. A notification is immediately sent to the user when a sensor picks up an unfamiliar outsider. In the outdoors, smartphones and wearable devices are used to monitor the user’s condition. The wearable devices collect an array of data such as the user’s movement and heart rate in real-time using embedded sensors and are thus extremely useful for not only detecting abnormal conditions but for users who require constant health monitoring from guardians.

While outdoor security cameras have limited range same as indoor sensors, wearable devices can be used to detect most emergencies in every environment. However, their performance is dependent on whether the user is currently wearing the device. To enable emergency detection in all environments, we devised an integrated system that utilizes indoor sensors, outdoor security cameras, and wearable devices. The proposed system expands the limited detection range of our previous study and helps users respond to emergencies both indoors and outdoors.

4. Conclusion

In this research, an algorithm and a system for detecting abnormalities in the outdoor environment were researched by expanding the findings of previous studies with a limited detection range of emergency situations requiring help. In previous studies, we examined convergence algorithm and system for detecting abnormalities by detecting a person’s movement using indoor sensors such as video, activity, and LiDAR and comparing the detected movement with daily patterns. The convergence algorithm proposed in previous studies can detect a variety of emergency situations occurring indoors but cannot detect abnormalities occurring outdoors due to a limited detection range. We compared the performance of outdoor security cameras and wearable devices that can detect emergency situations of users outdoors, and explore solutions for the limited detection range of the method researched in our previous studies. In the future, further research will be conducted on efficiently converging various sensors for detecting emergency situations to provide one solution for people requiring help in a variety of situations in all types of environments.