Browse > Article
http://dx.doi.org/10.14372/IEMEK.2022.17.4.191

Behavior Pattern Prediction Algorithm Based on 2D Pose Estimation and LSTM from Videos  

Choi, Jiho (Jeonbuk National University)
Hwang, Gyutae (Jeonbuk National University)
Lee, Sang Jun (Jeonbuk National University)
Publication Information
IEMEK Journal of Embedded Systems and Applications / v.17, no.4, 2022 , pp. 191-197 More about this Journal
Abstract
This study proposes an image-based Pose Intention Network (PIN) algorithm for rehabilitation via patients' intentions. The purpose of the PIN algorithm is for enabling an active rehabilitation exercise, which is implemented by estimating the patient's motion and classifying the intention. Existing rehabilitation involves the inconvenience of attaching a sensor directly to the patient's skin. In addition, the rehabilitation device moves the patient, which is a passive rehabilitation method. Our algorithm consists of two steps. First, we estimate the user's joint position through the OpenPose algorithm, which is efficient in estimating 2D human pose in an image. Second, an intention classifier is constructed for classifying the motions into three categories, and a sequence of images including joint information is used as input. The intention network also learns correlations between joints and changes in joints over a short period of time, which can be easily used to determine the intention of the motion. To implement the proposed algorithm and conduct real-world experiments, we collected our own dataset, which is composed of videos of three classes. The network is trained using short segment clips of the video. Experimental results demonstrate that the proposed algorithm is effective for classifying intentions based on a short video clip.
Keywords
Rehabilitation exercise; Behavior pattern; Human pose estimation; Image sequence; Deep learning;
Citations & Related Records
Times Cited By KSCI : 2  (Citation Analysis)
연도 인용수 순위
1 J. Ajay, C. Song, A. Wang, J. Langan, Z. Li, W. Xu, "A Pervasive and Sensor-free Deep Learning System for Parkinsonian Gait Analysis," In Proceedings of 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 108-111, 2018.
2 J. Wang, S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, L. Shao, "Deep 3D Human Pose Estimation: A Review," Computer Vision and Image Understanding, Vol. 210, pp. 103225, 2021.
3 Y. He, R. Yan, K., Fragkiadaki, S. I. Yu, "Epipolar Transformers," In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7779-7788, 2020.
4 K. Iskakov, E. Iskakov, V. Lempitsky, Y. Malkov, "Learnable Triangulation of Human Pose," In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7718-7727, 2019.
5 S. Hochreiter, J. Schmidhuber, "Long Short-Term Memory," Neural computation, Vol. 9, No. 8, pp. 1735-1780, 1997.   DOI
6 Z. Cao, T. Simon, S. E. Wei, Y. Sheikh, "Realtime Multi-person 2d Pose Estimation Using Part Affinity Fields," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291-7299, 2017.
7 A. Toshev, C. Szegedy, "Deeppose: Human Pose Estimation Via Deep Neural Networks," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1653-1660, 2014.
8 A. Kos, A. Umek, "Wearable Sensor Devices for Prevention and Rehabilitation in Healthcare: Swimming Exercise with Real-time Therapist Feedback," IEEE Internet of Things Journal, Vol. 6, No. 2, pp. 1331-1341, 2018.   DOI
9 M. Panwar, D. Biswas, H. Bajaj, M. Jobges, R. Turk, K. Maharatna, A. Acharyya, "Rehab-net: Deep Learning Framework for Arm Movement Classification Using Wearable Sensors for Stroke Rehabilitation," IEEE Transactions on Biomedical Engineering, Vol. 66, No. 11, pp. 3026-3037, 2019.   DOI
10 L. M. S. D. Nascimento, L. V. Bonfati, M. L. B. Freitas, J. J. A. Mendes Junior, H. V. Siqueira, S. L. Stevan Jr, "Sensors and Systems for Physical Rehabilitation and Health Monitoring-A Review," Sensors, Vol. 20, No. 15, pp. 4063, 2020.
11 Z. A. Zhu, Y. C. Lu, C. H. You, C. K. Chiang, "Deep Learning for Sensor-based Rehabilitation Exercise Recognition and Evaluation," Sensors, Vol. 19, No. 4, pp. 887, 2019.   DOI
12 T. Zhong, D. Li, J. Wang, J. Xu, Z. An, Y. Zhu, "Fusion Learning for sEMG Recognition of Multiple Upper-limb Rehabilitation Movements," Sensors, Vol. 21, No. 16, pp. 5385, 2021.
13 S. Qiu, L. Liu, H. Zhao, Z. Wang, Y. Jiang, "MEMS Inertial Sensors Based Gait Analysis for Rehabilitation Assessment via Multi-sensor Fusion," Micromachines, Vol. 9, No. 9, pp. 442, 2018.