KIPS Transactions on Software and Data Engineering (정보처리학회논문지:소프트웨어 및 데이터공학)

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Fall Detection Based on 2-Stacked Bi-LSTM and Human-Skeleton Keypoints of RGBD Camera

RGBD 카메라 기반의 Human-Skeleton Keypoints와 2-Stacked Bi-LSTM 모델을 이용한 낙상 탐지

  • Received : 2021.09.30
  • Accepted : 2021.10.17
  • Published : 2021.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

In this study, we propose a method for detecting fall behavior using MS Kinect v2 RGBD Camera-based Human-Skeleton Keypoints and a 2-Stacked Bi-LSTM model. In previous studies, skeletal information was extracted from RGB images using a deep learning model such as OpenPose, and then recognition was performed using a recurrent neural network model such as LSTM and GRU. The proposed method receives skeletal information directly from the camera, extracts 2 time-series features of acceleration and distance, and then recognizes the fall behavior using the 2-Stacked Bi-LSTM model. The central joint was obtained for the major skeletons such as the shoulder, spine, and pelvis, and the movement acceleration and distance from the floor were proposed as features of the central joint. The extracted features were compared with models such as Stacked LSTM and Bi-LSTM, and improved detection performance compared to existing studies such as GRU and LSTM was demonstrated through experiments.

본 연구에서는 MS Kinect v2 RGBD 카메라 기반의 Human-Skeleton Keypoints와 2-Stacked Bi-LSTM 모델을 이용하여 낙상 행위를 탐지하는 방법을 제안한다. 기존의 연구는 RGB 영상에서 OpenPose 등의 딥러닝 모델을 이용하여 골격 정보를 추출한 후 LSTM, GRU 등의 순환신경망 모델을 이용해 인식을 수행하였다. 제안한 방법은 카메라로부터 골격정보를 바로 전달 받아 가속도 및 거리의 2개의 시계열 특징을 추출한 후 2-Stacked Bi-LSTM 모델을 이용하여 낙상 행위를 인식하였다. 어깨, 척추, 골반 등 주요 골격을 대상으로 중심관절을 구하고 이 중심관절의 움직임 가속도와 바닥과의 거리를 특징으로 제안하였다. 추출된 특징은 Stacked LSTM, Bi-LSTM 등의 모델과 성능비교를 수행하였고 GRU, LSTM 등의 기존연구에 비해 향상된 검출 성능을 실험을 통해 증명하였다.

Keywords

Acknowledgement

이 논문은 2021년도 정부(산업통상자원부)의 재원으로 한국산업기술평가관리원의 지원을 받아 수행된 연구임(No.20009899, 지능형 케어 서비스 개발).

References

  1. W. Chen, Z. Jiang, H. Guo, and X. Ni, "Fall detection based on key points of human-skeleton using openpose," Symmetry, Vol.12, No.5, pp.744, May 2020. https://doi.org/10.3390/sym12050744
  2. H. Li, A. Shrestha, H. Heidari, J. Le Kernec, and F. Fioranelli, "Bi-LSTM network for multimodal continuous human activity recognition and fall detection," IEEE Sensors Journal, Vol.20, No.3, pp.1191-1201, 2020. doi: 10.1109/JSEN.2019.2946095
  3. J. W. Si, et al., "Fall detection using skeletal coordinate vector and LSTM model," Journal of Korean Institute of Information Technology, Vol.18, No.12, pp.19-29, Dec. 2020. http://dx.doi.org/10.14801/jkiit.2020.18.12.19
  4. S. Mekruksavanich and A. Jitpattanakul, "LSTM networks using smartphone data for sensor-based human activity recognition in smart homes," Sensors, Vol.21, Iss.5, pp.1636, 2021. https://doi.org/10.3390/s21051636
  5. Y. K. Kang, H. Y. Kang, and D. S. Weon, "Human skeleton keypoints based fall detection using GRU," Journal of the Korea Academia-Industrial Cooperation Society, Vol.22, No.2, pp.127-133, 2021.
  6. M. Waheed, H. Afzal, and K. Mehmood, "NT-FDS-A noise tolerant fall detection system using deep learning on weara ble devices," Sensors, Vol.21, Iss.6, Articles No.2006, 2021. https://doi.org/10.3390/s21062006
  7. Q. Xu, G. Huang, M. Yu, and Y. Guo, "Fall prediction based on key points of human bones," Physica A: Statistical Mechanics and its Applications, Vol.540, Feb. 2020.
  8. C. B. Lin, Z. Dong, W. K. Kuan, and Y. F. Huang, "A framework for fall detection based on openpose skeleton and LSTM/GRU models," Applied Sciences, Vol.11, No.1, pp.329, 2021. https://doi.org/10.3390/app11010329
  9. Z. Cao, T. Simon, S. E. Wei, and Y. Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields," In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition(CVPR), 2016.
  10. PoseNet [Internet], https://github.com/tensorflow/tfjs-models/tree/-master/posenet
  11. MS Kinect v2 [Internet], https://azure.microsoft.com/en-us/services/-kinect-dk/
  12. O. Yildirim, "A novel wavelet sequence based on deep bidirectional LSTM network model for ECG signal classification," Computers in Biology and Medicine, Vol.96, pp.189-202, 2018. https://doi.org/10.1016/j.compbiomed.2018.03.016
  13. A. Graves, N. Jaitly, and A.-R. Mohamed, "Hybrid speech recognition with deep bidirectional LSTM," in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp.273-278, 2013.
  14. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, Vol.521, No.7553, pp.436-444, 2015. https://doi.org/10.1038/nature14539
  15. A. Onan and M. A. Tocoglu, "A term weighted neural language model and stacked bidirectional LSTM based framework for sarcasm identification," IEEE Access, Vol.9, pp.7701-7722, 2021. https://doi.org/10.1109/ACCESS.2021.3049734
  16. D. Utebayeva, A. Almagambetov, M. Alduraibi, Y. Temirgaliyev, L. Ilipbayeva, and S. Marxuly, "Multi-label UAV sound classification using Stacked Bidirectional LSTM," Fourth IEEE International Conference on Robotic Computing (IRC), 2020.
  17. K. Jun, D. Lee, K. Lee, S. Lee, and M. S. Kim, "Feature extraction using an rnn auto-encoder for skeleton-based abnormal gait recognition," IEEE Access, Vol.8, pp.19196-19207, 2020. https://doi.org/10.1109/access.2020.2967845
  18. K. Adhikari, H. Bouchachia, and H. Nait-Charif, "Activity recognition for indoor fall detection using convolutional neural network," In Proceedings of the 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), Nagoya, Japan, pp.81-84, May 2017.
  19. Z. Cao, T. Simon, S. Wei, and Y. Sheikh, "Realtime multi-person 2D Pose Estimation Using Part Affinity Fields," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Honolulu, HI, USA, pp.1302-1310, Jul. 2017.
  20. A. Fares, S. H. Zhong, and J. Jiang, "EEG-based image classification via a region-level stacked bi-directional deep learning framework," IEEE International Conference on Bioinformatics and Biomedicine Madrid, Spain, Dec. 2018.
  21. Robust Scaler [Internet], https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html