Journal of Electrical Engineering and Technology

The Korean Institute of Electrical Engineers (대한전기학회)
권호 표지
DOI QR코드

DOI QR Code

Depth Images-based Human Detection, Tracking and Activity Recognition Using Spatiotemporal Features and Modified HMM

  • Kamal, Shaharyar (Dept. of Electronics and Radio Engineering, Kyung Hee University) ;
  • Jalal, Ahmad (Dept. of Computer Science and Engineering, POSTECH) ;
  • Kim, Daijin (Dept. of Computer Science and Engineering, POSTECH)
  • Received : 2015.09.14
  • Accepted : 2016.05.16
  • Published : 2016.11.01
Download PDF
⟨ Previous Next ⟩

Abstract

Human activity recognition using depth information is an emerging and challenging technology in computer vision due to its considerable attention by many practical applications such as smart home/office system, personal health care and 3D video games. This paper presents a novel framework of 3D human body detection, tracking and recognition from depth video sequences using spatiotemporal features and modified HMM. To detect human silhouette, raw depth data is examined to extract human silhouette by considering spatial continuity and constraints of human motion information. While, frame differentiation is used to track human movements. Features extraction mechanism consists of spatial depth shape features and temporal joints features are used to improve classification performance. Both of these features are fused together to recognize different activities using the modified hidden Markov model (M-HMM). The proposed approach is evaluated on two challenging depth video datasets. Moreover, our system has significant abilities to handle subject's body parts rotation and body parts missing which provide major contributions in human activity recognition.

Keywords

References

  1. X. Sun, H. Kashima and N. Ueda, "Large-scale personalization human activity recognition using online multitask learning," IEEE Trans. on knowl. and data engine. vol. 25, no.11, pp.2551-2563, Nov. 2013. https://doi.org/10.1109/TKDE.2012.246
  2. A. Jalal, Y. Kim, S. Kamal, A. Farooq and D. Kim, "Human daily activity recognition with joints plus body features representation using Kinect sensor," in Proceedings of ICIEV Conference, Fukuoka, Japan, pp.1-6, Jun. 2015.
  3. A. Jalal, N. Sharif, J. Kim and T. Kim, "Human activity recognition via recognized body parts of human depth silhouettes for residents monitoring services at smart homes," Indoor and Built Environment, vol. 22, pp. 271-279, 2013. https://doi.org/10.1177/1420326X12469714
  4. A. Jalal and I. Uddin, "Security architecture for third generation (3G) using GMHS cellular network," in Proceedings of IEEE Conference on Emerging Technologies, Islamabad, Pak, pp.74-79, Nov. 2007.
  5. A. Jalal, S. Lee, J. Kim and T. Kim, "Human activity recognition via the features of labeled depth body parts," in Proceedings of ICOST Conference, Artiminio, Italy, pp.246-249, June 2012.
  6. A. Jalal and S. Kamal, "Real-time life logging via a depth silhouette-based human activity recognition system for smart home services," in Proceedings of AVSS Conference, Seoul, Korea, pp.74-80, Aug. 2014.
  7. A. Jalal, S. Kamal and D. Kim, "A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments," Sensors, vol. 14, no. 7, pp. 11735-11759, 2014. https://doi.org/10.3390/s140711735
  8. L. Xia and J. Aggarwal, "Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera," in Proceedings of CVPR Conference, Portland, Oregon, pp.2834-2841, June 2013.
  9. A. Jalal, J. Kim and T. Kim, "Development of a life logging system via depth imaging-based human activity recognition for smart homes," in Proceedings of Inter. Symposium on Sustainable healthy buildings, Seoul, Korea, pp.91-95, 2012.
  10. O. Oreifej and Z. Liu, "HON4D: Histogram of oriented 4D normal for activity recognition from depth sequences," in Proceedings of CVPR Conference, Portland, Oregon, pp.716-723, June 2013.
  11. A. Jalal, S. Kamal and D. Kim, "Shape and motion features approach for activity tracking and recognition from Kinect video camera," in Proceedings of WAINA Conference, Gwangju, Korea, pp.445-450, Mar. 2015.
  12. A. Jalal, Y.-H. Kim, Y.-J. Kim, S. Kamal and D. Kim, "Robust human activity recognition from depth video using spatiotemporal multi-fused features," Pattern Recognition, 2016.
  13. A. Jalal and Y. Kim, "Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data," in Proceedings of AVSS Conference, Seoul, Korea, pp.119-124, Aug. 2014.
  14. A. Farooq, A. Jalal and S. Kamal, "Dense RGB-D Map-Based Human Tracking and Activity Recognition using Skin Joints Features and Self-Organizing Map," KSII Transactions on internet and information systems, vol. 9, no. 5, pp. 1856-1869, 2015. https://doi.org/10.3837/tiis.2015.05.017
  15. A. Jalal, J. Kim and T. Kim, "Human activity recognition using the labeled depth body parts information of depth silhouettes," in Proceedings of SHB symposium, Korea, pp.1-8, Oct. 2012.
  16. A. Jalal and S. Kim, "Advanced performance achievement using multi-algorithmic approach of video transcoder for low bit rate wireless communication," ICGST Journal of graphics, vision and image proc., vol. 5, no. 9, pp. 27-32, 2005.
  17. A. Jalal and M. Zeb, "Security and QoS Optimization for distributed real time environment," in Proceedings of CIT Conference, Aizu-Wakamatsu, Japan, pp. 369-374, Oct. 2007.
  18. A. Jalal and S. Kim, "The mechanism of edge detection using the block matching criteria for the motion estimation," in Proceedings of HCI Conference, Korea, pp. 484-489, Jan. 2005.
  19. A. Jalal and S. Kim, "A complexity removal in the floating point and rate control phenomenon," in Proceedings of KMS Conference, Korea, pp. 48-51, June 2005.
  20. A. Jalal, S. Kim and B. Yun, "Assembled algorithm in the real-time H.263 codec for advanced performance," in Proceedings of Healthcom Conference, Korea, pp. 295-298, June 2005.
  21. A. Jalal and A. Shahzad, "Multiple facial feature detection using vertex-modeling structure," in Proceedings of ICL Conference, Villach, Austria, pp.1-7, Sep. 2007.
  22. A. Jalal and M. Zeb, "Security enhancement for elearning portal," International Journal of Computer Science and Network Security, vol. 8, no. 3, pp. 41-45, 2008.
  23. A. Jalal, M. Uddin, J. Kim and T. Kim, "Daily human activity recognition using depth silhouettes and R transformation for smart home," in Proceedings of ICOST Conference, Canada, pp.25-32, June 2011.
  24. A. Jalal and S. Kim, "Global security using human face understanding under vision ubiquitous architectture system," World academy of science, engineering, and technology, vol. 13, pp. 7-11, 2006.
  25. A. Jalal and Y. Rasheed, "Collaboration achievement along with performance maintenance in video streaming," in Proceedings of ICL Conference, Villach, Austria, pp.1-8, Sep. 2007.
  26. A. Jalal, M. Zia, J. Kim and T. Kim, "Recognition of human home activities via depth silhouettes and R transformation for smart homes," Indoor and Built Environment, vol. 21, pp. 184-190, 2012. https://doi.org/10.1177/1420326X11423163
  27. A. Jalal, M. Zia and T. Kim, "Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home," IEEE Transaction on Consumer Electronics, vol. 58, pp. 863-871, 2012. https://doi.org/10.1109/TCE.2012.6311329
  28. A. Jalal, Y. Kim and D. Kim, "Ridge body parts features for human pose estimation and recognition from RGB-D video data," in Proceedings of ICCCNT Conference, Hefei, China, pp.1-6, July 2014.
  29. J. Wang, Z. Liu, Y. Wu and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras," in Proceedings of CVPR Conference, Providence, RI, pp.1290-1297, June 2012.
  30. S. Kamal and A. Jalal, "A hybrid feature extraction approach for human detection, tracking and activity recognition using depth sensors," Arabian Journal for Science and Engineering, pp. 1043-1051, 2016.
  31. A. Jalal, S. Kamal, A. Farooq and D. Kim, "A spatiotemporal motion variation features extraction approach for human tracking and pose-based action recognition," in Proceedings of ICIEV Conference, Fukuoka, Japan, pp.1-6, Jun. 2015.
  32. X. Yang and Y. Tian, "Super normal vector for activity recognition using depth sequences," in Proceedings of CVPR Conference, Columbus, pp. 804-811, June 2014.
  33. A. Jalal, S. Kamal and D. Kim, "Depth map-based human activity tracking and recognition using body joints features and self-organized map," in Proceedings of ICCCNT Conference, Hefei, China, pp. 1-6, July 2014.
  34. M. Muller and T. Roder, "Motion templates for automatic classification and retrieval of motion capture data," in Proceedings of ACM symposium on computer animation, Austria, pp. 137-146, Sep. 2006.
  35. X. Yang and Y. Tian, "Eigenjoints-based action recognition using naive-bayes-neartest-neighbor," in Proceedings of CVPR Conference, Providence, RI, pp. 14-19, June 2012.
  36. A. Jalal, S. Kamal and D. Kim, "Individual Detection-Tracking-Recognition using depth activity images," in Proceedings of URAI Conference, Goyang, Korea, pp. 450-455, Oct. 2015.
  37. A. Jalal, "IM-DailyDepthActivity dataset," imlab.postech.ac.kr/databases.htm, 2016, [Online; accessed February 5, 2016].
  38. A. Jalal, S. Kamal and D. Kim, "Depth Silhouettes Context: A new robust feature for human tracking and activity recognition based on embedded HMMs," in Proceedings of URAI Conference, Goyang, Korea, pp.294-299, Oct. 2015.
  39. A. Baak, M. Muller, G. Bharaj, H. Seidel and C. Theobalt, "A data-driven approach for real-time full body pose reconstruction from a depth camera," in Proceedings of ICCV Conference, Barcelona, Spain, pp. 1092-1099, Nov. 2011.

Cited by

  1. 3D Reconstruction Framework for Multiple Remote Robots on Cloud System vol.9, pp.4, 2017, https://doi.org/10.3390/sym9040055
  2. A Comprehensive Review on Handcrafted and Learning-Based Action Representation Approaches for Human Activity Recognition vol.7, pp.12, 2017, https://doi.org/10.3390/app7010110
  3. Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey vol.8, pp.4, 2017, https://doi.org/10.3390/info8040134
  4. Research on a Surface Defect Detection Algorithm Based on MobileNet-SSD vol.8, pp.9, 2018, https://doi.org/10.3390/app8091678
  5. Head Pose Detection for a Wearable Parrot-Inspired Robot Based on Deep Learning vol.8, pp.7, 2018, https://doi.org/10.3390/app8071081
  6. Real-Time Ground Vehicle Detection in Aerial Infrared Imagery Based on Convolutional Neural Network vol.7, pp.6, 2018, https://doi.org/10.3390/electronics7060078
  7. A Dynamic Bayesian Network for Vehicle Maneuver Prediction in Highway Driving Scenarios: Framework and Verification vol.8, pp.1, 2019, https://doi.org/10.3390/electronics8010040
  8. Depth Maps-Based Human Segmentation and Action Recognition Using Full-Body Plus Body Color Cues Via Recognizer Engine vol.14, pp.1, 2019, https://doi.org/10.1007/s42835-018-00012-w
  9. Human action recognition: a framework of statistical weighted segmentation and rank correlation-based selection pp.1433-755X, 2020, https://doi.org/10.1007/s10044-019-00789-0