Browse > Article
http://dx.doi.org/10.9717/kmms.2021.24.11.1481

Deep Learning-based Real-time Heart Rate Measurement System Using Mobile Facial Videos  

Ji, Yerim (Dept. of IT Engineering, Division of ICT Convergence Engineering, College of Engineering, Sookmyung Women's University)
Lim, Seoyeon (Dept. of IT Engineering, Division of ICT Convergence Engineering, College of Engineering, Sookmyung Women's University)
Park, Soyeon (Dept. of IT Engineering, Division of ICT Convergence Engineering, College of Engineering, Sookmyung Women's University)
Kim, Sangha (Dept. of IT Engineering, Division of ICT Convergence Engineering, College of Engineering, Sookmyung Women's University)
Dong, Suh-Yeon (Dept. of IT Engineering, Division of ICT Convergence Engineering, College of Engineering, Sookmyung Women's University)
Publication Information
Journal of Korea Multimedia Society / v.24, no.11, 2021 , pp. 1481-1491 More about this Journal
Abstract
Since most biosignals rely on contact-based measurement, there is still a problem in that it is hard to provide convenience to users by applying them to daily life. In this paper, we present a mobile application for estimating heart rate based on a deep learning model. The proposed application measures heart rate by capturing real-time face images in a non-contact manner. We trained a three-dimensional convolutional neural network to predict photoplethysmography (PPG) from face images. The face images used for training were taken in various movements and situations. To evaluate the performance of the proposed system, we used a pulse oximeter to measure a ground truth PPG. As a result, the deviation of the calculated root means square error between the heart rate from remote PPG measured by the proposed system and the heart rate from the ground truth was about 1.14, showing no significant difference. Our findings suggest that heart rate measurement by mobile applications is accurate enough to help manage health during daily life.
Keywords
Remote Photoplethysmography; Heart Rate; Deep Learning; Mobile Application; Healthcare system;
Citations & Related Records
연도 인용수 순위
  • Reference
1 M. Sabokrou, M. Pourreza, X. Li, M. Fathy, and G. Zhao, "Deep-HR: Fast Heart Rate Estimation from Face Video Under Realistic Conditions," Expert Systems with Applications, Vol. 186, Article 115596, 2021.
2 J. Carreira and A. Zisserman, "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
3 Z. Yu, X. Li, and G. Zhao, "Remote Photoplethysmograph Signal Measurement from Facial Videos using Spatio-temporal Networks," Proceedings of the British Machine Vision Conference, pp. 1-12, 2019.
4 J.-P. Lomaliza and H. Park, "Improved Heart-Rate Measurement from Mobile Face Videos," Electronics, Vol. 8, No. 6, pp. 663, 2019.   DOI
5 M.J. Lee, Y.M. Kim, and Y.M. Lim, "Masked Face Temperature Measurement System Using Deep Learning," Journal of Korea Multimedia Society, Vol. 24, No. 2, pp. 208-214, 2021.   DOI
6 S. Wang, A. Pohl, T. Jaeschke, M. Czaplik, M. Kony, and S. Leonhardt, et al., "A Novel Ultra-Wideband 80 GHz FMCW Radar System for Contactless Monitoring of Vital Signs," Proceeding of Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 4978-4981, 2015.
7 C. Yang, C. Veiga, J.J. Rodriguez-Andina, J. Farina, A. Iniguez, and S. Yin, "Using PPG Signals and Wearable Devices for Atrial Fibrillation Screening," IEEE Transactions on Industrial Electronics, Vol. 66, No. 11, pp. 8832-8842, 2019.   DOI
8 X. Niu, H. Han, S. Shan, and X. Chen, "VIPLHR: A Multi-Modal Database for Pulse Estimation from Less-Constrained Face Video," Proceedings of the Asian Conference on Computer Vision, pp. 562-576, 2018.
9 B.P. Yan, C.K. Chan, and C.K. Li, "Resting and Postexercise Heart Rate Detection From Fingertip and Facial Photoplethysmography Using a Smartphone Camera: A Validation Study," JMIR mHealth and uHealth, Vol. 5, No. 3, e33, 2017.   DOI
10 X. Chen, J. Cheng, R. Song, Y. Liu, R. Ward, and Z. J. Wang, "Video-Based Heart Rate Measurement: Recent Advances and Future Prospects," IEEE Transactions on Instrumentation and Measurement, Vol. 68, No. 10, pp. 3600-3615, 2019.   DOI
11 A. Dasari, S. Ananyananda, L.A. Jeni, and C.S. Tucker, "Evaluation of Biases in Remote Photoplethysmography Methods," NPJ Digital Medicine, Vol. 4, No. 1, p. 91, 2021.   DOI
12 V. Jindal, "Integrating Mobile and Cloud for PPG Signal Selection to Monitor Heart Rate during Intensive Physical Exercise," Proceeding of International Conference on Mobile Software Engineering and Systems (MOBILE Soft), pp. 36-37, 2016.
13 D. Qiao, F. Zulkernine, R. Masroor, R. Rasool, and N. Jaffar, "Measuring Heart Rate and Heart Rate Variability with Smartphone Camera," Proceeding of 22nd IEEE International Conference on Mobile Data Management (MDM), pp. 248-249, 2021.
14 K. Norton, L. Norton, and D. Sadgrove, "Position Statement on Physical Activity and Exercise Intensity Terminology," Journal of Science and Medicine in Sport, Vol. 13, No. 5, pp. 496-502, 2010.   DOI
15 M. Van Gastel, B. Balmaekers, S.B. Oetomo, and W. Verkruysse, "Near-continuous Noncontact Cardiac Pulse Monitoring in a Neonatal Intensive Care Unit in Near Darkness," Proceedings of Optical Diagnostics and Sensing XVIII, Vol. 10501, 2018.
16 P. Huang, B. Wu, and B. Wu, "A Heart Rate Monitoring Framework for Real-World Drivers Using Remote Photoplethysmography," IEEE Journal of Biomedical and Health Informatics, Vol. 25, No. 5, pp. 1397-1408, 2021.   DOI
17 G. de Haan and V. Jeanne, "Robust Pulse Rate from Chrominance-based rPPG," IEEE Transactions Biomedical Engineering, Vol. 60, No. 10, pp. 2878-2886, 2013.   DOI
18 W. Wang, A.C. den Brinker, S. Stuijk, and G. de Haan, "Algorithmic Principles of Remote PPG," EEE Transactions on Biomedical Engineering, Vol. 64, No. 7, pp. 1479-1491, 2017.   DOI
19 W. Chen and D. McDuff, "DeepPhys: Videobased Physiological Measurement using Convolutional Attention Networks," Proceeding of the European Conference on Computer Vision (ECCV), pp. 349-365, 2018.
20 G. Zhang, Z, Mei, Y. Zhang, X. Ma, B. Lo, and D. Chen, "A Non-invasive Blood Glucose Monitoring System Based on Smartphone PPG Signal Processing and Machine Learning," IEEE Transactions on Industrial Informatics, Vol. 16, No. 11, pp. 7209-7218, 2020.   DOI
21 C. Yao, S. Wang, J. Zhang, W. He, H. Du, and J. Ren, et al., "rPPG-Based Spoofing Detection for Face Mask Attack using Efficientnet on Weighted Spatial-Temporal Representation," Proceeding of IEEE International Conference on Image Processing (ICIP), pp. 3872-3876, 2021.
22 N. Malesevic, V. Petrovic, M. Belic, C. Antfolk, V. Mihajlovic, and M. Jankovic, "Contactless Real-time Heartbeat Detection via 24 GHz Continuous-Wave Doppler Radar Using Artificial Neural Networks," Sensors, Vol. 20, No. 8, pp. 2351, 2020.   DOI
23 D. Kim, J. Kim, and K. Lee, "Real-time Vital Signs Measurement System using Facial Image Data," Journal of Broadcast Engineering, Vol. 26, No. 2, pp. 132-142, 2021.   DOI
24 W. Verkruysse, L.O. Svaasand, and J.S. Nelson, "Remote Plethysmographic Imaging using Ambient Light," Optics Express, Vol. 16, No. 26, pp. 21434-21445, 2008.   DOI
25 D. Qiao, F. Zulkernine, R. Masroor, R. Rasool, and N. Jaffar, "Measuring Heart Rate and Heart Rate Variability with Smartphone Camera," Proceeding of IEEE International Conference on Mobile Data Management (MDM), pp. 248-249, 2021.
26 PyTorch(2019), https://pytorch.org/mobile/home (accessed September 17, 2021).
27 Y. Ostchega, K.S. Porter, J. Hughes, C.F. Dillon, and T. Nwankwo, "Resting Pulse Rate Reference Data for Children, Adolescents, and Adults: United States, 1999-2008," National Health Statistics Reports, Vol. 41, pp. 1-16, 2011.
28 N. Zhu, J.R. Suarez-Lopez, S. Sidney, B. Sternfeld, P.J. Schreiner, M.R. Carnethon, et al., "Longitudinal Examination of Age-Predicted Symptom-Limited Exercise Maximum HR," Medicine and Science in Sports and Exercise, Vol. 42, No. 8, pp. 1519-1527, 2010.   DOI
29 W. Wang and C. Shan, "Impact of Makeup on Remote-PPG Monitoring," Biomedical Physics and Engineering Express, Vol. 6, No. 3, 035004, 2020.   DOI
30 J.A. Dawson, C.O. Kamlin, and C. Wong, "Changes in Heart Rate in the First Minutes after Birth," Archives of Disease in Childhood. Fetal and Neonatal Edition, Vol. 95, No. 3, pp. 177-181, 2010.
31 N.V. Hoan, J. Park, S. Lee, and K. Kwon, "Real-time Heart Rate Measurement based on Photoplethysmography using Android Smartphone Camera," Journal of Korea Multimedia Society, Vol. 20, No. 2, pp. 234-243, 2017.   DOI
32 W. Wang, S. Stuijk, and G. de Haan, "A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation," IEEE Transactions Biomedical Engineering, Vol. 63, No. 9, pp. 1974-1984, 2016.   DOI