[KSCI] Korea Science Citation Index Service

http://dx.doi.org/10.9708/jksci.2022.27.06.043

Frontal Face Video Analysis for Detecting Fatigue States

Cha, Simyeong (VAIV Company Inc.)
Ha, Jongwoo (VAIV Company Inc.)
Yoon, Soungwoong (VAIV Company Inc.)
Ahn, Chang-Won (VAIV Company Inc.)

Publication Information

Journal of the Korea Society of Computer and Information / v.27, no.6, 2022 , pp. 43-52 More about this Journal

Abstract

We can sense somebody's feeling fatigue, which means that fatigue can be detected through sensing human biometric signals. Numerous researches for assessing fatigue are mostly focused on diagnosing the edge of disease-level fatigue. In this study, we adapt quantitative analysis approaches for estimating qualitative data, and propose video analysis models for measuring fatigue state. Proposed three deep-learning based classification models selectively include stages of video analysis: object detection, feature extraction and time-series frame analysis algorithms to evaluate each stage's effect toward dividing the state of fatigue. Using frontal face videos collected from various fatigue situations, our CNN model shows 0.67 accuracy, which means that we empirically show the video analysis models can meaningfully detect fatigue state. Also we suggest the way of model adaptation when training and validating video data for classifying fatigue.

Keywords

Fatigue measurement; Video analysis; Migration; Video classification; Deep learning; Machine learning;

Citations & Related Records

Times Cited By KSCI : 2 (Citation Analysis)

Reference
Cited By KSCI

1	Escobar-Linero, E., Dominguez-Morales, M., & Sevillano, J. L. (2022). Worker's physical fatigue classification using neural networks. Expert Systems with Applications, 198, 116784. DOI
2	Zhuang, Q., Kehua, Z., Wang, J., & Chen, Q. (2020). Driver fatigue detection method based on eye states with pupil and iris segmentation. Ieee Access, 8, 173440-173449. DOI
3	Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.,; Heigold, G.; Gelly, S.; et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv preprint arXiv:2010.11929.
4	Hochreiter, S.; and Schmidhuber, J. 1997. LONG SHORT-TERM MEMROY. Neural Computation, 9(8):1735-1780. DOI
5	Soomro, K.; Zamir, A. R.; and Shah, M. 2012. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv preprint arXiv:1212.0402.
6	Redmon, J.; Divvala, S.; Girshick, R.; and Farhadi, A. 2016. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779-788.
7	Yoo, S.; Kim, S.; Kim, D.; and Lee, Y. 2018. Development of Acquisition System for Biological Signals using Rasberry Pi. Journal of the Korea Institute of Information and Communication Engineering, 25(12): 1935-1941. DOI
8	Yan, P., Sun, Y., Li, Z., Zou, J., & Hong, D. (2020). Driver fatigue detection system based on colored and infrared eye features fusion. Computers, Materials & Continua, 63(3), 1563-1574. DOI
9	Kazemi, V.; and Sullivan, J. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1867-1874.
10	Qiang, Y., Liu, J., & Du, E. (2017). Dynamic fatigue measurement of human erythrocytes using dielectrophoresis. Acta biomaterialia, 57, 352-362. DOI
11	Dai, Z.; Liu, H.; Le, Q.; and Tan, M. 2021. CoAtNet: Marrying Convolution and Attention for All Data Sizes. Advances in Neural Information Processing Systems, 34.
12	Lee, Y.; Shin, S.; Cho, T.; Yeom, H.; and Kim, D. 2021. An Experimental study on Self-rated Fatigue Assessment Tool for the Fatigue Risk Groups. In Proceedings of 2021 KMIST Conference, 1755-1756.
13	Kim, K.; and Lim, C. 2016. Wearable Health Device Technology in the IoT Era. The Korean Institute of Electrical Engineers, 65(11): 18-22. DOI
14	Kim, D. 2020. A Study on the Pilot Fatigue Measurement Methods for Fatigue Risk Management. The Korean journal of aerospace and environmental medicine, 30(2): 54-60. DOI
15	Girshick, R.; Donahue, J.; Darrell, T.; and Malik, J. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 580-587.
16	He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778.
17	Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint arXiv:1704.04861
18	Tan, M.; and Le, Q. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning, 6106-6115. PMLR.
19	Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems, 30.
20	Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015. Going deeper with convolutions. In proceedings of the IEEE conference on computer vision and pattern recognition, 1-9.
21	Cho, K.; Van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078.
22	Dalal, N.; and Triggs, B. 2005. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volumne 1, 886-893. IEEE.
23	Kay, W.; Carreira, J.; Simonyan, K.; Zhang, B.; Hillier, C.; Vijayanarasimhan, S.; Viola, F.; Green, T.; Back, T.; Natsev, P.; et al. 2017. The Kinetics Human Action Video Dataset. arXiv preprint arXiv:1705.06950.
24	Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; and Berg, A. C. 2016. SSD: Single Shot MultiBox Detector. In European conference on computer vision, 21-37 Springer.
25	Sagonas, C.; Tzimiropoulos, G.; Zafeiriou, S.; and Pantic, M. 2013. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In Proceedings of the IEEE international conference on computer vision work-shops, 397-403.
26	Redmon, J.; and Farhadi, A. 2018. YOLOv3: An Incremental. arXiv preprint arXiv:1804.02767.
27	Chollet, F. 2017. Xception: Deep Learning with Depthwise Separable Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1251-1258.
28	Huang, G.; Liu, Z.; van der Maaten, L.; and Weinberger, K. Q. 2017. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700-4708.
29	Yang, S.; Luo, P.; Loy, C.-C.; and Tang, X. 2016. WIDER FACE: A Face Detection Benchmark. In Procedings of the IEEE conference on computer vision and pattern recognition, 5525-5533.
30	Lin, M., Chen, Q., & Yan, S. (2013). Network in network. arXiv preprint arXiv:1312.4400.
31	Simonyan, K.; and Zisserman, A. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556.
32	Lilleholt, L., Zettler, I., Betsch, C., & Bohm, R. (2020). Pandemic fatigue: Measurement, correlates, and consequences.
33	Dinges, D. F. 1995. An overview of sleepiness and accidents. Journal of sleep research, 4: 4-14. DOI
34	Smets, E.; Garsssen, B; Bonke, B. d.; and De Haes, J. 1995. The Multidimensional Fatigue Inventory (MFI) Psychometric Qualifies of an Instrument to Assess Fatigue. Journal of Psychosomatic Research, 39(3):315-325. DOI
35	Choe, J.-H,; Antoine, B. S. R.; and Kim, J.-H. 2014. Trend of Convergence Technology between Healthcare and the IoT. Information and Communications Magazine, 31(12):10-16.