Browse > Article
http://dx.doi.org/10.5909/JBE.2021.26.6.748

Deep Learning-based Gaze Direction Vector Estimation Network Integrated with Eye Landmark Localization  

Joo, Heeyoung (Korea Electronics Technology Institute, KETI)
Ko, Min-Soo (Korea Electronics Technology Institute, KETI)
Song, Hyok (Korea Electronics Technology Institute, KETI)
Publication Information
Journal of Broadcast Engineering / v.26, no.6, 2021 , pp. 748-757 More about this Journal
Abstract
In this paper, we propose a gaze estimation network in which eye landmark position detection and gaze direction vector estimation are integrated into one deep learning network. The proposed network uses the Stacked Hourglass Network as a backbone structure and is largely composed of three parts: a landmark detector, a feature map extractor, and a gaze direction estimator. The landmark detector estimates the coordinates of 50 eye landmarks, and the feature map extractor generates a feature map of the eye image for estimating the gaze direction. And the gaze direction estimator estimates the final gaze direction vector by combining each output result. The proposed network was trained using virtual synthetic eye images and landmark coordinate data generated through the UnityEyes dataset, and the MPIIGaze dataset consisting of real human eye images was used for performance evaluation. Through the experiment, the gaze estimation error showed a performance of 3.9, and the estimation speed of the network was 42 FPS (Frames per second).
Keywords
Gaze Estimation; Eye Landmark Localization; Eye Landmark Detection;
Citations & Related Records
연도 인용수 순위
  • Reference
1 R. Sherman, and Alan B. Craig, "Understanding Virtual Reality: Interface, Application, and Design, Second Edition," Morgan Kaufmann Series in Computer Graphics, Massachusetts, pp.3-58, 2018.
2 Oliver J. Muensterer, Martin Lacher, Christoph Zoeller, Matthew Bronstein, and Joachim Kubler, "Google Glass in pediatric surgery: An exploratory study," International Journal of Surgery, Vol.12, No.4, pp.281-289, 2014.   DOI
3 Simonyan, Karen, and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.
4 M. Tepper, L. Rudy, A. Lefkowitz Aaron, A. Weimer, M. Marks, S. Stern, and S. Garfein, "Mixed Reality with HoloLens: Where Virtual Reality Meets Augmented Reality in the Operating Room," Plastic and Reconstructive Surgery, Vol.140, No.5, pp.1066-1070, 2017.   DOI
5 The Market prediction for the virtual, augmented, and mixed reality technology by Statista https://www.statista.com/statistics/591181/global-augmented-virtual-reality-market-size (accessed Sep. 3, 2021).
6 Hyundai mobis' research development about driver's eye tracking, http://www.epnc.co.kr/news/articleView.html?idxno=91211 (accessed Sep. 3, 2021).
7 Erroll Wood, Tadas Baltrusaitis, Louis-Philippe Morency, Peter Robinson, Andreas Bulling, "Learning an appearance-based gaze estimator from one million synthesized images," In Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 131-138. 2016.
8 https://www.cl.cam.ac.uk/research/rainbow/projects/unityeyes/tutorial.html#:~:text=UnityEyes%20is%20a%20tool%20for,for%20other%20eye%20tracking%20systems (accessed Sep. 3, 2021).
9 Examples of MPIIGaze dataset, https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/gaze-based-human-computer-interaction/appearance-based-gaze-estimation-in-thewild (accessed Sep. 3, 2021).
10 Examples of UnityEyes dataset,
11 Seonwook Park, Xucong Zhang, Andreas Bulling, and Otmar Hilliges, "Learning to find eye region landmarks for remote gaze estimation in unconstrained settings," Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, pp.1-10, 2018.
12 J. Carmigniani, and B.Furht, "Augmented Reality: An Overview," Springer, New York, pp.3-46, 2011.
13 Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten, "Densely Connected Convolutional Networks," arXiv preprint arXiv:1608.06993, 2016.
14 Diederik P. Kingma, Jimmy Ba, "Adam: A method for stochastic optimization," arXiv preprint, 2014.
15 Park, Seonwook, Adrian Spurr, and Otmar Hilliges, "Deep pictorial gaze estimation," Proceedings of the European Conference on Computer Vision (ECCV), pp. 721-738, 2018.
16 Anuradha Kar, and Peter Corcoran, "A review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms," IEEE Access, Vol.5, pp.16495-16519, 2017.   DOI
17 An example of the visualization of the gaze estimation, https://www.hankyung.com/it/article/201701051859v (accessed Sep. 3, 2021).
18 Sunghyun Cho, "Introduction to eye-tracking technology," The Magazine of the IEEK, Vol.45, pp.23-32, 2018.
19 Laura Sesma, Arantxa Villanueva, and Rafael Cabeza, "Evaluation of pupil center-eye corner vector for gaze estimation using a web cam," Proceedings of the Symposium on Eye Tracking Research and Applications, pp.217-220, 2012.
20 Christian Nitschke, Atsushi Nakazawa, and Haruo Takemura, "Display-camera calibration using eye reflections and geometry constraints", Computer Vision and Image Understanding, Vol.115, No.6, pp.835-853, 2011.   DOI
21 Mariette Awad, Rahul Khanna, "Support vector regression," Efficient learning machines, pp.67-80, 2015.
22 Alejandro Newell, Kaiyu Yang, Jia Deng, "Stacked hourglass networks for human pose estimation," European conference on computer vision, pp.483-499, 2016.
23 Xucong Zhang, Yusuke Sugano, Mario Fritz, Andreas Bulling, "MPIIGaze: Real-world dataset and deep appearance-based gaze estimation," IEEE transactions on pattern analysis and machine intelligence Vol.41, pp.162-175, 2017.   DOI
24 A. Tsukada, M. Shino, M. Devyver, and T. Kanade, "Illumination-free gaze estimation method for first-person vision wearable device", 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp.2084-2091, 2011.
25 Noel Gorelick, Matt Hancher, Mike Dixon, Simon Ilyushchenko, David Thau, and Rebecca Moore,"Google Earth Engine: Planetary-scale geospatial analysis for everyone," Remote Sensing of Environment, Vol.202, pp.18-27, 2017.   DOI