Nuclear Engineering and Technology

Korean Nuclear Society (한국원자력학회)
권호 표지
DOI QR코드

DOI QR Code

Method for Inference of Operators' Thoughts from Eye Movement Data in Nuclear Power Plants

  • Ha, Jun Su (Khalifa University of Science, Technology & Research) ;
  • Byon, Young-Ji (Khalifa University of Science, Technology & Research) ;
  • Baek, Joonsang (Khalifa University of Science, Technology & Research) ;
  • Seong, Poong Hyun (Korea Advanced Institute of Science and Technology)
  • Received : 2015.07.02
  • Accepted : 2015.09.09
  • Published : 2016.02.25
Download PDF
⟨ Previous Next ⟩

Abstract

Sometimes, we need or try to figure out somebody's thoughts from his or her behaviors such as eye movement, facial expression, gestures, and motions. In safety-critical and complex systems such as nuclear power plants, the inference of operators' thoughts (understanding or diagnosis of a current situation) might provide a lot of opportunities for useful applications, such as development of an improved operator training program, a new type of operator support system, and human performance measures for human factor validation. In this experimental study, a novel method for inference of an operator's thoughts from his or her eye movement data is proposed and evaluated with a nuclear power plant simulator. In the experiments, about 80% of operators' thoughts can be inferred correctly using the proposed method.

Keywords

References

  1. M. Barriere, D. Bley, S. Cooper, J. Forester, A. Kolaczkowski, W. Luckas, G. Parry, A. Ramey-smith, C. Thompson, D. Whitehead, J. Wreathall, Technical Basis and Implementation Guidelines for a Technique for Human Event Analysis (ATHEANA), Rev.01, NUREG-1624, US NRC, 2000.
  2. J. Kemeny, The Need for Change: the Legacy of TMI. Report of the President's Commission on the Accident at Three Miles Island, Pergamon, New York, 1979.
  3. H. Yoshikawa, Human-machine interaction in nuclear power plants, Nucl. Eng. Technol. 37 (2005) 151-158.
  4. S.J. Lee, P.H. Seong, Development of an integrated design support system to aid cognitive activities of operators, Nucl. Eng. Technol. 39 (2007) 703-716. https://doi.org/10.5516/NET.2007.39.6.703
  5. T.B. Sheridan, Telerobotics, Automation, and Human Supervisory Control, MIT Press, Cambridge, MA, 1992.
  6. R.J. Mumaw, E.M. Roth, K.J. Vicente, C.M. Burns, there is more to monitoring a nuclear power plant than meets the eye, Hum. Factors 42 (2000) 36-55. https://doi.org/10.1518/001872000779656651
  7. A.L. Yarbus, Eye Movements and Vision, Plenum, New York, 1967 (originally published in Russian in 1962).
  8. M. Wang, Y. Maeda, Y. Takahashi, Human Intention Recognition via Eye Tracking Based on Fuzzy Inference, the 6th International Conference on Soft Computing and Intelligent Systems, and the 13th International Symposium on Advanced Intelligence Systems (SCIS-ISIS 2012), Kobe, Japan, November 2012, pp. 846-851.
  9. Z. Huiping, I. Makoto, I. Toshiyuki, Eye movement-based inference of truck driver's intent of changing lanes, SICE J. Control Meas. Syst. Integr. 2 (2009) 291-298. https://doi.org/10.9746/jcmsi.2.291
  10. J.S. Ha, P.H. Seong, Attentional-resource effectiveness measures in monitoring and detection tasks in NPPs, IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 40 (2010) 993-1008. https://doi.org/10.1109/TSMCA.2010.2044999
  11. C.D. Wickens, J. Lee, Y. Liu, S.G. Becker, An Introduction to Human Factors Engineering, second ed., Prentice Hall, 2004.
  12. C.D. Wickens, J.G. Hollands, Engineering Psychology and Human Performance, third ed., Prentice Hall, Upper Saddle River, NJ, 2000.
  13. R.E. Jones, J.L. Milton, P.M. Fitts, Eye Movements of Aircraft Pilots: I. A Review of Prior Eye Movement Studies and a Description of Eye Movements during Instrument Flight, USAF Tech. Rep. No. 5837, Wright-Patterson Air Force Base, Dayton, OH, 1949.
  14. J.L. Milton, R.E. Jones, P.M. Fitts, Eye Movements of Aircraft Pilots: II. Frequency, Duration, and Sequence of Fixation when Flying the USAF Instrument Low Approach System (ILAS), USAF Tech. Rep. No. 5839, Wright-Patterson Air Force Base, Dayton, OH, 1949.
  15. P.M. Fitts, R.E. Jones, J.L. Milton, Eye movements of aircraft pilots during instrument landing approaches, Aeronaut. Eng. Rev. 9 (1950) 1-5.
  16. J.W. Senders, The human operator as a monitor and controller of multidegree of freedom system,, IEEE Trans. Hum. Factors Electron 5 (1964) 3-5.
  17. J.R. Carbonell, A queueing model for many-instrument visual sampling, IEEE Trans. Hum. Factors Electron HFE-7 (1966) 157-164. https://doi.org/10.1109/THFE.1966.232984
  18. J.R. Carbonell, J.L. Ward, J.W. Senders, A queuing model of visual sampling: experimental validation, IEEE Trans. Man Mach. Syst. MMS-9 (1968) 82-87.
  19. R.D. Smallwood, Internal models of the human instrument monitor, IEEE Trans. Hum. Factors Electron HFE-8 (1967) 181-187. https://doi.org/10.1109/THFE.1967.233967
  20. S. Baron, D.L. Kleinman, The human as an optimal controller and information processor, IEEE Trans. Man Mach. Syst. MMS-10 (1969) 9-17.
  21. S. Baron, D.L. Kleinman, W.H. Levison, An optimal control model of the human response: part ii: prediction of human performance in a complex task, Automatica 6 (1970) 371-383. https://doi.org/10.1016/0005-1098(70)90052-X
  22. D.L. Kleinman, R.E. Curry, Some new control theoretic models for human operator display modelling, IEEE Trans. Syst. Man Cybern. SMC-7 (1977) 778-784.
  23. D.L. Kleinman, S. Baron, W.H. Levison, An optimal control model of the human response: part i: theory and validation, Automatica 6 (1970) 357-369. https://doi.org/10.1016/0005-1098(70)90051-8
  24. W.H. Levison, A Control Theory Model for Human Decision Making, NTIS No. N73-10104, 7th NASA Annual Conference on Manual Control, University of Southern California, 1971.
  25. P. Wewerinke, A Model of the Human Decision Maker Observing a Dynamic System, Tech. Rep. NLR TR 81062 L, National Lucht-en Ruimtevaartlaboratorium, Netherlands, 1981.
  26. C.D. Wickens, J. Helleberg, J. Goh, X. Xu, W.J. Horrey, Pilot Task Management: Testing an Attentional Expected Value Model of Visual Scanning, Tech. Rep. ARL-01-14/NASA-01-7, NASA Ames Research Center, Moffett Field, CA, 2001.
  27. T.B. Sheridan, On how often the supervisor should sample, IEEE Trans. Syst. Sci. Cybern. SSC-6 (1970) 140-145.
  28. T. Kvalseth, Quantitative Modeling of the Time-multiplexing Characteristics of Human Controllers, NSF GK-37419, 1975.
  29. T. Kvalseth, Human information processing in visual sampling, Ergonomics 21 (1977) 439-454.
  30. T. Kvalseth, The effect of cost on the sampling behavior of human instrument monitors, in: T.B. Sheridan, G. Johannsen (Eds.), Monitoring Behavior and Supervisory Control, Plenum, New York, 1977.
  31. N. Moray, Human Information Processing and Supervisory Control, Tech. Rep., Man-machine Systems Laboratory, MIT, Cambridge, MA, 1980.
  32. N. Moray, The role of attention in the detection of errors and the diagnosis of failure in man-machine systems, in: J. Rasmussen, W.B. Rouse (Eds.), Human Detection and Diagnosis of System Failures, Plenum, New York, 1981.
  33. N. Moray, Monitoring behavior and supervisory control, in: K. Boff, L. Kaufman, J. Thomas (Eds.), Handbook of Human Perception and Performance, John Wiley and Sons, New York, 1986.
  34. N. Moray, G. Neil, C. Brophy, The Behavior and Selection of Fighter Controllers, Tech. Rep., Ministry of Defense, London, 1983.
  35. E.R.F.W. Crossman, J.E. Cooke, R.J. Beishon, Visual attention and sampling of displayed information in process control, in: E. Edwards, F. Lees (Eds.), The Human Operator in Process Control, Taylor and Francis, London, 1974.
  36. J.W. Senders, Visual Scanning Process, University of Tilburg Press, Netherlands, 1983.
  37. A.X. Miao, G.L. Zacharias, S.P. Kao, A computational situation assessment model for nuclear power plant operations, IEEE Trans. Syst. Man Cybern. PART A 27 (1997) 728-742. https://doi.org/10.1109/3468.634636
  38. M.C. Kim, P.H. Seong, A computational model for knowledge-driven monitoring of nuclear power plant operators based on information theory, Reliab. Eng. Syst. Saf. 91 (2006) 283-291. https://doi.org/10.1016/j.ress.2005.01.017
  39. M.C. Kim, P.H. Seong, An analytic model for situation assessment of nuclear power plant operators based on Bayesian inference, Reliab. Eng. Syst. Saf. 91 (2006) 270-282. https://doi.org/10.1016/j.ress.2005.01.012
  40. J.S. Ha, P.H. Seong, Experimental investigation between attentional-resource effectiveness and perception and diagnosis in nuclear power plants, Nucl. Eng. Des. 278 (2014) 758-772. https://doi.org/10.1016/j.nucengdes.2014.08.025
  41. D. Gopher, The skill of attentional control: acquisition and execution of attention strategies, in: D.E. Meyer, S. Kornblum (Eds.), Attention and Performance, Vol. 14, MIT Press, Cambridge, MA, 1993.
  42. D. Kahneman, Attention and Effort, Prentice Hall, Englewood Cliffs, NJ, 1973.
  43. J.M. O'Hara, J.C. Higgins, W.F. Kramer, J. Kramer, Computerbased procedure systems: technical basis and human factors review guidance, NUREG/CR-6634, US NRC, 2002.
  44. J. Rasmussen, Diagnostic reasoning in action, IEEE Trans. Syst. Man Cybern. 23 (1993) 981-992. https://doi.org/10.1109/21.247883
  45. T.L. Saaty, The Analytic Hierarchy Process, McGraw-Hill, 1980.
  46. W.F. Stubler, J.M. O'Hara, J.C. Higgins, J. Kramer, Human Systems Interface and Plant Modernization Process: Technical Basis and Human Factors Review Guidance, NUREG/CR-6637, US NRC, 2000.
  47. J.S. Ha, P.H. Seong, A review of studies on operator's information searching behavior for human factors studies in NPP MCRs, Nucl. Eng. Technol. 41 (2009) 247-270. https://doi.org/10.5516/NET.2009.41.3.247
  48. G. Underwood, J. Everatt, The role of eye movements in reading: some limitations of the eye-mind assumption, in: E. Chekaluk, K.R. Llewellyn (Eds.), The Role of Eye Movements in Perceptual Process, Advances in Psychology, Vol. 88, Elsevier Science Publishers BV, Amsterdam, 1992, pp. 111-169.
  49. N. Moray, I. Rotenberg, Fault management in process control: eye movements and action, Ergonomics 32 (1989) 1319-1342. https://doi.org/10.1080/00140138908966910
  50. T.L. Hodgson, H.J. Muller, Evidence relating to premotor theories of visuspatial attention, in: J.M. Findlay, et al. (Eds.), Eye Movement Research, Elsevier Science BV, 1995.
  51. W.X. Shneider, H. Deubel, Visual attention and saccadic eye movements: evidence for obligatory and selective spatial coupling, in: J.M. Findlay, et al. (Eds.), Eye Movement Research, Elsevier Science BV, 1995.
  52. J.B. Best, Cognitive Psychology, West Publishing Company, St. Paul, MN, 1992.
  53. A. Droivoldsmo, J.G. Skraaning, M. Sverrbo, J. Dalen, T. Grimstad, G. Andresen, Continuous measure of situation awareness and workload, HWR-539, OECD Halden Reactor Project, 1988.
  54. H. Seok, H.C. No, Development of a software for the micro simulator for Kori Nuclear Power Plant Unit 2, Nucl. Technol. 106 (1994) 384-396. https://doi.org/10.13182/NT94-A34968
  55. http://www.seeingmachines.com/.
  56. D.E. Irwin, Visual memory within and across fixations, in: K. Rayner (Ed.), Eye Movements and Visual Cognition: Scene Perception and Reading, Springer-Verag, New York, NY, 1992.
  57. J. Duncan, R. Ward, K. Shapiro, Direct measurement of attentional dwell time in human vision, Nature 369 (1994) 313-315. https://doi.org/10.1038/369313a0
  58. C.M. Moore, H. Egeth, L.R. Berglan, S.J. Luck, Are attentional dwell times inconsistent with serial visual search? Psychon, Psychon. Bull. Rev 3 (1996) 360-365. https://doi.org/10.3758/BF03210761
  59. R. Ward, J. Duncan, K. Shapiro, Effects of similarity, difficulty, and non-target presentation on the time course of visual attention, Percept, Psychophys 59 (1997) 593-600. https://doi.org/10.3758/BF03211867
  60. J.M. O'Hara, W.F. Stubler, J.C. Higgins, W.S. Brown, Integrated System Validation: Methodology and Review Criteria, NUREG/CR-6393, US NRC, 1997.
  61. G.J. Skraning, The Operator Performance Assessment System (OPAS), HWR-538, OECD Halden Reactor Project, 1998.

Cited by

  1. The Neuroergonomics of Aircraft Cockpits: The Four Stages of Eye-Tracking Integration to Enhance Flight Safety vol.4, pp.1, 2018, https://doi.org/10.3390/safety4010008
  2. Eye-Tracking Studies Based on Attentional-Resource Effectiveness and Insights into Future Research vol.202, pp.2, 2016, https://doi.org/10.1080/00295450.2018.1428003
  3. Comprehensive evaluation method for user interface design in nuclear power plant based on mental workload vol.51, pp.2, 2016, https://doi.org/10.1016/j.net.2018.10.010
  4. Biosignal-Based Attention Monitoring to Support Nuclear Operator Safety-Relevant Tasks vol.14, pp.None, 2016, https://doi.org/10.3389/fncom.2020.596531
  5. Study on cognitive load of OM interface and eye movement experiment for nuclear power system vol.52, pp.1, 2016, https://doi.org/10.1016/j.net.2019.06.023
  6. Electroencephalography-Based Intention Monitoring to Support Nuclear Operators’ Communications for Safety-Relevant Tasks vol.207, pp.11, 2016, https://doi.org/10.1080/00295450.2020.1837583