1. Introduction
With the development of hardware and software and the widespread use of digital devices, the virtual reality (VR) device market continues to grow. The recent commercialization and increased availability of VR headsets, also referred to as head-mounted displays (HMDs), has sparked interest in the use of VR technology for research, military, medical, educational, and entertainment purposes [1]. experts predict that VR technology will change the way of life in the future, including how we work, entertain ourselves, communicate, and learn [2].
While the maturity and economic viability of VR technology are highly regarded with the emergence of various products and manufacturers over the past five years (the VR "second wave"), it is necessary to address certain aspects for increasing user adoption, such as user experience, usability, accessibility, and health effects [2]. cybersickness, a human factor affecting user experience, is of significant concern [3][4]. Cybersickness is a side effect of exposure to the virtual environment and is characterized by unpleasant physiological symptoms such as nausea and dizziness. Many studies have been aimed at uncovering the cause ofcybersickness, which is considered to be strongly related to the level of user experience [4][5][6].
With the continuous growth of the market for and availability of various VR devices, indicators are required for the systematic and rational evaluation ofthe comfort experience of VR HMDs. Such an evaluation is necessary to enhance the user experience, which is essential to the commercialization of VR devices. Thus, the development of evaluation indicators should be prioritized. While research on cybersickness- causing factors has been actively conducted, no study has ranked cybersickness-causing factors for HMDs. Studies that have identified cybersickness- causing factors in HMDs have not provided evaluation indicators for comfort experience assessment.
This study aimed to develop evaluation indicators for the comfort experience of HMDs. A Delphi survey was conducted with cybersickness experts; the survey was based on cybersickness-causing factors identified from previous studies. Furthermore, the relative importance of the evaluation indicators was determined by calculating weights for all dimensions and sub-factors. The results of this study can be used as basic data to evaluate the comfort experience of VR HMDs in future studies.
2. Litrature Review
The literature on experiencing motion sickness indicates that visual perception and motion are the primary factors determining its occurrence. In preceding studies related to experiencing cybersickness in virtual environments (the experience addressed in this study), inconsistencies in visual perception and motion have been commonly identified as causes of general cybersickness. Therefore, this section describes the main features of sensory mismatch and postural instability—two representative cybersickness theories.
2.1 Sensory Conflict Theory
Sensory conflict theory is a fundamental theory that defines motion sickness. It states that conflicts occur when information recorded by different senses of a person for perceiving the world does not match his or her movements [7][8]. For VR, a conflict exists between the perspectives of viewing content and the vestibular sense responsible for balancing it. Consequently, a mismatch occurs in the information delivered by the two concerned sensory organs.
Sub-concepts of the sensory conflict theory include "input conflict, " "output conflict, " and "expectancy violation." Input conflict occurs during information processing because of a mismatch in the information recorded by the input senses. In contrast, output conflicts result from inconsistencies between processed information as information expressed externally. Finally, expectancy violation refers to the conflict that occurs when information collected based on existing experience and knowledge differs from that expected.
2.2 Postural Instability Theory
Postural instability theory emerged from a criticism of the three sub-concepts of sensory conflict theory. The criticism was that, logically, input conflict, output conflict, and expectancy violation do not differ significantly—the theory is abstract, precluding a complete explanation of motion sickness [9]. it is challenging to explain individual differences in the degree of sensory conflicts experienced by individuals for a given stimulation situation.
In contrast to sensoryconflict theory, which postulates mismatches in sensory information as the cause of motion sickness, postural instability theory specifies the cause as a reduction in postural control ability—defined as persistent postural imbalance. Individuals may experience motion sickness when they lose the ability to control their posture under a closed-loop feedback system consisting of "environmental dynamics, " "goals of behavior, " and "constrain the control of posture, " from which they perceive the world [10].
2.3 Cybersickenss Studies on Hardware
Based on previous studies, cybersickness-causing factors can be divided into three categories: content factors, hardware factors, and user characteristic factors [11]. This study focused on hardware (HMD)-related cybersickness- causing factors.
Previous studies regarding hardware-related cybersickness- causing factors have verified the influence of display device types, including screens, monitors, and HMDs, on cybersickness [12][13] [14]. Studies on the changes in the degree of cybersickness by field of view (FoV) [15][16], the influence of time delay on cybersickness [17], the influence of framerate on cybersickness [18], and the influence of flickering on cybersickness have been conducted [19]. Because this study aims to develop evaluation indicators for the comfort experience of VR HMD, we comprehensively included the variables covered in various studies. Each variable is described in detail in the next section.
3. Research Framework
This study classified the evaluation indicators of the comfort experience of HMDs into six dimensions based on the prior studies on cybersickness, focusing on the sensory conflict and postural instability theories. From these dimensions, the critical sub-factors were identified and compared between standards for determining guidelines for the production of HMDs characterized by reduced cybersickness. The standards considered were IEEE standards, which are international standards, TTA, which are South Korea’s domestic standards, and factors presented in previous papers on cybersickness [20][21][22]. The dimensions of the evaluation indicators were FoV, latency, framerate, V-sync, rig, and resolution. The research framework is summarized in Figure 1.
(Figure 1) System framework
3.1 Head-Mounted Display
VR is defined as a computer-system-based convergence technology that provides users with a realistic and immersive sense of reality in an artificially-formed virtual space [23]. HMDs, which users wear on their heads for VR space experience, include high-resolution displays, GPS receivers, earth magnetic fields, and gyroscopes. The first HMD was developed in 1968 by the American computer scientist Ivan Edward Sutherland, who worked at Harvard University, and his student Bob Sproull [24][25]. Since thelaunch of Oculus Rift, HMDs have been actively produced by manufacturers (Table 1) such as Samsung and Oculus (Gear VR, jointly produced by the two companies), HTC (Vive), and others.
(Table 1) HMDs variety of specifications
HMDs in the market have several specifications that have been designed to provide users with an interface that offers a comfortable experience. HMDs artificially create VR and facilitate deep, three-dimensional perception through binocular parallax. This operation principle causes visual and behavioral conflicts, resulting in users experiencing cybersickness, as confirmed by many previous studies [26][27].
3.2 Field of View
FoV is the size of the area that users can observe. It is classified by the vertical/horizontal orientation of the display, viewing angle, and diagonal length of the FoV. If a display is used, a small FoV indicates a narrow viewing area, in which users must move the screen frequently. While a small FoV is characterized by reduced image immersion and visual cognitive ability, a high FoV can cause screen distortion, resulting in users feeling dizzy or uncomfortable. Furthermore, a large device weight because of a high FoV is likely to make users feel less comfortable and more fatigued [28].
Figure 2 illustrates the FoV of the human eye. The human eye vision-level differs for monocular vision, binocular vision, horizontal-line-of-sight view, and vertical-line-of-sight view. For the human eye, for a horizontal-line-of-sight view, monocular and binocular vision cover an FoV of approximately 160º and 120º, respectively. Similarly, the vertical-line-of-sight view has different FoVs for the two types of vision. The concentration of a person depends on the FoV of the human eye. Therefore, the FoV of VR content depends on the purpose of the content, such as therapy, training, or entertainment.
(Figure 2) FoV of human eye [29]
3.3 Latency
Latency refers to the difference between the time required for the VR device to respond to a behavior signal inputted by a user and the time at which the VR device presents the signal. VR latency can distract users and affect their comfort level and the intensity of cybersickness [28]. Studies have demonstrated that users experience dysentery and discomfort when the latency exceeds a certain level (60 ms) and a reduced sense of immersion and fatigue because of losing their sense of direction [30].
Figure 3 illustrates the systematic process by which latency occurs. VR latency is the sum of the times required for device location detection through motion tracking systems, rendering through game engines, scene transmission through graphics hardware, and transmission of pixel-based photons by the display [31].
(Figure 3) Motion to photon latency [32]
3.4 Framerate
Framerate refers to the rate of speed at which a stationary image is reproduced. A video consists of a collection of still pictures, each of which is a frame. The rate at which the frame is changed is the framerate. Figure 4 illustrates the dependence of the image processing frequency on the framerate. Low framerate can cause flickering. The flickering phenomenon can cause eye fatigue and is likely to cause headaches, fatigue, and photo-epileptic seizures. The cumulative effects of the flickering phenomenon include discomfort and cybersickness [28].
(Figure 4) Dependence of the image processing frequency on the framerate [33]
3.5 V-sync
V-sync refers to the number of images that a display projects per second. The higher the V-sync, the more scenes can be displayed in a second, and the lower the number of screen breaks. Typical digital images require an average V-sync of 60 Hz, whereas game content requires a relatively higher V-sync. If the device’s V-sync does not match the frequency of the content’s image, a tear occurs on the screen. The tearing phenomenon caused by a low V-sync affects the device user’s comfort level. V-sync is also a significant factor affecting users’ comfort experience.
Figure 5 illustrates the screen for different V-sync levels. The left monitor’s screen presents a horizontal tear caused by a low V-sync, while the right monitor’s screen displays a clean scene without any tear because the V-sync is adequate.
(Figure 5) Screen of different V-sync [34]
3.6 Rig
A rig is a device used to combine two or more cameras for filming. Several cameras are required for producing images, such as three-dimensional and VR images. There are different types of rigs, such as horizontal, vertical, and 360ºrigs (Figure 6), and the type used depends on the type of image required.
(Figure 6) Different types of rigs [35]
If a rig is used and if the rig’s cameras are not adjusted precisely or aligned close to the no-parallax point, HMD users could feel uncomfortable because of distortion between images. Therefore, it is essential to position the rig precisely when creating content. Because the size of cameras used for producing VR images is large due to their high performance, highly precise adjustments are required for these cameras; their size makes it difficult to align them close to the no-parallax point between optical instruments [28].
3.7 Resolution
The conceptual definition of resolution is the number of pixels on the screen. As illustrated in Figure 7, the number of pixels and display space vary with the resolution. ahigher resolution corresponds to a larger number of pixels on the screen, a larger space, and greater detail.
(Figure 7) Various resolutions
The sense of immersion and the comfort level of users of VR content depend on the resolution. Furthermore, in contrast to ordinary video devices, VR device optical systems involve convex lenses. Therefore, for a comfortable experience, VR device users require a higher resolution than universal image display environments [28].
4. Method
No previous study has ranked the major factors responsible for cybersickness with HMDs. In this study, six dimensions were identified and compared for different standards for determining guidelines for the production of HMDs characterized by reduced cybersickness. The standards considered were IEEE standards, which are international standards, TTA standards, which are South Korea’s domestic standards, and factors obtained from previous studies on cybersickness. The dimensions were FoV, latency, framerate, V-sync, rig, and resolution.
This study used the Delphi method, widely used in technology, education, and policy-making [36]. Because the Delphi method has a score-ranking weakness in which participants cannot identify differences between items with different ranks [37], the analytic hierarchical process (AHP) method was also used. In the AHP method, the entire decision-making process is divided into several stages. Each stage is then analyzed and interpreted stepwise for reasonable decision-making [38]. In this study, the AHP method was also used to identify and rank VR cybersickness-causing factors.
A total of 20 VR experts participated in this study. They comprised officials from a government-funded research institute, VR/AR content development companies, university professors, doctors, and media professionals actively engaged in research on cybersickness reduction. All participants had experienced and studied cybersickness problems directly or indirectly. These experts were chosen based on their knowledge of the cybersickness problem, nature of work, and experience.
This study obtained optimal results by considering the scope of HMD users’ experience. Accordingly, the number of VR/augmented reality contents used and the time taken to view each of the contents were considered, as depicted in Table 2. Furthermore, the type of content preferred by the participating experts was also considered to minimize the experts’ personal bias in evaluating cybersickness-casing factors.
(Table 2) Demographic characteristics of experts participated in Delphi survey
In this study, the measurement items were incorporated into email-based questionnaires to analyze and control data collected from the experts and increase survey result accuracy [39].
4.1 Delphi Survey
The Delphi research method, which involves collecting survey information from experts and collecting opinions, was introduced by Dalkey and Helmer (1963) [40]. It is a tool for experts to organize and prioritize collected factors for solving problems from a different perspective [41], and it has a logical basis. It is based on the decision-making principle that many opinions are more reliable than several. The Delphi survey is conducted mainly through the mail rather than physically from a gathering of experts. Consequently, opinions are provided and collected through email.
The Delphi research method involves repeated surveys from anonymous experts to evolve a collective consensus, without the experts gathering in person to debate. Usually, opinion coordinators in charge of the surveys obtain the opinions of 10–15 experts two or three times. The experts receive feedback from other experts, which produces the average or median value of each survey.
In this study, Delphi expert surveys were conducted four times to develop indicators for evaluating the comfort experience of HMDs. The surveys were conducted between October 1, 2018, and November 2, 2018. In the surveys, concept definitions, types of factors, and indicator pools obtained from a literature review were presented as examples to indicate research direction.
The Delphi research method was conducted using the process presented in Figure 8 and Table 3. Experts assessed the validity based on other experts’opinions collected through open-ended questions. Each expert answered a questionnaire with a seven-point scale. After a statistical analysis of the response results, box and whisker charts were prepared to identify the measurement variable, the average of the experts’ responses, the scores of the experts’ responses, and the quartiles of the experts’ responses. In this process, experts verified the feedback and statistical analysis results prepared by other experts and corrected their presented scores.
(Figure 8) Delphi survey outline
(Table 3) Delphi survey outline for developing
In the first Delphi survey, a validity analysis of an HMD’s dimension classification for cybersickness was conducted, and the definition of each factor was collected. In the second Delphi survey, the dimension classification was re-evaluated, factors were classified, and a validity analysis was conducted based on the dimension classification for HMD cybersickness reduction. In the third Delphi survey, factors based on the dimension classification were re-evaluated, the evaluation indicators of the classified factors were chosen, and a validity analysis was conducted. In the fourth Delphi survey, the factors were re-evaluated, the weights of the evaluation indicators were calculated, and a ranking survey was conducted.
The results of the Delphi survey were analyzed using SPSS 25 (Windows version). Statistics such as mean, percentage, standard deviation, and frequency were computed.
5. Results
5.1 Dimension classification and definition
The factors causing cybersickness in a VR environment were collected and compared against different standards for two purposes: 1) to determine guidelines for producing HMDs characterized by reduced cybersickness and 2) to classify and define the dimensions of the evaluation indicators for the comfort experience of HMDs. The standards considered were IEEE standards, which are international standards, TTA standards, which are South Korea’s domestic standards, and factors obtained from previous papers on cybersickness. In the Delphi surveys, the evaluation frame was presented in six dimensions: FoV, latency, framerate, V-sync, rig, and resolution. As depicted in Figure 9, the evaluation indicators were developed (Evaluating for Comfort of HMD; ECH). Table 4 presents the definitions of the factors.
(Figure 9) Evaluating for Comfort of HMD (ECH)
(Table 4) Definition of factors
Based on the Delphi surveys, the six dimensions of the evaluation indicators were defined. The detailed factors for the dimensions were identified as FoV–device FoV, latency –device latency, framerate–device framerate, V-sync– device V-sync, rig–camera-angle view, rig–no-parallax point, resolution–device resolution, and resolution–pixels per inch (PPI).
5.2 Factor Classification
In the HMD comfort experience evaluation, factor classification was the most fundamental component. It was presented in the Delphi surveys by assigning sub-factors for each dimension based on the literature review; the classification is depicted in Table 5.
(Table 5) Factors classification reference
5.3 Delphi Analysis Result
In the first Delphi survey, the dimensions of HMD-induced cybersickness were classified and defined. Moreover, a validity analysis was conducted based on opinions collected through open-ended questions. A total of 20 experts (100%) participated in the first Delphi survey.
The validity analysis results of the first Delphi survey for the dimension classification and definitions are depicted in Table 6. The experts agreed on the definitions of the evaluation indicator dimensions for the comfort experience evaluation for HMDs, with most central values being 6 (reasonable) and the minimum being 5 (slightly reasonable).
(Table 6) Dimension classification validity analysis
In the second Delphi survey, the HMD cybersickness dimension classification provided in the first Delphi questionnaire was reassessed. A validity analysis of the factor classification based on the dimension classification was performed. In this survey, 15 experts (75%) participated in the re-evaluation through the second Delphi questionnaire, while all 20 experts (100%) participated in the factor classification.
The experts agreed on the results of the revaluation of the HMD cybersickness dimension classification and definitions in the second Delphi survey, with the median value of most dimensions being 6 (reasonable), as depicted in Table 7. The value of "rig"was also relatively higher compared to the first survey. The second step of the survey confirmed the validity of HMD dimension classification and definition.
(Table 7) Re-evaluation of the dimension classification
The validity analysis results of the factor classification based on the dimension classification for HMD-induced cybersickness conducted in the second Delphi survey are presented in Table 8. The experts expressed their opinions on the factor classification for the evaluation indicators for the comfort experience of HMDs. The scored values were mostly 6 (reasonable), and the minimum value was 5 (slightly reasonable).
(Table 8) Factor classification validity analysis
In the third Delphi survey, the HMD cybersickness factor classification obtained in the second Delphi survey was reassessed. The reassessment was performed through the third Delphi questionnaire and involved 14 experts (70%).
The revaluation results in the third Delphi survey were mostly 6 (reasonable), and the minimum value was 5 (slightly reasonable), as depicted in Table 9. The re-evaluation analyses performed in the second step demonstrated the validity of the HMD cybersickness factor classification.
(Table 9) Re-evaluation of factor classification [3rd] Re-evaluation
5.4 Selection of Measurement Indicators
In the third Delphi survey, the indicator pool, which could be quantified and measured, was obtained through a literature review and used to select evaluation indicators and analyze validity. The validity of each factor’s evaluation criteria was assessed on a seven-point scale by referring to the indicator pool. The measurement criteria of the pool presented for each factor are depicted in Table 10.
(Table 10) Measurement criteria of the pool
All 20 experts (100%) participated in the selection and validity analysis of the evaluation indicators in the third Delphi survey.
In this survey, in the validity analysis performed to choose the measurement indicators, the median value for most of the measurement indicators presented was 6 (suitable), as depicted in Table 11.
(Table 11) Selection of measurement indicators delphi result
In the validity analysis of the evaluation indicators for "camera angle view, " a sub-factor of the rig, the median value was 5(slightly reasonable), which revealed relatively lower validity than the other factors. However, this value validates the measurement criteria indicator.
5.5 Calculation of Weights
Based on the indicators chosen through the survey analysis, the weights for each factor were calculated in the fourth Delphi survey, in which all 20 experts (100%) participated.
From the survey results, the evaluation indicators for the comfort experience of VR headsets were developed as follows.
The AHP analysis revealed a consistency index (C.I.) of 0.01, indicating that the weights were consistent by experts’responses. As depicted in Table 12, the final weights for the dimensions were as follows: latency, 0.24; framerate, 0.20; resolution, 0.20; FoV, 0.15; V-sync, 0.11; and rig, 0.11. Therefore, the ranking of the dimensions was as follows: first priority, latency; joint second priority, framerate and resolution; third priority, FoV; and joint fourth priority, V-sync and rig.
(Table 12) ECH ranks and weights
The weights for the factors of the dimensions were as follows: device latency, 0.19; device framerate, 0.16; device resolution, 0.14; PPI, 0.14; device FoV, 0.10; camera angle view, 0.09; no-parallax point, 0.09; and device V-sync, 0.08. Therefore, the ranking of the factors was as follows: first priority, device latency; second priority, device framerate; joint third priority, device resolution and PPI; fourth priority, device FoV; joint fifth priority, camera angle view and no-parallax point; and sixth priority, device V-sync.
6. Conclusion
This study aimedto develop indicators for the practical evaluation of the comfort experience of HMDs by classifying and prioritizing cybersickness-causing factors presented by previous studies and in standard documents. The priorities of the dimensions and factors were derived from the results of the four Delphi expert surveys.
Based on the priorities, latency is the most crucial consideration for the comfort experience of HMDs . the time interval for the user’s real-time operation to be displayed as a video signal in the HMD is the dimension that most affects the user’s comfort level. Next in the priority order are framerate and resolution, implying that device specification that determines the realistic expression of VR through displays strongly influences the user’s sense of immersion and the occurrence of cybersickness. The third priority is FoV, which indicates that HMD users are likely to experience cybersickness if they perceive the viewing angle differently from that in the real world. The lowest priorities are V-sync and the rig, suggesting that, if distortion occurs in the VR image of the HMD, its effect on the comfort experience is relatively smaller than the effects of the preceding dimensions. However, distortion induces cybersickness.
Despite many previous studies regarding VR cybersickness-causing factors, no study has ranked cybersickness- causing factors for HMDs. In this study, the relative importance of the evaluation indicators was determined by calculating weights for all dimensions and sub-factors. The results of this study can be used as a reference for future studies. Moreover, as a practical contribution, developers can conduct technical development while considering each factor’s ranking.
However, considering that this study is a first attempt to rank cybersickness-causing factors for HMDs, additional studies are required to objectify the VR HMDs evaluation index presented in this study. Furthermore, determining how to reflect individual differences due to personal experiences or content- related cybersickness-causing factors should be considered in future research.
참고문헌
- Caglar Yildirim, "Cybersickness during VR gaming undermines game enjoyment: A mediation model," Displays, vol. 59, pp. 35-43, September, 2019. https://doi.org/10.1016/j.displa.2019.07.002
- Andrej Somrak, Iztok Humar, M. Shamim Hossain, Mohammed F. Alhamid, M. Anwar Hossain and Joze Guna, "Estimating VR Sickness and user experience using different HMD technologies: An evaluation study," Future Generation Computer Systems, vol. 94, pp. 302-316, May, 2019. https://doi.org/10.1016/j.future.2018.11.041
- Robert S. Kennedy and Kay M. Stanney, "Postural instability induced by virtual reality exposure: development of a certification protocol," International Journal of Human-Computer Interaction, vol. 8, no. 1, pp. 25-47, January, 1996. https://doi.org/10.1080/10447319609526139
- Mark S. Dennison, A. Zachary Wisti and Michael D'Zmura, "Use of physiological signals to predict cybersickness," Displays, vol. 44, pp. 42-52, September, 2016. https://doi.org/10.1016/j.displa.2016.07.002
- Juno Kim, Wilson Luu and Stephen Palmisano, "Multisensory integration and the experience of scene instability, presence and cybersickness in virtual environments," Computers in Human Behavior, July, 2020. https://doi.org/10.1016/j.chb.2020.106484
- Seamas Weech, Sophie Kenny, Markus Lenizky and Michael Barnett-Cowan, "Narrative and gaming experience interact to affect presence and cybersickness in virtual reality," International Journal of Human-Computer, vol. 138, June, 2020. https://doi.org/10.1016/j.ijhcs.2020.102398
- J. T. Reason and J. J. Brand, Motion Sickness, Academic Press, London, 1975. https://nla.gov.au/nla.cat-vn247634
- M. Griffin, Handbook of Human Vibration, Academic Press, London, 1990. https://doi.org/10.1121/1.401606
- Thomas Stoffregen and Gary Riccio, "An Ecological Critique of the Sensory Conflict Theory of Motion Sickness," Ecological Psychology, vol. 3, no. 3, pp. 159-194, September, 1991. https://doi.org/10.1207/s15326969eco0303_1
- E. Lee and K. Gam, "Predictability of Visually Induced Motion Sickness by Orientation Illusion," Journal of Social Science, vol. 53, no. 2, pp. 261-280, December, 2014. https://www.dbpia.co.kr/Journal/articleDetail?nodeId=NODE06094044
- E. Chang, D. Seo, H.T. Kim and B. Yoo, "An Integrated Model of Cybersickness: Understanding User's Discomfort in Virtual Reality," Journal of KIISE, vol. 45, no. 3, pp. 251-279, March, 2018. http://doi.org/10.5626/JOK.2018.45.3.251
- B. Keshavarz, H. Hecht and L. Zschutschke, "Intravisual conflict in visually induced motion sickness," Displays, vol. 32, no. 4, pp. 1810188, October, 2011. https://doi.org/10.1016/j.displa.2011.05.009
- R. Vlad, O. Nahorna, P. Lardet and A. Guerin, "The influence of the visualization task on the simulator sickness symptoms - a comparative ssq study on 3dtv and 3d immersive glasses," Proc. Of the 3DTV-Conference: The True Vision- Capture, Transmission and Displays on 3D Video (3DTV-CON), 2013, pp. 1-4, 2013. https://doi.org/10.1109/3dtv.2013.6676647
- K. Kim, M.Z. Rosenthal, D. Zielinski and R. Brady, "Effects of virtual environment platforms on emotional responses," Computer methods and programs in biomedicine, vol. 113, no. 3, pp. 882-893, March, 2014. https://doi.org/10.1016/j.cmpb.2013.12.024
- M.L. van Emmerik, S.C. de Vries and J.E. Bos, "Internal and external fields of view affect cybersickness," Displays, vol. 32, no. 4, pp. 169-174, October, 2011. https://doi.org/10.1016/j.displa.2010.11.003
- A.S. Fernandes and S.K. Feiner, "Combating vr sickness through subtle dynamic field-of-view modification," Proc. of the 3D User Interfaces(3DUI), 2016 IEEE Symposium on, pp. 201-210, 2016. https://doi.org/10.1109/3dui.2016.7460053
- S. Davis, K. Nesbitt and E. Nalivaiko, "A systematic review of cybersickness," Proc. of the 2014 Conference on Interative Entertainment, pp. 1-9, 2014. https://doi.org/10.1145/2677758.2677780
- H. Renkewitz and T. Alexander, "Perceptual issues of augmented and virtual environments", 2007. http://www.dtic.mil/get-tr-doc/pdf?AD=ADA474154
- L.R. Rebentisch, "Cybersickness prioritization and modeling," Computer Science, Michigan State University, ProQuest Dissertations Publishing, 2015. https://www.semanticscholar.org/paper/Cybersickness-prioritization-and-modeling-Rebenitsch/477b2c1dbb7bfc48dfcb40c3bc7bff689832c91b
- TTA, TTAK.KO-10.1030, "Head Mounted Display based Virtual Reality Content Production Guide for Reducing of Virtual Reality Sickness", December, 2017. http://www.tta.or.kr/data/ttas_view.jsp?pk_num=TTAK.KO-10.1030
- Electronics and Telecommunications Research Institute, report no.TRKO201900003994, 1st annual report version, "(TTA, 2017) Development of a method for regulating human-factor parameters for reducing VR-induced sickness", Ministry of Science and ICT, November, 2017. https://scienceon.kisti.re.kr/srch/selectPORSrchReport.do?cn=TRKO201900003994
- Oculus, "Oculus Best Practices Version 310-30000-02," 2017. https://www.oculus.com/
- C. Choi, "The Fourth Industrial Revolution and Development Strategies and Implications of e-Business Companies," The e-Business Studies, vol. 18, no. 3, pp. 39-54, June, 2017. https://doi.org/10.20462/tebs.2017.06.18.3.39
- Jungyeob Han, "Study on the Feature of Mobile HMD-Based VR Experience Contents Design," Journal of the Korea Institute of Spatial Design, vol. 10, no. 5, pp. 199-207, October, 2015. https://doi.org/10.35216/KISD.2015.10.5.199
- Qiao Rongrong and Dongsoong Han, "A Study on the Virtual reality sickness Measurement of HMD-based Contents Using SSQ," Journal of Korea Game Society, vol. 18, no. 4, pp. 15-32, August, 2018. https://doi.org/10.7583/JKGS.2018.18.4.15
- Sue V. G. Cobb, Sarah Nichols, AmandaRamsey and John R. Wilson, "Virtual Reality-Induced Symptoms and Effects (VRISE)," Presence, vol. 8, no. 2, pp. 169-186, April, 1999. https://doi.org/10.1162/105474699566152
- Sarah Sharples, Sue Valerie Gray Cobb, Amanda Moody and John R. Wilson, "Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems," Displays, vol. 29, no. 2, pp.58-69, September, 2007. https://doi.org/10.1016/j.displa.2007.09.005
- TTA, TTAK.KO-10.1030/R1, "Head Mounted Display based Virtual Reality Content Production Guide for Reducing of Virtual Reality Sickness (Revision)", December, 2018. http://www.tta.or.kr/data/ttas_view.jsp?pk_num=TTAK.KO-10.1030
- What are survey accurate visual simulations?. Retrieved March 19, 2020. https://buildmedia.com/portfolio-items/what-are-survey-accurate-visual-simulations/.
- E. M. Kolasinski, "Simulator Sickness in Virtual Environments" (Technical Report. 1027) Alexandria, VA:US Army Research Institute for the Behavioral and Social Sciences, 1995. https://doi.org/10.21236/ADA295861
- Georgi Yordanov Kostov, "Fostersing Player Collaboration Within a Multimodal Co-Located Game," FachhochSchule, Hagenberg, 2015. https://doi.org/10.13140/rg.2.1.3865.7044
- B. Iribe, "Virtual Reality- A New Frontier in Computing", 2013. Retrieved March 19, 2020. http://www.slideshare.net/DevCentralAMD/keynote-virtual-reality-a-new-frontierin-computing-by-brendan-iribe-ceo.
- D. Brunner, "Frame Rate: A Beginner's Guide", 2017. Retrieved March 19, 2020. https://www.techsmith.com/blog/frame-rate-beginners-guide/.
- AMD, Enhanced Sync: Tech Talk, Retrieved March 19, 2020. https://www.amd.com/en/technologies/radeon-software-enhancedsync.
- [360 camera] Build Up Camera Rig for 360 stereo video, 2016. Retrieved March 19, 2020. http://vr-project.blogspot.com/2016/12/build-up-camera-rig-uncompleted.html.
- Gohar Feroz Khan, Junghoon Moon, C. Rhee and J.J. Rho, "E-government skills identification and development: toward a staged-based user-centric approach for developing countries," Asia Pacific Journal of Information Systems, vol. 21, no. 1, pp. 1-31, January, 2010. http://ksci.kisti.re.kr/search/article/articleView.ksci?articleBean.atclMgntNo=HGKJCF_2010_v20n1_1
- J. Daniel Couger, "Key human resource issues in IS in the 190-s: view of IS executives versus human resource executives", Information and Management, vol. 14, no. 4, pp.161-174, 1998. https://doi.org/10.1016/0378-7206(88)90055-9
- M.K. Chen and Shih-Ching Wang, "The critical factors of success for information service industry in developing international market: using analytic hierarchy process (AHP) approach", Expert Systems with Applications, vol. 27, no. 1, pp.694-704, January, 2010. https://doi.org/10.1016/j.eswa.2009.06.012
- Philip M. Podsakoff, Scott.B. Mackenzie, Jeongyeon Lee and Nathan P. Podsakoff, "Common method biases in behavioral research: a critical view of the literature and recommended remedies" Journal of Applied Psychology, vol. 88, no. 5, pp. 879-903, 2003. https://doi.org/10.1037/0021-9010.88.5.879
- Norman Dalkey and Olaf Helmer, "An experimental application of the Delphi method to the use of experts", Management Science, vol. 9, no. 3, pp. 458-467, April, 1963. https://doi.org/10.1287/mnsc.9.3.458
- M. Turoff, S.R. Hilts, "Computer based Delphi processes", 1995. Retrieved March 19, 2020. https://www.semanticscholar.org/paper/Computer-basedDelphi-processes-Turoff-Hiltz/11b67e214fd016c72d339ba3bfc5e075c757747b