Browse > Article
http://dx.doi.org/10.3837/tiis.2021.07.013

Overview of Motion-to-Photon Latency Reduction for Mitigating VR Sickness  

Ryu, Yeongil (Department of Computer Education, Sungkyunkwan University)
Ryu, Eun-Seok (Department of Computer Education, Sungkyunkwan University)
Publication Information
KSII Transactions on Internet and Information Systems (TIIS) / v.15, no.7, 2021 , pp. 2531-2546 More about this Journal
Abstract
For several years, virtual reality (VR) and augmented reality (AR) technologies have been improving. However, some hurdles remain that slow down the distribution of VR and AR devices, such as head-mounted display (HMD), and related consumer content. One issue is VR motion sickness, which has been experienced by users using 360 degree VR content via HMD. This paper discusses the related international standardization work that classifies the factors causing VR sickness, and proposes the process for VR sickness level evaluation. Among the factors causing VR sickness, many research institutes regard minimizing MTP (Motion-to-Photon) latency as the key enabler to mitigate VR sickness. Thus, this paper introduces research trends of MTP latency measurement and MTP latency mitigation. This paper categorizes the research on MTP latency measurement into 2 categories of hardware-based approach and software code-level approach. The 2 approaches have different pros and cons depending on use-case, purpose, and architecture of each multimedia system. The pros and cons are addressed in this paper. Additionally, the research on mitigating MTP latency with diverse strategies such as proactive computing, caching, and edge server technology is explained, and compared to conventional technologies, shows improved performance.
Keywords
Motion-to-Photon latency; tile; video coding; viewport; VR sickness; 360 degree videos;
Citations & Related Records
연도 인용수 순위
  • Reference
1 R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre, "Recent advances in augmented reality," IEEE Comput. Graph. Appl., vol. 21, no. 6, pp. 34-47, 2001.   DOI
2 J. M. Boyce, Y. Ye, J. Chen, and A. K. Ramasubramonian, "Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 1, pp. 20-34, 2016.   DOI
3 M. S. Elbamby, C. Perfecto, M. Bennis, and K. Doppler, "Toward Low-Latency and Ultra-Reliable Virtual Reality," IEEE Network, vol. 32, no. 2, pp. 78-84, 2018.   DOI
4 P. M. Sharkey, D. W. Murray, and J. J. Heuring, "On the kinematics of robot heads," IEEE Transactions on Robotics and Automation, vol. 13, no. 3, pp. 437-442, 1997.   DOI
5 G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand, "Overview of the High Efficiency Video Coding (HEVC) Standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649-1668, 2012.   DOI
6 Y. Bao, H. Wu, T. Zhang, A. A. Ramli, and X. Liu, "Shooting a moving target: Motion-prediction-based transmission for 360-degree videos," in Proc. of IEEE International Conference on Big Data, pp. 1161-1170, 2016.
7 S. W. Choi, S. Lee, Seo, M. W. Seo, and S. J. Kang, "Time sequential motion-to-photon latency measurement system for virtual reality head-mounted displays," Electronics, vol. 7, no. 9, pp. 171, 2018.   DOI
8 M. Domanski, O. Stankiewicz, K. Wegner, and T. Grajek, "Immersive visual media - MPEG-I: 360 video, virtual navigation and beyond," in Proc. of 2017 International Conference on Systems, Signals and Image Processing, Poznan, pp. 1-9, 2017.
9 M. Wien, J. M. Boyce, T. Stockhammer, and W. Peng, "Standardization Status of Immersive Video Coding," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 1, pp. 5-17, 2019.   DOI
10 B. Iribe, "Virtual reality-A new frontier in computing," in Proc. of the AMD Announces 2013 Developer Summit, San Jose, CA, USA, 2013.
11 J. Funda, R. H. Taylor, and, R. P. Paul, "On homogeneous transforms, quaternions, and computational efficiency," IEEE Transactions on Robotics and Automation, vol. 6, no. 3, pp. 382-388, 1990.   DOI
12 SM05PD Mounted Photodiodes - SM05 and SM1 Compatible Specifications.
13 de la Fuente, Y. Sanchez, R. Skupin, and T. Schierl, "Video processing for panoramic streaming using HEVC and its scalable extensions," Multimedia Tools and Applications, vol. 76, no. 4, pp. 5631-5659, 2017.   DOI
14 A. Zare, A. Aminlou, M. M. Hannuksela, and M. Gabbouj, "HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications," in Proc. of the 24th ACM international conference on Multimedia, pp. 601-605, 2016.
15 J. Son, D. Jang, and E. S. Ryu, "Implementing motion-constrained tile and viewport extraction for VR streaming," in Proc. of the 28th ACM SIGMM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 61-66, 2018.
16 X. Dong, Y. Yan, M. Tan, Y. Yang, and I. W. Tsang, "Late Fusion via Subspace Search with Consistency Preservation," IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 518-528, 2019.   DOI
17 S. Shi, V. Gupta, M. Hwang, and R. Jana, "Mobile VR on edge cloud: a latency-driven design," in in Proc. of the 10th ACM Multimedia Systems Conference, pp. 222-231, 2019.
18 E. Vig, M. Dorr, and D. Cox, "Large-scale optimization of hierarchical features for saliency prediction in natural images," in Proc. of the IEEE conference on computer vision and pattern recognition, pp. 2798-2805, 2014.
19 S. Shi, and C.-H. Hsu, "A Survey of Interactive Remote Rendering Systems," ACM Comput. Surv., vol. 47, no. 4, pp. 1-29, 2015.
20 T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, "Overview of the H.264/AVC video coding standard," IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560-576, 2003.   DOI
21 A. Mahzari, A. Taghavi Nasrabadi, A. Samiei, and R. Prakash, "FoV-aware edge caching for adaptive 360 video streaming," in Proc. of the 26th ACM international conference on Multimedia, pp. 173-181, 2018.
22 J. He, M. A. Qureshi, L. Qiu, J. Li, F. Li, and L. Han, "Rubiks: Practical 360-Degree Streaming for Smartphones," in Proc. of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pp. 482-494, 2018.
23 D. Ochi, Y. Kunita, A. Kameda, A. Kojima, and S. Iwaki, "Live streaming system for omnidirectional video," in Proc. of IEEE Virtual Reality, pp. 349-350, 2015.
24 M. Graf, C. Timmerer, and C. Mueller, "Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation," in Proc. of the 8th ACM on Multimedia Systems Conference, pp. 261-271, 2017.
25 P. R. Alface, J. Macq and, N. Verzijp, "Interactive omnidirectional video delivery: A bandwidth-effective approach," Bell Labs Technical Journal, vol. 16, no. 4, pp. 135-147, 2012.   DOI
26 IEEE Std 3079-2020, IEEE, 2020.
27 H. Chen and Y. Xiao, "Cache access and replacement for future wireless Internet," IEEE Communications Magazine, vol. 44, no. 5, pp. 113-123, 2006.   DOI
28 C. Wu, Z. Tan, Z. Wang, and S. Yang, "A dataset for exploring user behaviors in VR spherical video streaming," in Proc. of the 8th ACM on Multimedia Systems Conference, pp. 193-198, 2017.
29 S. Gul, D. Podborski, T. Buchholz, T. Schierl, and C. Hellge, "Low-latency cloud-based volumetric video streaming using head motion prediction," in Proc. of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 27-33, 2020.
30 F. Qian, B. Han, L. Ji, and V. Gopalakrishnan, "Optimizing 360 video delivery over cellular networks," in Proc. of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, pp. 1-6, 2016.
31 M. W. Seo, S. W. Choi, S. L. Lee, E. Y. Oh, J. S. Baek, and S. J. Kang, "Photosensor-based latency measurement system for head-mounted displays," Sensors, vol. 17, no. 5, pp. 1112, 2017.   DOI
32 M. Hosseini and V. Swaminathan, "Adaptive 360 VR Video Streaming: Divide and Conquer," in Proc. of 2016 IEEE International Symposium on Multimedia, San Jose, pp. 107-110, 2016.
33 C. Feldmann, C. Bulla, and B. Cellarius, "Efficient stream-reassembling for video conferencing applications using tiles in HEVC," in Proc. of the International Conferences on Advances in Multimedia, pp. 130-135, 2013.
34 J. Gutierrez, E. J. David, A. Coutrot, M. P. Da Silva, and P. L. Callet, "Introducing UN Salient360! Benchmark: A platform for evaluating visual attention models for 360° contents," in Proc. of 2018 Tenth International Conference on Quality of Multimedia Experience, pp. 1-3, 2018.
35 D. D. Salvucci, and J. H. Goldberg, "Identifying fixations and saccades in eye-tracking protocols," in Proc. of the symposium on Eye tracking research & applications, pp. 71-78, 2000.
36 S. Lee, D. Jang, J. Jeong, and E. S. Ryu, "Motion-constrained tile set based 360 degree video streaming using saliency map prediction," in Proc. of the 29th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 20-24, 2019.