KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Using Freeze Frame and Visual Notifications in an Annotation Drawing Interface for Remote Collaboration

  • Kim, Seungwon (The School of Information Technology and Mathematical Sciences, The University of South Australia) ;
  • Billinghurst, Mark (The School of Information Technology and Mathematical Sciences, The University of South Australia) ;
  • Lee, Chilwoo (School of Electoronics and Computer Engineering, Chonnam National University) ;
  • Lee, Gun (The School of Information Technology and Mathematical Sciences, The University of South Australia)
  • Received : 2018.05.04
  • Accepted : 2018.08.12
  • Published : 2018.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

This paper describes two user studies in remote collaboration between two users with a video conferencing system where a remote user can draw annotations on the live video of the local user's workspace. In these two studies, the local user had the control of the view when sharing the first-person view, but our interfaces provided instant control of the shared view to the remote users. The first study investigates methods for assisting drawing annotations. The auto-freeze method, a novel solution for drawing annotations, is compared to a prior solution (manual freeze method) and a baseline (non-freeze) condition. Results show that both local and remote users preferred the auto-freeze method, which is easy to use and allows users to quickly draw annotations. The manual-freeze method supported precise drawing, but was less preferred because of the need for manual input. The second study explores visual notification for better local user awareness. We propose two designs: the red-box and both-freeze notifications, and compare these to the baseline, no notification condition. Users preferred the less obtrusive red-box notification that improved awareness of when annotations were made by remote users, and had a significantly lower level of interruption compared to the both-freeze condition.

Keywords

References

  1. Ronald T. Azuma, "A survey of augmented reality," Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385, 1997. https://doi.org/10.1162/pres.1997.6.4.355
  2. Bhaskar Bhattacharya and Eliot Winer, "A method for real-time generation of augmented reality work instructions via expert movements," The Engineering Reality of Virtual Reality 2015, International Society for Optics and Photonics, Vol.9392 , March, 2015.
  3. S. Webel, U. Bockholt, T. Engelke, N. Gavish, M. Olbrich, and C. Preusche, "An augmented reality training platform for assembly and maintenance skills," Robotics and Autonomous Systems, vol. 61, no. 4, pp. 398-403, 2013. https://doi.org/10.1016/j.robot.2012.09.013
  4. Steven J. Henderson and Steven K. Feiner, "Augmented reality in the psychomotor phase of a procedural task," in Proc. of IEEE International Symposium on Mixed and Augmented Reality ,pp. 191-200, October 2011.
  5. Rafael Radkowski, Jordan Herrema, and James Oliver, "Augmented reality-based manual assembly support with visual features for different degrees of difficulty," International Journal of Human-Computer Interaction, vol. 31, no. 5, pp. 337-349, 2015. https://doi.org/10.1080/10447318.2014.994194
  6. Nils Petersen and Didier Stricker, "Learning task structure from video examples for workflow tracking and authoring," in Proc. of IEEE International Symposium on Mixed and Augmented Reality, pp. 237-246, November 2012.
  7. Guido Maria Re and Monica Bordegoni, "An augmented reality framework for supporting and monitoring operators during maintenance tasks," in Proc. of International Conference on Virtual, Augmented and Mixed Reality, pp. 443-454, June 2014.
  8. Dima Damen, Teesid Leelasawassuk, and Walterio Mayol-Cuevas, "You-Do, I-Learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance," Computer Vision and Image Understanding, vol. 149, pp. 98-112, August 2016. https://doi.org/10.1016/j.cviu.2016.02.016
  9. Susan R. Fussell, Leslie D. Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer and Adam D.I. Kramer, "Gestures over video streams to support remote collaboration on physical tasks," Human-Computer Interaction, vol. 19, no. 3, pp. 273-309. November 12 2009.
  10. J. R. Brubaker, G. Venolia, and J. C. Tang, "Focusing on shared experiences: moving beyond the camera in video communication" in Proc. of Designing Interactive Systems Conference, pp. 96-105, June 11-15, 2012.
  11. S. R. Fussell, R. E. Kraut, and J. Siegel, "Coordination of communication: Effects of shared visual context on collaborative work," in Proc. of the 2000 ACM conference on Computer supported cooperative work, pp. 21-30, December 2000. ACM.
  12. Seungwon Kim, Gun A. Lee, and Nobuchika Sakata, (2013, October). "Comparing pointing and drawing for remote collaboration," in Proc. of IEEE International Symposium on Mixed and Augmented Reality, pp. 1-6, 2013.
  13. S. Gauglitz, C. Lee, M. Turk, and T. Hollerer, "Integrating the physical environment into mobile remote collaboration," in Proc. of the 14th international conference on Human-computer interaction with mobile devices and services, pp. 241-250, September 2012.
  14. S. Gauglitz, B. Nuernberger, M. Turk, and T. Hollerer, "World-stabilized annotations and virtual scene navigation for remote collaboration," in Proc. of the 27th annual ACM symposium on User interface software and technology, pp. 449-459, October 2014.
  15. S. Kim, G. Lee, N. Sakata, and M. Billinghurst, "Improving co-presence with augmented visual communication cues for sharing experience through video conference," in Proc. of 2014 IEEE International Symposium on Mixed and Augmented Reality, pp. 83-92, September 2014.
  16. D. Kirk, T. Rodden, and D. S. Fraser, "Turn it this way: grounding collaborative action with remote gestures," in Proc. of the SIGCHI conference on Human Factors in Computing Systems, pp. 1039-1048, April 2007.
  17. R. S. Sodhi, B. R. Jones, D. Forsyth, B. P. Bailey, and G. Maciocci, "BeThere: 3D mobile collaboration with spatial input," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 179-188, April 2013.
  18. J. Rekimoto and K. Nagao, "The world through the computer: Computer augmented interaction with real world environments," in Proc. of Proceedings of the 8th annual ACM symposium on User interface and software technology, pp. 29-36, December 1995.
  19. S. Kim, G. A. Lee, S. Ha, N. Sakata, and M. Bllinghurst, "Automatically freezing live video for annotation during remote collaboration," in Proc. of Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1669-1674, April 2015.
  20. Joerg Hauber, "Understanding Remote Collaboration in Video Collaborative Virtual Environments," PhD Thesis, University of Canterbury, 2008.
  21. Gary M. Olson and Judith S. Olson, " Distance matters," Human-computer interaction, vol. 15, no. 2, pp. 139-178, December 09 2009. https://doi.org/10.1207/S15327051HCI1523_4
  22. R.E. Kraut, M. D. Miller and J. Siegel, "Collaboration in performance of physical tasks: Effects on outcomes and communication," in Proc. of Proceedings of the ACM conference on Computer supported cooperative work, pp. 57-66, November 1999. ACM.
  23. S. R. Fussell, L. D. Setlock, and R. E. Kraut, "Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks," in Proc. of Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 513-520, April 2003.
  24. P. Dourish, and V. Bellotti, "Awareness and coordination in shared workspaces," in Proc. of Proceedings of the ACM conference on Computer-supported cooperative work, pp. 107-114, December 1992.
  25. C. Gutwin and S. Greenberg, "Workspace awareness for groupware," in Proc. of Proceedings of the Conference on Human Factors in Computing Systems, pp.208-209, 1996.
  26. John C. Tang and Scott L. Minneman, "VideoDraw: a video interface for collaborative drawing," ACM Transactions on Information Systems (TOIS), vol. 9, no.2, pp.170-184, April 1991. https://doi.org/10.1145/123078.128729
  27. Hiroshi Ishii, Minoru Kobayashi and Kazuho Arita, "Iterative design of seamless collaboration media," Communications of the ACM, vol.37, no.8, pp.83-97, August 1994. https://doi.org/10.1145/179606.179687
  28. S. Chen, M. Chen, A. Kunz, A. E. Yantac, M. Bergmark, A. Sundin, and M. Fjeld, "SEMarbeta: mobile sketch-gesture-video remote support for car drivers," in Proc. of Proceedings of the 4th Augmented Human International Conference on - AH, pp.69-76, 2013
  29. W. L. Koh, J. Kaliappan, M. Rice, K. T. Ma, H. H. Tay, and W. P. Tan "Preliminary investigation of augmented intelligence for remote assistance using a wearable display," in Proc. of TENCON 2017-2017 IEEE Region 10 Conference, pp. 2093-2098. November 5-8 2017.
  30. M. Rice, S. C. Chia, H. H. Tay, M. Wan, L. Li, J. Ng, and J. H. Lim, "Exploring the use of visual annotations in a remote assistance platform," in Proc. of Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1295-1300, May 2016.
  31. H. Kato and M. Billinghurst, "Marker tracking and HMD calibration for a video-based augmentedreality conferencing system," in Proc. of Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality, October 1999.
  32. S. Guven, S. Feiner, and O. Oda, "Mobile augmented reality interaction techniques for authoring situated media on-site," in Proc. of IEEE International Symposium on Mixed and Augmented Reality, pp.235-236, October 2006.
  33. G. A., Lee, U. Yang, Y. Kim, D. Jo, J.H. Kim and J. S. Choi, "Freeze-Set-Go interaction method for handheld mobile augmented reality environments," in Proc. of Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology, pp.143-146, 2009.
  34. O. Fakourfar, K. Ta, R. Tang, S. Bateman, and A. Tang, "Stabilized annotations for mobile remote assistance," in Proc. of Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1548-1560, May 2016.
  35. Y.S Chang, B. Nuernberger, B. Luan and T. Hollerer, "Evaluating gesture-based augmented reality annotation," in Proc. of 2017 IEEE Symposium on 3D User Interfaces, pp. 182-185, March 2017.
  36. B. Nuernberger, K. C. Lien, T. Höllerer, and M. Turk, "Interpreting 2d gesture annotations in 3d augmented reality," in Proc. of 2016 IEEE Symposium on 3D User Interfaces, pp. 149-158, March 2016.
  37. B. Nuernberger, K. C. Lien, G. Grinta, C. Sweeney, M. Turk, and T. Hollerer, "Multi-view gesture annotations in image-based 3D reconstructed scenes," in Proc. of Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, pp. 129-138, November 2016.
  38. Saul Greenberg and Carl Gutwin, "Implications of we-awareness to the design of distributed groupware tools," Computer Supported Cooperative Work (CSCW), vol. 25, issue. 4-5, pp. 279-293, October 2016. https://doi.org/10.1007/s10606-016-9244-y
  39. Stephan Lukosch, Heide Lukosch, Dragos Datcu, and Marina Cidota, "Providing information on the spot: Using augmented reality for situational awareness in the security domain," Computer Supported Cooperative Work (CSCW), vol. 24, issue 6, pp. 613-664, December 2015. https://doi.org/10.1007/s10606-015-9235-4
  40. Jesus Gallardo, Crescencio Bravo, and Ana Isabel Molina, "A framework for the descriptive specification of awareness support in multimodal user interfaces for collaborative activities," Journal on Multimodal User Interfaces, vol. 12, no. 2 pp. 145-159, November 28 2017.
  41. C. Gutwin, O. Schneider, R. Xiao, and S. Brewster, "Chalk sounds: the effects of dynamic synthesized audio on workspace awareness in distributed groupware," in Proc. of Proceedings of the ACM 2011 conference on Computer supported cooperative work, pp. 85-94, March 2011.
  42. C. Gutwin, S. Bateman, G. Arora, and A. Coveney, "Looking Away and Catching Up: Dealing with Brief Attentional Disconnection in Synchronous Groupware," in Proc. of In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 2221-2235, February 2017.
  43. Marina Cidota, Stephan Lukosch, Dragos Datcu, and Heide Lukosch, "Comparing the effect of audio and visual notifications on workspace awareness using head-mounted displays for remote collaboration in augmented reality," Augmented Human Research, vol.1, no. 1, 10 October 2016.
  44. Simon Y. Li, Farah Magrabi, and Enrico Coiera, "A systematic review of the psychological literature on interruption and its patient safety implications," Journal of the American Medical Informatics Association, vol. 19, no. 1, pp.6-12, September 2011. https://doi.org/10.1136/amiajnl-2010-000024
  45. Chih-Yuan Ho, Mark I. Nikolic, and Nadine B. Sarter, "Supporting timesharing and interruption management through multimodal information presentation," in Proc. of Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 45, no. 4, pp. 341-345, October 2001.
  46. Michael Posner, "Orienting of attention," Quarterly journal of experimental psychology, vol. 32, no.1, pp.3-25, May 2007. https://doi.org/10.1080/00335558008248231
  47. Shameem Hameed, Thomas Ferris, Swapnaa Jayaraman, and Nadine Sarter, "Using informative peripheral visual and tactile cues to support task and interruption management," Human factors: The Journal of the Human Factors and Ergonomics Society, vol. 51, no. 2, pp. 126-135, June 2009. https://doi.org/10.1177/0018720809336434
  48. David D. Woods, "The alarm problem and directed attention in dynamic fault management," Ergonomics, vol. 38, no. 11, pp. 2371-2393, March 2007. https://doi.org/10.1080/00140139508925274
  49. W. Tan, H. Liu, Z. Dong, G. Zhang, H. Bao, "Robust monocular SLAM in dynamic environments," in Proc. of the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 209-218, May 2013.
  50. Seungwon Kim, Mark Billinghurst, and Gun Lee, "The Effect of Collaboration Styles and View Independence on Video-Mediated Remote Collaboration," Computer Supported Cooperative Work (CSCW), vol. 27, no. 3-6, pp 569-607, June 2018. https://doi.org/10.1007/s10606-018-9324-2
  51. G. A. Lee, S. Kim, Y. Lee, A. Dey, T. Piumsomboon, M. Norman, and M. Billinghurst, "Improving Collaboration in Augmented Video Conference using Mutually Shared Gaze," in Proc. of ICAT-EGVE 2017 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, pp.197-204, 2017.
  52. Kunal Gupta, Gun A. Lee, and Mark Billinghurst, "Do you see what i see? the effect of gaze tracking on task space remote collaboration," IEEE transactions on visualization and computer graphics, vol. 22, no. 11, pp.2413-2422, July 2016. https://doi.org/10.1109/TVCG.2016.2593778