Analysis of Intermediary Roles and Technical Components for Smartglass-Assisted Interactive Remote Collaboration

Abstract

Recent advances in Internet-of-Things and wearable computing have established computationally sufficient infrastructure to remove the barriers of physical locations. The recent COVID-19 pandemic has accelerated the use of remote collaboration, work-from-home, teleconferencing, online education, digital twin, and metaverse enriched with various configurations of augmented reality, virtual reality, mixed reality, and extended reality. Beyond specialized niche wearable applications in the medical and entertainment domains, we aim to identify the underlying characteristics of smartglass-enabled killer applications for everyday use. In this paper, we analyze intermediary roles in remote collaboration consisting of authoritative supervisors, synergetic collaborators, and speculative explorers. We also propose technical components for smartglass-assisted interactive remote collaboration composed of personal tracking, teleconference, and interaction components. More specifically, the application tasks of each intermediary role and objectives of three functional components are defined and elaborated.

I. INTRODUCTION

Personal computing has evolved from desktop computers to mobile devices such as smartphones to increase the mobility of computing activities. The next upcoming paradigm shift of personal computing is actively occurring in wearable computers such as smartwatches and smartglass as shown in Fig. 1.

Fig. 1. Wearable computing as the next level of personal computing.

Wearable computing is known for always-on, alwaysaccessible, and always-connected properties of computing that enable more intimate and frequent use regardless of the user’s whereabouts and intention. Commercial products such as Google Glass Enterprise Edition 2, Microsoft HoloLens 2, and Facebook Oculus Quest 2 have proven feasibility and effectiveness in the development of niche application domains such as in the medical and industrial sectors. Moreover, the recent outbreak of the COVID-19 pandemic has accelerated the use of non-contact remote collaboration technology with various configurations of augmented reality (AR), virtual reality (VR), mixed reality (MR), and extended reality (XR). The concept of remote collaboration has been explored in specialized medical and surgical fields [1], [2], [3]. Going beyond targeted and highly specialized niche wearable applications in the healthcare and entertainment domains, we seek to identify the underlying principles that can rapidly develop smart glasses-assisted killer applications for everyday use. We believe that smartglass-assisted interactive remote collaboration is a promising candidate for killer wearable applications when this concept is extended and deployed for more broad everyday applications and casual users.

In this paper, we analyze intermediary roles in remote collaboration and propose technical components for realizing smartglass-assisted interactive remote collaboration [4]. More specifically, we define various intermediary roles of remote collaboration participants and elaborate technical components between those interfaces.

II. RELATED WORK

The proliferation of mobile devices has introduced various frameworks specifically designed for mobile computing environments. For example, cameras on mobile devices are actively used in Personal Companion [5] and camera phone-based interaction techniques [6] to interact with the smart environment. The camera-based frameworks [5], [6] enabled the mobile devices to recognize visual codes or fiducial markers for interaction where a relative distance and angle between camera and visual tags are used as a set of commands.

Research on mobile AR has introduced sensor fusion that naturally connected wearable devices for extended usage and better performance. For these extended use cases, Yoon et al. introduced an augmented smart coach that benefited from using wearable devices for collecting personal big data and generating a personalized quantifiedself representation [7]. Moreover, developments in wearable devices have increased non-academic people to quantify their lives in quantified-self movements [8] aiming for personalized tracking [9]. Choe and Lee defined personalized tracking as, “an act of tailoring a tracking regimen to suit an individual’s specific situations, preferences, and needs” [9].

More recent studies used commercial smartglass such as Google Glass Enterprise Edition 2 for health professional training [10] and Vuzix Blade for nursing education [11]. Ong et al. discussed the potentials for XR telehealth for health care providers during the COVID-19 pandemic [12], while traditional “telementoring” in surgical and medical domains is summarized in survey and review papers [1], [2].

In comparison to related works, we attempt to polish previous remote collaboration scenarios to support three intermediary roles beyond medical and surgical contexts by including smartglass and their characteristics. The proposed intermediary roles and technical components can be used as a reference implementation to explore smartglass-assisted interactive remote collaboration applications.

III. INTERMEDIARY ROLES AND TASKS

In remote collaboration scenarios, we can define intermediary roles based on who takes initiatives with varying controllability and interaction expressivity (CIE). We propose three possible intermediary roles for remote collaboration as authoritative supervisor, synergetic collaborator, and speculative explorer as illustrated in Fig. 2.

Fig. 2. Possible intermediary roles for remote collaboration.

3.1. Authoritative Supervisor

The first intermediary role is authoritative supervisor where one participant has significantly greater CIE than other parties. The party with this authoritative CIE acts as a supervisor to other participants. This relationship is common in telementoring scenarios where mentors have greater knowledge and expertise than mentees [4]. The smartglasses of the supervisor displays visual data from other participants that help the supervisor to give concrete instructions or guidance. Non-supervisory participants basically play two functions. They play the role of passive sensors to deliver details captured with their smartglasses to their supervisor on the remote site. Furthermore, they play the role of the designated actuator to physically carry out a task on behalf of the supervisor. This intermediary role is depicted in the left figure of Fig. 2.

3.2. Synergetic Collaborator

The second intermediary role involves participants with approximately the same level of CIE. The smartglasses on these peers display the same visualization and they can zoom in or out visual data on one’s own device. However, the CIE of one device is on equal terms with the other device. Since collaborators or peers have a similar level of knowledge and expertise, their collaboration can create synergy. This intermediary role is depicted in the center figure of Fig. 2.

3.3. Speculative Explorer

The third intermediary role involves participants carrying out tasks in unknown or unfamiliar domains. In this role, smartglasses are used as an exploration device that recognizes, interprets, and augments additional visualization on the smartglass. A notable application on this role includes social and gaming applications that do not require prior knowledge or experience from the users. In this role, users are explorative in previously unknown or unexperienced tasks. This intermediary role is depicted in the right figure of Fig. 2.

3.4. Tasks for Different Intermediary Roles

The aforementioned three intermediary roles have already been examined in previous remote collaboration systems and applications. We have classified tasks and applications found in remote collaboration studies according to the three intermediary roles in Table 1.

Table 1. Tasks and applications found in various roles.

Surgery and medical tasks [10], [13], [14], [15] in remote collaboration scenarios include intermediary roles of authoritative supervisor (AS) as well as synergetic collaborator (SC). Since surgical and medical tasks require an expertise, speculative explorer (SE) is not common. In education and training tasks [11], [16], [17], all three intermediary roles are addressed. In AS, we have scenarios involving teachers and students. We can have a group of peers or team-based activities with prior knowledge (i.e., SC) or for completely different or new domain (i.e., SE). Similar to surgery and medica tasks, industrial and physical tasks [3], [18], [19], [20] also typically involve AS and SC roles. Lastly, in social and gaming [21], [22], [23] tasks, all three intermediate roles addressed where there exist various solutions to carry out such tasks.

IV. TECHNICAL COMPONENTS

We have previously defined a specialized use case of telementoring with three distinguishing characteristics of “(1) mentors and mentees of different levels, (2) sharing of first-person and eye-level view via smartglass, and (3) extended use cases with bi-directional interaction [3]”. To realize the three characteristics of remote collaboration, the following functional components are designed and included in our reference framework.

4.1. Components for Personal Tracking

Due to the always-(on, accessible, connected) characteristics of wearable devices, embedded sensors on these devices are used to monitor and analyze users physically via various physiological sensors and IMU (inertial measurement unit). A notable example includes location-based services where the user’s location is tracked for providing location-based notifications. Personal big data collected from smartglass and smartwatches are important sources for machine learning approaches. Moreover, quantified-self [6] and personal tracking [7] approaches have already been effectively deployed in healthcare, sleep, and food journaling. For example, users' activities can be tracked with motion sensors and physiological sensors to measure physical fatigue as well as a cognitive load of a task of interest. The lessons and insights from these studies pave the way for preparing successful remote collaboration. Power-consuming personal tracking components for wearable devices should be carefully designed to deal with the limited battery capacity on these compact and portable devices.

4.2. Components for Audio-visual Teleconference

By the definition of AS, SC, and SE, participants (including mentors, mentees, and peers) communicate with audio and video in real-time. At least one side (i.e., mentor or mentee) needs to wear smartglasses, but the remaining side can use non-smartglasses computers. To effectively communicate in terms of real-time feedback and communicative instructions, audio-visual teleconference functions are required. It is critical to provide and share users’ views when collaborating, while the actual resolution of the streaming video may differ in varioustasks of interest (i.e., medical/surgical, educational, and industrial).

Due to the COVID-19 pandemic, audio-visual teleconference functions are used globally in several platforms such as Zoom, Google MeetTM, Microsoft Teams and implemented with open sources such as Jitsi Meet (https://github.com/jitsi/jitsi-meet), BigBlueButton (https://bigbluebutton.org/), OpenVidu (https://openvidu.io/), Element (https://element.io/), and Wire (https://wire.com/en/) as well as using WebRTC API [3]. Audio-visual components should be designed and implemented considering various display sizes and display resolution.

4.3. Components for Supporting Various Interaction Styles

Participants in remote collaboration need tools and user interfaces for pointing or marking problematic areas that needs the other participants’ feedback. Mentors also need tools and user interfaces to guide and instruct the mentees. To support the needs of mentors, mentees, and collaborators, various interaction styles including touchbased, voice-based, gesture-based, and telestration (i.e., pointing, annotating, drawing) are required.

Wearable devices’ various embedded sensors can be used to improve the user experience and usability of the intermediary roles of AS, SC, and SE. For example, tactile physical feedback generated via vibration can be used to alert users in situations where the users’ attention level is low (i.e., performing some activities), hard-of-hearing users (i.e., physically, and noisy ambient sound) are involved, and multimodality are exploited for emphasis.

Fig. 3 shows our proposed technical components for the smartglass-based remote collaboration in three categories of personal tracking components, teleconference components, and interaction components. Personal tracking components are employed to track the wearer’s physiological, cognitive load, physical fatigue, and activity levels while managing and analyzing personal big data generated from the smartglass. Teleconference components are used to live-stream video and audio to mentors and mentees, respectively. Interaction components allow the user to pick and choose various interaction styles (i.e., touch-based, voice-based, gesture-based, telestration) that suit the user’s task and environment. All three components rely on embedded sensors such as motion sensors [24] and smartglass hardware. For example, on a Google Glass Enterprise Edition 2, activity levels in personal tracking components, video and audio streaming in teleconference components, and touch-based UI, voicebased UI, and gesture-based UI in interaction components can be exploited to implement remote collaboration applications. Similar to personal tracking components, power-consuming and computationally intensive interaction components such as voice-based UIs and gesture-based UIs must be carefully integrated to handle limited battery capacity.

Fig. 3. Technical components for SAIT.

V. DISCUSSION

This section illustrates how the previously mentioned technical components can or are being used in smartglassassisted applications. Table 2 shows three remote collaboration applications of VIRNECT Remote by VIRNECT (https://virnect.com), Spatial 4.0 by Spatial (https://spatial.io), and metaverse platform by MAXST (https://maxst.com). Personal tracking components are used by a metaverse platform by MAXST, for tracking the user’s physical location within the office space. All three applications use teleconference components to share camera views for remote collaboration. Lastly, various interaction styles including touch-based UI, voice-based UI, and gesture-based UI are used in these examples as well.

Table 2. Smartglass-assisted application examples.

VI. CONCLUSION

Wearable devices including smartglasses are getting popular among technical and healthcare professionals. More casual use cases involving gaming and work-fromhome applications are being developed and explored. Remote collaboration, a popular topic studied in medical and surgical domains, can be tailored to include smartglasses and various common functionalities to realize a killer application for wearables. In this paper, we analyzed intermediary roles in remote collaboration and proposed technical components for realizing smartglass-assisted interactive remote collaboration. More specifically, we defined various intermediary roles of remote collaboration participants and elaborated technical components between those interfaces that include three main components of personal tracking, teleconference, and interaction. Further studies are deserved for the practical development and deployment of the proposed reference components using commercially available smartglasses to iteratively gain insights and improve smartglasses-based interactive remote collaboration applications.