Mixed reality multi-person interaction research based on the calibration of the HoloLens devices

Abstract

Currently, the application of virtual reality technology is becoming more and more popular in all aspects of life. From virtual entertainment to industrial simulation, the new operation and working methods brought by virtual reality visualization technology have greater appeal and advantages. With the renewal and iteration of related equipment, more and more functions make its limitations continue to decrease, but its applicability continues to improve. Take the optically transparent head-mounted device as an example. It integrates more computer functions, presents and interacts in a virtual way, further integrates with daily behaviors, and shortens the distance between users and digital information.

1. INTRODUCTION

With the continuous update and iteration of technology and design, the amount of information we can process, receive and display is also increasing. Developers have made virtual reality technology further serve some of our current entertainment, work areas and daily life areas [1,2].

In recent years, the development of network and cloud information transmission technology has made network transmission relatively more optimized, more stable, and more efficient. The covid-19 outbreak in early 2020 has greatly increased the demand for information and communications in isolation. Due to this epidemic, people's daily lives have been greatly hindered. Therefore, online services such as online classrooms, online conferences, and online offices are all used to fill gaps in society in a short period of time.

For many activities that require interaction and social interaction in people's daily lives, the space and low-cost advantages of virtual reality technology can theoretically meet part of the activities of people restricted by epidemics [3]. In this way, people can interact with friends or other people at home for interactive entertainment, collaborative work and other content, free from space and epi- demics, and provide a safer way of social behavior. Therefore, the multi-person interaction research of virtual reality technology has important value and pioneering.

2. BACKGROUND

2.1 Mixed Reality

Mixed Reality(MR) [4] is a specific subset of Virtual Reality (VR) related technologies [5]. It involves merging the real and virtual worlds somewhere in the "virtual continuum", connecting the completely real environment with the completely virtual environment. And based on advances in computer vision, graphics processing capabilities, display technology, and input systems, the user, computer, and environment interact with each other (Fig. 1).

Fig. 1. The interactions between computers, humans, and environments.

2.2 HoloLens Devices

The Microsoft HoloLens is a self-contained system running a special version of Windows 10 that it generates and projects stereographic images onto the lenses of a headset [6]. The experiment in this paper mainly uses Microsoft HoloLens 1 and 2 devices (Fig. 2) as the main test content and research basis.

Fig. 2. HoloLens 1 (left) and HoloLens 2 (right).

2.3 Error and offset

Although the HoloLens device has a certain calibration function, the displacement and error caused by some subtle differences in the human body, movement, time and application always exist. In order to solve this problem in this paper, just refer to the traditional offline calibration method to reduce the error [7].

2.4 Calibration research

Based on the above errors, in order to prevent the actual experience from being excessively affected, we refer to the more popular calibration methods [8,9]. In the specific measures of this research, the method used is to use eyes instead of the camera, and calculate the projection matrix of the rendering camera based on photogrammetry technology with eyes calibration in HoloLens devices (Fig. 3).

Fig. 3. The left is Eye calibration in HoloLens devices. The right is camera-based calibration: C can be regarded as the imaging position of the human eye, and E is the coaxial camera relative to the display.

2.5 Multiplayer Server

The establishment of a qualified multiplayer server is the basis of multiplayer interaction. In this article, we will build a multiplayer server as a pre-interaction link. The conditions it needs to meet include providing users with sufficient information prompts and being able to synchronize information between users to ensure as much real-time synchronization as possible [10].

3. EXPERIMENTAL DESIGN

3.1 Offline Calibration Design

The main steps [11]:

1. Physical method: adjust the position of the device to ensure that the user can see the full screen range (completed with the aid of virtual markers and guidance).

2. Virtual camera adjustment: Using the target of image tracking as a reference, adjust the relative spatial position of the virtual camera to match the virtual content visible to the human eye. Refer to Fig. 4.

Fig. 4. The target based on AR image tracking serves as a calibration reference.

3.2 Server Framework Design

Photon Cloud [12] is the basis of the server, and the program script will always check that the link is normal. The second part is the "room lobby". Users can choose to enter other people's game rooms or create their own game rooms. The user who creates the game room will decide the start of the game and the closing of the game room. See Fig. 6.

In the Lobby section (Fig. 5). “Lobby” interface the user sees. “Create a room” is the first is used to create a list with limited capacity. “Join the room” is used to join the selected room (list). "Start game" is used to lead all users into the scene. “Ready/ None” is used to confirm the user's readiness.

Fig. 5. The Lobby logic structure.

Fig. 6. Multi-user connection logic structure.

3.3 Content Design

As for the choice of interactive content, we hope to have as many interactive methods as possible to satisfy the perception provided to users by mixed reality (such as sound, voice, color, space, motion, and touch feedback).

Considering that the substantive content must have enough sociality to promote the interaction between users, the main interactive part is finally presented in the form of chess. The reason is that the competitive nature of chess helps to stimulate the enthusiasm of users. Secondly, the more peaceful operation of chess can prevent some excessive movements from affecting the displacement of the user's head-mounted device. Finally, it is considered that it can be substantive The activities of things are compared for reference. The specific measure for presenting chess interaction is to realize "automatic" chess that can be controlled by voice and gesture interaction through various interactive functions of HoloLens. (Similar to chess with voice control in "Harry Potter")

At the same time, we made an animation of each chess piece, so that the chess piece will have corresponding action feedback when it is controlled to move, eat or be eaten, to show the fun of the virtual part in mixed reality, which is more interesting than pure chess movement. Can maintain the user's experience interest. Refer to Fig. 7.

Fig. 7. Virtual chessboard (“Pawn”).

3.4 Target Image Design

This part of the design mainly focuses on two objects. The former is used to provide a reference in the calibration part. The latter is used to place a virtual chessboard. Two image objects are designed according to the feature point standard of Vuforia (Table 1).

Table 1. Image Targets that possess the following attributes will enable the best detection and tracking performance from the Vuforia Engine.

4. TEST AND RESULTS

4.1 Test Process

The user enters the application through the HoloLens device to realize the import and installation, and then the calibration part appears first [13]. Pseudo-text will be used to guide the user, and "Next" will be used as the password to perform voice control to proceed to the next step. After completing the calibration part, the two users will directly enter the server scene. Here, the user can choose to join a "room" created by others or create a "room" by himself (refer to Fig. 5 for the process). The person who creates the room will act as the host server, and the person who joins the room will act as the guest server. Refer to Fig. 8.

Fig. 8. Multi-person online practical test.

The host server needs to ensure that the guest server is in a ready state before sending instructions to make both parties enter the chessboard scene together. In the chessboard scene, both users need to determine the position of the chessboard by finally identifying the corresponding image target to place the chessboard, and then the chess game can be started. White (host server) goes first, and both users can take turns using gesture control, touch control and Speech Control to control the movement and capture of the chess(the movable range of the pieces will be highlighted). Please refer to the test video content of [13].

4.2 Test Results

Judging from the results of multiple tests: According to the past experience of AR multi-person interaction on mobile terminals (please refer to the video [14]), image tracking usually causes deviation and tremor due to equipment and other reasons, but it can be identified by the space of HoloLens equipment And a more stable tracking system to optimize this function. The rest of the interactive functions basically meet the expected expectations, but there are some delays and low sensitivity issues with touch gestures, which will affect the overall interactive experience.

Secondly, because voice commands can be received by two devices at the same time, the script will turn off the voice receiving function of the other user during the corresponding game round of the user to avoid errors. However, Speech Contorl has a certain delay (Refer to Fig. 9), and after many tests, it is determined that there is a part of noise interference. Factors make the identification time longer. As an insurance, other gesture control and touch control can play a substitute role in time. The above content has been tested and feasible in HoloLens1 and HoloLens2.

Fig. 9. Voice recognition will affect the recognition speed due to noise interference. The picture shows a prompt for successful voice recognition.

In addition, when you need to pay attention, the device will occupy the use of the device camera during the screen recording, so the frame rate of the device will be reduced during the screen recording, and the actual operation of the device will be more smooth. In addition, due to the influence of the frame rate, some preset animations controlled by the number of frames will be relatively delayed, and we will find ways to improve the corresponding problems in the future.

5. CONCLUSION

According to the observation of the existing test results, although the expected results can be achieved, there are still some issues that need to be paid attention to:

- In the calibration phase : need to optimize the tracking loss problem.

- The visual data of the content scene needs to be optimized : add virtual visual information and optimize the UI.

- The voice control part needs further improvement : optimizing the recognition mode to avoid the impact of verbal errors.

In order to conduct further research on this experiment, we hope that we can interact with various types of devices (not only holographic devices, mobile phones, PCs, VR devices, etc.), and try to build a general virtual interactive system.