3D Motion Capture based Physical Fitness using Full Body Tracking Suit

Abstract

This paper presents an approach to exercise that utilizes motion capture through the Rokoko Smart Suit. With the emergence of Covid-19, physical fitness levels have declined due to restrictions on in-person fitness classes and gym closures. To maintain physical activity, many individuals have turned to mobile applications and streaming videos. However, home workouts often lack the motivation and experience found in gyms, classes, or community centers, particularly with the presence of coaches and instructors. Additionally, instructors find it challenging to convey precise postures to their online students, and vice versa. To address this issue, the researchers propose the use of a full-body tracking suit like the Rokoko Smart Suit, which enables instructors to present a more realistic approach to physical activity. The Rokoko Smart Suit offers a 3D view of the instructor, eliminating the limitations of camera scope when streaming on platforms like Zoom or MS Teams. This technology enhances the at-home workout experience, and the incorporation of 3D virtual reality features can further elevate the realism of a workout.

1. Introduction

In the context of the recent COVID-19 pandemic, many individuals are still dealing with the challenge of returning to their normal routines. Although the number of people afflicted by COVID-19 has significantly reduced, many remain concerned and some still occansionaly suffer from COVID-19. A few are still wearing masks in public and work places in allmost all the domains including fitness, training and rehabiliation. While many fitness clubs/spaces have resumed their normal operations, there are a number of people still hesitant to go to those packed places. To overcome this problem, many have turned to online platforms like YouTube videos, live streaming of an instructor, or Zoom to maintain a fitness routine. In person fitness classes have been adapted to suite online formats. Indeed, there is a wealth of recorded fitness videos available. However, findings have revealed that such recorded videos lack engagement since they are created for the mases and not an individual trainee. Hence, vidoes alone are not a viable option for individuals seeking personalized training or for home remedy to releive muscle/joint pain. In certain cases, just following a YouTube video without a coach or trainer could cause more damage to the muscles or joints thereby worsoning the pain, as it happened to the very first author of this article. Comparatively, live video streaming is a better personalized option but both the trainer and trainees have to overcome the limitations of the camera view. In such cases, current users may consider adapting new technologies.

One potential but expensive technique for this adaptation involves the use of motion capture technology with a Smart Suit. This approach allows users to partake in real-life fitness classes from the convenience of their own homes. In this research, we found the Smart Suit offers an advantage of real-time interaction with instructors and coaches. Rather than following a pre-recorded video, users can engage with their instructors in real-time using the Rokoko Suit. To investigate whether the users find real-time fitness classes more engaging and beneficial compared to following pre-recorded videos, the researchers of this paper conducted a study while wearing the Rokoko Smart Suit, as discussed in Section 4. The findings revealed that the Rokoko Smart Suit is a viable option that overcomes the limitations of a camera view in online Zoom or MS Teams classes. Our research demonstrated that the Rokoko Smart Suit accurately captures and displays users' movements in a 3D environment. One of the major limitations of this werable technology is the expense. At the time of this paper the usual price of a Rokoko Smart Suit is around $5,000, which is not an easy buy for a middle class user.

2. Motivation

Scientists have anticipated the COVID-19 will not be the last pandemic. A pandemic like COVID-19 could errupt in future but what, where, and when cannot be forseen [1]. Per this same article, “one ongoing threat is a poxvirus called monkeypox that is currently spreading throughout the world”. Regarldless of when the next outbreak will occur, undoubtly pandemics be with us in the future. Medical scientists are not only watching the evolution or modifications of the viruses but also thinking about potential solutions. Due to our increased interactions in our current world when a pandemic does breakout it will affect almost everybody. Therefore we need to think about potential problems and solutions beyond the medical world. It is obvious that there are many aspects of this problem and one such aspect is physical trainings and exercises, as many of us are becoming health conscious with some participating in group training and exercise sessions in gyms or clubs, or in our social gatherings. In case of pandemics, we are not able to be physically present in the same room. Instead our sessions are moved from the physical space to a cyber space. However, there are no good cyber options available to perform such exercises with precision. Options such as zoom, are oftern hampered by camera view and movement limitations. In such scenarios, 3D-based wearable devices could be a better option. This research work was conducted to look into the suitability of using Rokoko Smartsuit to perform exercises for the trainer and trainees.

3. Literature Review

This section discusses existing studies that focus on motion capture during body movement. For this purpose, we studied Teslasuit since it shows promising results from movement assessment and provides very useful feedback to users [1]. However, the proposed approach involes the use of large data sets of professional athletes. Such data would need to be collected from professional athletes and sports scientists and optimized for non-professional use. Prior to this work, motion capturing suit was an expensive task and created many discomforts. To address this issue, an accurate motion recognition approach was proposed using the Hidden Markov Model (HMM) trained on the specific movement [2]. The proposed approach did show good results in detecting the complex full-body gestures. Later approach also incoporates the feedback that helps users hear the exercise that has been executed correctly. However, this approach shows limitations in hearing the exercise if it was incorrectly performed.

In addtion to body suits, Virtual reality (VR) technology has applications for entertainment, education and rehabiliation. VR technology enhances the sense of users that they are present in a virtual environment. Therefore, a user’s motion can be synchronized with the avatar’s motion. However, most of the existing approaches do not show good performance or face the high latency. Futhermore, a user’s motion cannot be resasonable represented by current VR applications due to the lack the positional and rotational data. To overcome this issue, Vive Trackers have been used with VR-based applications to crerate the immersive VR experience [3]. The proposed approach is capable of tracking the joint rotation with a reasonable accuracy and sustains the low end latency.

An other aspect is human motion recognition that relies on the capture of limb motion. Due to varying limb activities, the existing inertial methods cannot resolve the serious drift and instability issues. A motion capturing method is proposed in [4] where a wearable device is incoprated with the micro tris-axis inertial and micro tri-axix flow sensors. Three dimensional motion with acceleration, velocity and attitude angle of human limbs is determined with the help of device. The daily activities, including prolonged excercises, and strenous are accurately measured by the support of the proposed approach.

Similarly, personal tracking techniques allow users for monitoring and show reflection on the physical activities as well as fitness. However, users are not sure about the accuracy of data sent or received by devices. A survey based study was conducted to obtain the feeback of users for different products such as tracking devices [5]. During this study, authors identified several challenges, including the users’ expectations, lifestyle, physiological characteristics and types of activities. In addition, due to unclear understanding about the accuracies, experimental designs and controls ultimately resulted in incorrect outcomes.

Middle aged adults benefit from the aquatic exercises for their health maintenance and motor recovery. Moreover, neurological and orthopedic patients can benefit by amplifying and accelerating the rehabilation process from immersive, personalization and biofeedback techniques. Despite these advantages, a therapist cannot accurately visulize rehabiliation process under the water. A wearable biofeedback suit during aquatic feedback is proposed to visulize the exercise activities [6]. Biofeedback modalities are used to visulize the feedback are compared with the optical tracking system. The proposed system correctly measures the relevant angles.

Other than the information in the eralier mentioned studies, we found that virtual reality (VR) technology has crucial and promising role in providing the support for motor intervention in autism spectrum disorder (ASD) [13] and training patient rehabiliation [14]. This may help youth with the ASD. Recent work introduces wearable electrical impedance tomography (EIT) [15] to monitor muscle movement and visualize on the virtual avatar.

Most recent studies [15-18] emapsize on the real time measurements during exercises. However, these studies vary in the proposal of the approaches. Study [16] uses a pentographic exoskeleton (PGE) sensor while in [17] authors used robots in weight shifting exercise. A Tello Edu drone based system [18] is introduced that shows a high accuracy for human movement detection.

Figure 1 illustrates the number of articles published between 2015 and June 2023. In the initial years (2015-2018), researchers paid relatively little attention to this research topic. The potential reasons could be the expensive technology of wearable suits as well as a not-in-demand trend. However, from 2015 the technology started being accepted by some niche markets and we saw some people around us wearing smart watches. However, in 2020-2021 it gained more attention in the reseach community. From 2021 to 2022 we see a drop of attention. Recently this is reversing we enter 2023. It seems, there are some unknown factors causing this fluctuation in Smart Suit research work. In case of market demand, we can easily say that price is the foremost factor, as it is not easy for a middle class income person to buy a pricy suit worth $5000.

(Figure 1) Research on motion capture

It is interesing to note the surge of activity is well aligned with the advent of COVID-19. In addition the use of 3D technology began to emerge in our daily lives, however, as we notice the general masses have not fully adopted 3D technology. We do not see general people wearing 3D devices or Smart Suits around us, except in Hollywood movies showing futuristic life styles, if at all. There are several video games that are making use of oculus devices. The users usually wear them only for a short period of time, like up to 30 to 60 minutes, because wearing them for a longer time is uncomfortable. It indicates the adoption of 3D wearable devices is still quite slow. Thus apart from the cost factor, another factor is poor User eXpereince (UX). The users are not very comfortable wearing heavy devices on their head that cover their eyes or wearing Rokoko Smartsuit that ditinguish them too much from other people around them. People naturally perform some multi-tasking like looking at one side of the street while listening to voices/sounds coming from the other side or eating or drinking at the same time. The similar situation is in the office or work spaces, multi-tasking for physical things (not for mental work ). In our experiments, we also noticed that while the Rokoko Smarsuit is not complicated to wear for a Computer Scince student, it would bring some risky-factor to a non-tech person’s mind. A non-tech person won’t feel it safe to wear or find it easy to setup the system, calibrate it with the virtual 3D character, wear the smart suit and perform exercise as they do in their normal lives.

To summerize, we conclude the main reasons for the low-popularity of wearable devices is: Cost and UX issues. UX and wearable devices is an open area of research and has a good potential for future experimentation, analysis and publications. Table 1 depicts the current systems or models to address the issue of motion monitoring and capturing during exercises. This paper identified six (6) significant smart systems highlighting their advantages and disadvantages. Various aspects of motion capturing such as cost and energy, 3D reconstruction and positioning and keeping errors’ information have been focused. However, these proposed approaches still suffer from high accuracy and precision issues, which can be worked out in future research.

(Table 1) Studies and their characteristics

4. Problems

4.1 Constraints due to COIVD-19

As previously mentioned, the COVID-19 pandemic has had a significant impact on society, with over 21.9 million individuals directly affected by the virus. In an attempt to limit the virus's spread, various limitations and restrictions have been implemented. Consequently, gyms and community centers were among the establishments that had to close, prompting a shift towards at-home physical activity. Individuals have had to adapt their fitness routines and methods to accommodate exercising at home. To support this transition, numerous applications have been developed, offering bodyweight exercises, diet tracking, and creative workouts utilizing common household furniture [21]. However, engaging in physical exercise at home differs greatly from the environment provided by gyms, fitness classes, or community centers. Many individuals struggle to replicate the same level of motivation and inspiration they experience in a communal setting. The social atmosphere and encouragement found in public fitness spaces are often lacking at home. The need for personal motivation becomes paramount when exercising in a home environment. A study was conducted to assess the usage of mobile applications by individuals at home, revealing an 18.2% decrease in physical activity minutes per session [13]. Thus, the COVID-19 pandemic has presented an inherent challenge, negatively impacting the physical activity levels of individuals in their own homes.

4.2 Limitations on Virtual Exercise Training

Moreover, virtual fitness training methods, such as video recording or live streaming, have inherent limitations due to the constraints of the camera's scope. With the shift towards online platforms for physical activity, fitness professionals and trainers have resorted to recording themselves performing exercises, allowing users to mimic their movements at home. However, this approach fails to replicate the benefits and similarities of an in-person class. Users do not experience the same sense of connection and immersion as they would in a physical class setting. Therefore, instructors have turned to live streaming to enhance the user experience. However, both video recording and streaming methods are restricted by the limitations of the camera's angle and view. Instructors are confined to working within the boundaries of the camera's perspective.

5. Use of Motion Capture and Virtual Reality

5.1 Rokoko Motion Capture Suit

In our research, we employed the Rokoko Motion Capture Suit (depicted in Figure 2) to record concise fitness videos. The Rokoko smart suit features locomotion Inertia Measurement Sensors (IMS) [20] and displays a compact design, enabling swift calibration and a comfortable fit. Additionally, the suit is accompanied by user-friendly Rokoko software, which proved to be remarkably accessible even for individuals lacking hands-on experience with motion capture. The software facilitated locomotion correction through various filters and allowed for manual adjustments. Furthermore, it seamlessly integrated with Blender, an open-source graphics software.

(Figure 2) Rokoko Smart Suit

5.2 Flowchart of Setup Implementation

In order to implement our experimental plan, we designed the setup shown in figure 3.

(Figure 3) Flowchart of Implementation Setup

The setup of the suit was uncomplicated, thanks to its sleek design. It offered Wi-Fi capabilities, expanding the range of movement for the user. However, a couple of limitations were encountered during our implementation plan. The first one was to identify the actors with the body size to fit our Rokoko Suit (the suit we had available was size small and it did not fit several of volunteer actors). Fortunately, we did find two undergraduate students at Virginia Military Institute (VMI) who could easily fit. Another limitation was the requirement to maintain distance from any metallic objects that could potentially cause interference.

With the Rokoko software installed on a Windows 10 PC, the filming process was straightforward. In the interest of research and to support the Human-Computer Interaction (HCI) class at VMI, the two students took turns recording short videos lasting between 15 to 60 seconds. The videos showcased various exercises such as push-ups, yoga poses, high-knees, and jumping-jacks, as depicted in Figure 4.

(Figure 4) Visualization of Rokoko character

Another positive outcome we obtained was the precision of movements. The instructor actor was able to point out the mistakes the client was making while performing the exercise as shown in Figure 5, the client’s knee was bent while lifting the left leg. This may be missed out in video streaming lesson.

(Figure 5) Spotting client’s mistake

There are seveal such scenarios where precise observation was possible due to the sensors used in those area of body.

6. Major Contributions of this Study

There are three major contributions of this study (i) to provide siginficant analysis of literature review on wearable devices in the context of motion capture used for exercises, and it has been noticed that there have not been many studies conducted in this area. And, if we narrow it down to the COVID-19 as well, our study seems to be one of the rare studies in this domain, (ii) to provide a novel idea of using Smart Suit for 3D-based exercises that is an appropriate and precise way to monitor and guide a remote trainee by a coach or trainer, and (iii) to provide step-by-step guiding model Figure 3 how to implement the entire setup for using the Smart Suit.

7. Further Considerable Solution

In years to come, many individuals would remain in fear of contracting the virus, leading them to be hesitant about returning to their previous fitness and exercise routines. Engaging in physical activity at home presents several limitations, including a lack of motivation, reduced interaction, and constraints imposed by the camera's scope. The incorporation of a wearble body suit equiped with sensors, such as the Rokoko Smart Suit, into physical fitness endeavors offers a solution that closely resembles pre-COVID-19 real-life fitness experiences. The Rokoko Smart Suit employs advanced motion tracking technology to deliver a 3D representation of the user within the Suit. This Suit comprises 19 sensors embedded throughout the garment, ensuring precise and accurate movement tracking [19][22]. The sensor placement in the clothing is shown in Figure 6.

(Figure 6) Sensor Placement in Rokoko Suit

The Rokoko systems operates through a WiFi connection and is recorded using the Rokogo Smart Suit Studio software. Unity, a popular software platform, allows users to control the visualizations of the 3D figure, granting them flexibility and creative control. Unlike traditional methods, the user is not tied by cords and is not confined to the limitations of the camera's view. Figure 4 illustrates how visualization edits within the Unity software can alter the background and physical appearance of the character.

By enabling users to transcend the constraints of the camera, the Rokoko Smart Suit significantly enhances the experience of live-streaming workouts. The incorporation of 3D visuals contributes to a more realistic and immersive representation of the user's movements, fostering a heightened sense of realism and engagement.

8. Future Work

Incorporating 3D technology offers a better alternative to online video streaming platforms like Zoom or MS Teams. One way to implement this is by integrating the Rokoko Smart Suit with virtual reality components. The medical field has already begun utilizing virtual reality in conjunction with the Rokoko Suit to enable doctors to view their patients in a 3D environment. This allows doctors to assess their patients' form and provide corrections when necessary, taking advantage of the Suit's WiFi connectivity [23]. Similarly, virtual reality elements can be leveraged to create a realistic simulation of a gym, community center, or fitness class. Through visualization features, users can immerse themselves in their original workout environment, fostering a sense of presence and familiarity. This combination of virtual reality and the Rokoko Suit empowers individuals to feel as though they are physically present in their preferred exercise settings, enhancing motivation and engagement during their home workouts. One aspect to consider is the UX and wearable devices as mentioned above. The wearable devices need to be comfortable and light weight to use. The users should not be conscious about it, like many of us wear eye-glasses without feeling them. Without addressing the UX issue, werable devices would not be easily acceptable by normal users.

9. Conclusion

Indeed, the impact of Covid-19 has been felt across all facets of our lives. Restaurants, stores, and gyms, among other establishments, have faced restrictions aimed at curbing the spread of the virus. Specifically, gyms and fitness centers have had to close their doors and cancel classes due to social distancing requirements. As a result, many individuals have adapted their fitness routines to the limited space of their homes. Mobile applications, recorded workouts, and live-streaming have emerged as resources to facilitate continued physical activity.

However, the social interaction and motivation derived from attending a gym or fitness class are difficult to replicate in a home environment. Covid-19 has essentially taken away away the benefits of such communal fitness experiences. The introduction of suits, such as the Rokoko Smart Suit, offer a solution that allows users to experience physical activity in a more realistic and engaging manner. With the Smart Suit, instructors can train clients, who can view the instructor in a 3D setting. Unlike traditional camera limitations, the instructor is not confined to a specific scope, and visualizations can be edited to enhance the workout environment. Furthermore, by incorporating the Rokoko Suit into virtual reality, it becomes possible to simulate a genuine fitness experience. Users can immerse themselves in a gym or fitness room alongside other participants and engage in real-time workouts guided by the instructor. In summary, the Rokoko Smart Suit empowers users to enjoy a more authentic and lifelike setting for their physical fitness endeavors.