---
title: "ReliaAvatar: A Robust Real-Time Avatar Animator with Integrated Motion Prediction"
authors:
  - "Bo Qian"
  - "Zhenhuan Wei"
  - "Jiashuo Li"
  - "Xing Wei"
doi: "10.24963/ijcai.2024/346"
url: "https://ijcai.org/proceedings/2024/346"
pdf: "https://ijcai.org/proceedings/2024/0346.pdf"
venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence"
year: 2024
pages: "3124-3132"
track: "Main Track"
keywords:
  - "Robotics: ROB: Human robot interaction"
  - "Humans and AI: HAI: Human-computer interaction"
  - "Humans and AI: HAI: Personalization and user modeling"
  - "Humans and AI: HAI: Applications"
---

# ReliaAvatar: A Robust Real-Time Avatar Animator with Integrated Motion Prediction

Bo Qian, Zhenhuan Wei, Jiashuo Li, Xing Wei

*Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 3124-3132. DOI: [10.24963/ijcai.2024/346](https://doi.org/10.24963/ijcai.2024/346)

## Abstract

Efficiently estimating the full-body pose with minimal wearable devices presents a worthwhile research direction.  Despite significant advancements in this field, most current research neglects to explore full-body avatar estimation under low-quality signal conditions, which is prevalent in practical usage. To bridge this gap, we summarize three scenarios that may be encountered in real-world applications: standard scenario, instantaneous data-loss scenario, and prolonged data-loss scenario, and propose a new evaluation benchmark. The solution we propose to address data-loss scenarios is integrating the full-body avatar pose estimation problem with motion prediction. Specifically, we present ReliaAvatar, a real-time, reliable avatar animator equipped with predictive modeling capabilities employing a dual-path architecture. ReliaAvatar operates effectively, with an impressive performance rate of 109 frames per second (fps). Extensive comparative evaluations on widely recognized benchmark datasets demonstrate ReliaAvatar's superior performance in both standard and low data-quality conditions. The code is available at https://github.com/MIV-XJTU/ReliaAvatar.

## Full text

## 1 Introduction

Virtual reality, augmented reality (AR), and mixed reality (MR) are rapidly evolving fields that offer new dimensions for human communication and interaction. A critical aspect of enhancing these immersive experiences lies in the ability of models to generate seamless and realistic avatar motions, ideally using user-friendly devices such as head-mounted displays (HMD) or non-wearable WiFi devices [Yan et al., 2024]. Despite notable advancements in using sparse observations to animate full-body avatars, a significant research gap exists to effectively drive avatars in scenarios plagued by low-quality signals.

In practical applications, poor-quality signals are a common challenge, particularly when wearable devices are intended to be as convenient as possible. Various factors, including network fluctuations, occlusion in motion capture systems, and limited visibility of interactive handles in HMDs, can degrade signal integrity. Previous works such as AGRoL [Du et al., 2023] and HMD-NeMo [Aliakbarian et al., 2023] have addressed some aspects of these challenges. However, systematic exploration and comprehensive solutions for diverse data-loss scenarios remain underdeveloped.

∗Corresponding author: Xing Wei.

Figure 1: Visualization in the context of prolonged data-loss scenario. We mask out the latter half of a sample, which consisted of 80 frames and depicted "crouching". The first row represents the ground truth, the second row represents the response of ReliaAvatar, and the third row represents the response of AvatarPoser. The visualization clearly indicates that ReliaAvatar can operate effectively in prolonged data loss scenarios with only minor distortions. In contrast, AvatarPoser completely fails to perform in this scenario.

![Image](https://ijcai.org/proceedings/2024/fulltext/346/image_000000_b99eae2caf4c7d8c3516cefe412f35168dd2bc859e2f784a1bf1e53b177f35f5.png)

Table 1: Ability to handle different scenarios.

| Method                    | Standard   | Instantaneous   | Prolonged   |
|---------------------------|------------|-----------------|-------------|
| Other Methods ReliaAvatar | ✓ ✓        | ✕ ✓             | ✕ ✓         |

In this work, we delve into potential data-loss scenarios that could occur in real-world applications. We specifically identify and focus on two scenarios: instantaneous data-loss and prolonged data-loss scenarios. Our response to these challenges is two-fold, encompassing both model architecture and training methodology innovations. We introduce Relia- Avatar, a real-time, robust, autoregressive Avatar Animator, integrating full-body joint motion prediction with an innovative training approach that simulates data-loss conditions.

We develop two distinct data-loss scenarios: instantaneous, resembling tracker signal loss like network packet loss, and prolonged, simulating long-last loss of specific motion capture joints, such as a hand-held controller's prolonged invisibility. Our model operates through two pathways: a regression pathway for conventional full-body avatar pose estimation tasks and a prediction pathway that predicts motions in the absence of tracker signals, ensuring continuity in avatar movements. Both pathways leverage GRU-based models for feature extraction. The combined features are then transformed into a decoding token sequence representing 22 SMPL joints, which will be used in a Transformer encoder to model the inter-joint relationships. We also propose an autoregressive training pipeline encompassing three preprocessing methods aligned with standard, instantaneous, and prolonged data-loss scenarios. During training, each signal sequence is processed using one of these preprocessing methods, enabling the model to adapt to the challenges posed by different data loss scenarios.

We compare ReliaAvatar with existing methods in all scenarios on the AMASS benchmark dataset [Mahmood et al., 2019]. The results showcase our model's state-of-the-art performance in both standard and data-loss scenarios. We also provide a comparison with AvatarPoser [Jiang et al., 2022] in Figure 1 in the prolonged data-loss scenario. These qualitative and quantitative results collectively demonstrate that we are the first method, as stated in Table 1, that has robustness to data quality. Our model also demonstrates impressive performance during the online inference stage, achieving a remarkable 109 fps. This outstanding speed surpasses other avatar pose estimation methods, highlighting the superiority of our model in real-time applications.

Our contribution can be summarized as follows:

- We pioneer a comprehensive investigation into practical data-loss scenarios, identifying and focusing on two key scenarios: instantaneous and prolonged data loss.
- We propose ReliaAvatar, a real-time, robust avatar animator with integrated full-body joint motion prediction, and an autoregressive training pipeline tailored for dataloss scenarios. This approach significantly enhances the model's adaptability and performance.
- Our experimental results demonstrate that ReliaAvatar not only achieves top-tier performance in the standard scenario but also represents, to our knowledge, the first method adept at managing various data-loss scenarios effectively. Furthermore, ReliaAvatar outperforms other methods in terms of computational efficiency.

## 2 RelatedWork

## 2.1 Full-body Avatar Pose Estimation

In the past, motion capture systems required users to wear numerous devices, as seen in early implementations [Xsens, 2013; Vlasic et al., 2007]. However, this requirement often compromises the immersive experience, highlighting the need for more compact approaches. Consequently, the focus shifted to generating a full-body pose using sparse observations, a topic that has garnered significant attention and led to numerous advancements [Von Marcard et al., 2017; Huang et al., 2018; Yi et al., 2021; Yang et al., 2021; Jiang et al., 2022; Zheng et al., 2023; Du et al., 2023; Aliakbarian et al., 2023]. These innovative methods allow users to wear tracking devices on only a limited number of joints, such as the head, hands, and pelvis, thus improving the overall user experience. Among these developments, AvatarPoser [Jiang et al., 2022] introduced a Transformer-based architecture combined with an inverse kinematics (IK) module, paving the way for more sophisticated avatar control techniques. AGRoL [Du et al., 2023] utilized diffusion models to drive avatars, showing notable robustness against instantaneous data loss. Similarly, HMD-NeMo [Aliakbarian et al., 2023] made strides in avatar control by focusing on scenarios where the hands are partially or entirely obscured. Building on these foundational works, our research delves deeper into the realm of full-body avatar estimation under the conditions of low-quality data.

## 2.2 3DHumanMotion Prediction

A key innovation of our model is the motion prediction pathway in the architecture, which is used to provide additional cues to avatar pose estimation. There are many motion prediction methods available for reference, ranging from nonlinear Markov models [Lehrmann et al., 2014] and Restricted Boltzmann Machines [Taylor et al., 2006], to more recent developments like Graph Convolutional Networks [Ma et al., 2022; Mao et al., 2020; Mao et al., 2019] and Recurrent Neural Networks [Jain et al., 2016; Liu et al., 2019; Martinez et al., 2017]. In the prediction pathway, we use a GRU module[Cho et al., 2014] to extract features and a Transformer [Vaswani et al., 2017] module to model the interjoint relationships. The prediction pathway can be aggregated with the regression pathway at the joint-relation Transformer to form our reliable avatar animator.

## 3 Method

## 3.1 ProblemDefinition

Task. The core objective of our task is to accurately predict the 3D full-body human poses yt at any given time-step t, using sparse joint signals xt and supplementary information et . This supplementary information, et , may include data derived from historical movements or other available motion signals at time-step t. Therefore, our objective is formally defined as follows.

<!-- formula-not-decoded -->

Traditionally, this task is approached as an upsampling challenge, where the goal is to learn the mapping from sparse joint signals to a complete full-body joint configuration. In these conventional methods, inputs are configured as a series of continuous sparse signals over a time window to facilitate the extraction of temporal features, leading to the following model formulation.

<!-- formula-not-decoded -->

Here, the supplementary information et is represented as et = {xi}t-1 i=t-T , with T denoting the window size.

Those approaches neglect the continuity of the prediction process, which has been proven to be crucial in visual tracking[Wei et al., 2023; Bai et al., 2024]. Unlike them, we treat the task as an integrative process of both upsampling and full-body joint motion prediction. To this end, we incorporate historical trajectory states as an additional input variable. We utilize a Gated Recurrent Unit (GRU) to extract temporal features from both the tracker signals and the historical trajectory states. This approach allows us to redefine et as a composite of previous pose predictions and historical hidden states, i.e., et = {ˆ yt-1 , ht-1 x , ht-1 y }. Consequently, our model's formulation is as follows:

<!-- formula-not-decoded -->

Here, htx , hty represent the hidden states generated by the GRU, essential for capturing the complex temporal dynamics of human motion.

Representation. ReliaAvatar's input signals consist of four parts: the tracker signals xt , hidden tracker states ht-1 x , the historical trajectory states ˆ yt-1 , and hidden historical states ht-1 y . Here, ˆ yt-1 represents the output of the model at timestep t - 1, and ht-1 x and ht-1 y are the hidden states returned by the GRU module.

The input tracker signals xt at time-step t are composed of the orientation 1 xr ∈ R|J|×6 , rotation velocity ∆xr ∈ R|J|×6 , position xp ∈ R|J|×3 , and linear velocity ∆xp ∈ R|J|×3 of the trackers. J represents the set of all trackers, ∆xp = xtp-xt-1 p , ∆xr = f6D(R-1 (xt-1 r )R(xtr )), and R and R-1 , respectively, represent converting the 6D representation of rotation into the corresponding rotation matrix and inverse matrix, where f6D represents converting the rotation matrix into a 6D representation. Therefore, xt = {xtr ,∆xtr , xtp ,∆xtp } ∈ R|J|×18 .

The historical trajectory states ˆ yt at time t + 1 are also composed of the four parts: 22 SMPL joints' local rotation relative to their parent joints ˆ ytr ∈ R22×6 , position ˆ ytp ∈ R22×3 and corresponding velocity ∆ˆ ytr ∈ R22×6 and ∆ˆ ytp ∈ R22×3 . Therefore, the overall output can be represented as ˆ yt = {ˆ ytr , ∆ˆ ytr , ˆ ytp , ∆ˆ ytp } ∈ R396 .

Scenarios. Current models often operate under the assumption that tracker signals are consistently available, promptly received, and accurate. However, real-world applications present more complex and varied scenarios in which the quality and consistency of tracker signals cannot always be guaranteed. To address this, we have identified three primary scenarios in which our model and similar models may need to operate:

- Standard scenario: This scenario represents ideal conditions where the model successfully receives all required tracker signals without any loss or degradation. This is the standard operational scenario for most existing methods.

1 The orientation is represented by the 6D representation of rotation [Zhou et al., 2019].

- Instantaneous data-loss scenario: Real-world applications are prone to transient disruptions, such as network fluctuations, leading to the momentary loss of specific tracker signals. To simulate these sudden and brief interruptions in signal transmission in this scenario, we make each tracker signal at any given time t subject to a probability p of being lost, as illustrated in Fig 3.A.
- Prolonged data-loss scenario: There are instances where certain tracker signals may be consistently unavailable over a long-lasting period 2 . Such situations can arise from continuous occlusion of infrared markers or sustained invisibility of hand-held controller signals in an HMD kit. This scenario is characterized by a continuous loss of all tracker signals for a specific duration, as depicted in Fig 3.B. It represents challenges posed by prolonged data unavailability.

The detailed configurations and implications of these scenarios will be further explored in Section 3.4 for the training stage and Section 4.3 for the testing stage, respectively.

## 3.2 Overview

ReliaAvatar adopts an autoregressive pipeline during both the training and testing stages. The process of ReliaAvatar's handling continuous tracker signal sequences can be represented by the following equations:

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

<!-- formula-not-decoded -->

...

During the training stage, given a tracker signal sequence {xt }L-1 t=0 as input, the training process is divided into L sequential steps with each step building upon the previous one. L is the length of the sequence.

As depicted in Figure 2, our model features two distinct yet interconnected pathways: the regression pathway (Regression Encoder→Joint-relation Transformer→Decoder) and the prediction pathway (Prediction Encoder→Joint-relation Transformer→Decoder). The regression pathway operates similarly to the traditional avatar animator. Its primary function is to regress full-body poses from sparse joint information. This process can be formulated as:

<!-- formula-not-decoded -->

The prediction pathway, on the other hand, aligns more closely with motion prediction models. It utilizes known sequences of past motion to predict the current full-body motion. The formulation for this pathway is as follows:

<!-- formula-not-decoded -->

Both pathways converge at the Joint-Relation Transformer. This convergence results in the complete model represented by Eq. (3). The Joint-Relation Transformer and the decoder are shared modules utilized by both pathways, while each pathway has its unique components - the regression encoder for the regression pathway and the prediction encoder for the prediction pathway.

2 The term "long-lasting period" refers to a period that is significantly longer than the frame interval, for example, 1 second.

Figure 2: Illustration of our dual-pathway, autoregressive framework. ReliaAvatar has two pathways: the regression pathway (Regression Encoder→Joint-Relation Transformer→Decoder) and the prediction pathway(Prediction Encoder→Joint-Relation Transformer→Decoder). The output of ReliaAvatar at time-step t forms a part of the input at time-step t +1. The blocks with diagonal lines as foreground ( , , ) represent tokens that signify SMPL joints, e.g., pelvis, left wrist, right ankle.

![Image](https://ijcai.org/proceedings/2024/fulltext/346/image_000001_318d44f206cac9de2d37d896ed0bd13d5cb7c5733401824154ff6c400ccd16ab.png)

Figure 3: Illustration of data-loss scenarios. represents that the signals have been received properly, while represents that the signals are lost.

![Image](https://ijcai.org/proceedings/2024/fulltext/346/image_000002_7a8f0e42fb370ab63d0fab51e007e9cc2d1ba516c1ae049bb2547a109ac6cb73.png)

## 3.3 Model Architecture

Regression Encoder and Prediction Encoder. As shown in Figure 2, tracker signals are initially processed through a sparse fully connected layer (SFC) to extract initial features Et0 ∈ R|J|×d , where d is the embedding dimension of the model. The purpose of SFC is to merge multiple linear layers into a sparse linear layer, thereby reducing the number of loops in the forward process and improving the runtime efficiency. In the regression encoder, SFC is formed by merging four linear layers that handle orientation xr, rotation velocity ∆xr, position xp, and linear velocity ∆xp. This module can be formulated as:

<!-- formula-not-decoded -->

Then, the GRU module will extract the temporal features, as described in the following formula:

<!-- formula-not-decoded -->

After the regression encoder, the features of all tracker signals Etx ∈ R|J|×d are obtained. Subsequently, an upsampling module will be used to increase the dimension to 22×d, thus obtaining the decoding token sequence {tix }21 i=0 which will be input to the Joint-Relation Transformer. Each token represents a SMPL joint.

The prediction encoder is similar to the regression encoder. After passing through an SFC and a GRU module, ˆ yt-1 is mapped to a token Ety ∈ Rd . Then a fully connected layer is applied to expand it to 22 joint tokens {tiy }21 i=0 .

Joint-Relation Transformer. In human kinematics, the motions of the entire body are not simply a combination of independent motions of multiple joints. In contrast, the actions performed by the human body are strongly correlated with the relationship between joints. Therefore, it is evident that equipping the model with the ability to model joint relationships can enhance the driving effectiveness. A Transformer encoder is used to model joint relationships, which has been proven to be effective in PETR [Shi et al., 2022].

The input of the Transformer is a token sequence {ti }21 i=0 = Concat (tx, ty) ∈ R22×(2d) representing 22 SMPL joints, which is fused from the tokens generated by the previous two encoders.

Decoder. The decoder is responsible for decoding rotation ˆ ytrot and position ˆ ytpos directly from the output of the token sequence of the Transformer. The decoder is composed of two layers of MLP. There are three decoder variants:

- SharedDecoder. Since the pelvic joint parameter represents the global orientation of the entire body, while the other joint parameters represent local rotations relative to their parent joints, a dedicated decoder is used for the pelvic joint, while the remaining joints share a common decoder.
- Multi-FC Decoder. Each joint has its own dedicated decoder. During the inference process, the decoding of each joint is done sequentially.

- SFC Decoder. Use an MLP composed of SFCs as a unique decoder. As a result, all joints can be decoded simultaneously in parallel.

We adopted SFC decoder due to its advantages of high accuracy and high efficiency. The comparison of the three decoders will be mentioned in Sec 4.5.

## 3.4 Simulation Training

We simulate the three scenarios mentioned in Sec during training ReliaAvatar.

- Standard scenario. In the standard scenario, all tracker signals can be fully and correctly transmitted to the model. So in this scenario, there is no need for any masking of the signals.
- Instantaneous data-loss scenario. In the instantaneous data-loss scenario, we use a dropout layer to randomly mask the signals of each tracker in each frame with a probability of 0.1.
- Prolonged data-loss scenario. In the prolonged dataloss scenario, we mask out the latter half of each input sequence {xt}L-1 t=0 . That is to say, {xt}L-1 t=L/2 = 0.

Each sequence has an equal probability (1/3) to undergo the above three simulation treatments.

## 3.5 Loss Function

We adopt orientation loss Lori, rotation loss Lrot, SMPL position loss L SMPL pos , decoded position loss L dec pos , and velocity loss Lvec during the optimization process. All losses are computed using the L1 loss. Taking the rotation loss as an example, its loss function is as follows:

<!-- formula-not-decoded -->

Here, ˆ yrot, yrot represent the model output and corresponding ground truth, respectively. Other losses are computed in the same way. The whole loss function can be formulated as:

<!-- formula-not-decoded -->

Here, λori, λrot, λSMPL pos , λdec pos , λvec are weights of corresponding losses. The difference between L SMPL pos and L dec pos is that L SMPL pos is computed based on the position SMPL(ˆ yrot) inferred by SMPL and the ground truth ypos, while L dec pos is computed based on the position ˆ ypos decoded by the decoder and ypos.

## 4 Experiments

## 4.1 Implementation Details

For all GRUs in the model, the number of layers is set to 1 and the dimension is set to 256. The joint-relation Transformer has 4 layers with a dimension of 512. The initial learning rate is set to 5 × 10-4 , and is halved every 15000 iterations. The length of the input sequence L is set to 32. {λori, λrot, λSMP L pos , λdec pos , λvec} = {0.02, 1, 1, 1, 0.5}. The model is trained on two GeForce RTX 3090 GPUs for a total of 90000 iterations, with a batch size of 32 on each GPU. The training process costs approximately 32 hours to complete.

## 4.2 EvaluationMetrics

We adopt the following evaluation metrics.

- MPJPE: Mean Per Joint Position Error [cm].
- MPJRE: Mean Per Joint Rotational Error [degree].
- MPJVE: Mean Per Joint Velocity Error [cm/s].
- fps: Frames Per Second [frame].

## 4.3 Evaluation Protocols

Standard scenario. Following the experimental setup outlined in AvatarPoser [Jiang et al., 2022], we partition the CMU [graphics lab., 2000], BMLrub [Troje, 2002], and HDM05 [M¨uller et al., 2007] subsets into 90% training data and 10% testing data. For comparison, we evaluate the results using both three inputs (head and wrists) and four inputs (head, wrists, and pelvis). It is worth noting that, unless explicitly stated otherwise, all other results are based on the usage of three inputs.

Instantaneous data-loss scenario. AGRoL [Du et al., 2023] proposed a setting for instantaneous data-loss, which randomly masks out 10% of the input sequence. However, in practical applications, an instantaneous data-loss does not necessarily mean the complete loss of all tracker signals. Instead, it might indicate the loss of information from a specific joint. Therefore, we propose a new setting where signals of each tracker signal have a certain probability p of being lost at each moment. We evaluated each model five times at p = 0.1, 0.5 and 0.9, and took the average as the result.

Prolonged data-loss scenario. In the scenario of prolonged data-loss, data is continuously lost for a long-last period. For evaluation purposes, we set a protocol to mask out subsequent M frames every 80 frames. We evaluated each model in M = 20, 40 and 60.

## 4.4 Comparison to the State-of-the-art

The evaluation results of each model in the standard scenario are extracted from the reports in their respective papers. For comparison in data-loss scenarios, we select several publicly available state-of-the-art methods (AvatarPoser [Jiang et al., 2022], AGRoL [Du et al., 2023] and AvatarJLM [Zheng et al., 2023]). We conduct a fair comparison by retraining their publicly available code and evaluating them under the protocols for data-loss scenarios using the retrained parameters.

Standard scenario. To verify the effectiveness of our model, we perform a fair comparison with the state-of-the-art methods in the standard scenario. The results, as shown in Table 2 and Table 3, demonstrate that our model achieves state-ofthe-art performance in three- and four-input conditions. This indicates that ReliaAvatar, despite being primarily designed to tackle data-loss scenarios, surpasses other models even in the standard scenario.

Instantaneous data-loss scenario. We compare ReliaAvatar with other models under three conditions: p = 0.1, p = 0.5, and p = 0.9. As shown in Table 4, ReliaAvatar demonstrates its superior robustness compared to other models in the instantaneous data-loss scenario. Under the conditions of p = 0.1 and 0.5, ReliaAvatar is hardly affected. Even when the signals of each tracker have only a 10% probability of being received, ReliaAvatar can still operate normally.

Table 2: State-of-the-art comparison on AMASS [Mahmood et al., 2019] in the standard scenario using three inputs (head and wrists). Best in bold.

| Method                           | MPJRE   |   MPJPE |   MPJVE |
|----------------------------------|---------|---------|---------|
| FinalIK [Unity, 2018]            | 16.77   |   18.09 |   59.24 |
| CoolMoves [Ahuja et al., 2021]   | 5.20    |    7.83 |  100.54 |
| LoBSTr [Yang et al., 2021]       | 10.69   |    9.02 |   44.97 |
| VAE-HMD [Dittadi et al., 2021]   | 4.11    |    6.83 |   37.99 |
| AvatarPoser [Jiang et al., 2022] | 3.21    |    4.18 |   29.40 |
| EgoPoser [Jiang et al., 2023]    | -       |    4.14 |   25.95 |
| AGRoL [Du et al., 2023]          | 2.66    |    3.71 |   18.59 |
| DAP [Di et al., 2023]            | 2.69    |    3.68 |   24.30 |
| AvatarJLM [Zheng et al., 2023]   | 2.90    |    3.35 |   20.79 |
| ReliaAvatar (Ours)               | 2.53    |    3.18 |   18.30 |

Table 3: State-of-the-art comparison on AMASS [Mahmood et al., 2019] in the standard scenario using four inputs (head, wrists, and pelvis. Best in bold.

| Method                           |   MPJRE |   MPJPE |   MPJVE |
|----------------------------------|---------|---------|---------|
| FinalIK [Unity, 2018]            |   12.39 |    9.54 |   36.73 |
| CoolMoves [Ahuja et al., 2021]   |    4.58 |    5.55 |   65.28 |
| LoBSTr [Yang et al., 2021]       |    8.09 |    5.56 |   30.12 |
| VAE-HMD [Dittadi et al., 2021]   |    3.12 |    3.51 |   28.23 |
| AvatarPoser [Jiang et al., 2022] |    2.59 |    2.61 |   22.16 |
| AvatarJLM [Zheng et al., 2023]   |    2.40 |    2.09 |   17.82 |
| ReliaAvatar (Ours)               |    2.16 |    1.94 |   14.32 |

Prolonged data-loss scenario. We compare our model with the previous state-of-the-art models in the prolonged dataloss scenario under three conditions: M = 20, 40, and 60. Table 5 indicates that existing models are unable to handle scenarios where there is a continuous loss of more than 40 frames of signals. In contrast, ReliaAvatar exhibits superior robustness compared to other methods, as it can function nearly normally even with a continuous loss of 20 frames. It is capable of operating even when the signal is continuously lost for 60 frames.

Table 4: Comparison in instantaneous data-loss scenario. Best in bold and second best underlined.

| p Method                        |   MPJRE |   MPJPE |   MPJVE |
|---------------------------------|---------|---------|---------|
| 0.1 AvatarPoser AGRoL AvatarJLM |    9.01 |   20.02 | 1532.62 |
|                                 |    6.59 |   12.31 |  101.71 |
|                                 |    6.65 |   13.40 |  887.24 |
| ReliaAvatar(Ours)               |    2.66 |    3.32 |   20.44 |
| AvatarPoser AGRoL               |   15.47 |   53.07 | 3651.32 |
| 0.5                             |    9.96 |   20.04 |  114.12 |
| AvatarJLM                       |   16.86 |   43.02 | 2512.13 |
| ReliaAvatar(Ours)               |    2.77 |    3.51 |   26.51 |
| AvatarPoser                     |   17.98 |   63.53 | 2582.13 |
| 0.9 AGRoL                       |   13.08 |   27.59 |  117.26 |
| AvatarJLM                       |   25.16 |   65.50 | 1154.16 |
| ReliaAvatar(Ours)               |    5.12 |    7.86 |   45.00 |

Table 5: Comparison in prolonged data-loss scenario. Best in bold and second best underlined.

| M Method                       |   MPJRE |   MPJPE |   MPJVE |
|--------------------------------|---------|---------|---------|
| AvatarPoser                    |    7.43 |   14.46 |  102.18 |
| 20 AGRoL                       |    6.59 |   12.31 |  101.71 |
| AvatarJLM                      |    8.96 |   18.71 |  133.92 |
| ReliaAvatar (Ours)             |    2.84 |    3.67 |   24.69 |
| 40 AvatarPoser AGRoL AvatarJLM |   11.52 |   23.57 |  123.33 |
|                                |    9.96 |   20.24 |  114.12 |
|                                |   14.52 |   34.39 |  154.17 |
| ReliaAvatar (Ours)             |    3.48 |    5.17 |   34.01 |
| AvatarPoser                    |   15.05 |   31.36 |  118.73 |
| 60 AGRoL                       |   13.08 |   27.59 |  117.26 |
| AvatarJLM                      |   20.14 |   49.95 |  181.36 |
| ReliaAvatar (Ours)             |    4.70 |    7.69 |   45.02 |

Table 6: Ablation experiments of the input signals. Best in bold.

| Input signals               |   MPJRE |   MPJPE |   MPJVE |
|-----------------------------|---------|---------|---------|
| {X}                         |    2.69 |    3.49 |   23.49 |
| {X, yrot}                   |    2.54 |    3.25 |   19.44 |
| {X, ypos}                   |    2.58 |    3.28 |   18.39 |
| {X, yrot, ypos}             |    2.53 |    3.25 |   19.07 |
| {X, yrot, ypos,∆yrot,∆ypos} |    2.53 |    3.18 |   18.30 |

ReliaAvatar's robustness to low-quality signals stems from three aspects: 1) Our model abandons windowed inputs. Models that solely rely on a window of tracker signals as input (e.g., AvatarPoser with a window size of 40) exhibit obvious malfunctions when facing prolonged data-loss since the signals within the window are all set to zero. 2) Our model integrates motion prediction into the avatar animator. When the regression pathway fails to function properly, the prediction pathway can still predict the current full-body motion based on the historical trajectory states. 3) The autoregressive training paradigm allows for simulating abnormal scenarios that may occur in applications.

## 4.5 Ablation Studies

To illustrate the roles of various designs in our model, we conduct the following ablation experiments.

Input Signals. As shown in Table 6, we validate the performance gain of incorporating historical trajectory states into the input. Here, X represents the input used by most current methods, i.e. X = {xr,∆xr, xp,∆xp}.

If the historical trajectory states are not used as input, the regression pathway remains inactive. The experimental results demonstrate that incorporating yrot,∆yrot, ypos,∆ypos into the input leads to performance gains. Due to the continuity of motion, the current full-body is naturally influenced by and aligned with the previous trajectories. So this improvement can be attributed to the previous trajectories providing cues for the generation of the current full-body motion.

Decoder. As shown in Table 7, we compare the performance of different decoder designs in our model. The model using a

Table 7: Ablation experiments of the design of decoder. Best in bold and second best underlined.

| Decoder   |   MPJRE |   MPJPE |   MPJVE |    fps |
|-----------|---------|---------|---------|--------|
| Shared    |    2.64 |    3.38 |   19.51 | 112.22 |
| Multi-FC  |    2.51 |    3.21 |   18.64 |  46.73 |
| Sparse-FC |    2.53 |    3.18 |   18.30 | 109.65 |

Table 8: Ablation experiments of the fusion method. Best in bold.

| method             |   MPJRE |   MPJPE |   MPJVE |
|--------------------|---------|---------|---------|
| add                |    2.66 |    3.31 |   19.70 |
| concat (joint-dim) |    2.66 |    3.31 |   19.83 |
| concat (feat-dim)  |    2.53 |    3.18 |   18.90 |

shared decoder has larger errors but shorter runtime while the model using multi-FC decoders has smaller errors but suffers from poor real-time performance. The model using a Sparse-FC decoder has the advantages of both low error and fast running speed.

Fusion Method. The methods to integrate features from two pathways are worth exploring. We have explored three different methods for feature integration: addition, concatenation along the feature dimension, and concatenation along the joint dimension. As shown in Table 8, the fusion method of concatenating features along the embedding dimensions achieves the best performance.

Simulation Training. An important distinction of ReliaAvatar compared to other methods is its ability to simulate various scenarios during the training process. This is attributed to the model architecture with joint motion prediction and the autoregressive training paradigm. The results presented in Table 9 demonstrate that simulation training for data-loss scenarios does not adversely affect performance in the standard scenario.

## 4.6 Analysis

Ability of motion prediction. An important capability that we expect ReliaAvatar to achieve is the ability to predict fullbody movements of the current frame using historical trajectory states when tracker signals are lost. This enables us to drive avatars smoothly even in the absence of real-time tracker information. We calculate MPJPE for the 40 frames after losing real-time tracker signals. The results presented in Table 10 demonstrate that ReliaAvatar is capable of predicting relatively accurate poses within several tens of frames after the disappearance of tracker signals.

Table 9: Ablation experiments of simulation training. Best in bold and second best underlined.

| method           |   Standard. |   Instantaneous |   Prolonged |
|------------------|-------------|-----------------|-------------|
| no mask          |        3.17 |           51.41 |       24.30 |
| + random mask    |        3.18 |            3.92 |       12.26 |
| + prolonged mask |        3.18 |            3.51 |        5.17 |

Table 10: The statistical errors after the interruption of tracker signals. #n represents the n-th frame after the interruption.

| Frame   |   #1 |   #3 |   #7 |   #10 |   #20 |   #30 |   #40 |
|---------|------|------|------|-------|-------|-------|-------|
| MPJPE   | 3.41 | 3.51 | 3.91 |  4.34 |  6.34 |  8.94 | 11.62 |

Inference Time. To ensure a fair comparison of the operational efficiency of different models, we conduct speed tests on state-of-the-art models under identical operating modes and hardware conditions. For each model, we construct a zero input and run it 10,000 times to obtain the average time as its inference time. All tests are conducted on a single GeForce RTX 3090 GPU. As shown in Figure 4, ReliaAvatar surpasses other models in both operational efficiency and accuracy.

Figure 4: Inference time comparison.

![Image](https://ijcai.org/proceedings/2024/fulltext/346/image_000003_0ba1fcdbafc6388c46c2fb5175e8bc62a9cc291bd2dffb2004c4c24e0145ede7.png)

## 5 Conclusion

We conducted a comprehensive exploration of the full-body avatar pose estimation problem under low-quality signal scenarios, which had not been systematically investigated before. We summarize three scenarios that may be encountered in practical applications: standard scenario, instantaneous dataloss scenario, and prolonged data-loss scenario. To address these challenges, we proposed ReliaAvatar, a real-time, robust, autoregressive avatar animator. We consider the fullbody avatar pose estimation problem as a combination of joint upsampling and motion prediction. Therefore, ReliaAvatar possesses an upsampling pathway and a prediction pathway. Furthermore, we incorporate simulation training for data-loss scenarios on top of autoregressive training. Experimental results demonstrate that ReliaAvatar not only outperforms other methods in the data-loss scenarios but also achieves state-ofthe-art performance in the standard scenario. In addition to its outstanding accuracy and robustness, ReliaAvatar also exhibits superior running efficiency compared to other methods. These advantages reduce the hardware requirements for animating a full-body avatar. As a result, more affordable and convenient devices can be used to drive avatars, which contributes to the wider adoption of related technologies.

## Acknowledgements

This work was supported by National Key R&amp;D Program of China under Grant No. 2021ZD0110400, the Fundamental Research Funds for the Central Universities No. xxj032023020, and sponsored by the CAAI-MindSpore Open Fund, developed on OpenI Community.

## References

1. [Ahuja et al., 2021] Karan Ahuja, Eyal Ofek, Mar Gonzalez- Franco, Christian Holz, and Andrew D Wilson. Cool- moves: User motion accentuation in virtual reality. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2):1-23, 2021.
2. [Aliakbarian et al., 2023] Sadegh Aliakbarian, Fatemeh Saleh, David Collier, Pashmina Cameron, and Darren Cosker. Hmd-nemo: Online 3d avatar motion gener- ation from sparse observations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9622-9631, 2023.
3. [Bai et al., 2024] Yifan Bai, Zeyang Zhao, Yihong Gong, and Xing Wei. Artrackv2: Prompting autoregressive tracker where to look and how to describe. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
4. [Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
5. [Di et al., 2023] Xinhan Di, Xiaokun Dai, Xinkang Zhang, and Xinrong Chen. Dual attention poser: Dual path body tracking based on attention. In 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Work- shops (CVPRW), pages 2795-2804. IEEE, 2023.
6. [Dittadi et al., 2021] Andrea Dittadi, Sebastian Dziadzio, Darren Cosker, Ben Lundell, Thomas J Cashman, and Jamie Shotton. Full-body motion from a single head- mounted device: Generating smpl poses from partial ob- servations. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 11687- 11697, 2021.
7. [Du et al., 2023] Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 481-490, 2023. [graphics lab., 2000] CMU graphics lab. Cmu graphics lab motion capture database. http://mocap.cs.cmu.edu/, 2000. Accessed: 2024-05-08.
8. [Huang et al., 2018] Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J Black, Otmar Hilliges, and Ger- ard Pons-Moll. Deep inertial poser: Learning to recon- struct human pose from sparse inertial measurements in real time. ACM Transactions on Graphics (TOG), 37(6):1- 15, 2018.
9. [Jain et al., 2016] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the ieee conference on computer vision and pattern recogni- tion, pages 5308-5317, 2016.
10. [Jiang et al., 2022] Jiaxi Jiang, Paul Streli, Huajian Qiu, An- dreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In European Conference on Computer Vision, pages 443-460. Springer, 2022.
11. [Jiang et al., 2023] Jiaxi Jiang, Paul Streli, Manuel Meier, Andreas Fender, and Christian Holz. Egoposer: Robust real-time ego-body pose estimation in large scenes. arXiv preprint arXiv:2308.06493, 2023.
12. [Lehrmann et al., 2014] Andreas M Lehrmann, Peter V Gehler, and Sebastian Nowozin. Efficient nonlinear markov models for human motion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 1314-1321, 2014.
13. [Liu et al., 2019] Zhenguang Liu, Shuang Wu, Shuyuan Jin, Qi Liu, Shijian Lu, Roger Zimmermann, and Li Cheng. Towards natural and accurate future motion prediction of humans and animals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10004-10012, 2019.
14. [Ma et al., 2022] Tiezheng Ma, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. Progressively gen- erating better initial guesses towards next stages for high- quality human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6437-6446, 2022.
15. [Mahmood et al., 2019] Naureen Mahmood, Nima Ghor- bani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442-5451, 2019.
16. [Mao et al., 2019] Wei Mao, Miaomiao Liu, Mathieu Salz- mann, and Hongdong Li. Learning trajectory dependen- cies for human motion prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9489-9497, 2019.
17. [Mao et al., 2020] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion pre- diction via motion attention. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 474-489. Springer, 2020.
18. [Martinez et al., 2017] Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using re- current neural networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2891-2900, 2017.
19. [Müller et al., 2007] Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Mocap database hdm05. Institut für Informatik II, Universität Bonn, 2(7), 2007.
20. [Shi et al., 2022] Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, and Wenming Tan. End-to-end multi-person pose estima- tion with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11069-11078, June 2022.
21. [Taylor et al., 2006] Graham W Taylor, Geoffrey E Hinton, and Sam Roweis. Modeling human motion using binary latent variables. Advances in neural information process- ing systems, 19, 2006.
22. [Troje, 2002] Nikolaus F Troje. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of vision, 2(5):2-2, 2002.
23. [Unity, 2018] Unity. The final inverse kinematics solu- tion for unity. https://assetstore.unity.com/packages/tools/ animation/final-ik-14290, 2018. Accessed: 2024-05-08.
24. [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
25. [Vlasic et al., 2007] Daniel Vlasic, Rolf Adelsberger, Gio- vanni Vannucci, John Barnwell, Markus Gross, Wojciech Matusik, and Jovan Popović. Practical motion capture in everyday surroundings. ACM transactions on graphics (TOG), 26(3):35-es, 2007.
26. [Von Marcard et al., 2017] Timo Von Marcard, Bodo Rosen- hahn, Michael J Black, and Gerard Pons-Moll. Sparse in- ertial poser: Automatic 3d human pose estimation from sparse imus. In Computer graphics forum, volume 36, pages 349-360. Wiley Online Library, 2017.
27. [Wei et al., 2023] Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual track- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 9697-9706, 2023.
28. [Xsens, 2013] MVN Xsens. Full 6dof human motion track- ing using miniature inertial sensors. Daniel Roetenber- gLuingeHenk, 2013.
29. [Yan et al., 2024] Kangwei Yan, Fei Wang, Bo Qian, Han Ding, Jinsong Han, and Xing Wei. Person-in-wifi 3d: End- to-end multi-person 3d pose estimation with wi-fi. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
30. [Yang et al., 2021] Dongseok Yang, Doyeon Kim, and Sung- Hee Lee. Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals. In Computer Graphics Forum, volume 40, pages 265-275. Wiley On- line Library, 2021.
31. [Yi et al., 2021] Xinyu Yi, Yuxiao Zhou, and Feng Xu. Transpose: Real-time 3d human translation and pose es- timation with six inertial sensors. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.
32. [Zheng et al., 2023] Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, and Xiaojie Jin. Realistic full-body tracking from sparse observations via joint-level modeling. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 14678-14688, 2023.
33. [Zhou et al., 2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753, 2019.