MCM: Multi-condition Motion Synthesis Framework

1 Introduction

Human motion synthesis finds extensive applications in fields such as film production, game development, and simulation. However, traditional manual animation techniques are notably constrained in terms of efficiency. Following the advent of neural network-based generative models, a variety of these models, such as Variational Autoencoders (VAEs) [Kingma and Welling, 2013], Generative Adversarial Networks [Goodfellow et al., 2014], Denoising Diffusion Probabilistic Models (DDPM) [Ho et al., 2020] have been adapted and refined for the specific domain of human motion generation to achieve high-fidelity results.

Current conditional human motion generation, such as text-to-motion [Guo et al., 2022a; Zhang et al., 2024] and music-to-dance [Siyao et al., 2022; Tseng et al., 2023] focuses on generating human motion from a singular modality condition. The integration of condition information from different modalities is an area that has not been extensively explored. The dual-modality-driven 3D motion generation task presents the following primary challenges:

• Current HMS datasets predominantly encompass unimodal conditions. For instance, HumanML3D [Guo et al., 2022a] and AIST++ [Li et al., 2021] only consider text modality and music modality, respectively.
• Text and audio, being temporally non-aligned modalities, pose significant challenges in ensuring both temporal and semantic correlation with 3D human motion spontaneously.
• Fine-tuning directly from an existing single-condition pretrained model may necessitate structural adjustments to the existing model architecture on one hand, and, on the other hand, fine-tuning may lead to catastrophic forgetting of the pre-trained conditional association capabilities.

Typically, TM2D [Gong et al., 2023a] and UDE [Zhou and Wang, 2023a] tokenize motion from HumanML3D [Guo et al., 2022a] and AIST++ [Li et al., 2021] using Motion VQVAE [Van Den Oord et al., 2017]. In the generation phase, they use text and sound respectively to drive GPT [Hudson and Zitnick, 2021] to generate motion token sequences and then integrate these sequences by using weighted merging or replacement methods to achieve multi-condition driven. Employing the post-fusion approach circumvents the need to construct large-scale text-sound-motion data pairs; however, it harbors a critical drawback: each motion token in the sequence is essentially generated under the influence of a single modality condition. Tokens generated based on text do not conform to auditory conditions, while those generated from sound fail to align with semantic information.

To address the above challenges, we propose a novel endto-end framework MCM (Multi-Condition Motion synthesis), which is tailored for multi-conditon driven 3D human motion synthesis. MCM adopts a dual-branch structure, comprising the main branch and the control branch. Initially, the main branch leverages an arbitrary pre-trained text2motion DDPM network like MotionDiffuse [Zhang et al., 2024] and MDM [Tevet et al., 2022], ensuring the motion quality and semantic coherence during multi-condition motion synthesis as shown in Figure 1. Subsequently, the control branch initializes its parameters, mirroring the structure of the main branch, and assumes the responsibility of modifying the motion in accordance with the auditory conditions. During the training of tasks conditioned on sound, MCM freezes the parameters of the main branch and activates those of the control branch. This approach is designed to preserve the motion quality and semantic correlation capabilities of the main branch without compromise. This methodology obviates the need for collecting a text-audio-motion dataset, yet enables both audio and text conditions to simultaneously influence every part of the motion sequence.

Regarding the structural design of the main branch, existing text2motion models [Zhang et al., 2024; Tevet et al., 2022] employ self-attention along the temporal dimension to grasp the sequential associations of motion sequences and perform cross-attention with textual descriptions to capture semantic information. However, when considering motion sequences, it is imperative to recognize that the channel dimension holds valuable spatial information and inter-joint relationships within the human body, aspects that have often been underappreciated. In this work, we propose a Transformer-based DDPMnetwork, termedMWNet, that incorporates a Multi-Wise self-attention mechanism (channelwise self-attention) to better learn spatial information.

In summary, our core contributions are as follows:

• We introduce a unified MCM framework for 3D human motion synthesis based on multiple conditions. Remarkably, without necessitating structural reconfiguration of the network, MCM extends the capabilities of DDPMbased methods to sound-conditional inputs.
• We propose a Transformer-based architecture MWNet, enriched with a multi-wise attention mechanism, which leverages spatial information within motions to achieve better motion generation quality and comparable semantic matching capacity.
• Extensive experiments show that our method demonstrates competitive performance in the single-conditiondriven tasks (text2motion and music2dance) and multicondition-driven tasks. Furthermore, we also present an ablation analysis to elucidate the contribution of each component, enhancing understanding of their individual and collective impact on the system's performance.

2 RelatedWork

2.1 Text-to-motion

Text-to-motion converts descriptive text into motion sequences using various generative models. VAE-based methods [Guo et al., 2022a; Guo et al., 2022b; Petrovich et al., 2022] learn motion distribution in latent space but are limited by VAE's Gaussian posterior estimation, resulting in suboptimal results. Recently, diffusion models have shown impressive image synthesis capabilities, prompting increased research into their application for motion generation.

MotionDiffuse [Zhang et al., 2024] and MDM [Tevet et al., 2022] employ DDPM for human motion synthesis. MotionDiffuse features a linear self-attention mechanism, whereas MDM uses a vanilla Transformer Encoder and predicts the original motion sequence at each time step, instead of step-specific noise. MLD [Chen et al., 2023] utilizes a latent vector DDPM for forward noising and reverse denoising in motion latent space. MAA [Azadi et al., 2023] enhances performance on out-of-distribution data by pretraining a diffusion model on a large dataset of (text, static pseudo-pose) pairs. ReMoDiffuse [Zhang et al., 2023c] introduces a retrieval-enhanced DDPM, significantly improving in-distribution text-to-motion capabilities.

With advancements in Large Language Models, autoregressive generative models are now used to link two different modalities. TM2T [Guo et al., 2022b], T2MGPT [Zhang et al., 2023a], and MotionGPT [Jiang et al., 2023] approach text-to-motion by using VQ-VAE [Van Den Oord et al., 2017] to discretize motion sequences and employ GRU [Chung et al., 2014], GPT [Vaswani et al., 2017; Radford et al., 2019], or T5 [Raffel et al., 2020] models to learn the correlations between motion and text tokens. HumanTomato [Lu et al., 2023] proposes using hierarchical VQ-VAEs and GPT for the generation of full-body motion sequences.

2.2 Music-to-dance

The music-to-dance task involves generating dance movements synchronized with music beats. DanceNet [Zhuang et al., 2022] uses an LSTM-based auto-regressive method to create dance movements. DeepDance [Sun et al., 2020] employs Generative Adversarial Networks for human motion generation. FACT [Li et al., 2021] introduces the Full-Attention Cross-modal Transformer that utilizes selfattention among music, dance movements, and combined music-dance tokens. Bailando [Siyao et al., 2022] utilizes a VQ-VAE encoding for separate upper and lower body parts, integrating them into full-body dance sequences using GPT. EDGE [Tseng et al., 2023] employs a Transformer-based DDPM for precise music-rhythm matching and introduces a contact consistency loss to address physical errors in foot movements. EnchantDance [Han et al., 2023] develops a robust dance latent space to train a dance diffusion model.

2.3 Multi-condition HMS

The aforementioned method has a limitation in that it can only accept a single modality condition as input, which is insufficient to meet the requirements of many scenarios. Some methods have made efforts to integrate multiple conditions and accommodate various scenarios. TriModal [Yoon et al., 2020] and CaMN [Liu et al., 2022] generate co-speech motion based on speech and spoken text, with CaMN integrating additional conditions such as facial expressions. MoFusion [Dabral et al., 2023] for the first time, implements a unified UNet-based DDPM architecture, which, through two different types of conditional encoders, can effectively function in both text-to-motion and music-to-dance tasks. UDE [Zhou and Wang, 2023b] and TM2D [Gong et al., 2023b] both adopt a model structure based on VQ-VAE and GPT and propose a similar approach to integrate music and text conditions. They generate motion token sequences conditioned on sound and text respectively, and then either weight or concatenate the generated logits to obtain a smooth motion sequence that blends motion tokens conditioned on both text and audio.

3 Method

3.1 ProblemDefinition

The objective of the multi-condition human motion synthesis task is to generate a motion sequence X ∈ RT×D under a set of constraint conditions C. X is an array of xi, where i ∈ {1, 2, . . . , T}, and T denotes the number of frames. Each xi ∈ RD represents the D-dimensional pose state vector at the i-th frame. cj ∈ C could be textual description, speech voice, or background music.

3.2 MCMFramework

As depicted in Figure 2, MCM framework employs a dualbranch architecture, comprising a pre-trained main branch with frozen parameters and a trainable control branch. The main branch is composed of any neural network grounded in DDPM, such as MotionDiffuse or MDM. In this study, we have devised and implemented a DDPM network named MCM as the main branch of MCM, which will be comprehensively detailed in Section 3.3. The control branch mirrors the structural framework of the main branch and is initialized utilizing the parameters derived from the main branch.

Drawing inspiration from ControlNet [Zhang et al., 2023b], we implement a strategy of optimizing the main branch and control branch independently during distinct stages of the training process. We pre-train the main branch on the text-to-motion task, with the objective of this phase being to endow the MCM with foundational motion quality and semantic association capabilities. In the training phase of audio-to-motion tasks, all parameters, excluding those assigned to the control branch and bridge module, are maintained in a fixed state. This strategy is implemented to guarantee the retention of the main branch's generative quality and its capabilities in semantic association.

For the input to the Control branch, we perform elementwise addition of the Jukebox features with the motion latent vector to incorporate audio information into MCM. The output of each layer of the Control branch is added to the input of the main branch through the bridge module, thereby causing a slight offset in the output of the main branch under the control of audio conditions. By performing zero-initialization on the bridge module, we ensure that the initial output of MCM is identical to that of the main branch and gradually adjust the parameters according to the audio conditions during the iterative process.

3.3 MWNet Architecture

As previously elucidated, the quality of the main branch exerts a substantial impact on the final quality and semantic association capacities of the motions generated by MCM. Consequently, we delved deeper into the structural aspects of the main branch in the context of the text-to-motion task and proposed the MWNet model.

As delineated in Section 3.1, the channel dimension of the motion sequence encompasses the entire spatial information pertinent to the motion sequence. Nevertheless, current DDPM-based motion generation models, including MotionDiffuse and MDM, predominantly concentrate on time-wise self-attention and cross-attention mechanisms. These are employed to model the correlations at the temporal level and the semantic level between motions and their respective conditions. In the spatial dimension, information is currently modeled exclusively through linear layers and activation functions. We posit that this approach is inadequate for capturing the complex spatial relationships present in intricate motion sequences, such as the correlations between joints. Consequently, we advocate for the use of channel-wise selfattention [Ding et al., 2022] and introduce the Multi-Wise attention mechanism, specifically designed to model these critical aspects of spatial information. MWNet is composed of layers arranged in the configuration depicted in Figure 3 (a).

Similar to Stable Diffusion [Rombach et al., 2022] and GLIDE [Nichol et al., 2021], we use FiLM [Perez et al., 2018] blocks to furnish timestamp information to MWNet after every attention or Feed Forward Network (FFN) modules. In the FiLM module, the weights and biases for the affine transformation performed on the motion latent vector are derived from the timestep mapping, as shown in the following:

LN denotes layer normalization module [Ba et al., 2016]. W1, and W2 are two projection matrices. I denotes a matrix wherein all elements are 1, possessing a shape congruent with that of x. ⊙ denotes the element-wise multiplication.

With projection weights W Q, W K , W V , X are projected to Q = XW Q, K = XW K , V = XW V and split into Nh heads or Ng groups. We denote Qi, Ki, and Vi for each head or group. Time-wise self-attention can be denoted as follows:

whereas, channel-wise self-attention can be denoted as:

where Ch and Cg denote the channel numbers for head and group, respectively.

4 Experiments

4.1 Dataset

We evaluate the proposed method using the HumanML3D [Guo et al., 2022a], AIST++ [Li et al., 2021], and BEAT [Liu et al., 2022] dataset. In all experiments, we process the motion of all involved datasets to the same format with a 22-joint skeleton (the first 22 joints of the SMPL skeleton) and 20 FPS. Subsequently, following HumanML3D, we use a 263-dimension representation x = concat( ˙ra , ˙rx , ˙rz , ry , j p , j v , j r , cf ) to represents motions at every frame. ˙ra ∈ R is root angular velocity along the Y-axis; ˙rx , ˙rz ∈ R are root linear velocities on XZ-plane; ry is root height; j p j v ∈ R3j and j r ∈ R6j are the local joints positions, velocities, and rotations in root space, with j denoting the number of joints; cf ∈ R4 is binary features obtained by thresholding the heel and toe joint velocities to emphasize the foot ground contacts.

We also processed AIST++ dataset and BEAT dataset to generate pseudo text descriptions, thereby forming audiotext-motion sample pairs for training in the control stage.

4.2 EvaluationMetrics

For text2motion evaluation, we employ the established metrics [Guo et al., 2022a] to assess the generated motion sequences across various dimensions. (1) Motion quality: Frechet Inception Distance (FID) [Onuma et al., 2008] evaluates the dissimilarity between two distributions by calculating the difference between feature vector (extracted by feature extractor) statistical measures (mean and covariance). (2) Diversity: The Diversity and MultiModality metrics respectively assess the degree of variation in the generated motions with different text inputs and with the same text input. (3) Semantic matching: The motion-retrieval precision (R Precision) evaluates the accuracy of matching between texts and motions using Top 1/2/3 retrieval accuracy. Multi-modal Distance (MM Dist) measures the distance between motions and text features extracted by the feature extractor trained with contrastive learning.

For music2dance evaluation, we adopt the commonly used evaluation metrics following [Dabral et al., 2023]. (1) FID: utilizing kinetic and geometry features implemented within fairmotion [Gopinath and Won, 2020]. (2) Diverisy: it computes the average pairwise Euclidean distance of the kinetic and geometry features of the motions generated from music in the validation and test set. (3) Beat Alignment Score (BAS): a metric that quantifies the congruence between kinematic beats and musical beats. Kinematic beats correspond to the local minima of kinetic velocity within a motion sequence, signifying points where motion momentarily halts.

4.3 Implementation Details

We conduct training of MCMs utilizing distinct DDPM-like main branch architecture, including MotionDiffuse [Zhang et al., 2024], MDM [Tevet et al., 2022], and our MWNet. The conditioning inputs from diverse modalities are pre-processed through the employment of pre-trained condition encoders. We use CLIP [Radford et al., 2021] to extract features from text conditions. Similar to EDGE, we use the prior layer of Jukebox [Dhariwal et al., 2020] to extract features from all audio conditions(music and human voice) and downsample them to the same frame rate as the motion samples (20 FPS). Regarding the diffusion model, we set the number of diffusion steps at 1000, while the variances βt follow a linear progression from 0.0001 to 0.02. We employ the Adam optimizer for training the model, employing a learning rate of 0.0002 throughout both training phases. In concordance with MDM, we adopt the strategy of predicting xstart as an alternative to the prediction of noise. This approach is aimed at achieving an enhanced quality of motion.

4.4 Music-to-Dance Evaluation

During the training phase on the AIST++ dataset, we processed the training data into motion segments with a maximum length of 196 frames, equivalent to 9.8 seconds. In the evaluation stage, prior studies were assessed using the complete dance motion sequences from the validation and test sets of AIST++ dataset, maintaining the original frame rate of 60 FPS. To facilitate an equitable comparison with these studies, we linearly interpolated the generated motion sequences, initially at 20 FPS, to upscale them to 60 FPS. Table 4 presents the evaluation outcomes of MCM on the AIST++ dataset. When utilizing MCM, three distinct DDPM methodologies, previously trained on the text2motion dataset, all demonstrated performance on par with state-of-the-art (SOTA) methods in the music2dance task. As demonstrated in Table 4, our approach achieved the best F IDk, while UDE achieved the best FIDg, and the diversity of the dance movements generated by each method was found to be remarkably similar. Furthermore, our methodology, while preserving a comparable level of dance motion quality and diversity, attains SOTA metrics in beat alignment exclusively via the elementary element-wise integration of Jukebox music features. Among various implementations utilizing the MCM framework, MWNet + MCM demonstrates relatively superior dance motion quality and beat alignment. Conversely, MotionDiffuse + MCM excels in achieving enhanced diversity in its outputs.

4.5 Text-to-motion Evaluation

We train and evaluate our main branch model MWNet on HumanML3D dataset. The quantitative results are shown in Table 1. Overall, we achieved the best motion quality and next to best semantic relevance among all methods. Moreover, it surpasses all non-DDPM methods. Owing to its retrieval-based mechanism, ReMoDiffuse remains the model with the most robust semantic correlation capabilities. Compared to the TM2D and MoFusion approach, which similarly accommodates the integration of auditory conditions, our method exhibits a distinct advantage in both semantic relevance and the quality of motion. A robust capability for semantic association lays the groundwork for maintaining strong semantic relational abilities in multi-condition HMS scenarios. Our method exhibits moderate performance on the MultiModality metric. The underlying causes of this phenomenon were explored in our ablation studies.

Table 2: We request users to compare the quality of the multicondition HMS from three aspects: beat alignment, dance motion quality, and semantic match. Each cell in the table represents the win rate of MCM in each comparative aspect.

4.6 Text-soundMulti-condition Generation

The outputs of text-sound multi-condition generation are illustrated in Figure 4. Under the multi-condition HMS scenario, we generate corresponding dance, or co-speech movements, under the joint control of textual and auditory conditions. For instance, we adjust specific gestures during a speech and particular dance moves in a dance through text, achieving interesting control effects.

In order to conduct a fair comparison with existing methods capable of integrating textual and auditory conditions. We sampled 10 music segments from the AIST++ dataset and 5 segments from outside the AIST++ dataset, and designed distinct dance motion descriptions for each piece of music. We employed TM2D and MCM to generate dance movements under multi-conditional constraints, specifically, to produce dance sequences that conform to textual descriptions and align with musical rhythms. Subsequently, we invited 20 users to conduct a comparative assessment of the generated outcomes. The subjects were asked to make comparisons from three aspects: beat alignment, text match, and motion quality. Importantly, the entire evaluation process was meticulously designed to be anonymous, thereby ensuring that participants were unable to discern the origins of the data samples, which in turn guaranteed an unbiased assessment of the different models. As shown in Table 2, the results of the user study reveal a distinct preference among users for our outcomes on the AIST++ dataset. Moreover, our results also demonstrate a relative advantage in external datasets. This is attributed to the fact that in MCM, both text and audio conditions concurrently influence the entire motion sequence. In contrast, TM2D's post-fusion approach results in each motion token being driven effectively by only a single modality. Consequently, the actions generated from the text do not align with the rhythm of the music. Simultaneously, our superiority in semantic relevance is largely attributable to the significantly stronger semantic correlation capabilities of our main branch compared to TM2D, as evident from the evaluation results on the HumanML3D dataset.

4.7 Ablation Study

Dual-branch vs Single-branch

In this section, we further explore the superiority of the Dualbranch structure in transferring from text-condition scenarios to sound-condition scenarios. The dual branch structure trains the main branch on the text2motion task, and the control branch on the sound condition; whereas for the single branch structure, we directly optimize the parameters of the main branch during the control stage, and the audio condition is added directly to the motion latent vector as input. In the lower half of Table 4, we compare the performance differences on the AIST++ dataset between dualbranch and single-branch structures equipped with different main branches. The results show that all dual-branch outcomes surpass their single-branch counterparts. Additionally, the single-branch structure no longer possesses the semantic association obtained from text-to-motion pretraining.

Multi-wise Self-attention Module Design

We designed experiments to verify the necessity and superiority of multi-wise self-attention module. Our Multi-wise self-attention module consists of a time-wise self-attention (denoted as TA), a channel-wise self-attention (CS), a crossattention (CA), and two FFNs (F). By modifying or reordering these modules, we compared the performance metrics of different configurations on the HumanML3D benchmark. In this experiment, each trial was limited to training for only 500 epochs to conserve experimental time. We experimented with arranging or adjusting the attention modules for different categories. As shown in Table 3, comparing T/CA/F/T/F and T/CA/F/CS/F, it can be observed that substituting CS for T results in significantly improved motion quality, as CS compensates for the spatial information deficiencies of T. It was found that using both T and CS in the module simultaneously helps significantly in enhancing semantic relevance and motion quality. Additionally, we observed that placing CS before T and CA can improve motion quality with a slight loss in semantic relevance. We also found that the downside of CS is that it leads to reduced MultiModality, which we aim to explore and address in future work. Ultimately, we chose the option that offered the highest motion quality as the module design of MWNet.

5 Conclusion

We propose MCM, a novel framework for the multiconditioned motion synthesis method that spans multiple scenarios. Utilizing MCM, methods based on DDPM for text2motion tasks can concurrently adapt to multi-conditions without necessitating any structural modifications. Additionally, we introduce a Transformer-based architecture MWNet that incorporates multi-wise self-attention, enhancing the modeling of spatial information and inter-joint correlations. We quantitatively evaluate our approach across tasks based on various modal conditions. In the text-to-motion task, our results surpass all existing methods in terms of motion quality and are comparable in semantic correlation capabilities. In the music-to-dance task, we achieved the best beat alignment and F IDk metrics, along with comparable dance diversity and quality. In the multi-condition HMS domain, our approach accomplishes the concurrent generation of dance movements based on both text and sound, as well as cospeech motion.

Acknowledgments

This work was supported in part by the Zhejiang Province (China) Research Project 2023C01045 and National Natural Scientific Foundation of China 61972346. Corresponding author: Weidong Geng. † Equal contribution.