IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation

1 Introduction

In computer vision, automatic visual content generation has experienced a remarkable evolution due to Generative Adversarial Networks (GANs) [Goodfellow et al., 2014]. Many improvements are made to reach better results including CAGAN [Ni et al., 2018] , MEGAN [Park et al., 2018] and so on [Zhang and Peng, 2018b]. GANs are capable of more complex tasks like muti-domain synthesis [Mao and Li, 2018; Hao et al., 2018] and multi-view generation [Tian et al., 2018; Song et al., 2018].

∗These authors contributed equally.

†Corresponding author.

In this paper, we focus on generating videos from text, namely text-to-video generation. There are two critical issues in such task. First, the generated frames need to be both realistic and temporally coherent. Second, the video content needs to be in accordance with text. In brief, text-to-video generation has two key components: visual quality and semantic consistency.

A straightforward idea to handle such issues is to use conditional generative models like c-GAN [Goodfellow et al., 2014] and c-VAE [Doersch, 2016] as most current methods do, which are mainly adapted from image synthesis like [Zhang et al., 2017; Zhang and Peng, 2018a]. When extending image generation to videos, there is an inevitable problem that videos have one more dimension than images. In order to tackle this, [Pan et al., 2017] replaces the 2D convolutional layers [Zeiler et al., 2010] with 3D layers, while [Mittal et al., 2016] chooses to simply use 2D layers to generate the video frame by frame. However, 3D layers may have poorer frame quality than 2D layers [Saito et al., 2017] while 2D layers fail to take temporal dependency into account. Due to these reasons, generated videos may suffer from either low frame quality or poor temporal coherence. On the other hand, simply treating text as conditions cannot concretely measure the semantic consistency between videos and text and thus may struggle to precisely coordinate videos with text.

In order to overcome the difficulties of video generation from text, we present a novel Introspective Recurrent Convolutional GAN as shown in Figure 1, which mainly has following contributions:

• Recurrent Transconvolutional Generator (RTG): Frame quality and temporal coherence are two aspects of visual quality of the generated videos. To improve these, our generator integrates LSTM cells with 2D transconvolutional networks. Such structure endows the generator with memory of history information so that each frame is generated on the basis of previous frames, which may lead to better coherence. Besides, as 2D transconvolutional layers put more emphasis on the details of each frame than 3D ones, the definition of each frame is boosted. In this way, our generator can synthesize videos with better quality.
• Mutual-information Introspection (MI): Instead of simply judging matched or not, mutual information is introduced to measure semantic similarity quantitatively. In this way, our model introspects how far the generated video is semantically from the given text during a two-stage training process. In the first stage, the text encoder is trained with a seq2seq auto-encoder and an introspective network extracting mutual information between the text and the corresponding video. In this way, semantic distances among different text are established. In the second stage, our model introspects the semantic distance between the generated videos and the corresponding text and try to minimize it to boost the semantic consistency. By doing so, our model can generate videos precisely matched with the given text.

To verify the effectiveness of our IRC-GAN method, we collect several datasets and modify them into 3 datasets: Single Moving Mnist-4, Double Moving Mnist-4 and KTH-4, comparing with 4 state-of-the-art approaches.

2 Introspective Recurrent Convolutional GAN

Our IRC-GAN attempts to synthesize a temporal coherent and plausible frame sequence semantically aligned with the given text. As shown in figure 2, there are three components: the text encoder network, the recurrent transconvolutional generator network and the introspective discriminator network.

2.1 Text Encoder Network

The text encoder is designed to encode the given text into text latent codes ztext for video generation. First, each word is represented as a one-hot vector. So a sentence of length l can be denoted as {w1, w2, . . . , wl}. wt is the t-th word's one-hot vector. The sentence is then fed into a bidirectional LSTM network [Schuster and Paliwal, 1997] to contextually embed each word into ht. Finally, we input the contextually embedded word sequence {h1, h2, . . . , hl} into a LSTM-based encoder and treat the final LSTM output as the text latent code ztext ∈ Rdtext .

2.2 Recurrent Transconvolutional Generator Network

Overview of Structure

We prefer 2D-transconvolution-based networks to 3D networks for two reasons: 1) Flexibility. 2D networks can handle any arbitrary length of frame sequences. 2) Frame quality. 3D networks construct the whole video in one go, whose kernels are distracted, while 2D networks do the job frame by frame, whose kernels focus on the details of each frame. So the synthesized frames by 2D networks tend to be of higher definition.

But such method can't deal with temporal coherence, because each frame is generated independently. To break such independence, we introduce LSTM units into the network so that each frame can be created on the basis of previous frames. Furthermore, the text latent code ztext is additionally fed into the LSTM units at each time step to remind the generator of what the given text is.

Details of Network

First, an LSTM-based feature generator Gf(z) : Rdtext+dnorm → Rdl×df is designed to transform the latent code z ∈ Rdtext+dnorm into a frame-wise feature sequence [y1, y2, · · · , ydl ]. The latent code z is the concatenation of the text latent code ztext and a normal random vector znorm. And yi ∈ Rdf denotes the frame-wise feature which will be utilized to generate the i-th frame later.

Then each yi will go through a series of 2D transconvolutional layers, denoted as TransConv2Dk, k = 1, 2, · · · , 5, and will finally turn into a video frame with the size of dc × dh × dw. As is mentioned above, in order to take temporal dependency into account, we integrate LSTM cells with each TransConv2Di to endow the generator with the memory of history information. In particular, the output of TransConv2Di, denoted as Vi ∈ Rdci ×dhi ×dwi , is first reshaped as ˆ Vi ∈ Rdci ×(dhi ×dwi ) and we send ˆ Vi to a LSTMbased memory unit Mi : R(dhi ×dwi ) → R(dhi ×dwi ) channel by channel. We collect all the channels that go through Mi and put together to form Li ∈ Rdci ×dhi ×dwi . Then Li, concatenated with the original Vi in the channel dim, is treated as the input of the next 2D transconvolution layer, TransConv2Di+1. The states of these memory units are passed across frames and thus each frame can be constructed on the basis of history information which can contribute to the temporal coherence.

Besides, we additionally feed the text latent code ztext into the memory units at each time step to further guide the video generation so the mapping of the LSTM-based memory unit is actually R(dhi ×dwi )+dtext → R(dhi ×dwi ) . At the end, we put the generated frames together and join them into a whole video with the size of dl × dc × dh × dw.

2.3 Introspective Discriminator Network

Basic Discriminators

Inspired by [Pan et al., 2017], the discriminator network D is designed to distinguish real videos from synthetic ones from three perspectives: (1) the whole video, (2) each video frame, (3) the motion across adjacent frames. To implement these three points, two types of basic discriminators are required:

• 3D discriminator D1(v) : Rdl×dc×dh×dw → [0, 1]. D1 first extracts video-level features mv from the input video v ∈ Rdl×dc×dh×dw via 3D convolutional layers and then feeds mv into a fully-connected layer with softmax to discriminate whether the input video is real or fake from the perspective of the whole video.
• 2D discriminator D2(v) : Rdl×dc×dh×dw → [0, 1]. D2 also gets a video as its input but processes it in a framewise way. In order to obtain information about the temporal coherence, we get the output after four 2D convolutional layers and subtract the output of the previous frame fi-1 from that of the current one fi. The details are the same as [Pan et al., 2017].

Mutual-information Introspection

The discriminator network described above is only able to help synthesize videos that look like real ones. In order to semantically align the generated videos to the text, we propose mutual-information introspection, which is inspired by InfoGAN [Chen et al., 2016]. We similarly argue that there should be high mutual information between the videos and the corresponding text. In information theory, mutual information between X and Y measures the "amount of information" learned from knowledge of random variable Y about the other random variable X. The mutual information can be expressed as the difference of two entropy terms:

This definition has an intuitive interpretation: I(X; Y ) is the reduction of uncertainty in X when Y is observed. This interpretation makes it easy to formulate a cost in the text-to-video problem: give any z′ ∼ Pztext (z), we want PG(G(ztext, znorm)|ztext = z′ ) to have a small entropy. In other words, the uncertainty of the generated video is minimized when the corresponding text is given. In formalization, mutual-information introspection can be expressed as the following optimization problem:

Similarly with [Chen et al., 2016], the mutual information term I(ztext;G(ztext, znorm)) is hard to maximize directly and we first find its lower bound:

The above inequation makes sense because the KLdivergence is always non-negative. It is obvious that this lower bound becomes tight as the auxiliary ditribution Q approaches the posterior distribution P(·|v) where v obeys the distribution of videos , making the KL-divergence between Q and P decreases to zero.

In implementation, we parameterize the auxiliary Q as a neural network named as introspective network and perform supervised learning with video-text pairs to make it converge to the true posterior distribution P(·|v). Assume we have trained Q to an approximation of P and then, as a result, since the entropy of ztext has nothing to do with the video generation, mutual-information introspection becomes the maximization of Ez′∼P (ztext),x∼G(z′ ,znorm)Q(z′ |x).

For simplicity, we assume the posterior distribution as a parameterized normal distribution. Then the problem becomes:

where z0 is the mean of the normal posterior distribution P (·|v = x) which depends on the video x.

From the above mathematical derivation, an intuitive interpretation for mutual-information introspection is to introspect the generation by reconstructing what it is made from, namely the process of video-to-text, and use the reconstruction error as the cost function.

In order to reduce the computation cost, the Q introspective network can share all the convolutional layers with the discriminators. So we just modify the two discriminators described above by adding a fully-connected layer respectively to output the reconstructed text latent code.

We find by experiments that preserving the conditional structure of the 3D discriminator like [Pan et al., 2017] can speed up the convergence of the model. Therefore, the integrated discriminator network is composed of the following two subnetworks:

• 3D-convolution-based subnet D1(v, ztext) : Rdl×dc×dh×dw × Rdtext → [0, 1] × Rdtext . From the perspective of the whole video, D1 distinguishes whether the input video looks like a real one and matches the given text, as well as introspectively outputs the reconstructed text latent code z1 0 . Of course, the input ztext will not participate in the reconstruction of the latent code.
• 2D-convolution-based subnet D2(v) : Rdl×dc×dh×dw × Rdtext → [0, 1] ×Rdtext . From the perspective of frame quality and motion, D2 just distinguishes whether the input video is real or fake and also introspectively outputs the reconstructed text latent code z2 0 .

In the following paper, we still use D1(v, ztext) and D2(v) to represent the true-or-false answer the discriminator network outputs, and we denote the introspectively reconstructed text latent codes from video v as Q1(v) and Q2(v) for convenience.

2.4 Optimization

According to the discussion above, the overall optimization problem of the discriminator network is as follows:

where LD1 , LD2 , LQ1 , LQ2 are defined as:

vreal and vsyn denote real videos and synthetic videos respectively while zmatched and zunmatched denote the text latent code that match and does not match the video.

Equations (6) and (7) are cost functions of typical GANs. Equations (8) and (9) are meant to learn the true posterior distribution P (·|v) by means of supervised learning, which is part of our mutual-information introspection.

On the other hand, the overall optimization problem of the generator network is as follows:

where LG1 , LG2 , Linfo are defined as:

Equations (11) and (12) are also cost functions of typical GANs. Equation (13) is the regularization that makes the generator synthesize videos from which the reconstructed text latent code can be as close to the ground-truth as possible, which means aligning the generated videos to the given text.

As discussed above, we can see that both the recurrent transconvolutional generator and the mutual-information introspection rely on an adequate distribution of the text latent code. So it is natural to perform a two-stage training process: first pre-training the text encoder and then training the whole model with the encoder fixed.

In the first stage, the text encoder is trained with a seq2seq auto-encoder [Dai and Le, 2015] and a "Q-style" introspective network which recovers the text latent code from videos. In order to better perform mutual-information introspection later, the cost function includes not only the term of the autoencoder reconstruction error, but also the mean square distance between the encoded ztext and the reconstructed text latent code from the corresponding video. In the second stage, only the text encoder in the previous training process is preserved. And we train the generator and discriminator network with the cost functions of (5) and (10).

3 Experiments and Results

3.1 Datasets

We have adopted 3 datasets of progressively increasing complexity: Single Moving Mnist-4, Double Moving Mnist-4 and KTH-4. They are constructed according to [Mittal et al., 2016]. Each video in these datasets has 16 frames and each frame has the size of 64 × 64.

• Single Moving Mnist-4: [Mittal et al., 2016] first constructed Single Moving Mnist dataset for the text-tovideo task. As there are merely two motions: up-down and left-right in the original dataset, we find it too simple to distinguish the effectiveness of the current methods. So we slightly improve the dataset by introducing four moving directions : move left then right, move right then left, move up then down and move down then up while the original dataset doesn't tell apart "up then down" and "down then up". Each video is accompanied with a single sentence describing the digit and its moving direction.
• DoubleMovingMnist-4: [Mittal et al., 2016] also constructed a more complicated version of Moving Mnist which contains two bouncing handwritten digits. Similarly, we extend the dataset by introducing four moving directions for each digit.
• KTH-4: Based on the original KTH dataset by [Laptev et al., 2004], [Mittal et al., 2016] first select some video clips and accompany each video clip with a descriptive caption. Since [Mittal et al., 2016] didn't make open their dataset or source codes to construct it, we construct our own KTH-4 according to the method mentioned in [Mittal et al., 2016]. The original KTH dataset contains 600 videos, classified into 6 actions and 25 people. We pick out 4 of the actions: walking, running, boxing, waving hands, and then extend them into 6 motions: walking from left to right, walking from right to left, running from left to right, running from right to left, waving hands and boxing.

3.2 ComparedMethods

We compare our IRC-GAN with the following state-of-theart methods: Sync-DRAW [Mittal et al., 2016], VGAN-c [Vondrick et al., 2016], TGANs-c [Pan et al., 2017] and MoCoGAN-c [Tulyakov et al., 2018]. Like [Pan et al., 2017], we also modify VGAN and MoCoGAN as VGAN-c and MoCoGAN-c to adapt to the text-to-video task.

3.3 EvaluationMetrics

In text-to-video generation, we need to evaluate both the visual quality and the semantic match. In addition to visually examining the results, we adopt two quantified evaluation metrics to evaluate the effectiveness.

Generative Adversarial Metric (GAM)

Generative Adversarial Metric [Im et al., 2016] can directly compare two generative adversarial models by having them engage in a "battle" against each other. Given two generative models M1 = ( ˜G1, ˜D1) and M2 = ( ˜G2, ˜D2), two kinds of ratios between the discriminators of the two models are calculated:

where ϵ(·) denotes the average classification error rate and xtest is the testing set. If rtest is close to 1, which means the two models have almost the same ability to recognize the real videos, the relationship between rsample and 1 can reveal which model can fool the other model more easily. For example, rtest ≈ 1 and rsample < 1 mean that G1 is a better generative model than G2.

Since this method is restricted for GANs, Sync-DRAW is excluded from this comparison. As for our IRC-GAN, our discriminator network additionally takes the mutual information between the generated videos and the text into consideration. So we add a term about the mutual information when calculating the classification error rate of our discriminator network. In order to be fair, when comparing our IRC-GAN with other methods, we adjust the weights of the mutual information term to make sure rtest is close to 1 so that we can compare the two models by checking rsample.

Human Evaluation

Additionally, we conduct a user study to evaluate both of the visual quality and semantic consistency. To compare the visual quality of our method and others, the testee will be shown 2 gifs, one randomly chosen from the generated videos of our method and the other from one of the compared methods. The testee is then required to point out the gif he thinks to have better quality. On the other hand, to compare the semantic match, we show the testee two gifs generated from the same text but from different methods and we require the testee to report which gif is more in line with the text. A total number of 20 evaluators are invited as testees. Each testee is given 30 inquiries on visual quality and 30 inquiries on semantic match. The testee can prefer either of the two given gifs or he can claim a draw, which means choosing the two at the same time. The percentage user preference shown in Tables 2 and 3 is the proportion of the chosen times of two methods.

3.4 Qualitative Analysis

Figure 3 shows some samples of our results on these datasets. VGAN-c doesn't seem to reach convergence and performs poorly. By traditional 2D transconvolutional layers, SyncDRAW can generate relatively clear frames but its temporal coherence remains to be improved: The shape of the digit varies across frames and its motion is not coherent. Although our IRC-GAN performs roughly the same as TGANsc and MoCoGAN-c on the simple Moving Mnist dataset, our IRC-GAN generates better results on the real-world KTH-4 dataset. The results demonstrate that our IRC-GAN can generate videos of both fine definition and coherence.

In Table 1, the GAM rsample scores are all less than 1, which means our IRC-GAN can generate videos that can fool the discriminators of other GAN-based methods. In other words, compared with the videos generated by other methods, those generated by ours are more consistent with the text and more similar to the real ones.

(a) Results on Single Moving Mnist-4

(b) Results on Double Moving Mnist-4

(c) Results on KTH-4

Table 2: User preference score on video generation quality

In Tables 2 and 3, we can see that more testees prefer the generated videos of our method both on visual quality and semantic match. The advantage of our model on visual quality suggests that our RTG is capable of synthesizing more coherent and more vivid videos by memorizing the previous generated frames. And exceeding in semantic match indicates that measuring the semantic distance between videos and text by MI can effectively improve the video-text consistency.

3.5 Ablation Analysis

Two ablation experiments are conducted to further show the effectiveness of the two components in our approach, namely MI and RTG. 1) RCGAN-c: Our approach without MI. 2) TGANs-info: RTG is replaced with the commonly-used generator of TGANs.

As shown in Table 4, these two baseline models are compared with our IRC-GAN via GAM metrics. The rsample scores are less than 1 on all of the 3 datasets, which means the videos generated by our IRC-GAN can fool these two baseline models. In other words, our model is better.

4 Conclusion

In this paper, we have proposed an Introspective Recurrent Convolutional GAN (IRC-GAN) to generate videos from text. Such task needs to consider both visual quality and semantic consistency. To improve visual quality, we propose a recurrent transconvolutional generator where LSTM cells are integrated with 2D transconvolutional layers. Such generator boosts both the definition of each video frame and temporal coherence across the whole video. On the other hand, semantic consistency is ensured by mutual-information introspection. In this way, the semantic distances between videos and text can be learnt, which helps to synthesize corresponding videos. Experiments on three datasets compared with several state-of-art methods verify the effectiveness of our method. In the future work, we will introduce cross-media techniques to further establish relations between text and video.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61771025.