A Structured Latent Variable Recurrent Network With Stochastic Attention For Generating Weibo Comments

1 Introduction

Generating realistic comments for social applications is attaching lots of attention [Shen et al., 2019; Holtzman et al., 2019; Zhang et al., 2018]. Compared with rulebased [Goddeau et al., 1996] and retrieval-based methods [Eric and Manning, 2017], generative sequence-tosequence (Seq2Seq) [Sutskever et al., 2014; Liu et al., 2018; Yang et al., 2019; Zha et al., 2019] models using encoderdecoder architectures have been widely used due to their super capacity for sequential data. As Figure 1a, traditional Seq2Seq models directly maximize the likelihood between input and output. They tend to generate "safe" and meaningless comments of high-frequency, and such comments lack I have no hobby

∗Corresponding Author

( |

step t ：

p y y x

t

t

, )

'watching'

theme space theme space (b) Our proposed SARN

Figure 1: Illustrations of comment generation process. (a) Traditional Seq2Seq models predict the t-th word yt via p(yt|y<t, x) given input x, where stochastic variation is parameterized by this conditional output distribution. (b) We introduce latent variables zd and zw with stochastic attention to capture the discourse-level and word-level stochastic variations. Our model estimates the distribution of p(yt|z≤t w , zd, y<t, x). Diverse comments are produced conditioned on different assignments of latent variables.

the diversity [Li et al., 2016]. Closely related to response generation, generating diverse and coherent comments is important for improving the user experience of intelligent agents.

Recently, some methods are proposed to improve comment diversity, which can be divided into two groups. The first one focuses on improving the word generation process, including designing new variations of generation procedures and finding new types of decoding objectives. [Mou et al., 2016] proposed a backward-forward decoding process which generated a reply start from diversified middle keyword. [Vijayakumar et al., 2016] and [Shao et al., 2017] proposed diversity-enforced beam-search procedures. Moreover, [Yao et al., 2016] incorporated Inverse Document Frequency into the decoder objective to promote the comment diversity. The

movies second group emphasizes that comments are diversified in terms of high-level properties or attributes, such as language style, topic, intention. These properties are explored as context information to generate more specific comments. [Xing et al., 2017] fed topic information into a joint attention module to generate topic-related comments. [Zhou et al., 2017b] added emotion context to build emotional chatting agents. Moreover, another series of works implicitly modeled the high-level attributes using stochastic latent variables [Zhao et al., 2017; Cao and Clark, 2017; Serban et al., 2017; Shen et al., 2017; Liu et al., 2019]. In such way, diverse comments are able to be sampled conditioned on the latent variables. However, most of above models treat comment generation as a single-level process, neglecting the hierarchy in natural comments.

In practice, the variabilities of comments arise from such the hierarchy, including both global discourse-level properties and word-level selections. Given one post, first, a number of comments can be generated conditioned on discourselevel diversity (i.e. different emotions or language styles). Then, for one specific discourse-level selection, various local word-trajectories are able to be generated conditioned on word-level diversity. How to formulate the above procedure remains an open question.

In this paper, we propose a structured latent variable Recurrent Network with Stochastic Attention (SARN), a probabilistic model that exploits both hierarchical-structured latent variables and the stochastic attention to promote multi-level diversity of comments. First, we introduce both discourselevel variable and word-level latent variables to build the hierarchy of comments. In detail, the discourse-level variable is used to capture the high-level properties such as style, topic and intention. Word-level variables are leveraged by the stochastic attention to infer the variations of the key-words of interest in the input post. Second, we endow these latent variables with hierarchical and temporal dependencies, which are specifically designed to model the complex structure of comments. As result, diverse comments are able to be produced conditioned on different assignments of these latent variables. The proposed SARN is trained using the Stochastic Gradient Variational Bayesian framework [Kingma and Welling, 2013] which maximizes the variational lower bound of the conditional log likelihood. Experiments on open-domain Weibo dataset showed that the proposed SARN yields significantly more diverse comments at both discourse-level and wordlevel compared to other methods.

2 Preliminary

Given an input x = (x1, ..., xT ) and a comment sequence y = (y1, ..., yT′), where ytis the t-th caption word, Seq2Seq model is trained to maximize the probability of p(y|x). A bidirectional recurrent network (B-RNN) first summarizes the input sequence x into hidden states ht.

Soft-attention. A decoder RNN calculates the context ct, and estimates the probability of y. In detail, the decoder RNN takes a context ct and the previously decoded word yt-1to update its state st as st = f(st-1, yt-1, ct), and ctis a weighted average of the encoder hidden states as,

where g is inner-product function and αti can be viewed as the similarity score between encoder's state ht and decoder's state st-1. Finally, the probability distribution of candidate words at time-step t is calculated as,

The final objective is to maximize the (log) likelihood of p(y|x) for all timesteps as p(y|x) = ∏T ′ t=1 p(yt|y<t, x).

This Seq2Seq model lacks the parametrization of stochastic variations of generation process, where only source of stochastic variation is provided by conditional distribution of p(yt|y<t, x)(the softmax layer). This model generates highfrequency comments with low-diversity [Li et al., 2016].

3 Methodology

To improve comment diversity, we reveal inherent multi-level hierarchy in comment generation, and propose a probabilistic model that exploits latent variables and a stochastic attention.

Hierarchical structure. As Figure 2, first, a decision-level latent variable zd is used to characterize the choice of highlevel properties, such as topic, intention. Second, a series of word-level latent variables zw = {ztw}T ′ t=1 are introduced to characterize the variations of word-level selections, generate diverse comments by focusing on different input key-words.

To induce a two-level hierarchy, we endow the latent variables zd and zw with structured dependencies by defining,

i) discourse-level stochastic variations are modeled by the prior distribution of p(zd|x). ii) word-level stochastic variations are modeled by the distribution of p(ztw |z<t w , y<t, zd). Specifically, each word-level variable ztw is not only conditioned on x and zd, but also conditioned on the previous latent trajectories of z<t w . These structured dependencies promote the model capacity for capturing multi-level diversities. Rather than directly maximizing p(y|x) , we define the joint probability distribution p(y, zw, zd|x) and maximize p(y|x) by marginalizing out all the latent variables as,

Stochastic attention. One key hypothesis is that diverse comments can be generated by focusing on different keywords of input. Different from deterministic Soft-attention (Eq. 1), we explicitly inject stochastic variations in the context vector ct, by leveraging the word-level variables ztw as,

where the variations of the focused input-words are characterized by the latent distribution of zw. Stochastic attention helps better capture the 1-to-n mappings. For one input, more diverse input-words can be focused to generate informative and meaningful predictions.

Generation. As Figure 2, given an input x, a comment sentence y is generated as follows.

1) Calculate the encoding states {ht}Tt=1using a B-RNN encoder (see preliminary).

2) Sample a discourse-level variable zd from a multivariate Gaussian distribution p(zd|x)as,

The mean µd and variance σ2d are produced by a MLP network, which takes the last hidden states hT as input as,

1. Iteratively run step 4), 5), 6) for each step t = 1, 2, ..., T ′ .

4) Update the hidden state st of the decoder RNN as,

where f is a LSTM unit, y0 is the '<start>' token, and z0w and s0 are initialized as z0w = zd and s0 = MLPs(zd).

5) Stochastic-attention: Sample the t-th variable ztw from another Gaussian distribution p(ztw|z<t w , y<t, zd)with mean µt and diagonal covariance diag(σ2t )as

where the mean and variance are also produced by a MLP network with input st as [µt ,σt] = MLPw(st). Then, compute the context vector ct according to Eq. 3.

6) Sample a word yt with probability p(yt|y<t, z≤t w , zd, x) from vocabulary set V as,

Inference. Directly optimizing objective of Eq. 2 involves a marginalization over the latent variables of zd and zw, which introduces an intractable inference of the posterior p(zd, zw|y, x). Exploiting Stochastic Gradient Variational Bayes (SGVB) [Kingma and Welling, 2013], we introduce an auxiliary distribution q(zd, zw|y, x) to approximate the true posterior as,

where q(zd|y, x)and q(ztw|z<t w , y≤t, zd)are multivariate Gaussian distributions with diagonal covariance, parameterized by MLP networks as,

As shown in Figure 3, hyT is the last hidden state of the BRNN encoder with input y. Via variational inference, we derive the objective function which is the lower bound of the conditional log likelihood of Eq. 2 as,

where KL is the Kullback-Leibler divergence, regularizing the approximated posteriors to be close to the priors.

Training. In Figure 3, the model is trained by maximizing the variational lower bound L of Eq. 11. In practice, the expectation term can be approximated using M Monte Carlo samples {ˆz(m) d ,ˆz(m) w }Mm=1 sampled from q(zd, zw|y, x). The reparametrization trick [Kingma and Welling, 2013] is adopted to reduce the variance of the approximation as,

where ε(m) is a vector of standard Gaussian samples, and ⊙ is dot production. In practice, we set M = 1 and maximize the approximated variational lower bound ˆ L as,

Analogous to existing latent variable models [Bowman et al., 2015; Zhao et al., 2017], our model suffers from the vanishing latent variable problem. During training, the KL terms

…

Figure 3: The network architecture for training. Some of the MLP subnetworks are omitted for abbreviation.

• Discourse-level latent variable d z

nothing working <end> but You like

attention T t t=1 {h } … Stochastic attention t c t s MLP Word-level latent variable t w z in Eq. 13 are easily crushed to zero, which means that latent variables fail to capture meaningful information. Since simply applying the KL-annealing [Bowman et al., 2015] is unsatisfactory, we introduce extra strategies for zd and zw. build 1 million post-comments pairs, and each post is coupled with 15 different comments. The dataset is randomly split into training, validation and testing sets, with 980k, 10k and 10k samples respectively.

x T h d MLP z d  t z Prior estimate Discourse-level Word-level t+1 z LSTM LSTM LSTM LSTM 1) For zd, the bag-of-word method [Zhao et al., 2017] is adopted. An auxiliary objective log p(ybow|zd) is used to force zd to capture global bag-of-words information ybow as,

d

where pyb is the probability vector over vocabulary V . 2) For zw, we propose an Auxiliary-path method with objective log ˜ p(yt|ztw ), which forces the model to make good predictions only conditioned on the latent variables zw. The auxiliary probability vector over the vocabulary for ˜ p(yt|ztw ) can be calculated by an auxiliary network MLPaux as,

The final objective function is rewritten as in Eq. 16, where α and β are hyper-parameters to balance two auxiliary terms.

There are two differences between SARN and existing methods [Park et al., 2018; Serban et al., 2017; Shen et al., 2017] which also exploit hierarchical structure. (1) they model different hierarchies, i.e., sentence-word hierarchy for SARN and dialog-sentence hierarchy for existing methods. (2) we leverage stochastic attention with hierarchical structure, which gives more capacity to model the complex variations in word selection.

4 Experiments

Data collection. We collected training corpus from Sina Weibo including Chinese post-comment data. To construct diverse training set, we compute scores of Tf-Idf cosine distance between a given post and other posts in the corpus, and then we select the nearest neighbor comments from the top15 similar posts. We empirically find that these selected comments are coherent enough with the selected post. Finally, we d 0 s <start> nothing but working w s MLP t s w LSTM LSTM LSTM LSTM … Compared methods. 1) Seq2Seq [Sutskever et al., 2014] Basic encoder-decoder model. 2) Seq2Seq-MMI [Li et al., 2016] Seq2Seq model with Maximum Mutual Information objective function. We used the MMI-bidi version with parameters λ = 0.5, γ = 1. 3) VRNN [Chung et al., 2015] Latent variable Seq2Seq model using time-step dependency. 4) CVRNN [Bowman et al., 2015] Latent variable Seq2Seq model using global dependency. 5) SpaceFusion [Gao et al., 2019] Seq2Seq model using latent space fusion 1 .

Settings. We adopted the Jieba Chinese tokenizer [Jieba, 2018], and constructed a vocabulary with 40k. For compared models, we adopted a LSTM for encoding and decoding. The dimension of the word embedding, attention state, latent variables, and LSTM hidden state were respectively set to be 100, 200, 500 and 500. The network parameters were initialized from normal distribution N (0, 0.01). We used Adam as optimizer with a fixed learning rate of 0.0001, and the batch size was set to 20. Gradient clipping was adopted with a threshold of 10. We selected parameters with the best validation performance. For the vanishing latent variable problem, we adopted the KL-annealing for models of VRNN, CVRNN and SARN, by linearly increasing training weight of KL-term from 0 to 1 after running 60,000 batches. We also adopted bag-of-word method for CVRNN, and the proposed auxiliary-path method for VRNN. The source code is released 2 .

Evaluation methods. Given one post, we sample 5 comments for each model for evaluation. We adopt beam search in decoding process for Seq2Seq and Seq2Seq-MMI, where beam size was set to 10. For VRNN, CVRNN and SARN, we randomly sample 5 comments using greedy decoding (beam size is 1), where the randomness comes from the latent variables [Bowman et al., 2015]. We apply three evaluation metrics, including human judgments, embedding-based evaluation, and n-gram based evaluation (BLEU-4). As [Zhou et al., 2017a], human judgments is performed via crowd-sourcing.

1 https://github.com/golsun/SpaceFusion

2 https://github.com/ysjakking/weibo-comments

We conduct blind evaluation, and outputs are randomly presented to 10 evaluators who have experiences of NLP. For each test input, every model generates 5 responses as a group. For each response the evaluators are asked to score its quality with Bad (NOT grammatically correct or relevant), Normal (grammatically correct and relevant to the input) and Good (beyond grammatically correct and relevant, the response is interesting and meaningful). Further, we define a comment is 'Acceptable' if it is scored Normal or Good. Besides, we conduct pairwise comparisons, where a better comment is chosen from two compared models. Numbers of Wins, Losses and Ties are reported. Compared with n-gram based metrics, embedding-based evaluation focuses more on measuring the semantic similarity. As [Serban et al., 2017], three metrics are adopted, including Average, Extrema and Greedy.

4.1 Performance Analysis

For human judgements, we evaluate the consistency of agreement using Fleiss' Kappa criterion, which are 0.34 (Fair agreement) for Table 1 and 0.30 (Fair agreement) for Table 2 respectively. We observe that, first, SARN outperforms other models with respect to both the Diversity value and Acceptable ratio. Compared with models only improving single-level diversity, such as the word-level VRNN and the discourse-level CVRNN, Seq2Seq-MMI and SpaceFusion, our method achieved higher Diversity value and Acceptable ratio. The improvements shows the effectiveness of leveraging the multi-level diversity. Second, the statistics of Bad, Normal and Good show that our model tends to generate more Good comments than Normal or Bad comments. It reveals that promoting the multi-level diversity also helps to improve the comment quality, generating more meaningful and interesting comments. Third, our method does not achieve superior BLEU-4 score. The reason is that BLEU-4 calculates the n-gram overlapping with ground-truth sentences, and such measurement is not positively related to the comment diversity. Because of such limitations of n-gram based measurement, we provide embedding-based evaluations. According to the Average, Extrema and Greedy scores in Table 1, we see SARN generates more coherent and informative comments in terms of high-level semantics. Finally, the pairwise comparison results in Table 2 show that the comments of our SARN are preferred in the majority of the experiments, and our method achieves comparable performance with SpaceFusion. It further validates that our SARN produces more diverse comments with better quality.

Figure 4 shows comment examples. We can see that SARN generates more informative comments with high diversity while other methods tend to produce phrases with similar meanings. As Eq. 3, comments are generated based on the understanding of key-words of interest. Color histogram in Figure 4 denotes the accumulated attention weights on the post word, where we find (1) SARN can focus on different post words so that it generates diverse comments; (2) Seq2Seq focuses on post words uniformly so that their comments are very similar. This is because that Seq2Seq uses deterministic soft-attention while SARN uses stochastic attention.

4.2 Parameter Analysis

To tackle the vanishing latent variable problem, we respectively add bag-of-word and auxiliary-path objectives for zd and zw in Eq. 16, introducing hyper-parameters α and β to balance the auxiliary terms. As shown in Table 3 and 4, we find that, first, zd and zw have great influence on Diversity degree and Acceptable rate. zd is to characterize the global properties of sentencesand and zw mainly controls the wordchoice. Second, both of KL cost and Diversity are gradually increased with the increment of α and β. This indicates that the vanishing latent variable problem is alleviated. Third, Acceptable rate is first improved with the increase of α and β.

Post

每ཙ上班来回真心累 I am really tired from going and returning from work every day

Comment1 每ཙ都很累

Tired every day

Comment2 我也是很累

I am also very tired

Comment3 我也很累啊

I am also very tired, ah~

Post

我也是每ཙ上班的

I go to work every day too

每ཙ都很累

Tired every day

我也是好累

I am also really tired

每ཙ都会累

I am tired every day

工作是多么的累啊

How tired is the work

我也要上班了

I have to go to work too

我每ཙ都在上班

I am working every day

我也累啊

I am also very tired

累了就休息一下

Take a little break

when you are tired

ཙ气这么好做点什么好呢 What to do with the good weather

Figure 4: Comment examples. The color histograms represent the accumulated attention weights on the post word of the same color.

I want to do something

Comment1 我也很想干点啥呢

今ཙཙ气很好

The weather today is good

Acceptable=61.7

那就做点晒ཚ阳

Then do some sun exposure

对啊希望有好ཙ气

Yes, I hope there is good weather

晚上是会热的吧

It's going to be hot at night

世界上就没有那么多废话

There are not so many

nonsense in the world

As long as there is a way

There is always a start

when you walk

Then the performance degradation is showed when α ≥ 0.3 and β ≥ 0.05 respectively. The main reason is that too large values of α and β overemphasize the auxiliary objectives, which gives the model much freedom to generate ungrammatical sentences. These results show that we can control the trade-off between improving diversity and maintaining grammatical sentence by using different values of α and β.

4.3 Multi-level Diversity Analysis

Parameter analysis shows that the values of α and β are proportional to the 'Diversity' degree. We further investigated how these two parameters influence the results. First, we tested different α by fixing β = 0.01 as shown in Figure 5. We can see that with the increase of α, the generated comments contain more different meanings, which indicates that the latent variable zd successfully catches discourse-level variations. Second, we tested different β by fixing α = 0.3. We found that the 'Diversity' degree of local words is more sensitive to β than α. As shown in Figure 6, our model begins to generate ungrammatical sentences consisting of diverse key words using large values of β > 0.1. This observation indicates that latent variables zw capture the word-level variations, and large value of β makes zw dominate the worddecisions, neglecting the long-term sequence dependency of LSTM hidden units. The above observations confirm our assumption that the two kinds of latent variables respectively catch discourse-level and word-level variations.

4.4 Large β And Search-basedModel

Interestingly, we found the ungrammatical sentences caused by large β (β > 0.1) usually contain highly diverse but coherent key words with respect to the post (see bolded words

ཙ气不错心情不错

Good weather, good mood

上班的人伤不起

People who work can't

afford to hurt

累并快乐着

Tired and happy

累了就休息

Take a break

when you are tired

心情好请我们吃饭

Good mood, you

should treat us a meal

in the middle column of Figure 6). Based on this observation, we built an effective search-based model based on these generated key words. In detail, we simply conducted Tf-Idf nearest search and selected the most similar comments from the training set. As shown in the right column of Figure 6, the obtained comments are highly coherent with the input post, and the final Diversity score and Acceptable ratio are further improved. Therefore, this search-based model provides us an effective way to combine generative and search-based generative model for practical industry use.

5 Conclusion

As a probabilistic model, the proposed SARN exploits both hierarchical-structured latent variables and the stochastic attention to promote multi-level diversity of comments. SARN is highly related to encoder-decoder models, while the main difference is that we inject multi-level stochastic variations in the generation process with both hierarchical and temporal dependencies. Experiments show that our model generates more diverse and realistic comments than other methods.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grand: 2018AAA0102003, in part by National Natural Science Foundation of China: 61771457, 61732007, 61672497, 61772494, 61931008, U1636214, 61622211, 61702491, and in part by Key Research Program of Frontier Sciences, CAS: QYZDJ-SSW-SYS013.