Knowledge-Aware Dialogue Generation via Hierarchical Infobox Accessing and Infobox-Dialogue Interaction Graph Network

1 Introduction

Open-domain dialogue systems aim to generate human-like responses; however, the inadequate knowledge carried by the queries dramatically constrains the ability of dialogue systems to understand the intrinsic semantics of the queries and generate user-friendly dialogues [Ghazvininejad et al., 2018]. The frequently generated boring responses (e.g., " I don't know") often lead to frustrating user experience [Li et al., 2016]. Recently, researchers have recognized that introducing external knowledge to dialogue generation systems has great potential for further improving performance. Knowledge is regarded as the awareness and understanding of the dialogue context [Yu et al., 2020]; namely, it can effectively assist the dialogue systems in understanding the intrinsic semantics and fabricating informative responses.

∗Corresponding author.

Dialogue systems can gather knowledge from various information sources, which can be broadly classified into (1) Knowledge texts, which can provide rich semantic information, are easy to access on the Internet, such as online encyclopedia (e.g., Wikipedia and Answers.com), news websites, and search engines [Tam, 2020]; (2) Knowledge bases, such as knowledge graphs [Zhang et al., 2020], and spread-sheet tables [Qin et al., 2019]; their knowledge items are wellorganized, and thus can be easily integrated into dialogue systems; and (3) Topics and keywords, which can promote the dialogue towards a specific and consistent direction [Wang et al., 2018; Wu et al., 2020b]. However, knowledge texts are unstructured, knowledge bases are hard to construct/collect, and topic words/keywords are not informative. Therefore, there is a question, can we utilize a new type of knowledge that is easy-collected, informative, structured, and consistent for further enhancing dialogue systems?

Recently, a new kind of knowledge, infobox tables, has attracted much attention in the data-to-text tasks [Bao et al., 2018; Chen et al., 2020]. Figure 1 provides an illustrating example of an infobox table that specifies an entity with multiple attribute key-values. Infobox knowledge integrates the advantages of the above three types of knowledge. First, massive infobox tables can be easily obtained from the Internet, such as Wikipedia articles or web pages of Google search results. Second, infobox tables are easy to use since infobox knowledge is well extracted and organized as informative attributes. Third, an infobox table focuses on one target entity without the interference of irrelevant entities, guaranteeing knowledge consistency. The infobox knowledge offers a great opportunity to employ one kind of reliable knowledge source to generate high-quality dialogue systems. Integrating the above three types of knowledge one by one often results in knowledge inconsistency and computational inefficiency.

To our best knowledge, using the infobox knowledge to improve the quality of open-domain dialogue generation has not been fully investigated yet. Compared to the usage of the infobox tables in the data-to-text tasks, conducting the infobox knowledge-aware dialogue generation is much more difficult to study. Unlike the data-to-text tasks that rephrase infobox attributes in an orderly fashion, dialogue generation tasks conduct attribute order-irrelevance knowledge selection, inference, and dialogue-infobox fusion. In addition, existing data-to-text studies usually only consider the infobox tables within limited domains. However, open-domain dialogue generation has to use infobox tables in a wide range of domains for maintaining enough knowledge coverage, which dramatically increases the difficulty of unambiguous knowledge representation. These challenges indicate that infobox knowledge-enhanced dialogue generation cannot directly utilize the techniques used in the existing data-to-text works.

With these challenges in mind, we propose HITA-Graph, a novel infobox knowledge-aware dialogue generation approach. (1) We propose an order-irrelevance Hierarchical Infobox Table Encoder (HITE) to represent an infobox table at three levels of granularity. To alleviate the sparsity of the word distribution of the open-domain infobox attributes, besides the usual word embedding, HITE further integrates the embedding learned from the char sequence of a word, the part-of-speech tag embedding, and the locally positional embedding into an intra-attribute level representation of infobox table. Subsequently, a multi-head attention network is adopted to compute an attribute-level representation. In the context-level, we propose an Infobox-Dialogue Interaction Graph Network (IDCI-Graph) to capture both the infobox and the dialogue context by building and inferring on an infobox-dialogue interaction graph. (2) We develop a Hierarchical Infobox Attribute Attention (HIAA), which allows the Decoder to access the encoded infobox knowledge in a coarse-to-fine manner. (3) When predicting the next target token, the Decoder is equipped with three modes: generating a vocabulary word, copying a word from the query, and copying an attribute word from the infobox. Such three modes are fused via the Dynamic Mode Fusion strategy (DMF).

We collected about 895K Chinese infobox tables from Wikipedia. Extensive experiments are conducted on a Chinese corpus [Cai et al., 2019b]. Both the automatic and human evaluation results demonstrate that the proposed approach HITA-Graph outperforms the representative competitors in almost all experiments. Our contribution is three-fold: (1) To our best knowledge, this work is the first to study the infobox knowledge-aware dialogue generation; (2) A novel Hierarchical Infobox Table Encoder and a novel InfoboxDialogue Interaction Graph Network are designed to integrate infobox knowledge. (3) AHierarchical Infobox Attribute Attention mechanism is proposed to access the infobox knowledge at different levels of granularity.

2 Our Approach

2.1 ProblemFormulation

Let D = {〈X, Y, T〉} denote the corpus, where X = (x1, · · · , xn) is a query, Y = (y1, · · · , ym) is a response, and T = {〈f k , f v 〉}l is an infobox table that consists of a set of attribute key-values and can be retrieved from the infobox base T . The goal of our task is: Y ∗ = argmaxY ′ P(Y ′ |X, T).

2.2 Context Encoder

A query X is firstly encoded into dialogue context states H = (h1, ..., hn) by a bi-directional GRU [Cho et al., 2014]:

where x is the embedding of x, and ht is the concatenation [hft ; hbt ]. hn is regarded as the context summary.

2.3 Hierarchical Infobox Table Encoder

For each attribute f kv i = 〈f k i , f v i 〉 ∈ T, its key f k i is an noun word, its value f v i = (wi,1, · · · ,wi,|f v i |) is a word sequence.

Intra-Attribute Encoding

For an attribute key f k i or a value wordwi,j, its word distribution is sparse; namely, there are rare words. If only represent them at the word-level, many words can not be recognized. Thus, we design a hybrid-level method. Taking f k i as an example, the corresponding embedding f k,hybrid i is given by:

where f k i is obtained by looking up the word-level embedding matrix, f k,char i is computed from the char sequence of f k i with the CharCNN Encoder [Kim et al., 2016], and MLPhf is a MLP network. The incorporation of chars can alleviate the issue of the sparse word distribution. Subsequently, each attribute f kv i is represented as a set of key-word embedding:

where f kw i,j is the j-th key-word pair of f kv i , wpos i,j is the partof-speech tag of wi,j, and pfi,j and pbi,j are local positions counted from the beginning (i.e., j) and the end (i.e., |f v i | - j + 1), respectively.

Then, the multi-head self-attention (denoted as MHA, [Vaswani et al., 2017] ) is used to compute the intra-attribute level representations Fkw,ia i = {f kw,ia i,j }:

Attribute-Level Encoding

For each key-value attribute f kv i , its attribute-level embedding f kv,a i is the weighted sum of Fkw,ia i = {f kw,ia i,j }:

where fs is a learn-able parameter, serving as a special query to compute the weight; thus, the infobox table T can be represented as a set of attribute embedding Fkv,a = {f kv,a i }.

知识的一致性较差，并且在各方面的时间开销都较大。

• 本文借用近年来在Infobox-to-Text任务中被广泛研究的Infobox作为知识源，其同时具备上述三种知识的特点。在生成式对话领域，尚无利用Info 作为知识源的工作。同时，本文与Infobox-to-Text工作中的方法也有较大差异，对话生成利用Infobox的模式显著不一样（不是按顺序访问，要 话上下文集成）。 Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)

Infobox-Dialogue Interaction Graph Network

An attribute-level representation f kv,a i interacts with neither the dialogue context nor other attributes (i.e., the infobox context). Inspired by the success of GATs in knowledgeenhanced text generation [Zhang et al., 2020], we design an Infobox-Dialogue Interaction Graph Network to capture the interaction information among attributes and the dialogue context. For each infobox-dialogue instance, we first construct a graph G = 〈V, E,R〉. The node set is V = {vkv }∪{vg }∪{vq }, where attribute nodes {vkv } correspond to the attributes {f kv } of T , vg is a virtual node serving as the global node, and vq is another virtual node representing the dialogue context. Meanwhile, E = {(vi, ri-→j, vj)} is edge set, where ri-→j ∈ R is the relational edge relation. For promoting the knowledge flow, G is fully-connected and directional in this paper. For each two nodes vi, vj, their directional edge ri-→j depends on the node types of vi, vj. Therefore, the relation set R consists of 3 self-connection types {selfvkv , selfvg , selfvq}, 3 types starting from an attribute node {vkv a -→ vkv b̸=a , vkv -→ vg , vkv -→ vq }, 2 types starting from the global node {vg -→ vkv , vg -→ vq }, and 2 types starting from the dialogue context node {vq -→ vg , vq -→ vkv }. Our graph network can stack multi-layers, assume vlt,i is the representation of vi at the layer lt:

where each W is a learn-able parameter, {vkv 0,i } are initialized by {f kv,a i }, vq 0 is initialized by hn, vg 0 is regarded as a learnable parameter. For simplicity, we use vlast,i to denote the representations outputted by the last layer.

2.4 Response Generation

Decoder State Initialization and Updating

Decoder is a GRU-based, whose initial state is given by z0 = Wbrg[vg last ; hn], and the following state zt is updated as:

where cht is the context attention,cft is the infobox attention, and yt-1 is the embedding of the last predicted token.

Hierarchical Infobox Attribute Attention

We enable the Decoder to access the infobox knowledge in a coarse-to-fine manner. The hierarchical infobox attribute attention cft is dynamically computed at each time step:

The coarse-grained αkv t-1,i is of attribute level, which measures the relevance between the {vkv last } and zt-1:

The fine-grained αkw t-1,i,j is of intra-attribute level; it considers the weight of each key-word pair in the attribute :

Dynamic Mode Fusion

To minimize the impact of unknown words and diversify the generated responses, three modes are computed and fused when predicting the next token at time step t:

Vocab mode. The next token can be a word in the predefined vocab V , the probability distribution is given by:

Copy mode. The Decoder can copy a word from the query X. To obtain a more accurate copy probability distribution, we reuse the parameter Wh of the query attention Eq. 9.

Infobox mode. The Decoder can also select a relevant keyword pair, and extracts its word as the output. Here, we apply the same hierarchical technique as Eq. 10:

Mode fusion. subsequently, we fuse the above three distributions into one distribution Pt with a MLP network:

Training

The training follows the maximum likelihood estimation process, which minimizes the negative log likelihood:

where I(·) is an indicator function to alleviate the unknown words issue in the dialogue generation, it equals 0/1 when the target token yt is an unknown/known word, respectively.

3 Experiments

3.1 Dataset

We use a previous open-released Chinese corpus [Cai et al., 2019b], and we have crawled about 895k infobox tables from Wikipedia. We use TF-IDF to rank the words of the query. According to the ranked order, we iteratively pick up a query word as the key to retrieve an infobox based on the pre-learned inverted index until a valid infobox has been found. If there is no matched infobox, a special blank infobox is adopted. Finally, the dataset is divided into train/validation/test sets where the size is 855K/30K/30K. https://github.com/pku-sixing/IJCAI2021-HITA-Graph.

3.2 Settings

Models. We first select 4 conversational models: Seq2Seq: The attentive Seq2Seq [Luong et al., 2015]. Pointer-Gen: Based on the Seq2Seq, it allows the decoder to copy a word from the query [See et al., 2017]. CCM: A commonsense knowledge-aware model that adopts graph attention [Zhou et al., 2018]; ConKADI: One of the latest SOTA commonsense knowledge-aware generation models, which proposes multiple methods to better select the knowledge [Wu et al., 2020a]. Then, we select 3 infobox-to-text baselines. LSTMGate: It proposes an LSTM-based Infobox Encoder [Liu et al., 2018b], HiLSTM: It further proposes a Hierarchical LSTM Infobox Encoder [Liu et al., 2019b]. Trans: The latest Transformer-based Infobox Encoder [Bai et al., 2020]. Such three baselines similarly use the Dual Attention to access the infobox knowledge. To adapt to the dialogue generation, we integrate their infobox encoder and infobox attention modules into the Seq2Seq. For better comparison, we further integrate our proposed Dynamic Mode Fusion into such three infoboxto-text baselines, the variants are denoted as 'X+DMF'.

Implementations. For CCM and ConKADI, we use their official codes and the commonsense knowledge released by ConKADI. For the others, we use our PyTorch implementations. Hyper-parameters are kept the same among models as possible. The word/char embedding dimension is 200; the part-of-speech tag embedding dimension is 10; the positional embedding dimension is 5; GRU's hidden size is 512. In our HITE, the multi-head attention has 4 heads and 2 layers; the interaction graph has 2 layers. Adam is used to optimizing parameters; the batch size is set to 50. The initial learning rate lr is 0.0001; after each epoch, lr will be halved if the perplexity on the validation set starts to increase. The training will be stopped if the perplexity on the validation set increases in two successive epochs. In the inference, beam search (k = 10 is adopted). The training of HITA-Graph consumes about 1 day on an Nvidia Titan-RTX GPU.

Metrics. For measuring the relevance to the ground-truth [Liu et al., 2016], we employ 3 embedding-based metrics, Embedding-Average (EmA), Embedding-Greedy (EmG), Embedding-Extrema (EmX), and 5 overlapping-based metrics, ROUGE-L, BLEU-1/2/3/4. For measuring the diversity and the informativeness, we report the ratio of distinct 1/2grams (DIST1/2) in generated words [Li et al., 2016], and the 4-gram entropy (Ent4) [Zhang et al., 2020].

3.3 Experimental Results and Analyses

Automatic evaluation. As shown in Table 1. HITA-Graph achieves the best overall performance; HITA-Graph wins first place in 9 metrics, and is comparable to first place in the remaining 2 metrics. Compared with the naive baseline Seq2Seq, knowledge-enhanced baselines have improvements more or less, indicating the necessity of incorporating knowledge. The improvements of three infobox-to-text baselines (i.e., LSTMGate, HiLSTM, Trans) are not notable. Unlike CCM, ConKADI, and our HITA-Graph, we find the reason is the inability to copy words from the query/infobox, so we subsequently equip our Dynamic Mode Fusion to such three baselines (i.e., '+DMF'). The enhanced variants achieve more notable improvements, especially in the aspect of diversity/informativeness. It shows the importance to design more generation modes for the Decoder. However, the enhanced variants are still behind our HITA-Graph, because our infobox-accessing solution is more suitable in the context of dialogue generation. The proposed HITA-Graph also outperforms two knowledge-enhanced conversational baselines, CCM and ConKADI, demonstrating: (1) Infobox can be a knowledge source in dialogue generation; (2) The proposed HITA-Graph is effective.

Human evaluation. Three volunteers are invited to annotate 200 sampled cases (1,200 pairs in total). The judgment is pair-wise, and follows two criteria: (1) Rationality: measuring the fluency and the relevance); (2) Informativeness: checking how much relevant knowledge is provided. As reported in Table 2, HITA-Graph outperforms all baselines. Compared to the commonsense-based ConKADI, HITA-Graph has a notable advantage in terms of rationality, and a slightly better performance in terms of informativeness. The agreement among annotators is highly consistent: (1) for the rationality, 94%/62% cases are given the same label by at least 2/3 volunteers; (2) for the informativeness, 94%/58% cases are given the same label by at least 2/3 volunteers.

Ablation study. As reported in Table 3, we designed and evaluated 5 HITA-Graph variants: (1) In case #1 and #2, we remove two methods that can alleviate the issue of unknown words that appears in the infobox and the dialogues, respectively. The decreased performance indicates the importance of handling unknown words in HITA-Graph explicitly; (2) In the next case #3-5, we evaluate the performance of HITAGraph if it incorporates less or no infobox knowledge. DMF brings more notable improvement than HIAA, especially in the aspect of diversity (i.e., DIST2). We find HIAA mainly improves the word overlap-based relevance (BLEU3) if we compare #3 and #0, or #5 with #4.

Attribute-order sensitivity study. Previous infobox-totext works show the order of infobox attributes can be a crucial factor to the performance [Bai et al., 2020]. However, unlike them, dialogue generation does not access the infobox in a specific order. Therefore, in dialogue generation, the infobox encoder should not be affected by order of attributes, and we subsequently propose an order-irrelevance infobox encoder, HITE. In this study, we demonstrate this feature by checking the performance (perf.) on the shuffled test set, which has the same attributes as the original test set, but with attribute orders randomly shuffled. As reported in Figure 3: (1) 'LSTMGate' and 'HiLSTM' employ the sequential LSTMs to encode attributes, and thus their performance

Query: 最近发现林宥嘉的'突 然想起你'。

Recently, I discovered Yoga Lin's 'Suddenly Thinking of You'.

Seq2Seq: 我也喜欢 林宥嘉。 I like Yoga Lin, too.

然想起我了。

Pointer-Gen: 我也是 突 然想起了 我也是 突 I also suddenly think of me, I also suddenly think of me.

Trans: 我也喜欢 林宥嘉。I like Yoga Lin, too.

Trans+DMF: 我也喜欢

林宥嘉。 I like Yoga Lin, too.

CCM: 我也想看我的偶像。 I also want to see my idol.

我 最喜欢的歌了。

ConKADI: 林宥嘉是 Yoga Lin is my favorite song.

HITA-Graph: 我也很喜欢

林宥嘉的歌 。

I also like Yoga Lin's songs.

Related Attributes:

(表: 林宥嘉)(Infobox: Yoga Lin), (职业: 歌手)(Job: Singer), (演奏乐器: 歌唱)(Musical Instruments: Singing)...

Table 4: Case Study. '林宥嘉' ('Yoga Lin') is a pop singer, who has sang a song '突 然想起你' ('Suddenly Thinking of You').

can be seriously impacted by the order. They only have 8387% of performance on the shuffled test set compared with the normal test set. (2) Our approach would not be impacted by order of attributes, and has a notable leadership compared with baselines; (3) The transformer-based 'Trans' is similarly irrelevant to the order because its attribute re-ordering module is inactivated in the dialogue [Bai et al., 2020]. However, the performance is worse than ours. In summary, this study proves our approach is much robust and powerful than baselines, demonstrating our approach can achieve better performance on the more complex real scenarios.

Case study. We report a real case in Figure 4. In the query, the user says he/she found a song, 'Suddenly Thinking of You ', performed by the singer 'Yoga Lin'. Except for the ConKADI and our HITA-Graph, other approaches did not realize 'Yoga Lin' is a singer. Although ConKADI realized this, it failed to generate a fluent response. This case indicates the proposed approach HITA-Graph is able to generate a high-quality and informative response with the incorporation of the infobox knowledge.

4 RelatedWork

Knowledge-aware dialogue generation. Suffering from outputting dull responses (such as 'I don't know') [Li et al., 2016], many efforts have been made to diversify the generations; for example, new objective [Li et al., 2016], latent variable [Gao et al., 2019], back-translation [Su et al., 2020], etc. Unlike human beings who can enhance their dialogue understanding and inference with various learned knowledge, machines only receive a query with limited knowledge [Ghazvininejad et al., 2018; Yu et al., 2020]. Therefore, the generation quality is always far from satisfactory. To bridge the gap of accessing knowledge, various types of knowledge are explored: 1) Knowledge texts, such as encyclopedia texts [Dinan et al., 2019], documents [Meng et al.,

2019], prototype dialogues [Cai et al., 2019a], and web pages [Tam, 2020]; 2) Structured knowledge bases, such as commonsense knowledge [Liu et al., 2018a; Wu et al., 2020a; Zhang et al., 2020] and spread-sheet tables [Qin et al., 2019]; 3) Topic words and keywords [Wang et al., 2018], which are also regarded as knowledge guidance knowledge. Different from such approaches, the proposed HITA-Graph uses the infobox knowledge, which is seldom used in the context of open-domain dialogue response generation.

Data-to-text. Infobox-to-text has been well studied [Chen et al., 2019; Chen et al., 2020]. The basic paradigm is similar to the Seq2Seq-based dialogue generation: An infobox table is first encoded into hidden states, and then a decoder generates a text by attentively accessing the hidden states [Nema et al., 2018; Liu et al., 2019a]. The representative approach LSTMGate [Liu et al., 2018b] proposes an LSTM-based encoder to encode an infobox table, and a dual-attention mechanism to access the infobox knowledge during the decoding. Subsequently, to better encode an infobox, pretrained language models (such as GPT-2 [Chen et al., 2020]), hierarchical LSTM encoders [Liu et al., 2019b], and the transformerbased encoder [Bai et al., 2020]) are successively proposed. Compared to them, HITA-Graph is notably different in encoding/accessing the infobox knowledge: (1) The encoding process of HITE is independent of the attribute order. The attribute order is a crucial factor to the generated text in the infobox-to-text task, and many efforts are devoted to it [Puduppully et al., 2019; Bai et al., 2020]. However, unless dialogue generation rephrases the infobox in an orderly fashion, the attribute order is meaningless. (2) Infobox-totext approaches usually consider only one or two vertical domains, but HITA-Graph is an open-domain approach. (3) HITA-Graph proposes a novel Infobox-Dialogue Interaction Graph Network to conduct the context-level infobox encoding, which not only promotes the data flow, but also interacts with the dialogue context more easily and effectively.

5 Conclusion

This paper proposes an infobox knowledge-aware dialogue generation approach, HITA-Graph. An infobox table specifies an entity with multiple attributes. The advantages of the infobox tables include: (1) easy to be collected from the Internet; (2) knowledge has been well-organized as attributes; (3) each infobox focuses on only one entity, bringing high knowledge consistency. In HITA-Graph, we propose a novel orderirrelevance Hierarchical Infobox Table Encoder and a novel Infobox-Dialogue Interaction Graph Network to encode the infobox knowledge better. We also propose a Hierarchical Infobox Attribute Attention mechanism to support the access of the encoded knowledge at different levels of granularity, and a Dynamic Mode Fusion strategy to generate more informative responses. Both the automatic and human evaluation demonstrate the proposed HITA-Graph outperforms representative competitors in almost all experiments.

Acknowledgments

This work is partly supported by ICBC Technology.