RDF-to-Text Generation with Graph-augmented Structural Neural Encoders

1 Introduction

RDF-to-text generation is to transform a set of Resource Description Framework (RDF) triples into informative and faithful text. The task is challenging since RDF sub-graph structure needs to be well modeled in addition to capturing semantic information in RDF triples. Figure 1 illustrates an RDF triples graph with corresponding descriptive text, in which the nodes (such as "FOOD-1" and "INGREDIENT") represent the entities and edges (such as "dishVariation" and "ingredient") represent the relations between the connected entities. It has many fact-aware applications such as knowledgebased question answering [Hao et al., 2017], entity summarization [Pouriyeh et al., 2017], and data-driven text generation [Liu et al., 2018].

Traditionally, an RDF-to-text generation system mainly focuses on the pipeline process of content selection and surface realization with human-crafted features. However, a critical issue for error propagation has been largely overlooked, which does harm to the quality of generated text. Recently, as the end-to-end deep learning has made great progress in natural language processing, RDF-to-text generation has achieved promising performance by using various sequence-to-sequence (Seq2Seq) models [Gardent et al., 2017b; Jagfeld et al., 2018]. Conceptually, RDF triple elements need to be processed and concatenated into a sequence in order to feed into Seq2Seq models.

∗Both authors contributed equally to this research.

†Corresponding author.

However, simply transforming the RDF triples into a sequence may lose important higher-order information. Since RDF triples can be represented as a knowledge graph, two graph-based approaches have been proposed recently for RDF-to-text generation. Trisedya et al. presented a graphbased triple encoder GTR-LSTM that captures both intratriple and inter-triple entity relationships by sampling different meta-paths to preserve graph structure in the encoder. However, since the encoder component is still based on recurrent neural networks, it often fails to capture rich local complex structure information between entities and relationships.

On the other hand, Marcheggiani and Perez-Beltrachini proposed a graph-to-sequence model (Graph2Seq) based on a modified Graph Convolutional Networks (GCN) [Kipf and Welling, 2016], which directly encodes graph-structured RDF triples and decodes a text sequence. However, it is well known that GCN often gets overfitting quickly when using multiple layers (>= 3), which weakens its capability of learning longer range dependence. Therefore, this model usually performs better on capturing local structure information of the graph than capturing global information between the RDF triples.

To address the aforementioned issues, we propose a novel neural network architecture by exploiting graph-augmented structural neural encoders for RDF-to-text generation. To this end, we first present to exploit both power of graph encoder and graph-based meta-paths encoder to jointly learn structure information locally and globally. We then present separated attentions on two different inputs i.e., graphs and meta-paths to jointly learn the final hidden representation of RDF triples in order to better decode the text sequence.

The combined encoders can focus on multiple perspectives of the input RDF graph. A novel bidirectional-GCN (bi-GCN) encoder is used to explicitly model the local structure information between the intra-triple relationships, while a new bidirectional Graph-based Meta-Paths (bi-GMP) encoder mainly focuses on modeling global long-range dependency between the inter-triple relationships.

We highlight our main contributions as follows:

• We propose a novel graph-augmented structural neural encoders model by combining a new bi-GCN encoder and a new bi-GMP encoder for explicitly modeling the global and local structure information of the input RDF triples.
• We further present separated attentions on each of graph encoders and fuse their corresponding context vectors to better decode the descriptive text.
• The experimental results on two WebNLG datasets (a challenge dataset and a supplementary dataset) corroborate the advantages of our model over state-of-the-art models in BLEU, METEOR and TER metrics.

2 RelatedWork

Our approach is highly related with existing works in structured data to text generation and graph neural networks.

2.1 RDF-to-Text Generation

RDF-to-text generation aims to generate a grammatically correct, fluent, informative and faithful description for graph structured input data. Early attempts for Web Ontology Language to text generation were done by [Bontcheva and Wilks, 2004] and verbalizing a knowledge base [Banik et al., 2012]. Pipeline approach is generally used to solve these data-to-text generation tasks, which includes two main steps: (1) content selection [Barzilay and Lapata, 2005] defines what contents should be described in the generated text; (2) surface realization [Deemter et al., 2005] implements the generation process word by word. In the past few years, diverse NLG tasks have achieved promising performance by using Seq2Seq model with attention mechanism [Bahdanau et al., 2014] and copy mechanism [Gu et al., 2016; See et al., 2017]. As shown in [Gardent et al., 2017b; Jagfeld et al., 2018], Seq2Seq model and its variants perform promisingly on RDF-to-text generation by concatenating RDF triple elements into a sequence. To reduce the probability of generating low-quality sentences, Zhu et al. proposed a framework based on Seq2Seq, which optimizes the inverse Kullback-Leibler (KL) divergence between the distributions of the real and generated sentences. By combining the pipeline system and neural networks, Moryossef et al. presented a method for matching reference texts to their corresponding text plans to train a plan-to-text generator.

2.2 Graph Neural Networks for Text Generation

Recently, there has been a surge of interests to exploit graphbased neural networks for graph-structured data to text generation. Researchers employed a bidirectional graph encoder to embed an input graph to a sequence of node embeddings and then used an attention-based LSTM method to decode the target sequence from these vectors [Xu et al., 2018; Chen et al., 2020; Gao et al., 2019]. Song et al. applied an LSTM over the state transitions of the graph-based encoder outputs to capture longer-range dependencies. Marcheggiani and Perez-Beltrachini exploited relational graph convolutional networks, introduced and extended in [Bruna et al., 2013; Defferrard et al., 2016; Kipf and Welling, 2016], to encode a node in a relational graph with its neighbor nodes, edge labels and edge directions at the same time. Although these graph neural networks consider different perspectives of graph structure information, none of graph-based encoders could fully capture both global and local dependencies in a knowledge graph.

3 ProblemDefinition

Formally, the RDF-to-text generation task is defined as follows. The input contains a set of RDF triples, denoted as S = {t1, t2, ...tn} where ti is a triple consisting of subject, relationship and object. All the input triples are represented as directed graphs. The aim is to generate a natural language descriptive text Y = 〈w1,w2, ...wT 〉 that represents the correct and concise semantics of entities and their relationships in the given RDF triple inputs. In this study, the bi-GMP encoder consumes input graph G1 = (V1, E1), where V1 represents all of the entity nodes and E1 denotes original relationships between these entity nodes. Differently, the bi-GCN encoder consumes input graph G2 = (V2, E2), where V2 contains entity nodes and relationship nodes since relationships are regarded as additional nodes instead of edges. E2 refers to a newly defined set of edges that describe the relationship between entity nodes and relationship nodes, or between multiple words of a node.

4 Our ProposedModel

In this section, we present our graph-augmented structural neural encoders including two graph-based encoders, and describe the approach to combine them for better capturing global and local relationships between and within the triples.

4.1 Bidirectional Graph-basedMeta-Paths Encoder

In order to encode information according to different metapaths and capture long-range dependency in graph G1, we present a new bidirectional Graph-based Meta-Paths encoder (bi-GMP), which is an improved variant of GTR-LSTM triple encoder [Trisedya et al., 2018]. Compared to GTR-LSTM, the bi-GMP model applies hidden state masking between different meta-paths to keep the paths encoding non-interfering.

Moreover, copy mechanism [See et al., 2017] is introduced into our meta-paths encoder to improve the performance.

The meta-paths of graph G1 are calculated by a combination algorithm of topological sort and single-source shortest path. We first calculate the nodes with zero in-degree VIN and nodes with zero out-degree VOUT. Then, we calculate a single-source shortest path using a node in VIN as the source node and a node in VOUT as the destination node with both the entity nodes and the relationships preserved in the path. The graph G1 is transformed to a sequence composed of a set of meta-paths Sp = {p1, p2, ...}, in which pk = 〈wk,1, wk,2, ...wk,nk 〉.

Inspired by the advantages of BiLSTM over LSTM, the proposed meta-paths encoder calculates the representation of each token in each meta-path from two directions and fuse them together in the final step. The hidden state ri is computed as:

where wi represents a token of entity node or relationship. f(·) and g(·) are single LSTM units. Finally, the bi-GMP encoder generates a set of entity node representations R1. We then use max pooling on R1 to compute the graph embedding of graph G1:ZG1 = Wrmaxpool(R1), where Wr denotes weight matrices.

4.2 Bidirectional Graph Convolutional Networks Encoder

For the graph G2 = (V2,E2), relationships are regarded as additional graph nodes, so G2 contains entity nodes and relationship nodes. Then, for the similar reason above, we present the bi-GCN encoder calculating the vector representations H(l) = {h(l) v0 , h(l) v1 , ...} ∈ RD×V at layer l:

����

�����

where ˆ A⊢ = A⊢ +I and ˆ A⊣ = A⊣ + I denote the source-totarget and target-to-source adjacency matrices of the directed graph G2 with inserted self-loops. I is an identity matrix and ˆDii = ∑j=0 ˆ Aij is a diagonal degree matrix. W (l-1) 1⊢ and W (l-1) 1⊣ are layer-specific trainable weight matrices. Wf denotes trainable weight matrix, and σ is a non-linearity function. H(0) is initialized with word embeddings. The bidirectional node embeddings at layer l are concatenated and inputed to a single-layer perceptron before input to the next bi-GCN layer. Bi-GCN encoder is stacked to L layer and R2 = H(L) is a set of entity and relationship node representations .

We then use average pooling on R2 to compute the graph embedding of graph G2: ZG2 = ϕ(avgpool(R2)) , where ϕ(·) is a single-layer perceptron.

4.3 Combining bi-GMP and bi-GCN Encoders

We propose a combination strategy to integrate the above biGMP encoder and the bi-GCN encoder for the input RDF triples, which aims to jointly learn the local and global structure information of the RDF triples input. The overall architecture of the combined encoders is depicted in Fig. 2.

Both encoders generate a set of node representations. In general, R2 = {h0, h1, h2...} captures the local structure information within the RDF triples better, since each node representation is directly modeled by all its one-hop neighbors at one layer. To avoid overfitting, bi-GCN encoder is stacked to 2 layers in this study. In this way, the output of bi-GCN encoder hi preserves at most two-hop information of the graph, which is limited to one or two triples. Whilst, R1 = {r0, r1, r2, ...} mainly focuses on the global structure information between the RDF triples, since the bi-GMP encoder computes hidden states following a traversal order calculated by a combination of topological sort and singlesource shortest path algorithm over the whole graph. The output of the bi-GMP encoder ri can preserve information from one triple to seven triples (an input contains at most seven triples in the WebNLG datasets) if there exists a meta-path.

Then, the combined graph embedding ZG is computed as ZG = ZG1 ⊕ ZG2 , where ⊕ is the component-wise addition.

4.4 Decoder

An attention-based LSTM decoder [Luong et al., 2015] is used for text generation. However, the plain top-down attention is not appropriate since we have two input graphs and it can not fully exploit different semantic token information from two quite different encoders. Therefore, we present separated attention mechanism on each of the graph encoders and fuse their corresponding context vectors to better decode the descriptive text.

ZG is fed into the decoder as initial hidden state. For each time step t, the decoder feeds the concatenation of the embedding of the current input (previously generated word) et and previous step context vector ct-1 as new input and previous hidden state st-1 to update its hidden state st. Then, we further apply the separate attentions by computing the align weights vectors at decoding time step t as follows:

where ri ∈ R1 and hj ∈ R2. M = |R1| and V = |R2| are the lengths of representations sequence. st is the hidden state of decoder. The score(·) function estimates the similarity of ri, hj and st.

Then, we compute bi-GMP level context vector cu and biGCN level context vector cv respectively:

Next, we concatenate cu, cv and decoder hidden state st to compute final attentional hidden state at this time step as:

where ct is the concatenated context vector at time step t and Wc and bc are learnable parameters. After calculating a sequence of attentional hidden states 〈 ˜ s1, ˜ s2, ..., ˜ sT 〉 recurrently, the decoder generates an output sequence 〈y1, y2, ..., yT 〉. The output probability distribution over a vocabulary at the current time step is calculated by:

where Wv and bv are learnable parameters. Finally, we employ the loss function with a negative log likelihood:

5 EntityMasking and Graph Constructions

In this section, we first discuss the importance of entity masking and then describe how to transform RDF triples set to input graphs for two encoders.

5.1 EntityMasking

Entity masking can improve the generalization ability of our model. We use the officially provided dictionary and DBpedia lookup API to map the subjects and objects in the triples set to corresponding entity types. Considering that multiple entities in a set of triples may belong to the same type, we assign an entity-id (eid) to each entity in the set. Therefore, each entity is replaced with its eid and type. For example, the entity "Bakewell pudding", namely FOOD-1 in Fig. 3, is replace by ENTITY-1 FOOD while the entity "Bakewell tart"(FOOD-2) is replaced by ENTITY-2 FOOD.

5.2 RDF Triples toMeta-paths

In Fig. 3(a), we first choose a source node with zero in-degree FOOD-1 and a destination node with zero out-degree PERSON. The shortest path from FOOD-1 to PERSON is FOOD1→region→ PLACE→leaderName→PERSON. Similarly, the other two meta-paths are:

• FOOD-1→region→PLACE→county→COUNTY
• FOOD-1→dishVariation→FOOD-2→ingredient →INGREDIENT

The meta-paths are concatenated to a sequence and then fed to the bi-GMP encoder. The hidden state of the last token in path pt is not forwarded to the first token in path pt+1, which is depicted in Fig. 2.

5.3 RDF Triples to bi-GCNGraph

For bi-GCN encoder, we treat relationships as additional nodes similar to [Marcheggiani and Titov, 2017] and the new relationship node is connected to the subject and object by two new directed edges, respectively. This graph construction method makes the new bi-GCN graph (Fig. 3(b)) two times larger than the original RDF triples graph (Fig. 3(a)) in hops, making it more difficult to capture long-range dependency. As shown in Fig. 3(b), the nodes in the graph are divided into entity nodes and relationship nodes. The original triple (FOOD-1, region, PLACE) is separated into two new triples region→FOOD-1 and region→PLACE with "region" being a node in the input graph. For nodes having more than one word, each word is separated into an independent node that is connected to the core node (usually the "ENTITY-id" node) with a new edge. In this way, the original entity "ENTITY1 FOOD INGREDIENTS" is separated into two new triples ENTITY-1→FOOD and ENTITY-1→INGREDIENTS.

6 Experiments

6.1 Datasets

We use two different WebNLG datasets 1 [Gardent et al., 2017a] which are designed for the task of mapping RDF triples to text. Each example is a 〈triples, text〉 pair. Here, one triple set may correspond to multiple reference texts. Each RDF triple is represented as 〈subject, relationship, object〉, where the subject and object are constants or entities. The first dataset is the WebNLG 2017 challenge dataset, consisting of 18102 training pairs, 2268 validation pairs, and 2495 test pairs in 10 categories (Astronaut, Building, Monument, University, SportsTeam, WrittenWork, etc.). The second supplementary dataset is extracted from an enriched version of the WebNLG 2017 challenge dataset. The enriched dataset consists of 31969 training pairs, 4030 validation pairs and 4222 test pairs. The second supplementary dataset contains 13867 training pairs, 1762 validation pairs, and 1727 test pairs, which does not overlap the first dataset. The second supplementary dataset belongs to 5 other categories (Athlete, Artist, MeanOfTransportation, CelestialBody, Politician).

6.2 Experimental Settings and EvaluationMetrics

We build a vocabulary list based on the training set, which is shared between the encoders and the decoder. For model hyperparameters, we set 300-dimension source and target word embeddings and 300-dimension hidden state for biGCN encoder, meta-paths encoder and decoder. We use Adam [Kingma and Ba, 2014] as the optimization method with an initial learning rate 0.001 and learnable parameters are updated every 64 instances.

1 https://gitlab.com/shimorina

For our experiments, we adopt the standard evaluation metrics of the WebNLG challenge, including BLEU [Papineni et al., 2002], METEOR [Denkowski and Lavie, 2011] and TER. The metric of BLEU suggested by the WebNLG challenge is multi-BLEU. For metrics BLEU and METEOR, the higher the better, while for metric TER, the lower the better.

6.3 Baselines

We compared our model against the following baselines:

Sequential Model. Sequential model contains an attentionbased bidirectional LSTM encoder and an LSTM decoder. The RDF triples are transformed into a sequence. Moreover, the results of MELBOURNE and PKUWRITER are both sequential models, which are reported in [Gardent et al., 2017b].

DGCN Model. We rerun the code of [Marcheggiani and Perez-Beltrachini, 2018] to get the results of Deep Graph Convolutional Encoder (DGCN).

Bi-GMP Model. Bi-GMP model consists of a meta-paths encoder calculating the representation of each token in the meta-paths from two directions and a one-layer LSTM decoder. Copy mechanism [Gu et al., 2016] is incorporated into the model to improve the performance.

Bi-GCN Model. Bi-GCN model consists of a bi-GCN encoder (GCN with two layers) and a one-layer LSTM decoder. We exploit bi-GCN with- and without- copy mechanism.

6.4 Experimental Results

As shown in Tables 1 and 2, our proposed model consistently outperforms other baselines on all three evaluation metrics. For instance, our full model achieves about 1.0 multiBLEU points higher than those of the other baselines on two WebNLG test datasets. This is because our full model could better capture the global and local graph structure of the RDF triples. In addition, Table 3 shows our proposed model achieve higher scores on BLEU-1, BLEU-2, BLEU-3 and BLEU-4 compared to other baselines, indicating that our full model can better encode multi-perspective information. Our code is publicly available for research purpose. 2

6.5 Ablation Study

As shown in Table 1 and Table 2, there are three key factors in our proposed model that may affect the quality of generated text. The first two are the bi-GCN and bi-GMP encoders. Experimental results shows the model combining graph-augmented structural neural encoders performs better than the models with one single graph-based encoder. This result is expected, since it is difficult for one single graph-based encoder to fully encode both global and local structure information completely. The third factor is copy mechanism. We only apply copy mechanism on the bi-GMP encoder in our full model. Interestingly, we found that when applying copy mechanism on both the bi-GMP encoder and the bi-GCN encoder, it actually leads to worse performance compared with our full model.

2 https://github.com/Nicoleqwerty/RDF-to-Text.

6.6 Case Study

We further manually inspect the outputs of different models and conduct case studies for better understanding the model performances. As shown in Table 4, we find that the models involving bi-GCN encoder perform better on covering correct relationships between entities. This result is expected since the bi-GCN encoder pays more attention to the local structure information and can predict the intra-triple relationship more accurately and effectively. Meanwhile, the bi-GMP encoder mainly focuses on the inter-triple relationships, which helps model long-range dependency with more context.

6.7 Human Evaluation

To further evaluate the quality of these generated text examples, we presented a number of original RDF triples and corresponding generated text pairs to three human evaluators. Three evaluators are shown 200 outputs, a quarter references, a quarter generated by bi-GCN model, a quarter generated by bi-GMP model and a quarter generated by full model. The evaluators were asked to evaluate the generated texts from three perspectives (each perspective scale from 1 to 5): I) Grammatical rates each text sample with respect to the coherence, no redundancy, and no grammatical errors; II) Informativity (Global) scores based on how well the information in input triples are covered; III) Conciseness (Local) evaluates whether the entities and corresponding relationships appearing in the generated sentence are accurate. We averaged the results from three evaluators for all three tasks. Table 5 indicates that text examples generated by our proposed model are indeed improved on the respects of informativity (global information) and conciseness (local information).

7 Conclusion

In this paper, we propose a novel approach via exploiting graph-augmented structural neural encoders for RDF-to-text generation. Our approach jointly learns structure information locally and globally via the combination of a bidirectional graph encoder and a bidirectional graph-based meta-paths encoder to learn intra-triple and inter-triple relationships. The experimental results show that our proposed model outperforms the state-of-the-art baselines. One of the future works is to further extend our proposed method to develop a knowledge graph question answering system.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No.61402191), and the Thirteen Fiveyear Research Planning Project of National Language Committee (No.WT135-11).