Neural Entity Summarization with Joint Encoding and Weak Supervision

1 Introduction

Semantic Web and knowledge graphs (KGs) provide entity descriptions in triples (i.e., node-arc-node), supporting many and various applications. Figure 1 illustrates a KG containing descriptions of entities such as Everest, which could be presented in a knowledge card in Google search. The size of an entity description can reach hundreds of triples, exceeding the capacity of a compact knowledge card and hence requiring a summarization method for generating a summary-an optimal subset of triples. This established research problem is referred to as entity summarization [Liu et al., 2019].

Challenges. In this work we focus on computing generalpurpose entity summaries. Existing methods are mostly unsupervised [Cheng et al., 2011; Sydow et al., 2013; Thalhammer et al., 2016; Gunaratna et al., 2015]. They use various heuristics for ranking and selecting triples. We attempt to develop a supervised approach where we face two challenges. (1) KG is a hybrid of graph structure and short text. Extracting and combining useful features from them to represent a set of triples is a non-trivial task. (2) It is expensive to manually label a large set of general-purpose entity summaries. The availability of training data for supervision is a bottleneck in this research.

∗Contact Author

Contributions. To address these challenges, we develop NEST, 1 the abbreviation for Neural Entity SummarizaTion, where our contributions are summarized as follows. (1) Our neural encoder jointly extracts structural and textual features from KGs. Manual engineering is obviated. (2) We fine-tune with weak supervision from programmatically labeled data. Manual labeling is not needed.

Approach. The full model of NEST is shown in Figure 2. It relies on the assumption that a high-quality general-purpose summary should satisfy two natural conditions. (1) Each selected triple should describe an important feature of the entity. (2) The selected triples should diversely span different aspects of the entity. Accordingly, NEST incorporates two neural scorers: a salience-based triple scorer (STS) and a diversity-based summary scorer (DSS). Input KGs are fed into a neural encoder, which is firstly pre-trained in an unsupervised way to generate contextualized representations of elements, and then is linked to STS and DSS to be fine-tuned in a supervised way.

1 https://github.com/nju-websoft/NEST

Outline. The remainder of the paper is organized as follows. We formulate the problem in Section 2 and present our approach in Section 3. Experiments are described in Section 4, and the results are given in Section 5. Related work is discussed in Section 6 before we conclude in Section 7.

2 Problem Statement

The Resource Description Framework (RDF) is a popular format for interchanging KGs. Our formulation uses RDF, but our approach can be applied to KGs in other formats.

RDF. Let I, B, L be the disjoint sets of all Internationalized Resource Identifiers (IRIs), blank nodes, and literals in RDF, respectively. They are collectively known as RDF terms. An RDF term r has a textual form text(r), e.g., a humanreadable label of an IRI or blank node, or the lexical form of a literal. RDF data is a set of subject-predicate-object triples:

KG can be straightforwardly formatted in RDF. Each triple 〈s, p, o〉 ∈ T represents an arc from subject s to object o labeled with predicate p. Figure 1 illustrates a KG where for RDF terms we show their textual forms.

Entity description. An entity is identified by an IRI or a blank node. The description of entity e is a set of triples:

Each triple t ∈ D(e) represents a property-value pair of e:

where p-1 represents the inverse of p. In Figure 1, the description of Everest contains four triples (i.e., incident arcs).

Entity summarization. Given an integer size constraint k, a summary of entity e is a subset of triples S ⊆ D(e) such that |S| ≤ k. Entity summarization is to find an optimal summary for an entity. Optimality depends on the application. We study the generation of general-purpose summaries.

3 Approach

NEST is fed with a sequence of RDF terms as input, which represents a random walk, a triple, or an entity summary in a KG, depending on the phase of our approach. The sequence is processed by a KG encoder to generate a vector representation for each RDF term in the sequence (Section 3.1). Based on the representations, an entity summary scorer assesses the quality of a summary from two perspectives: STS for salience and DSS for diversity (Section 3.2). In the training phase, we pre-train the KG encoder in an unsupervised way and then fine-tune the entire model in a supervised way. In the test phase, we search for an optimal summary where we use the trained model to evaluate candidate summaries (Section 3.3).

3.1 KGEncoder

We convert KGs into a low dimensional space to facilitate downstream scorers. A KG is a graph labeled with textual forms of RDF terms. Our novel neural encoder jointly extracts structural and textual features fromKGs.

Model. The input is a sequence of RDF terms r1r2 · · · rn. We extract the textual features of each ri in a standard way, by averaging the word embeddings of its textual form text(ri):

where wj is the fixed embedding of word wj, and rwi is the initial representation of ri using textual features. For wj, we use pre-trained fastText embeddings [Bojanowski et al., 2017]. Further, to extract structural features of the sequence, we feed text-featured representations rw1 rw2 · · · rwn into a forward model and a backward model, both using two layers of Long Short-Term Memory networks (LSTMs). They output

where --→ Θlstm and ←-- Θlstm are LSTM parameters. Each -→ Rri (resp. ←- Rri ) is the forward (resp. backward) representation of ri contextualized by r1r2 · · · ri-1 (resp. rnrn-1 · · · ri+1). The output representation of ri concatenates -→ Rri and ←- Rri :

Pre-training. We pre-train the KG encoder in an unsupervised way. We follow a common practice to sample random walks from a KG to capture its graph structure [Cai et al., 2018]. For each node r, we generate a set of undirected random walks of fixed length d starting from r. Such a walk is a sequence of d RDF terms, alternating between nodes (i.e., subjects or objects of triples) and arcs (i.e., predicates). Walks are constrained to not pass through any literals; that is, a walk may start or end at a literal but cannot have literals as intermediate nodes, to prohibit meaningless connections via polysemous literals, e.g., numbers. In Figure 1, given d = 5, an example random walk is:

Everest known-for Alex height 1.8m .

For a sampled random walk r1r2 · · · rd, the objective of pretraining is to maximize the existence probability of the walk. The forward and backward models compute this probability in opposite directions:

Specifically, --→ Rr1 --→ Rr2 · · · ---→ Rri-1 in the forward model are used to predict ri with a softmax layer, and ←-- Rrd ←--- Rrd-1 · · · ←--- Rri+1 predict ri in the backward model. We jointly maximize the log likelihood of the two directions:

where Θsoftmax are tied parameters for the softmax layer.

3.2 Entity Summary Scorer

To assess the quality of an entity summary, our novel neural scorer considers two perspectives: STS for the salience of a triple, and DSS for the diversity of a summary. STS and DSS work with the same KG encoder and share its parameters, but they are fine-tuned in succession with different training data.

STS-model. STS measures the salience of a triple t = 〈s, p, o〉. We feed t as a sequence of 3 RDF terms spo into the KG encoder, and concatenate the output representations of these RDF terms to represent t:

Based on t, we use a two-layer dense neural network (MLP) to compute a score representing the salience of t:

where Θdense (STS) are parameters for the MLP.

STS-fine-tuning. To train sal to compute salience, we fine-tune STS and the KG encoder with labeled entity descriptions where each triple t has a binary label Yt indicating whether t is in an ideal summary (Yt = 1) or not (Yt = 0). The objective of fine-tuning is to minimize the mean squared error over a set of labeled triples TSTS for training:

DSS-model. DSS measures the diversity of a summary S = {〈s1, p1, o1〉, . . . , 〈sk, pk, ok〉} through the diversity of predicates {p1, . . . , pk}. A diverse summary should not contain predicates describing similar aspects of an entity, e.g., height and elevation of Everest in Figure 1. We feed S as a sequence of 3k RDF terms s1p1o1 · · · skpkok into the KG encoder, and sum the output representations of the predicates to represent S:

Here for the KG encoder, we reset the cell state of LSTMs for every 3 RDF terms (i.e., every triple) in the input sequence, to make S invariant to the order of triples in S as such kind of invariance is reasonable for a set. Based on S, we use a two-layer dense neural network (MLP) to compute a score representing the diversity of S:

where Θdense (DSS) are parameters for the MLP.

DSS-fine-tuning. To train div to compute diversity, we fine-tune DSS and the KG encoder with a heuristically computed diversity score, namely the average cosine distance between pairs of predicates in S using their representations output from the KG encoder:

The objective of fine-tuning is to minimize the mean squared error over a set of entity summaries SDSS for self-training:

Compared with directly using ZS as div(S), when we finetune DSS to approximate ZS, we also tune the parameters of the KG encoder which is shared with STS, thereby having the potential to achieve better overall performance.

3.3 Entity Summary Generation

In the test phase, given entity description D(e) and each candidate summary S ⊆ D(e), we evaluate S using STS and DSS to generate its overall score:

For a size constraint k, there are (|D(e)| k ) candidate summaries. It may be computationally impracticable to evaluate all of them. We adopt the simulated annealing heuristic to search for an optimal summary. Specifically, we randomly sample an initial summary S0 ⊆ D(e) with |S0| = k and compute score(S0). Then in the i-th iteration, we replace a random triple in Si-1 with a new triple in D(e) \ Si-1 to form Si and compute score(Si). We will definitely continue iterations if score(Si) > score(Si-1). Otherwise, we will continue with an acceptance probability that is decreased after each iteration. Finally, Si in the last iteration is output.

4 Experiments

We compared NEST with 11 baseline methods on two public benchmarks for general-purpose entity summarization.

4.1 Datasets and EvaluationMetrics

We evaluated with the two largest available benchmarks for general-purpose entity summarization: the Entity Summarization BenchMark (ESBM) [Liu et al., 2020] and the FACES Evaluation Dataset (FED) [Gunaratna et al., 2015]. ESBM v1.0 2 consists of ESBM-D and ESBM-L. ESBM-D provides 1,200 ground-truth summaries for 100 entities in DBpedia v2015-10. Each entity description is labeled with 6 summaries for k = 5, and 6 summaries for k = 10, created by different human experts. Similarly, ESBM-L provides 480 ground-truth summaries for 40 entities in LinkedMDB. In FED, 3 50 entities in DBpedia v3.9 are labeled with 373 summaries for k = 5, and 373 summaries for k = 10.

Following ESBM, we compared a computed summary with a ground-truth summary and calculated F1 score. For each method to evaluate, we calculated its mean F1 over all the entities and ground-truth summaries in each dataset.

4.2 Programmatic Generation of Labeled Data

In the absence of a large set of manually labeled entity descriptions for training, we programmatically generated labeled data from the correspondence between DBpedia and Wikipedia. Such labels may contain noise and hence our supervision was weak but free of manual work. Methods trained with this data were then tested on ESBM and FED where entities were from not only DBpedia but also LinkedMDB, thereby also evaluating the generalizability of a method.

Specifically, for each entity e in DBpedia we found its corresponding article in Wikipedia, which typically began with an untitled section A(e) containing key factual information about e. We labeled triple t ∈ D(e) as positive, i.e., Yt = 1 in Eq. (11), if t was mentioned in A(e), otherwise negative and Yt = 0. We programmatically determined whether t was mentioned in A(e) as follows. We transformed the textual form of the property value, i.e., text(v(t)), into a normalized word sequence Wt in four steps: splitting the text using non-letters and non-digits as delimiters, converting each month word (e.g., February) into two digits (e.g., 02), lowercasing all the letters, and removing stop words. In the same way we transformed A(e) into WA(e). A mention of t in A(e) was recognized if every word in Wt was a number appearing in WA(e), or if more than half of the words in Wt appeared or had an approximate match in WA(e). We recognized an approximate match between word wi ∈ Wt and word wj ∈ WA(e) if wi consisted of more than γ characters and the edit distance between wi and wj was smaller than δ. We empirically set γ = 7 and δ = 3 in the experiments.

2 https://w3id.org/esbm/

3 http://wiki.knoesis.org/index.php/FACES

To evaluate the quality of the generated labels, we randomly sampled 200 entities from DBpedia and manually labeled 6,025 triples in their descriptions as the gold standard. The error matrix is shown in Table 1. The true positive rate is 92.93%, and the true negative rate is 98.16%, demonstrating the good quality of our labeled data.

For the subsequent evaluation, from the programmatically labeled data we randomly sampled 11,672 entity descriptions containing a total of 264,998 triples. The sampling was controlled to be disjoint with the test entities in ESBM and FED. We used 80% of the labeled data for training methods and 20% for validation, e.g., tuning hyperparameters.

4.3 Configuration of NEST

For each KG we pre-trained and fine-tuned a separate model.

Pre-training KG encoder. Starting from each node in a KG we sampled 100 random walks with d = 7. Our LSTMs contained 4,096 units, 300 dimension projections, and a residual connection from the first layer to the second layer. We trained for 10 epochs with batch size 256.

Fine-tuning STS. From each labeled entity description D(e), we randomly sampled 5 triples as a batch and we sampled |D(e)| 2 times. Our MLP had 4,096 units in each layer and applied ReLU activations. We trained for 10 epochs.

Fine-tuning DSS. From the labeled triples we randomly sampled 100,000 subsets of size 5 as summaries. Our MLP had 4,096 units in each layer and applied ReLU activations. We trained for 10 epochs with batch size 5.

Generating summary. In simulated annealing, we initialized the acceptance probability to 0.1, and decreased it by 0.1 |D(e)| after each iteration.

4.4 Baseline Methods

We compared with 11 baseline methods and an oracle method as a reference point used for comparisons.

Unsupervised methods. We compared with 6 unsupervised methods for general-purpose entity summarization: RELIN [Cheng et al., 2011], DIVERSUM [Sydow et al., 2013], FACES [Gunaratna et al., 2015], FACESE [Gunaratna et al., 2016], CD [Xu et al., 2016], and LinkSUM [Thalhammer et al., 2016].

Supervised methods. We also implemented 5 supervised neural methods as baselines, including 4 methods based on different graph embeddings: TransE [Bordes et al., 2013], Jointly (A-LSTM) [Xu et al., 2017], RotatE [Sun et al., 2019], RDF2Vec (K2V SG 300) [Ristoski et al., 2019]; and 1 state-of-the-art method for document summarization: BERTSUMEXT (large) [Liu and Lapata, 2019]. In each graph embedding based method, we represented a triple by concatenating the embedding vectors of the property and the value in the triple. Then we performed pointwise learning-torank (L2R) and chose the k top-ranked triples as a summary. We experimented with 3 L2R models: XGBoost (xgb), random forest (rf), and naive Bayes (nb). To adapt the document summarization method BERTSUMEXT to solve entity summarization, we generated a pseudo sentence from each triple by concatenating the textual forms of its three RDF terms. All these methods and our NEST were trained and tuned on the same training and validation sets.

Oracle method. We implemented an oracle method to show the highest achievable F1 as a reference point used for comparisons. Recall that ESBM and FED provided multiple ground-truth summaries for each entity. ORACLE selected k triples that appeared in the most ground-truth summaries.

5 Results and Analysis

We firstly compared NEST with baselines and then conducted an ablation study. We tested three research hypotheses. RH1: NEST with weak supervision can outperform existing methods for general-purpose entity summarization. RH2: The novel neural model of NEST can outperform existing neural models in the entity summarization task. RH3: The joint encoding in NEST can be more effective than separate structure encoding and text encoding.

5.1 Comparison with ExistingMethods for Entity Summarization and ORACLE

We compared NEST with 6 methods for general-purpose entity summarization. F1 scores are presented in Table 2. NEST achieved new state-of-the-art results on all the three datasets for both k = 5 and k = 10. It significantly (p < 0.01) outperformed all the baselines on ESBM-D (k = 5) and ESBM-L (k = 5 and k = 10). In other cases, differences were mostly significant. When k = 5, compared with the best-performing baseline on each dataset, NEST raised F1 from 0.299 to 0.354 by 18% on ESBM-D, from 0.252 to 0.332 by 32% on ESBML, and from 0.236 to 0.272 by 15% on FED. The results supported research hypothesis RH1.

Although these baseline methods were unsupervised while NEST was supervised, our supervision was weak as our labeled data was programmatically generated and hence free of manual work. Further, the generation used DBpedia, while NEST also showed the best performance on LinkedMDB (i.e., ESBM-L), demonstrating its generalizability.

The F1 scores of ORACLE were in the range of 0.5300.721. It was impossible for ORACLE or any other method to reach F1 = 1, because for each entity there were multiple different ground-truth summaries which could not simultaneously match a computed summary. NEST stepped closer to ORACLE than the baselines, but the gap was still considerable and suggested room for improvement.

5.2 Comparison with Supervised Neural Methods

We compared NEST with 5 supervised neural methods. F1 scores are presented in Table 3. Among these baselines, RotatE with XGBoost (xgb) generally achieved the best result on ESBM-D, while BERTSUMEXT topped on ESBM-L and FED. However, none of these methods were comparable with NEST. When k = 5, compared with the best-performing baseline on each dataset, NEST raised F1 from 0.297 to 0.354 by 19% on ESBM-D, from 0.248 to 0.332 by 34% on ESBML, and from 0.185 to 0.272 by 47% on FED. The results supported research hypothesis RH2.

All these methods were supervised and were trained and tuned on the same data. Therefore, we mainly attributed the superiority of NEST to the novelty of our KG encoder which jointly extracted and tightly fused structural and textual features from KGs by keeping track of the structural dependencies between textualized RDF terms. By contrast, TransE, RotatE, and RDF2Vec only encoded graph structure, BERTSUMEXT only encoded text, and Jointly loosely combined structure and text in a linear way.

Furthermore, among all the 11 baselines, supervised methods did not perform better than unsupervised methods. Therefore, developing an effective supervised method for generalpurpose entity summarization was not a trivial task. It helped to demonstrate the significance of our work.

5.3 Ablation Study

We implemented two variants of NEST to analyze the effectiveness of its novel joint encoding. The first variant (w/o structure) was not pre-trained with random walks in KGs but directly fine-tuned, thus focusing on textual features. In the second variant (w/o text), the initial representation of an RDF term was given by its identifier rather than using word embeddings, thus focusing on structural features.

F1 scores are presented in Table 4. Compared with NEST, the performance of both variants decreased notably. Text had a stronger influence than graph structure. The results supported research hypothesis RH3. In fact, the two variants still outperformed all the baselines on ESBM-D (k = 5) and ESBM-L (k = 5 and k = 10), demonstrating the effectiveness of our scorer and weak supervision.

6 RelatedWork

Extensive research has been focused on general-purpose entity summarization. RELIN [Cheng et al., 2011] computes the informativeness of a triple. DIVERSUM [Sydow et al., 2013] improves the diversity of a summary by choosing triples about different properties. FACES [Gunaratna et al., 2015] and FACES-E [Gunaratna et al., 2016] cluster similar triples to improve diversity and rank triples by their informativeness and value frequency. CD [Xu et al., 2016] formulates entity summarization as a quadratic knapsack problem to select the most informative while dissimilar triples. LinkSUM [Thalhammer et al., 2016] considers PageRank and backlinks. All of these methods are unsupervised, using various heuristics for ranking and selecting triples. By contrast, our approach is supervised, and it features a novel neural model to neatly encode structure and text in KGs.

There are also methods for summarizing entity descriptions for a particular task, e.g., entity linking [Cheng et al., 2015b], entity matching [Cheng et al., 2015a], entity search [Hasibi et al., 2017], and document enrichment [Gunaratna et al., 2017]. They focus on task-specific techniques, thus not comparable with our work. Although some of these methods also perform supervised learning, they rely on manually defined task-specific features [Zhang et al., 2012; Hasibi et al., 2017], while we employ a novel neural model to jointly extract structural and textual features from KGs. Besides, for their search task, labeled data is relatively easy to obtain from relevance judgment. However, for general-purpose entity summarization, we have to programmatically generate labeled data. Rather than extracting a subset of triples as a summary, some methods generate a textual summary using a sequence-to-sequence framework [Lebret et al., 2016; Hachey et al., 2017; Vougiouklis et al., 2018]. Their problem and techniques are fundamentally different from ours.

Our KG encoder can be viewed as a graph embedding method [Cai et al., 2018; Cui et al., 2019; Gesese et al., 2019]. Similar to RDF2Vec [Ristoski et al., 2019] and KGloVe [Cochez et al., 2017], we sample random walks to capture the graph structure of a KG. Whereas they encode walks as sequences of RDF term identifiers, we further exploit the textual form of each RDF term to jointly extract structural and textual features. This way of fusion is tighter than straightforward linear combination adopted by existing methods [Xie et al., 2016; Xu et al., 2017].

7 Conclusion

We presented NEST, a novel neural approach to generalpurpose entity summarization. Its KG encoder is pre-trained to jointly extract structural and textual features fromKGs. Its summary scorer is fine-tuned to generate high-quality diversified summaries based on programmatically labeled data. Extensive experiments have demonstrated the effectiveness and generalizability of NEST, which significantly outperforms the state of the art on two public benchmarks.

Our KG encoder has the potential to support other downstream tasks such as entity clustering and link prediction. Besides, it is possible to fine-tune our model to generate entity summaries for specific tasks (e.g., entity search) based on task-specific training data (e.g., relevance judgments).

Acknowledgements

This work was supported partially by the National Key R&D Program of China (2018YFB1005100) and the Six Talent Peaks Program of Jiangsu Province (RJFW-011).