Learning Embeddings from Knowledge Graphs With Numeric Edge Attributes

1 Introduction

Knowledge graphs are graph-based knowledge bases whose facts are modeled as labeled, directed edges between entities. Whether it is a social network, a bioinformatics dataset, or retail purchase data, modelling knowledge as a graph lets organizations capture patterns that would otherwise be overlooked. Research led to broad-scope graphs such as DBpedia [Auer et al., 2007], WordNet, and YAGO [Suchanek et al., 2007]. Countless domain-specific knowledge graphs have also been published on the web, giving birth to the so-called Web of Data [Bizer et al., 2011].

Knowledge graph embeddings (KGE) are a family of graph representation learning methods that learn vector representations of nodes and edges of a knowledge graph. They are applied to graph completion, knowledge discovery, entity resolution, and link-based clustering, just to cite a few [Nickel et al., 2016].

In multimodal knowledge graphs, entity nodes are associated to attributes from different data modalities, such as numbers, text, images. In this paper we deal with a specific flavor of multimodal knowledge graphs, that is graphs with numeric-enriched edges, either at predicate-level (i.e. a number assigned to each predicate type), or specific to each triple (this latter case is shown in Figure 1). Numericenriched triples have a prominent role in a growing number of applicative scenarios, from protein networks to workforce knowledge bases. Such numeric values may represent edge uncertainty (i.e. uncertain knowledge graphs) as in ConceptNet [Speer et al., 2017]; trust in the automated or semi-automated data extraction workflow used to build the graph [Mitchell et al., 2018]; out-of-band knowledge from wet lab experiments [Szklarczyk et al., 2016]; edge importance or link strength.

A number of works in knowledge graph representation learning literature support multimodal information and leverage numeric values associated to node entities to generate better embeddings for enhanced link prediction [Garcia-Duran and Niepert, 2017; Kristiadi et al., 2019; Wu and Wang, 2018]. Nevertheless, such models are not designed to learn from numeric values associated to edges of a knowledge graph. With the notable exception of [Chen et al., 2019], that however is designed for uncertain graphs only, supporting numeric is to date still an under-researched direction.

In this work, we focus on the task of predicting probability estimates of missing links in knowledge graphs with numericenhanced triples. We claim that a model must take into account such numeric literals. Regardless of their semantics, we operate under the assumption that such numeric values intensify or mitigate the probability of existence of a link. That implies treating triples with low-numeric values as "pseudonegatives". Traditional embedding models put at the same level low-valued and high-valued triples. However, in many real-world scenarios it is important to make such distinction (protein interactions [Szklarczyk et al., 2016]).

We propose FocusE, an add-on layer for knowledge graph embeddings to enhance link prediction with edge-related numeric literals. FocusE works with any existing KGE model that adopts the standard negatives generation protocol [Bordes et al., 2013]. We use edge numeric literals to modulate the margin between the scores of true triples and their corresponding negative corruptions. Inspired by Focal Loss [Lin et al., 2017] that aims at sparser hard examples by modulating the loss function, we leverage numeric literals to "focus" traditional KGE models on triples with higher numeric values.

Experiments show that models trained using FocusE outperform numeric-unaware baselines, in particular in discriminating triples with high-numeric attributes from those associated to low values.

2 RelatedWork

Knowledge Graph Embeddings. Although a comprehensive survey is out of the scope of this work (recent surveys provide a good coverage of the landscape [Bianchi et al., 2020]), it is worth listing the most popular knowledge graph embedding models proposed to date. TransE [Bordes et al., 2013] is the forerunner of distance-based models, and inspired a number of models commonly referred to as TransX. The symmetric bilinear-diagonal model DistMult [Yang et al., 2015] paved the way for its asymmetric evolutions in the complex space, ComplEx [Trouillon et al., 2016] and RotatE [Sun et al., 2019]. Some models such as RESCAL [Nickel et al., 2011], TuckER [Balaˇzevi´c et al., 2019], and SimplE [Kazemi and Poole, 2018] rely on different tensor decomposition techniques. Models such as ConvE [Dettmers et al., 2018] or ConvKB [Nguyen et al., 2018] leverage convolutional layers. Attention is used by [Nathani et al., 2019].

Numeric-aware models (node attributes). None of the models listed above leverage numeric attributes of any kind. However, a number of recent works support multimodal knowledge graphs and learn from numeric values associated to node entities. LiteralE enriches node embeddings with numeric information before scoring the triples [Kristiadi et al., 2019]. KBLRN combines latent, relational and numeric features using product of experts model [Garcia-Duran and Niepert, 2017]. TransEA learns a vanilla structural model using TransE scoring, and an attribute model for attributed triples, using regression over the attribute values, which is jointly trained [Wu and Wang, 2018]. Such models are not designed to learn from numeric values associated to edges.

Numeric-aware models (edge attributes). To the best of our knowledge 1 , the only work designed to work with numeric-aware edges is UKGE [Chen et al., 2019]. UKGE generates confidence scores for known triples by squashing numeric values in the [0-1] interval. It then uses probabilistic soft logic [Kimmig et al., 2012] to predict probability estimates for unseen triples, by jointly training a model to regress over the confidence values. A limitation of this approach is that out-of-band logical rules are required as additional input. It is also worth noting that UKGE design rationale aims at supporting uncertain knowledge graphs, i.e. graphs whose edge numeric values represent uncertainty. In this paper we aim at supporting generic numeric values, regardless of their semantics (uncertainty, strength, importance, etc). Moreover, we claim that triples with high numeric values should have higher impact, and this should be explicitly modeled in the scoring or loss computation.

1 We limit to Knowledge Graph Embeddings literature and knowledge graphs, i.e. directed, labeled, multi graphs. Other numeric-aware models such as graph neural networks that operate on different data modalities are out of the scope of this paper (they are either designed for tasks other than link prediction, most of them do not support multi-relational graphs, and cannot currently be applied at the scale KGE models operate).

3 Preliminaries

Knowledge Graph. A knowledge graph G = {(s, p, o)} ⊆ E ×R× E is a set of triples t = (s, p, o) each including a subject s ∈ E, a predicate p ∈ R, and an object o ∈ E. E and R are the sets of all entities and relation types of G.

Knowledge Graph with numeric-enriched triples. In a numeric-enriched knowledge graph G, each triple is assigned a numeric attribute w ∈ R, leading to G = {t = (s, p, o, w)}. It is worth noting that we are not tied to a specific numeric attribute semantics, as these numbers may encode importance, uncertainty, strength, etc. For example, Figure 1 assumes that numeric values indicate the importance of a link. The triple (MLEngineer, requiresSkill, Python, 0.98) is therefore more important than (MLEngineer, requiresSkill, JavaScript, 0.13).

Note w can be either defined at a predicate level or at triple level. In this paper, we assume that w is triple-specific.

Knowledge Graph Embedding Models. Knowledge graph embedding models (KGE) are neural architectures designed to predict missing links between entities. KGE encode both entities E and relations R into low-dimensional, continuous vectors ∈ Rk (i.e, the embeddings). Knowledge graph embeddings are learned by training a neural architecture over a training knowledge graph: an input layer feeds training triples to an embedding lookup layer that retrieves embeddings of entities and relations.

A scoring layer f(t) assigns plausibility scores to each triple. The scoring layer is designed to assign high scores to positive triples and low scores to negative corruptions. Most of literature differs in the design rationale of f(t). For example, the scoring function of TransE [Bordes et al., 2013] computes a similarity between the embedding of the subject es translated by the embedding of the predicate ep and the embedding of the object eo.

Corruptions are synthetic negative triples generated by a corruption generation layer that follows the protocol proposed in [Bordes et al., 2013]: we define a corruption of t as t- = (s, p, o′ ) or t- = (s′ , p, o) where s′ , o′ are respectively subject or object corruptions (i.e. other entities randomly selected from E). We generate synthetic negatives by corrupting one side of the triple at a time to comply with the local closed world assumption [Nickel et al., 2016].

Finally, a loss layer optimizes the embeddings by maximizing the margin between positive triples t and corruptions t-. In other words, the goal of the optimization procedure is learning optimal embeddings, such that at inference time the scoring function f(t) assigns high scores to triples likely to be correct and low scores to triples unlikely to be true.

Link Prediction. The task of predicting unseen triples in knowledge graphs is formalized in literature as a learning to rank problem, where the objective is learning a scoring function f(t = (s, p, o)) : E × R × E → R that given an input triple t = (s, p, o) assigns a score f(t) ∈ R proportional to the likelihood that the fact t is true. Such predictions are ranked against predictions from synthetic corruptions, to gauge how well the model tells positives from negatives.

Link Prediction with numeric-enriched triples. In this paper we predict probability estimates for unseen numericenhanced triples t = (s, p, o,w). The task is formalized as the same learning to rank problem described above for conventional link prediction.

4 FocusE

We present FocusE, an add-on layer for knowledge graph embedding architectures designed for link prediction with numeric-enriched triples. FocusE takes into account numeric literals associated to each link. Regardless of their semantics, we operate under the assumption that numeric values intensify or mitigate the probability of existence of a link. For example, given numeric values w in the [0 - 1] range, we assume that high values identify triples with higher chances of being true, low scores single out weak or unlikely relations, and w = 0 triples are considered negative samples.

FocusE consists in a plug-in layer that fits between the scoring and loss layers of a conventional KGE method and it is designed to be used during training (Figure 2). Unlike traditional architectures, before feeding the scoring layer to a loss function, we modulate its output based on numeric values associated to triples, to obtain "focused" scores. We leverage numeric values associated to triples so that during training the model focuses on triples with higher numeric values. We want our model to learn from training triples with high numeric values, and at the same time use edge numeric values to maximise the margin between scores assigned to true triples and those assigned to their corruptions. This increases the loss of the model and helps it focus on triples with higher values. Our contribution is described below in detail.

Let t = (s, p, o) be a positive triple and w > 0 its numericvalue. We define a corruption of t as t- = (s, p, o′ ) or t- = (s′ , p, o). where s′ , o′ are respectively subject or object corruptions.

Let f(t) be the scoring function of a KGE model. In the case of TransE [Bordes et al., 2013] this is:

where es, rp and eo are the embeddings of the subject s, predicate p, and object o.

We use a softplus non-linearity σ to make sure the scores returned by f(t) are greater or equal to zero, without introducing excessive distortion:

To take into account the effect of numeric values associated to triples, we define a modulating factor α ∈ R which is responsible to strike a balance between the influence of the structure of the graph and the impact of the numeric values associated to each triple:

where β ∈ [0, 1] is the structural influence, an hyperparameter that modulates the influence of graph topology, and w ∈ R is the numeric value associated to the positive triple t. β is used to re-weigh the triple value w. If β = 0 the original numeric values w are used. If β = 1, numeric values w are ignored and the model is equivalent to a conventional KGE architecture. Note that positive and negative triples are assigned different α equations. This is done to lower the margin between the scores for triples and their respective corruptions when the triple numeric value is high.

Finally, the FocusE layer h(t) is defined as:

Putting all together, the FocusE layer h(t) is then used in the loss function 2 L. This is a modified, more numerically stable version of the negative log-likelihood of normalized softmax scores proposed in [Kadlec, 2017]:

2 Note that the FocusE layer described in Eq.4 is compatible with other losses used in KGE literature (e.g. pairwise, negative loglikelihood, etc.)

As seen in equation 5, the modulation between structural influence and numeric values increases the margin between high and low-valued triples. Hence the model learns to focus on triples with higher numeric attributes.

During training, we decay the structural influence β from 1 to 0. This is controlled by the hyperparameter λ (decay). Initially the model gives equal importance to all training triples. When β decays to zero (linearly, over λ epochs), the model exclusively relies on numeric attributes (w). Section 5.3 shows empirical evidence of the effectiveness of λ.

5 Experiments

We assess the predictive power of FocusE on the link prediction task with numeric-enriched triples. Experiments show that FocusE outperforms conventional KGE models and its closest direct competitor UKGE [Chen et al., 2019] in discriminating low-valued triples from high-valued ones.

Datasets

We experiment with three publicly available benchmark datasets originally proposed by [Chen et al., 2019]. Triples are associated to numeric values w interpreted as uncertainty values. We also introduce a fourth dataset, where numeric values w encode the importance of each link. All datasets include triple-specific w values. Table 1 shows the statistics of all the datasets used for the experiments.

• CN15K [Chen et al., 2019]. A subset of ConceptNet [Speer et al., 2016], a common sense knowledge graph built to represent general human knowledge. Numeric values on triples represent uncertainty.
• NL27K [Chen et al., 2019]. A subset of the NeverEnding Language Learning (NELL) [Mitchell et al., 2018] dataset, which collects data from web pages. Numeric values on triples represent link uncertainty.
• PPI5K [Chen et al., 2019]. Knowledge graph of proteinprotein interactions [Szklarczyk et al., 2016]. Numeric values represent the confidence of the link based on existing scientific literature evidence.
• O*NET20K 3 . We introduce a subset of O*NET 4 , a dataset that includes job descriptions, skills and labeled, binary relations between such concepts. Each triple is labeled with a numeric value that indicates the importance of that link. Unlike the other datasets, we built a test set that includes a single predicate type that connects jobs to skills, under the assumption that we are interested in predicting that type of links only. This is done to mock up single-target link prediction tasks adopted in applied knowledge discovery scenarios.

Implementation Details and Baselines. FocusE and all baselines are implemented with the AmpliGraph library [Costabello et al., 2019] version 1.4.0, using TensorFlow 1.15.2 and Python 3.7. Code and experiments are available at https://github.com/Accenture/AmpliGraph. We experiment with three popular KGE baselines: TransE, DistMult, ComplEx. For each baseline and for FocusE, we carried out extensive grid search, over the following ranges of hyperparameter values: embedding dimensionality k = [200 - 600], with a step of 100; baseline losses={negative log-likelihood, multiclass-NLL, self-adversarial}; synthetic negatives ratio η = {5, 10, 20, 30}; learning rate= {1e-3, 5e-3, 1e-4}; epochs= [100-800], step of 100; L3 regularizer, with weight γ = {1e-1, 1e-2, 1e-3}. For FocusE we also tuned the decay λ = [100 - 800], with increments of 100.

3 https://docs.ampligraph.org/en/latest/ampligraph.datasets.html

4 https://www.onetonline.org/

For experiments with UKGE we modified 5 the original codebase 6 provided by the authors to generate learning to rank metrics required by our experimental protocol. We used the best hyperparameter configuration proposed by the authors. Note we do not evaluate UKGE on O*NET20K, since the model requires additional logical rules which are not available for this dataset. All experiments were run under Ubuntu 16.04 on an Intel Xeon Gold 6142, 64 GB, equipped with a Tesla V100 16GB.

Evaluation protocol. We adopt the standard evaluation protocol described by [Bordes et al., 2013] to all datasets described above. We predict whether each triple t = (s, p, o) ∈ T is a positive fact, where T is a disjoint held-out test set that includes only positives triples. We cast the problem as a learning-to-rank task: for each t = (s, p, o) ∈ T , we generate synthetic negatives t- ∈ Nt by corrupting one side of the triple at a time (i.e. either the subject or the object). We predict a score for each t and all its negatives t- ∈ Nt. We then rank the only positive t against all the negatives Nt 7 . We report learning to rank metrics such as mean rank (MR), mean reciprocal rank (MRR), and Hits at n (where n = 1, 10) by filtering out spurious ground truth positives from the list of generated corruptions (i.e. "filtered" metrics).

5 The original UKGE codebase does not perform a fair rank computation: the authors assign rank=1 when t and t- have the same score, whereas we assign a lower rank, as per agreed upon practice in the community. For example, if we have η = 1, 000 corruptions twhich are all assigned the same score as t, we assign t a rank=1,000 whereas UKGE ranks it first. To guarantee a fair comparison, we aligned UKGE to our procedure.

6 https://github.com/stasl0217/UKGE

7 As for standard protocol, we used distinct entities twice, leading to CN15K=∼30k, NL27K=∼54k, PP15K=10k, O*NET20K=∼40k synthetic negatives (Table 1).

5.1 Predicting High-Valued Links

First, we assess how well FocusE predicts triples associated to high numeric values. This is an important task in many applicative scenarios: high numeric values usually imply low uncertainty, high strength or importance, thus leading to more valuable newly-discovered knowledge. Table 3 reports, for each dataset, the best mean rank, MRR, and Hits@{1,10} computed over test sets that include only triples associated to the top-10% numeric attributes. Our goal is learning to assign low ranks to test triples (i.e. rank=1 being the best outcome).

Results show that FocusE brings better or very similar MRR to traditional, numeric-unaware baselines: on O*NET20K, FocusE increases MRR for all models, and it outperforms the best baseline by 14 base points (MRR marginally differs on CN15K, NL27K, and PPI5K).

Experiments show that FocusE outperforms UKGE: on CN15K, MRR is 15 base points higher, 19 points higher on NL27K, and up to 30 points better for PPI15K (UKGErect fails to provide actionable results 8 on PPI15K). Besides, UKGE requires additional external rules, hence the absence of results for O*NET20K. FocusE achieves better predictive power, without requiring additional out-of-band rules.

5.2 Discriminating High and Low-Valued Links

The primary goal of FocusE is achieving a clear separation between scores assigned to high and low-valued triples. To assess how well FocusE can tell high-valued links from lowvalued, we look at two complementary tasks: how the model performs on test triples with high values (top 10%) and on triples with low values (bottom 10%). We collect the respective MRR results and we compute the difference ∆MRR. The higher the difference, the better. Note this task differs from what traditionally done by KGE models, which are designed to tell positives from synthetic negatives only, thus not being able to discriminate between low and high-valued triples.

Such prior art shortcoming is evident from results in Table 4. Models trained using FocusE have a much wider ∆MRR compared to traditional baselines. Using link numeric values of training triples, FocusE better discriminates triples with higher values in the test set from the ones with lower values (see also Table 2 for details on the bottom 10% links). On O*NET20K, NL27K and PPI5K, FocusE models perform exceptionally better, with ∆MRR being consistently higher than traditional KGE baselines. FocusE also outperforms UKGE by a large margin (75%) on CN15K. It provides a large margin of 40 base points on PPI5K as well, i.e. a 411% increase (UKGErect does not provide actionable results).

8 Due to a problem in the UKGE codebase, UKGE predictive power is MRR=0, hence "not actionable".

5.3 Decay Impact

The structural influence β affects the impact of numeric values w. If β = 1, FocusE falls back to the underlying KGE model. If β = 0, the original numeric values w have maximum impact throughout the entire training. During training we decay β linearly, until it reaches 0.

In this experiment we vary the decay λ, while freezing the other hyperparameters to (k = 300, η = 30, NLL of normalized softmax scores loss (Equation 5), Adam optimizer with learning rate 1e-3, L3 regularizer, with weight γ = 1e-2. We choose different decay epoch values λ = {0, 100, 200, 300, 400, 500, 600, 700, 800}. We report changes in MRR on NL27K's top 10% test set by experimenting on TransE, DistMult, and ComplEx. Figure 3 shows that performance improves if λ increases. In most cases model performance saturates when λ > 400 epochs.

6 Conclusion

We show that by plugging an additional layer we can make a conventional KGE architecture aware of numeric values associated to triples. This leads to models that better discriminate high-valued and low-valued triples, regardless of the semantics of the numeric attributes, and without requiring additional out-of-band rules (unlike UKGE).

Future work will investigate the capability of predicting numeric values associated to unseen triples. We will also extend our approach to support multiple numeric attributes associated to the same triple.