EXPRESSIVE: A SPATIO-FUNCTIONAL EMBEDDING FOR KNOWLEDGE GRAPH COMPLETION

Abstract

Knowledge graphs are inherently incomplete. Therefore substantial research has been directed toward knowledge graph completion (KGC), i.e., predicting missing triples from the information represented in the knowledge graph (KG). KG embedding models (KGEs) have yielded promising results for KGC, yet any current KGE is incapable of: (1) fully capturing vital inference patterns (e.g., composition), (2) capturing prominent patterns jointly (e.g., hierarchy and composition), and (3) providing an intuitive interpretation of captured patterns. In this work, we propose ExpressivE, a fully expressive spatio-functional KGE that solves all these challenges simultaneously. ExpressivE embeds pairs of entities as points and relations as hyper-parallelograms in the virtual triple space R 2d . This model design allows ExpressivE not only to capture a rich set of inference patterns jointly but additionally to display any supported inference pattern through the spatial relation of hyper-parallelograms, offering an intuitive and consistent geometric interpretation of ExpressivE embeddings and their captured patterns. Experimental results on standard KGC benchmarks reveal that ExpressivE is competitive with state-of-the-art KGEs and even significantly outperforms them on WN18RR.

1. INTRODUCTION

Knowledge graphs (KGs) are large collections of triples r i (e h , e t ) over relations r i ∈ R and entities e h , e t ∈ E used for representing, storing, and processing information. Real-world KGs such as Freebase (Bollacker et al., 2007) and WordNet (Miller, 1995) lie at the heart of numerous applications such as recommendation (Cao et al., 2019) , question answering (Zhang et al., 2018) , information retrieval (Dietz et al., 2018) , and natural language processing (Chen & Zaniolo, 2017). KG Completion. Yet, KGs are inherently incomplete, hindering the immediate utilization of their stored knowledge. For example, 75% of the people represented in Freebase lack a nationality (West et al., 2014) . Therefore, much research has been directed toward the problem of automatically inferring missing triples, called knowledge graph completion (KGC). KG embedding models (KGEs) that embed entities and relations of a KG into latent spaces and quantify the plausibility of unknown triples by computing scores based on these learned embeddings have yielded promising results for KGC (Wang et al., 2017) . Moreover, they have shown excellent knowledge representation capabilities, concisely capturing complex graph structures, e.g., entity hierarchies (Nickel & Kiela, 2017) . Inference Patterns. Substantial research has been invested in understanding which KGEs can capture which inference patterns, as summarized in Table 1 . For instance, KGEs such as TransE (Bordes et al., 2013) and RotatE (Sun et al., 2019) can capture fundamental patterns such as composition. Recently, however, it was discovered that these two models can only capture a fairly limited notion of composition (Zhang et al., 2019; Abboud et al., 2020; Lu & Hu, 2020; Gao et al., 2020) , cf. also Appendix K.1. Thus, multiple extensions have been proposed to tackle some of these limitations, focusing, e.g., on modeling non-commutative composition (Lu & Hu, 2020; Gao et al., 2020 ). Yet, while these extensions solved some limitations, the purely functional nature of TransE, RotatE, and any of their extensions still limits them to capture solely compositional definition, not general composition (see Table 1 for the defining formulas, and cf. also Appendix K.1 for details). Therefore, capturing general composition is still an open problem. Even more, composition patterns describe paths, which are fundamental for navigation within a graph. Hence, the ability to capture general composition is vital for KGEs. In contrast, approaches such as SimplE (Kazemi & Poole, 2018) , ComplEx (Trouillon et al., 2016), and BoxE (Abboud et al., 2020) have managed to capture other vital patterns, such as hierarchy, yet are unable to capture any notion of composition. Contribution. This paper focuses on solving all the stated limitations simultaneously. In particular: : r 1 (X, Y ) ⇒ r 1 (Y, X) ✓ ✓ ✓ ✗ ✓ ✓ Anti-symmetry: r 1 (X, Y ) ⇒ ¬r 1 (Y, X) ✓ ✓ ✓ ✓ ✗ ✓ Inversion: r 1 (X, Y ) ⇔ r 2 (Y, X) ✓ ✓ ✓ ✓ ✗ ✓ Comp. def.: r 1 (X, Y ) ∧ r 2 (Y, Z) ⇔ r 3 (X, Z) ✓ ✗ ✓ ✓ ✗ ✗ Gen. comp.: r 1 (X, Y ) ∧ r 2 (Y, Z) ⇒ r 3 (X, Z) ✓ ✗ ✗ ✗ ✗ ✗ Hierarchy: r 1 (X, Y ) ⇒ r 2 (X, Y ) ✓ ✓ ✗ ✗ ✓ ✓ Intersection: r 1 (X, Y ) ∧ r 2 (X, Y ) ⇒ r 3 (X, Y ) ✓ ✓ ✓ ✓ ✗ ✗ Mutual exclusion: r 1 (X, Y ) ∧ r 2 (X, Y ) ⇒ ⊥ ✓ ✓ ✓ ✓ ✓ ✓ Challenge. • We introduce the spatio-functional embedding model ExpressivE. It embeds pairs of entities as points and relations as hyper-parallelograms in the space R 2d , which we call the virtual triple space. The virtual triple space allows ExpressivE to represent patterns through the spatial relationship of hyper-parallelograms, offering an intuitive and consistent geometric interpretation of ExpressivE embeddings and their captured patterns. • We prove that ExpressivE can capture any pattern listed in Table 1 . This makes ExpressivE the first model capable of capturing both general composition and hierarchy jointly. • We prove that our model is fully expressive, making ExpressivE the first KGE that both supports composition and is fully expressive. • We evaluate ExpressivE on the two standard KGC benchmarks WN18RR (Dettmers et al., 2018) and FB15k-237 (Toutanova & Chen, 2015) , revealing that ExpressivE is competitive with state-of-the-art (SotA) KGEs and even significantly outperforms them on WN18RR. Organization. Section 2 introduces the KGC problem and methods for evaluating KGEs. Section 3 embeds ExpressivE in the context of related work. Section 4 introduces ExpressivE, the virtual triple space, and interprets our model's parameters within it. Section 5 analyzes our model's expressive power and inference capabilities. Section 6 discusses experimental results together with our model's space complexity and Section 7 summarizes our work. The appendix contains all proofs of theorems.

2. KNOWLEDGE GRAPH COMPLETION

This section introduces the KGC problem and evaluation methods (Abboud et al., 2020) . Let us first introduce the triple vocabulary T , consisting of a finite set of entities E and relations R. We call an expression of the form r i (e h , e t ) a triple, where r i ∈ R and e h , e t ∈ E. Furthermore, we call e h the head and e t the tail of the triple. Now, a KG G is a finite set of triples over T and KGC is the problem of predicting missing triples. KGEs can be evaluated by means of an: (1) experimental evaluation on benchmark datasets, (2) analysis of the model's expressiveness, and (3) analysis of the inference patterns that the model can capture. We will discuss each of these points in what follows. Experimental Evaluation. The experimental evaluation of KGEs requires a set of true and corrupted triples. True triples r i (e h , e t ) ∈ G are corrupted by replacing either e h or e t with any e c ∈ E such



This table lists patterns that several KGEs can capture. Specifically, ✓ represents that the pattern is supported and ✗ that it is not supported. Furthermore, "Comp. def." stands for compositional definition and "Gen. comp." for general composition.

