DECENTRALIZED KNOWLEDGE GRAPH REPRESENTATION LEARNING

Abstract

Knowledge graph (KG) representation learning methods have achieved competitive performance in many KG-oriented tasks, among which the best ones are usually based on graph neural networks (GNNs), a powerful family of networks that learns the representation of an entity by aggregating the features of its neighbors and itself. However, many KG representation learning scenarios only provide the structure information that describes the relationships among entities, causing that entities have no input features. In this case, existing aggregation mechanisms are incapable of inducing embeddings of unseen entities as these entities have no pre-defined features for aggregation. In this paper, we present a decentralized KG representation learning approach, decentRL, which encodes each entity from and only from the embeddings of its neighbors. For optimization, we design an algorithm to distill knowledge from the model itself such that the output embeddings can continuously gain knowledge from the corresponding original embeddings. Extensive experiments show that the proposed approach performed better than many cutting-edge models on the entity alignment task, and achieved competitive performance on the entity prediction task. Furthermore, under the inductive setting, it significantly outperformed all baselines on both tasks.

1. INTRODUCTION

Knowledge graphs (KGs) support many data-driven applications (Ji et al., 2020) . Recently, learning low-dimensional representations (a.k.a. embeddings) of entities and relations in KGs has been increasingly given attentions (Rossi et al., 2020) . We find that existing models for KG representation learning share similar characteristics to those for word representation learning. For example, TransE (Bordes et al., 2013) , a well-known translational KG embedding model, interprets a triple (e 1 , r, e 2 ) as e 1 + r ≈ e 2 , where e 1 , e 2 , r denote subject, object and their relationship, respectively, and the boldfaces denote the corresponding embeddings. If we view e 1 as a word in sentences, and e 2 as well as many other objects of e 1 as the context words, then TransE and many KG embedding models (Wang et al., 2014; Dettmers et al., 2018; Nguyen et al., 2018; Kazemi & Poole, 2018; Sun et al., 2019) , learn representations in a form simaliar to that used in Skip-gram (Mikolov et al., 2013a) , where the input representation is learned to predict the context (i.e., neighbors) representations. Recently, many graph neural networks (GNNs) based models for KG representation learning (Wang et al., 2018; Schlichtkrull et al., 2018; Cao et al., 2019; Wu et al., 2019; Sun et al., 2020; Vashishth et al., 2020) have achieved state-of-the-art performance in KG-related tasks such as entity alignment and entity prediction. Those models learn KG representations in a CBOW (continuous bag-of-words) (Mikolov et al., 2013a) manner, in which the context entities are aggregated to predict the target. But they also consider the representation of an entity itself when aggregating the neighborhood information. This nature prevents those models (e.g., GCN (Kipf & Welling, 2017) and GAT (Velickovic et al., 2018) ) to be generalized to represent unseen entities. In many cases, the entities in prevalent KG-related tasks do not have self features. This motivates us to learn entity representations from and only from their context neighbors. We propose a decentralized KG representation learning approach, decentRL. The key idea of decentRL is to decentralize the semantic information of entities over only their neighbors (i.e., distributed context vector in CBOW (Mikolov et al., 2013b) ), which can be easily implemented by representing each entity through averaging its neighbor embeddings. In this paper, we look for 1

