TILP: DIFFERENTIABLE LEARNING OF TEMPORAL LOGICAL RULES ON KNOWLEDGE GRAPHS

Abstract

Compared with static knowledge graphs, temporal knowledge graphs (tKG), which can capture the evolution and change of information over time, are more realistic and general. However, due to the complexity that the notion of time introduces to the learning of the rules, an accurate graph reasoning, e.g., predicting new links between entities, is still a difficult problem. In this paper, we propose TILP, a differentiable framework for temporal logical rules learning. By designing a constrained random walk mechanism and the introduction of temporal operators, we ensure the efficiency of our model. We present temporal features modeling in tKG, e.g., recurrence, temporal order, interval between pair of relations, and duration, and incorporate it into our learning process. We compare TILP with state-of-the-art methods on two benchmark datasets. We show that our proposed framework can improve upon the performance of baseline methods while providing interpretable results. In particular, we consider various scenarios in which training samples are limited, data is biased, and the time range between training and inference are different. In all these cases, TILP works much better than the state-of-the-art methods.

1. INTRODUCTION

Knowledge graphs (KGs) contain facts (e s , r, e o ) representing relation r between subject entity e s and object entity e o , e.g., (David Beckham, plays for, Real Madrid) . In real world, many relations are time-dependent, e.g., a player joining a team for a season, a politician holding a position for a certain period of time, and two persons' marriage lasting for decades. To represent the evolution and change of information, temporal knowledge graphs (tKGs) have been introduced. An interval I, indicating the valid period of the fact, is utilized by tKGs to extend the triples (e s , r, e o ) into quadruples (e s , r, e o , I), e.g., (David Beckham, plays for, Real Madrid, [2003 , 2007] ). Automatically reasoning over KGs such as link predication, i.e., inferring missing facts using existing facts, is a common task for real-world applications. However, the introduction of temporal information makes this task more difficult. The important dynamic interactions between entities can not be captured by learning methods developed for static KGs. Recently, a few embedding-based frameworks have been proposed to address the above limitation, e.g., HyTE (Dasgupta et al. (2018) ), TNTComplEx (Lacroix et al. (2020)), and DE-SimplE (Goel et al. (2019) ). The common principle adopted by these models is to create time-dependent embeddings for entities and relations. Alternatively, first-order inductive logical reasoning methods have some desirable features relative to embedding methods when applied to KGs, as they provide interpretable and robust inference results. Since the resulting logical rules contain temporal information in tKGs, we call them temporal logical rules. Some recent works, e.g., StreamLearner (Omran et al. ( 2019)), and TLogic (Liu et al. ( 2021)), have introduced a framework for temporal KG reasoning. However, there are still several unaddressed issues. First, these statistical methods count from graph the number of paths that support a given rule as its confidence estimation. As such, this independent rule learning ignores the interactions between different rules from the same positive example. For instance, given certain rules, the confidence of some rules might be enhanced, while that of others can be diminished. Sec-ond, these methods cannot deal with the similarity between different rules. Given a reliable rule, it is reasonable to believe that the confidence of another similar rule, e.g., with the same predicates but slightly different temporal patterns, is also high. However, its estimated confidence with these methods can be quite low if it is infrequent in the dataset. Finally, the performance of these timestamp-based methods on interval-based tKGs is not demonstrated. It should be noted that the temporal relations between intervals are more complex than those of timestamps. All these problems are solved by our neural-network-based framework. In this paper, we propose TILP, a differentiable inductive learning framework. TILP benefits from a novel mechanism of constrained random walk and an extended module for temporal features modeling. We achieve comparable performance to the state-of-the-art methods, while providing logical explanations for the inference results. More specifically, our main contributions are summarized as follows: • TILP, a novel differentiable and temporal inductive logic framework, is introduced based on constrained random walks on temporal knowledge graphs and temporal features modeling. It is the first differentiable approach that can learn temporal logical rules from tKGs without restrictions. • Experiments on two benchmark datasets, i.e., WIKIDATA12k and YAGO11k, are conducted, where our framework shows comparable or improved performance relative to the state-of-the-art methods. For test queries, our framework has the advantage that it provides both the ranked list of candidates and explanations for the prediction. • The superiority of our method compared to existing methods is demonstrated in several scenarios such as when training samples are limited, data is biased, and time range of training and testing are different.

2. RELATED WORKS

Embedding-based methods. Recently, embedding-based methods for tKGs started emerging for a more accurate link prediction. The common principle of these methods is to create time-dependent embeddings for entities and relations, e. 



g., HyTE (Dasgupta et al. (2018)), TA-ComplEx (García-Durán et al. (2018)), TNTComplEx (Lacroix et al. (2020)), and DE-SimplE (Goel et al. (2019)). These embeddings are plugged into standard scoring functions in most cases. Further, other works, e.g, TAE-ILP (Jiang et al. (2016)), and TimePlex (Jain et al. (2020)), have investigated the explicit temporal feature modeling, and merged it into the embedding algorithms. The main weakness of embedding-based methods is lack of interpretability, as well as their failure when previously unobserved entities, relations, or timestamps present during inference. Logical-rule-based methods. Logical-rule-based methods for link prediction on tKGs is mainly based on random walks. Although these works show the ability of learning temporal rules, they perform random walks in a very restricted manner, which impairs the quality of learned rules. For example, Dynnode2vec (Mahdavi et al. (2018)) and Change2vec (Bian et al. (2019)) both process tKGs as a set of graph snapshots at different times where random walks are performed separately. DynNetEmbedd (Nguyen et al. (2018)) requires the edges in walks to be forward in time. Stream-Learner (Omran et al. (2019)) first extracts rules from the static random walk, and then extend them separately into time domain. The consequence is that all body atoms in the extended rules have the same timestamp. TLogic (Liu et al. (2021)) is the most recent work which extracts temporal logical rules from the defined temporal random walks. The temporal constraints for temporal random walks are built on timestamps instead of intervals, and are fixed during the learning. This inflexibility impairs its ability in temporal constraints learning. Furthermore, both StreamLearner and TLogic, which can truly learn temporal logical rules, are statistical methods which estimates the rule confidence by counting the number of rule groundings and body groundings. Differentiable rule learning. Several works utilize neural network architectures for rule learning, e.g., Neural-LP (Yang et al. (2017)), NTP (Rocktäschel & Riedel (2017)), DeepProblog (Manhaeve et al. (2018)), ∂ILP (Evans & Grefenstette (2018)), RuLES (Ho et al. (2018)), IterE (Zhang et al. (2019)), dNL-ILP (Payani & Fekri (2019)) and NLProlog (Weber et al. (2019)). These works mainly focus on static KGs, lacking the ability of capturing temporal patterns. Converting from static KGs to temporal KGs is not a trivial extension. For example, Neural-LP reduces the rule learning problem to

