RELATIONAL CURRICULUM LEARNING FOR GRAPH NEURAL NETWORK

Abstract

Graph neural networks have achieved great success in representing structured data and its downstream tasks such as node classification. The key idea is to recursively propagate and aggregate information along the edges of a given graph topology. However, edges in real-world graphs often have varying degrees of difficulty, and some edges may even be noisy to the downstream tasks. Therefore, existing graph neural network models may lead to suboptimal learned representations because they usually consider every edge in a given graph topology equally. On the other hand, curriculum learning, which mimics the human learning principle of learning data samples in a meaningful order, has been shown to be effective in improving the generalization ability and robustness of representation learners by gradually proceeding from easy to more difficult samples during training. Unfortunately, most existing curriculum learning strategies are designed for independent data samples and cannot be trivially generalized to handle data with dependencies. In order to address these issues, in this paper we propose a novel curriculum learning method for structured data to leverage the various underlying difficulties of data dependencies to improve the quality of learned representations on structured data. Specifically, we design a learning strategy that gradually incorporates edges in a given graph topology into training according to their difficulty from easy to hard, where the degree of difficulty is measured by a self-supervised learning paradigm. We demonstrate the strength of our proposed method in improving the generalization ability and robustness of learned representations through extensive experiments on nine synthetic datasets and nine real-world datasets with different commonly used graph neural network models as backbone models.

1. INTRODUCTION

Learning powerful representations of data samples with dependencies has become a core paradigm for understanding the underlying network mechanisms and performing a variety of downstream tasks such as social network analysis (Wasserman et al., 1994) and recommendation systems (Ying et al., 2018; Fan et al., 2019) . As a class of state-of-the-art representation learning methods, graph neural networks (GNNs) have received increasing attention in recent years due to their powerful ability to jointly model data samples and their dependencies in an end-to-end fashion. Typically, GNNs treat data samples as nodes and their dependencies as edges, and then follow a neighborhood aggregation scheme to learn data sample representations by recursively transforming and aggregating information from their neighboring samples. On the other hand, inspired by cognitive science studies (Elman, 1993; Rohde & Plaut, 1999 ) that humans can benefit from the sequence of learning basic (easy) concepts first and advanced (hard) concepts later, curriculum learning (CL) (Bengio et al., 2009; Kumar et al., 2010) suggests training a machine learning model with easy data samples first and then gradually introducing more hard samples into the model according to a designed pace, where the difficulty of samples can usually be measured by their training loss. Many previous studies have shown that this easy-to-hard learning strategy can effectively improve the generalization ability of the model (Bengio et al., 2009; Jiang et al., 2018; Han et al., 2018; Gong et al., 2016; Shrivastava et al., 2016; Weinshall et al., 2018) . Furthermore, previous studies (Jiang et al., 2018; Han et al., 2018; Gong et al., 2016) have shown that CL strategies can increase the robustness of the model against noisy training samples. An intuitive

