REVISITING EMBEDDINGS FOR GRAPH NEURAL NET-WORKS

Abstract

Current graph representation learning techniques use Graph Neural Networks (GNNs) to extract features from dataset embeddings. In this work, we examine the quality of these embeddings and assess how changing them can affect the accuracy of GNNs. We explore different embedding extraction techniques for both images and texts; and find that the choice of embedding biases the performance of different GNN architectures and thus the choice of embedding influences the selection of GNNs regardless of the underlying dataset. In addition, we only see an improvement in accuracy from some GNN models compared to the accuracy of models trained from scratch or fine-tuned on the underlying data without utilising the graph connections. As an alternative, we propose Graph-connected Network (GraNet) layers to better leverage existing unconnected models within a GNN. Existing language and vision models are thus improved by allowing neighbourhood aggregation. This gives a chance for the model to use pre-trained weights, if possible, and we demonstrate that this approach improves the accuracy compared to traditional GNNs: on Flickr v2, GraNet beats GAT2 and GraphSAGE by 7.7% and 1.7% respectively.

1. INTRODUCTION

Graph Neural Networks (GNNs) have been successful on a wide array of applications ranging from computational biology (Zitnik & Leskovec, 2017) to social networks (Hamilton et al., 2017) . The input for GNNs, although sourced from many different domains, is often data that has been preprocessed to a computationally digestible format. These digestible formats are commonly known as embeddings. Currently, improvements made to GNN architecture are tested against these embeddings and the state of the art is determined based on those results. However, this does not necessarily correlate with the GNNs accuracy on the underlying dataset and ignores the influence that the source and style of these embeddings have on the performance of particular GNN architectures. To test existing GNN architectures, and demonstrate the importance of the embeddings used in training them, we provide three new datasets each with a set of embeddings generated using different methods. We further analyse the benefit of using GNNs on fixed embeddings. We compare GNNs to standard models that have been trained or fine-tuned on the target raw data; these models treat each data point as unconnected, ignoring the underlying graph information in data. This simple unconnected baseline surprisingly outperforms some strong GNN models. This then prompts the question: Will mixing the two approaches unlock the classification power of existing unconnected models by allowing them to utilize the graph structure in our data? Based on the question above, we propose a new method of mixing GNNs with unconnected models, allowing them to train simultaneously. To achieve this we introduce a variation of the standard message passing framework. With this new framework a subset of the unconnected model's layers can each be graph-connected -exploiting useful graph structure information during the forward pass. We demonstrate that this new approach improves the accuracy of using only a pre-trained or fine-tuned model and outperforms a stand-alone GNN on a fixed embedding. We call this new approach GraNet (Graph-connected Network), and in summary, this paper has the following contributions:

