REG-NAS: GRAPH NEURAL NETWORK ARCHITEC-TURE SEARCH USING REGRESSION PROXY TASK

Abstract

Neural Architecture Search (NAS) has become a focus that has been extensively researched in recent years. Innovative achievements are yielded from the area like convolutional neural networks (CNN), recurrent neural networks (RNN) and so on. However, research on NAS for graph neural networks (GNN) is still in a preliminary stage. Because of the special structure of graph data, some conclusions drew from CNN cannot be directly applied to GNN. At the same time, for NAS, the models' ranking stability is of great importance for it reflects the reliability of the NAS performance. Unfortunately, little research attention has been paid to it, making it a pitfall in the development of NAS research. In this paper, we proposed a novel NAS pipeline, ReG-NAS, which balances stability, reliability and time cost to search the best GNN architecture. Besides, for the first time, we systematically analyzed factors that will affect models' ranking stability in a given search space, which can be used as a guideline for subsequent studies. Our codes are available at https://anonymous.4open.science/r/ReG-NAS-4D21 

1. INTRODUCTION

Graph neural networks (GNN) have received a lot of attention for their broad applications in social networks (Guo & Wang, 2020; Gao et al., 2021; Zhong et al., 2020) , molecule properties prediction (Shui & Karypis, 2020; Ma et al., 2020; Yang et al., 2021 ), traffic prediction (Diehl et al., 2019; Bui et al., 2021; Zhang et al., 2021) and so on. With the goal of "faster and more accurate", people always have a pursuit to find a better structure of GNN. However, similar to neural networks like CNN and RNN, searching an ideal GNN architecture manually also is challenging. Neural architecture search (NAS) for GNN is absolutely a key point to the future development of GNN. To design a NAS architecture, an intuitive yet primitive idea is to enumerate all models in a given search space, and evaluate each model's performance according to the metric specified by the downstream task (Ying et al., 2019; Dong & Yang, 2019; You et al., 2020) . However, it is extremely timeconsuming and needs a huge amount of computational resources. To make NAS more efficient, several searching methods are proposed. Most GNN NAS architectures can be divided into five classes. (1) Reinforcement-learning-based methods (Zhou et al., 2019; Gao et al., 2020; Zhao et al., 2020a) , where these architectures have controllers defined as a neural network that dynamically change the parameters according to the evaluation of the performance of the generated model; (2) Bayesian-optimization-based methods (Yoon et al., 2020; Tu et al., 2019) , which builds a probability distribution over sampled candidates and uses a surrogate function to test; (3) Evolution-learningbased methods (Shi et al., 2022; Li & King, 2020) , among which the genetic algorithm is the most commonly used for GNN NAS frameworks (Oloulade et al., 2021) . ( 4) Differentiable-search-based methods (Zhao et al., 2020b; Huan et al., 2021; Ding et al., 2021; Li et al., 2021b; Cai et al., 2021) , which learns one or two blocks that are repeated in the whole neural network, and for GNN block is generally represented as a direct acyclic graph consisting of an ordered sequence of nodes. (5) Random-search-based methods (Gao et al., 2020; Zhao et al., 2020a; Tu et al., 2019) , which generates random submodels from the search space. However, these methods are still time consuming and can take hours to days. To reduce the search time, a popular way in NAS is to use a proxy-task, usually much smaller than the groundtruth task (e.g., Cifar10 is a proxy for ImageNet). The representativeness of the proxytask is crucial, i.e., how similar results can be obtained from proxy-task and from groundtruth task.

