EQUIVARIANT HYPERGRAPH DIFFUSION NEURAL OP-ERATORS

Abstract

Hypergraph neural networks (HNNs) using neural networks to encode hypergraphs provide a promising way to model higher-order relations in data and further solve relevant prediction tasks built upon such higher-order relations. However, higher-order relations in practice contain complex patterns and are often highly irregular. So, it is often challenging to design an HNN that suffices to express those relations while keeping computational efficiency. Inspired by hypergraph diffusion algorithms, this work proposes a new HNN architecture named ED-HNN, which provably approximates any continuous equivariant hypergraph diffusion operators that can model a wide range of higher-order relations. ED-HNN can be implemented efficiently by combining star expansions of hypergraphs with standard message passing neural networks. ED-HNN further shows great superiority in processing heterophilic hypergraphs and constructing deep models. We evaluate ED-HNN for node classification on nine real-world hypergraph datasets. ED-HNN uniformly outperforms the best baselines over these nine datasets and achieves more than 2%↑ in prediction accuracy over four datasets therein.

1. INTRODUCTION

Machine learning on graphs has recently attracted great attention in the community due to the ubiquitous graph-structured data and the associated inference and prediction problems (Zhu, 2005; Hamilton, 2020; Nickel et al., 2015) . Current works primarily focus on graphs which can model only pairwise relations in data. Emerging research has shown that higher-order relations that involve more than two entities often reveal more significant information in many applications (Benson et al., 2021; Schaub et al., 2021; Battiston et al., 2020; Lambiotte et al., 2019; Lee et al., 2021) . For example, higher-order network motifs build the fundamental blocks of many real-world networks (Mangan & Alon, 2003; Benson et al., 2016; Tsourakakis et al., 2017; Li et al., 2017; Li & Milenkovic, 2017) . Session-based (multi-step) behaviors often indicate the preferences of web users in more precise ways (Xia et al., 2021; Wang et al., 2020; 2021; 2022) . To capture these higher-order relations, hypergraphs provide a dedicated mathematical abstraction (Berge, 1984) . However, learning algorithms on hypergraphs are still far underdeveloped as opposed to those on graphs. Recently, inspired by the success of graph neural networks (GNNs), researchers have started investigating hypergraph neural network models (HNNs) (Feng et al., 2019; Yadati et al., 2019; Dong et al., 2020; Huang & Yang, 2021; Bai et al., 2021; Arya et al., 2020) . Compared with GNNs, designing HNNs is more challenging. First, as aforementioned, higher-order relations modeled by hyperedges could contain complex information. Second, hyperedges in real-world hypergraphs are often of large and irregular sizes. Therefore, how to effectively represent higher-order relations while efficiently processing those irregular hyperedges is the key challenge when to design HNNs. In this work, inspired by the recently developed hypergraph diffusion algorithms (Li et al., 2020a; Liu et al., 2021b; Fountoulakis et al., 2021; Takai et al., 2020; Tudisco et al., 2021a) we design a novel HNN architecture that holds provable expressiveness to approximate a large class of hypergraph diffusion while keeping computational efficiency. Hypergraph diffusion is significant due to ) and so on. However, traditional hypergraph diffusion needs to first handcraft potential functions to model higher-order relations and then use their gradients or some variants of their gradients as the diffusion operators to characterize the exchange of diffused quantities on the nodes within one hyperedge. The design of those potential functions often requires significant insights into the applications, which may not be available in practice. We observe that the most commonly-used hyperedge potential functions are permutation invariant, which covers the applications where none of the nodes in a higher-order relation are treated as inherently special. For such potential functions, we further show that their induced diffusion operators must be permutation equivariant. Inspired by this observation, we propose a NN-parameterized architecture that is expressive to provably represent any permutation-equivariant continuous hyperedge diffusion operators, whose NN parameters can be learned in a data-driven way. We also introduce an efficient implementation based on current GNN platforms Fey & Lenssen ( 2019 To the best of our knowledge, we are the first one to establish the connection between the general class of hypergraph diffusion algorithms and the design of HNNs. Previous HNNs were either less expressive to represent equivariant diffusion operators (Feng et al., 2019; Yadati et al., 2019; Dong et al., 2020; Huang & Yang, 2021; Chien et al., 2022; Bai et al., 2021) or needed to learn the representations by adding significantly many auxiliary nodes (Arya et al., 2020; Yadati, 2020; Yang et al., 2020) . We provide detailed discussion of them in Sec. 3.4. We also show that due to the capability of representing equivariant diffusion operators, ED-HNN is by design good at predicting node labels over heterophilic hypergraphs where hyperedges mix nodes from different classes. Moreover, ED-HNN can go very deep without much performance decay. As an extra theoretical contribution, our proof of expressiveness avoids using equivariant polynomials as a bridge, which allows precise representations of continuous equivariant set functions by compositing a continuous function and the sum of another continuous function on each set entry, while previous works (Zaheer et al., 2017; Segol & Lipman, 2020) have only achieved an approximation result. This result may be of independent interest for the community. We evaluate ED-HNN by performing node classsification over 9 real-world datasets that cover both heterophilic and homophilic hypergraphs. ED-HNN uniformly outperforms all baseline methods across these datasets and achieves significant improvement (>2% ↑) over 4 datasets therein. ED-HNN also shows super robustness when going deep. We also carefully design synthetic experiments to verify the expressiveness of ED-HNN to approximate pre-defined equivariant diffusion operators.

2. PRELIMINARIES: HYPERGRAPHS AND HYPERGRAPH DIFFUSION

Here, we formulate the hypergraph diffusion problem, along the way, introduce the notations.



Figure1: Hypergraph diffusion often uses permutation-invariant hyperedge potentials to model higher-order relations. The gradients of those potentials determine the diffusion process and are termed diffusion operators. Our ED-HNN can universally represent such operators by feeding node representations into the message from hyperedges to nodes. Such small changes make big differences in model performance.

); Wang et al. (2019): We just need to combine a bipartite representation (or star expansion Agarwal et al. (2006); Zien et al. (1999), equivalently) of hypergraphs and the standard message passing neural network (MPNN) Gilmer et al. (2017). By repeating this architecture by layers with shared parameters, we finally obtain our model named Equivariant Diffusion-based HNN (ED-HNN). Fig. 1 shows an illustration of hypergraph diffusion and the key architecture in ED-HNN.

availability

//github.com/

