CHASING ALL-ROUND GRAPH REPRESENTATION RO-BUSTNESS: MODEL, TRAINING, AND OPTIMIZATION

Abstract

Graph Neural Networks (GNNs) have achieved state-of-the-art results on a variety of graph learning tasks, however, it has been demonstrated that they are vulnerable to adversarial attacks, raising serious security concerns. A lot of studies have been developed to train GNNs in a noisy environment and increase their robustness against adversarial attacks. However, existing methods have not uncovered a principled difficulty: the convoluted mixture distribution between clean and attacked data samples, which leads to sub-optimal model design and limits their frameworks' robustness. In this work, we first begin by identifying the root cause of mixture distribution, then, for tackling it, we propose a novel method GAME -Graph Adversarial Mixture of Experts to enlarge model capacity and enrich the representation diversity of adversarial samples, from three perspectives of model, training, and optimization. Specifically, we first propose a plug-and-play GAME layer that can be easily incorporated into any GNNs and enhance their adversarial learning capabilities. Second, we design a decoupling-based graph adversarial training in which the component of the model used to generate adversarial graphs is separated from the component used to update weights. Third, we introduce a graph diversity regularization that enables the model to learn diverse representation and further improves model performance. Extensive experiments demonstrate the effectiveness and advantages of GAME over the state-of-the-art adversarial training methods across various datasets given different attacks.

1. INTRODUCTION

Graph neural networks (GNNs) have been demonstrated to be effective at learning from graphs. They explore a message-passing mechanism to update node representations by iteratively aggregating information from their neighbors, allowing GNNs to achieve state-of-the-art performance (Kipf & Welling, 2017; Veličković et al., 2018; Hamilton et al., 2017) . Many real-world applications are based on GNNs, such as modeling over social networks (Fan et al., 2022; Zhang et al., 2019; Hu et al., 2020) , scene graph reasoning (Chen et al., 2020; Zhang et al., 2022) , and biological molecules (Jin et al., 2018; Xu et al., 2019; Guo et al., 2022) . Nevertheless, despite their outstanding performance, GNNs are susceptible to perturbations (Zügner et al., 2018b; Zügner & Günnemann, 2019; Zheng et al., 2021; Yue et al., 2022) , which necessitate techniques to leverage GNN's robustness against adversarial attacks. Attackers can downgrade the performance of GNNs from multiple perspectives, such as adding or removing edges (Geisler et al., 2021; Chen et al., 2023) , perturbing node properties (Zügner & Günnemann, 2019; Sun et al., 2020; Tian et al., 2023) , and injecting malicious nodes (Zou et al., 2021; Ju et al., 2023) . To enhance GNN's robustness, multiple defense methods against graph attacks have been proposed (Jin et al., 2020; Entezari et al., 2020; Zhang & Zitnik, 2020) . However, most existing methods have not uncovered the principled difficulty (i.e., the convoluted mixture distribution between clean and attacked data samples), which results in sub-optimal model design, poor robustness, and limited performance. In light of this, we study the robustness of GNNs from a more fundamental perspective by discovering the key pattern behind the adversarial attacks that jeopardizes the performance of GNNs. We begin by comparing the statistical differentiation between the latent representations of nodes on the clean graph and the adversarially generated graph, as shown in Figure 1 . We observe that the distributions of node representations for clean and adversarial graphs before the message passing are highly similar (i.e., Figure 1 (a)). However, as the model gets deeper, these two distributions get increasingly distinct, as demonstrated by the progressively larger shift shown from Figure 1 (a) to (c). This demonstrates that adversarial attacks imperil GNN's performance by generating adversarial graphs belonging to a distribution different from the clean graph, and the GNN model fails to transfer the knowledge learned from the clean graph to the generated adversarial graph. To address the above challenge, we propose Graph Adversarial Mixture of Experts (GAME), a novel framework that enhances the robustness for GNN by expanding the model capacity and increasing the representation diversity for adversarial graphs. Specifically, we design GAME from three perspectives: (i) To strengthen the model capacity, we propose a plug-and-play GAME layer to accommodate the adversarial graphs with diverse mixture distributions by dynamically routing multiple assembled expert networks. (ii) From the training perspective, we present a decoupling graph adversarial training strategy, namely DECOG, where each expert network is trained by adversarial graphs generated by maximizing the gradient of other experts. DECOG enforces each expert to learn distinct distributions that all other experts under-perform at. (iii) From the optimization perspective, we incorporate a graph diversity regularization (GRADIV) to further enhance the diversity of knowledge learned via all expert networks such that GAME is capable of handling various adversarial graphs. GAME is an all-round robust framework that not only improves GNN's resilience to adversarial attacks, but also without too much extra cost compared with normal GNN, since GAME dynamically activates only one subset of the experts to participate in the computation. The contributions of this paper can be summarized as follows: • To the best of our knowledge, this is the first work to improve GNN's robustness from the perspective of distribution differentiation. According to our empirical studies, existing GNNs fail to transfer the knowledge learned from one clean graph's distribution to another generated adversarial counterpart, which results in vulnerabilities to adversarial attacks. • To solve this challenge, we propose an all-round framework, namely Graph Adversarial Mixture of Experts (GAME), from the perspectives of model design (i.e., GAME layer to bolster the model capacity), training (i.e., DECOG to diversify the adversarial graphs), and optimization (i.e., GRADIV to further diversify the experts' knowledge). • Comprehensive experiments are performed on multiple benchmark datasets across varying scales, demonstrating that the robustness contributed to our proposed all-around GAME. The suggested method beats other common baselines in a variety of attack evaluations and natural evaluations, demonstrating that the all-around robust design of GAME handles intricate mixture distribution well and cleverly addresses a fundamental difficulty in graph adversarial training.

2. RELATED WORK

Graph Neural Networks. Graph Neural Networks have recently attracted a great deal of interest due to their effectiveness in learning non-Euclidean data and their remarkable performance in a vast array of graph mining tasks (Hamilton et al., 2017; Battaglia et al., 2018; Wu et al., 2020) . Graph convolutional network (GCN) is proposed in the early stage of GNN research to apply the concept of convolution from image to graph data (Kipf & Welling, 2017; Gao et al., 2018; Wu et al., 2019a) . Instead of simply averaging the features of neighboring nodes, graph attention networks (Veličković et al., 2018; Wang et al., 2019) use the attention module to value each neighboring node and learn more important nodes during message passing. Simultaneously, skip connection is introduced to construct deeper GNNs and learn comprehensive representations in order to overcome the oversmoothing phenomenon (Li et al., 2019; 2021a; b) . A concurrent work with ours exploits the sparsity property of the Mixture of Experts (MoE) mechanism to learn fairer representations on graphs (Liu et al., 2023) . Unlike prior works focusing on improving model's standard accuracy or fairness, to the best of our knowledge, our GAME is the first attempt to improve the robustness of adversarial graphs by introducing the Mixture of Experts mechanism to strengthen GNN capacity. Adversarial Learning on Graphs. It is demonstrated that deep learning models are susceptible to inputs with small adversarial perturbations, and several methods are proposed to improve the model's robustness (Goodfellow et al., 2015; Kurakin et al., 2017; Xie & Yuille, 2020) . Recent research has shown that GNNs are susceptible to adversarial attacks without exception (Zheng et al., 2021) , highlighting the urgent need to improve their robustness. Several methods are proposed for attacking graph data, including inserting or removing connections (Du et al., 2017; Chen et al., 2018; Waniek et al., 2018) , perturbing node features (Zügner et al., 2018a; Zügner & Günnemann, 2019; Sun et al., 2020) , or adding virtual nodes (Wang et al., 2020; Zou et al., 2021) . In the meantime, numerous defense methods against graph attacks have been developed for learning robust GNNs (Zhu et al., 2019; Feng et al., 2020; Jin et al., 2020) or removing the attacked input during preprocessing (Entezari et al., 2020; Zhang & Zitnik, 2020) . Go beyond prior works, this paper addresses a fundamental issue in graph adversarial learning (i.e., overly complex mixture distributions between clean and attacked nodes), subsequently improving the model capacity and representation diversity.

3. PRELIMINARIES

Mixture of Experts. In deep learning, the Mixture of Experts (MoE) constructs big neural networks with a dynamic routing strategy, which facilitates superior model capacity and attains better data parallelism (Shazeer et al., 2017) . Given the input x ∈ R d , the current expert layer E = {E i (•)} n i=1 with n experts, and the gating network P (•) = {p i (•)} n i=1 , the output of MoE module can be formulated as follows: y = i∈T p i (x)E i (x), where T represents the set of activated top-k expert indices. In the above equation, the gating module P (x) makes the activated portion of the model the same size as the normal network, hence enabling the efficient training of a larger neural network. Specifically, we calculate the gate-value p i (x) for i-th expert as follows: p i (x) = exp(h(x) i ) N j=0 exp(h(x) j ) , where h(x) is a linear transformation to compute the logits of experts given input x, and h(x) i reflects the i-th value of the obtained logits, which weights the i-th expert in current layer. Adversarial Training. Given the data distribution D over inputs x ∈ R d and their labels y ∈ R c , a standard classifier (e.g., a neural network) f : R d → R c with parameter θ maps an input to labels for classification, utilizing empirical risk minimization min θ E (x,y)∼D L(f (x; θ), y), where L(•, •) represents the cross-entropy loss. Numerous strategies have been proposed to improve neural network adversarial robustness, with adversarial training-based methods being the most effective. The majority of cutting-edge adversarial training algorithms optimize a hybrid loss consisting of a standard classification loss and an adversarial loss term: L cls = L(f (x; θ), y), L adv = max δ∈B(ϵ) L(f (x + δ; θ), y), where L cls denotes the classification loss over standard (or clean) inputs, while L adv is is the loss that encourages the model to learn from adversarial data samples, and B(ϵ) = {δ | ∥δ∥ ∞ ≤ ϵ} is the perturbation set. Popular adversarial training methods such as PGD (Madry et al., 2018) employs the same L cls as in Equation 3, but substitute L adv with a soft logits-pairing term. Thus, the overall goal of adversarial training is to minimize the following objective: 1 1 1 0   0 1 0 1   1 1 0 0   0 0 1 1   1 1 1 0   0 1 1 1   1 1 1 0   0 0 0 1 where λ is a fixed weight hyper-parameter. To balance standard and robust accuracies, the hyperparameter λ must be set to a constant value during training for all of these contemporary adversarial learning works. min θ E (x,y)∼D [(1 -λ)L cls + λL adv ], … ! * + + Mixture of Experts Gradient Update Adversarial Graph Generation maximizing ℒ !"# " ! " * ∇ $ ℒ !"# Activating Different Partial Experts ℒ !"# ℒ %&'!( ℛ )*!+,# minimizing ℒ )-./ … Gate × … % & N Layers Σ 1 1 0 1 1 0 0 0 1 " 1 1 1 0 0 1 0 1 1 1 0 0 0 0 1 1 … … " * (1) GAME layer (b) … … … … … … … Gate × … % & N Layers Σ … (2) DeCoG (3) GraDiv ∇ -ℒ !"# ℒ !"# ℒ )-./

4. METHODOLOGY

In this section, motivated by the prior analysis of adversarial mixture distribution in Section 1, we present a novel framework GAME to increase the adversarial robustness of GNNs. Figure 2 illustrates the overall framework of the proposed model. Specifically, first, our model is developed using GAME layers, which increases model capacity on adversarial mixture distribution by introducing the MoE mechanism. Second, based on the GAME layer, we design a novel DECOG training strategy that augments more varied graphs to facilitate GNNs' adversarial training. Third, in order to classify adversarial samples from mixture distribution, we suggest graph diversity regularization for learning more distinguishable representations. The details of different parts are described in the following.

4.1. EXPANDING MODEL CAPACITY: GAME LAYER

To improve the learning capacity of GNNs and account for the overly complex mixture distribution, each GNN layer in our model includes a GAME layer at the model level. To update the feature of a target node, our model employs GAME layer to transform and aggregate the features of the neighboring nodes. Subsequently, the GAME combines the feature of target node and neighboring nodes to formulate node representation in the graph: h (l) v = COMB (l) GAME (l) (h (l-1) v ), AGGR GAME (l) (h (l-1) u ), ∀u ∈ N v , where h (l) v denotes representation of node v at l-th layer; AGGR(•) and COMB(•) represent the neighbor aggregation and combination functions, respectively. In Equation 5, the GAME layer are constructed by W = {W i (•)} n i=1 and P (•), which represent the set of n expert networks and the gate module, respectively. Then, the GAME layer is formulated as follows: GAME(h) = i∈T p i (h)W i (h), where T represents the set of activated top-k experts in each GAME layer. In comparison to conventional GNNs, the proposed sparse GAME layer is able to build a wider model with low computational cost, resulting in a greater capacity to express mixed distributions, as shown in Figure 2 (1).

4.2. BOOSTING ADVERSARIAL GRAPHS: DECOG

For boosting the diversity of attacked node attributes and attacked adjacency matrices, we present Decoupling-based Graph Adversarial Training Strategy (DECOG) . DECOG aims to deliver more robust performance than traditional graph adversarial training by transferring adversarial features from all experts to each individual expert. Specifically, given a clean graph G and the node labels y with data distribution D, we first calculate the loss function over both clean graph G and attacked graph G * between input G's target nodes and their ground truth label y. Then, we learn a robust GAME model f θ with parameters θ: min θ E (G,y)∼D (1 -λ)L clean ce (f θ (G), y) + λL adv ce (f θ (G * ), y), where L clean ce (•, •), L adv ce (•, •) indicate the losses (e.g., cross-entropy for node classification) on clean graph and attacked graph, respectively, and λ regulates the weight of the adversarial loss. The adversarial graph G * is generated by GAME's customized PGD attack: during the learning procedure of PGD, GAME activates partial experts to compute multiple iterations of adversarial gradients. When the computation is finished, the gradients are added to the original graph as the final adversarial input for GNNs to minimize the adversarial learning loss. Therefore, each expert acquires the adversarial features generated by others. From a high-level vantage point, DECOG enables our GAME layer to implicitly transfer the aggregated knowledge of all experts to each individual expert. To formally describe the aforementioned pipeline, given the initial clean graph as G (0) , we first extract the initial adjacency matrix A (0) and node features X (0) from G (0) . Then, we dynamically sample the index set of activated experts T 0 using {A (0) , X (0) } to determine the GAME model's current activated part f θ ′ (0) | T 0 (•), where θ ′ (0) | T 0 is a subset of the GAME model f θ with parameters θ at the current iteration. To maximize L ce , f θ ′ (0) | T 0 (•) utilizes {A (0) , X (0) } as input. The gradients com- puted for A (0) and X (0) are represented as ∇ A (0) L ce (f θ ′ (0) (G (0) ) and ∇ X (0) L ce (f θ ′(0) (G (0) ), y), respectively. Both types of gradients are considered as adversarial noises, which are later incorporated to the current input {A (0) , X (0) } to derive the perturbed adjacency matrix and node features {A (1) , X (1) }. The procedure can be formulated as: A (t+1) = Π B(A,ϵ) (A (t) + α • sign(∇ A (t) L ce (f θ ′(t) (G (t) ), y))), X (t+1) = Π B(X,ϵ) (X (t) + α • sign(∇ X (t) L ce (f θ ′ (t) (G (t) ), y))), where B(A, ϵ) is the ℓ 0 sphere with radius around clean A, B(X, ϵ) is the ℓ ∞ sphere with radius around clean X, Π B(A,ϵ) means the nearest projection to B(A, ϵ), Π B(X,ϵ) means the nearest projection to B(X, ϵ), and α is the step size. With the aim of acquiring the adversarial graph G * = {A ( * ) , X ( * ) }, we calculate the gradients T times as in Equation ( 8) and treat final gradients as perturbations. Later, G * is utilized to train a different subset of activated experts. In general, DECOG enables the dynamic activation of each expert in GAME and facilitates the computation of more diverse attacked graph adjacency matrices and node features, as depicted in Figure 2 (2).

4.3. ENHANCING CLASSIFIABILITY IN ROBUST REPRESENTATION: GRADIV

For empowering GAME in learning more distinguishable representation from the complex mixture distribution, we design Graph Diversity Regularization (GRADIV) from the optimization level. This regularization term penalizes the model to maximize the distance between node embeddings and enforces the model to learn distinct representations. The regularization is formulated as: R GRADIV = -log N i=1 N j=1 exp(SIM(h i , h j )/τ ) N k=1 exp(SIM(h i , h k )/τ ) , k ̸ = i, where SIM(•, •) calculates the cosine similarity between two node representation, and N denotes the number of nodes. R GRADIV increases the distance between any pairs of learned node embeddings, which offers explicit supervision signal to learn node representations with more variety, and hence improving the model learning capacity. Finally, the overall learning objective function L GAME is defined as a weighted combination of L clean ce , L adv ce , and R GRADIV : L GAME = L clean ce + L adv ce + R GRADIV . (10) Consequently, L GAME enables GAME to learn distinguishable node embeddings and preserve robustness to attacked node features, thereby improving the representation quality for downstream task.

5. EXPERIMENT

In this section, we perform comprehensive experiments on the graph robustness benchmarks to demonstrate the effectiveness of our proposed GAME model against adversarial graphs with complex distributions. This section is guided by answers to the following five research questions: RQ-1: Can GAME achieve better robustness compared to other SOTAs? RQ-2: How does each component in our all-around framework contribute to the robustness improvement? RQ-3: Can GAME separate the mixed distribution of clean and attacked nodes? RQ-4: Does GAME generate more diversified training adversarial graphs compared to baselines? and RQ-5: Can GAME learn distinguishable node representations?

5.1. EXPERIMENTAL SETUP

Datasets. We utilize Graph Robust Benchmark (Zheng et al., 2021) dataset to evaluate our model's performance by graphs with varying scales, including grb-cora (small-scale), grb-citeseer (smallscale), grb-flickr (medium-scale), grb-reddit (large-scale), and grb-aminer (large-scale). Baseline Methods. We compare GAME with various baseline methods, spanning multiple perspectives. For models that specifically focus on robustness, we explore R-GCN (Zhu et al., 2019) , GNN-SVD (Entezari et al., 2020) , and GNNGuard (Zhang & Zitnik, 2020) . In addition, we incoporate general GNN models (i.e., GCN (Kipf & Welling, 2017) , GAT (Veličković et al., 2018) , GIN (Xu et al., 2019) , APPNP (Gasteiger et al., 2019a) , TAGCN (Du et al., 2017) , GraphSAGE (Hamilton et al., 2017) , SGCN (Wu et al., 2019a) ) with two generic defense approaches (i.e., layer normalization (Ba et al., 2016) (LN) and adversarial training (AT) (Madry et al., 2018) ). Attacking Strategies. We explore five effective yet diverse node injection attack methods to imperil the victim GNNs: RND (Zheng et al., 2021) , FGSM (Goodfellow et al., 2015; Zheng et al., 2021) , PGD (Madry et al., 2018) , SPEIT (Zheng et al., 2021) , TDGIA (Zou et al., 2021) . These node injection methods have been proven to deliver scalable and transferable attacks (Zheng et al., 2021) . Details of attacking strategies and adversarial training are described in Appendix B.1 and B.2, respectively. Also, we include the statistics of datasets in Appendix C. In addition, edge modification attack evaluation based on Soft-Medoid/Soft-Median (Geisler et al., 2020; 2021) is in Appendix F.

5.2. OVERALL PERFORMANCE ON GRAPH ROBUST BENCHMARK

To answer RQ-1, we conduct the experiments to evaluate the robustness and report the performance in Figure 3 (We run 10 times for mean results/standard deviation and the train:val:test split is 0.6:0.1:0.3. Due to the page limitation, we include the full results Table 2 with all numerical values in Appendix A). As shown in Figure 7 , GAME comprehensively achieves better robust accuracy than other baselines under five distinct attack assessments. For instance, on the small-scale grb-citeseer, the average accuracy of GAME outperforms the second-place R-GCN+AT by 2.96%; on the mediumscale grb-flickr, the average accuracy of our GAME outperforms the runner-up GAT+AT by 2.11%; on the large-scale grb-aminer, GAME outperforms the second-best GAT+AT by 1.32%. These results demonstrate the outstanding effectiveness of GAME against the distribution differentiation across five GRB graphs with different scales. Besides, under the w.o. attack setting, GAME significantly outperforms baselines by a large margin, proving that GAME equipped with multiple expert networks has stronger learning capability as well as higher accuracy on clean graphs. Though more parametrized than other baselines, GAME still enjoys the efficiency like regular GNNs and retains remarkable robustness against adversarial attacks, thanks to our proposed dynamic routing strategy that only activates partial experts to approximate the forward and backward of a normal GNN.

5.3. CONTRIBUTION OF INDIVIDUAL COMPONENT IN THE ALL-ROUND FRAMEWORK

GAME integrates three different components into a comprehensive graph robust learning framework, and to answer RQ-2, we conduct experiments on the performance without one of the individual components in GAME, denoted as (a) w/o GAME layer, (b) w/o DECOG, and (c) w/o GRADIV, as shown in Table 1 . We observe that the performance in both adversarial and clean graphs decreases after removing each component, demonstrating the contribution of each design in increasing the model's performance. For (a) w/o GAME layer, removing GAME layer from our framework results Figure 3 : Overall assessments of all framework across graph with different scales. We apply TDGIA, SPEIT, RND, PGD-based, and FGSM-based graph attacks to evaluate the robustness of different frameworks. w.o. attack refers to the performance on clean graphs. For better clarity, we only include strong baselines in each figure . in GAME being vanilla GCN on the model level. Such a removal disables the GRADIV training technique, as GRADIV depends on the GAME layer. Removing GAME layer decreases the performance on clean graphs (under w.o. attack) more than that on adversarial graphs. Specifically, under wo attack, this removal causes the model lose 2.58% accuracy, which is higher than the accuracy losses under two representative attacks (0.98% of PGD-based and 0.98% of FDSM-based) on grb-citeseer, respectively. For (b) w/o DECOG, disabling the DECOG training strategy results in the generated adversarial graphs tightly coupled with the activated experts at the current iteration (i.e., the activated experts for adversarial gradient computation are identical to those updated by minimizing the loss). As a result, this removal causes a more severe performance deterioration for adversarial graphs than it does for clean graphs. Specifically, under PGD-based and FDSM-based attacks, this removal causes the model on grb-citeseer to lose 0.56% and 0.75% accuracy respectively, which is comparable to the removal of GAME layer. However, the accuracy loss on wo attack is only 1.56%, which is significantly less than the accuracy loss caused by GAME layer. It demonstrates that DECOG improves the performance of GNNs against adversarial attacks by generating diverse yet effective adversarial graphs, which is also compatible with the subsequent additional studies in Section 5.5. And for (c) w/o GRADIV, GAME without the use of R GRADIV in Equation 10 leads to a scenario where actively diversifying node embeddings is no longer an explicit supervision signal during the optimization process. As a result, this removal leads to performance downgrade on both clean graph and adversarial graph (0.65% on clean grb-citeseer and 0.42% on FGSM-attacked grbciteseer, respectively), showing that the effectiveness of GRADIV in assisting GAME model to learn distinguishable representations on both clean and adversarial graphs.

5.4. PERFORMANCE AGAINST DISTRIBUTION DIFFERENTIATION

We evaluate the performance of GAME and the vanilla GCN against the attacks from graphs generated through the adversarial distributions that are extremely divergent from the distribution of clean graphs, as shown in Figure 4 . To answer RQ-3, we visualize the distributions of node representations from GCN and GAME at three stages (i.e., input layer (left column) and after first and second message passing layers (middle and right column respectively)). We observe that for the vanilla GCN, the distribution shift between the node representations in the clean and adversarial graphs still exists, even when the learning model forwards. In addition, the distribution difference between adversarial graphs and clean graphs is small. This phenomenon demonstrates that vanilla GNNs such as GCN suffer from mixture distributions between adversarial graphs and clean graphs, which prevents GCN from learning distinguishable representations. Compared with the vanilla GCN, GAME can successfully distinguish the node representations of the adversarial graphs from those of the clean graphs. Besides, as node representations pass through deeper layers, GAME still maintains the ability of discrimination, demonstrating GAME's outstanding effectiveness against adversarial graphs.

5.5. DIVERSITY OF ADVERSARIAL GRAPHS GENERATED BY GAME

(b) Ours (a) GCN To answer RQ-4, we visualize the distributions of both clean and adversarial graphs generated by GAME and GCN in Figure 5 . We observe that the adversarial graphs generated from GCN are similarly distributed to the clean graph. On the contrary, GAME generates adversarial graphs whose distribution is statistically distinct from that of the clean graphs, further demonstrating the effectiveness GCN (standard) Ours (standard) GCN (adversarial) Ours (adversarial) Figure 7 : The loss landscapes of GAME and a vanilla GCN over clean graphs (the 1st and 2nd figures) and adversarial graphs (the 3rd and 4th figures). Under both settings, we visualize the same set of nodes randomly selected from the test set of the grb-cora dataset. of DECOG and GRADIV. Diverse experts enable GAME to learn distinguishable node representations for robust performance, which significantly mitigates the GNN's training difficulties on the graphs with distinct distributions. To answer RQ-5, on grb-cora dataset, we visualize the node representations generated by GAME and a vanilla GCN, shown in Figure 6 . We observe that the node representations generated by GCN are generally entangled and intertwined with each other, while those generated by GAME exhibit a very well-clustered representation space with clear inter-cluster difference. This phenomenon demonstrates that GAME can distinguish the complex distributions and further learn distinguishable node representations, thanks to the all-around design of GAME.

5.7. ANALYSES ON OPTIMIZATION LANDSCAPE

To further validate the robustness of GAME, we analyze whether our all-round framework GAME reduces the difficulty of learning adversarial graphs by plotting its loss landscape (Li et al., 2018) w.r.t. both input and weight spaces. According to Figure 7 , we observe that GAME leads to a flatter minima optimization landscape than the adversarially trained GCN on both clean graphs and adversarial graphs, indicating that the method advantageously alleviates the learning difficulty on the adversarial graph. our GAME reduces the complexity of learning adversarial graphs, allowing adversarial training model weights to be as simple as in a standard environment rather than GCN.

6. CONCLUSION

In this paper, we first identify the fundamental issue in adversarial graph learning: the mixture distribution between clean and attacked data samples. Motivated by this problem, we propose Graph Adversarial Mixture of Experts (GAME), a novel method to improve the model capacity, augment adversarial graphs, and enrich the graph representation diversity. For acquiring these triple improvements, GAME contains three innovative components, including a plug-and-play GAME layer, a decoupling graph adversarial training strategy DECOG, and a graph diversity regularization strategy GRADIV. GAME outperforms other baselines when evaluated on multiple datasets under different attack methods. Additional experimental analysis demonstrates the effectiveness of GAME in handling the complex mixture distribution, generating distinct adversarial graphs, and learning distinguishable representations.

ETHICS STATEMENT

GAME enhances the robustness of GNN models against adversarial attacks, and therefore we believe that no ethical issues can be raised by our approach. In general, we should be very careful when applying machine learning models to ensure that there is no negative societal impact.

REPRODUCIBILITY STATEMENT

To ensure the reproducibility of our experiments, we include the link of the source code in Appendix B.2. In addition, the hyper-parameters and other factors to reproduce our method are also provided in the Appendix B.2.

A FULL RESULTS OF PERFORMANCE COMPARISON

We conduct extensive experiments on all five datasets in Table 2 . Here we display the results of graph injection scenario with Top-5 attacks v.s. Top-10 defenses plus our GAME model. Since we have chosen strong defense methods as baselines, it is generally hard for attacks to be all effectives. The best performance is bolded and the runner-up is underlined. Table 2 : Main results on Graph Robust Benchmark datasets (i.e., grb-cora, grb-citeseer, grb-flickr, grb-reddit and grb-aminer). This table is a comprehensive supplementary to Figure 3 . Partial results are cited from GRB (Zheng et al., 2021) We evaluate all the methods using five most common graph attack methods provide by GRB benchmark (Zheng et al., 2021) , including random, fast gradient sign method, projected gradient descent, SPEIT, and topological defective graph injection attack. The details of these five attack methods are as follows: • RND (Random): a method that injects random noises generated by Gaussian distribution (Zügner et al., 2018a) . • FGSM (Fast Gradient Sign Method): a method that linearizes the loss function around the current value of parameters to get an optimal max-norm constrained perturbation (Goodfellow et al., 2015) . • PGD (Projected Gradient Descent): a first-order adversary method that generates strongest assault using local first-order information about the network (Madry et al., 2018) . • SPEIT: a winning solution of KDD-CUP 2020 Graph Adversarial Attack & Defense competition, which is a global black-box graph injection attack with adversarial adjacent matrix and feature gradient attacks (Qinkai et al., 2020) . • TDGIA (Topological Defective Graph Injection Attack): a powerful graph injection attack that injects malicious nodes progressively around topologically vulnerable nodes in the graph (Zou et al., 2021) . Following the prior work (Zheng et al., 2021) , using a vanilla GCN as the surrogate model brings more transferable and better black-box node injection attack effects than other GNN models. Therefore, we choose GCN as the surrogate model for all attacks in our experiments.

B.2 HYPER PARAMETERS AND ADVERSARIAL TRAINING DETAILS

The hyper-parameters of GAME are shown in Table 3 . The Hyper-parameters for adversarial training used in DECOG are included in Table 4 . We then show the the adversarial training (AT) procedure of GAME as follows: Step ① Initialization: The warm-up step. The training graph is utilized to train GAME model for a few iterations. Step ② PGD attack: The PGD attack is employed to inject the malicious nodes and edges that assault the training nodes by message passing and create an attacked graph. Step ③ Update gradients: The model parameters are updated based on the gradients that are calculated by training on the attacked graph and minimizing the node classification loss. Step ④ Repetition: This adversarial training procedure is repeated until the training loss converges. Finally, we can obtain a GAME model with better learning capability and outstanding robustness. According to Table 2 , we intriguingly discover that GAME with PGD can also defend against other types of attacks, demonstrating the superior generality and applicability of GAME. Notably, we discover that the GAME layer's favorable attribute, i.e., sparsely activating partial experts, can be employed to design DECOG training strategy, which augments more diverse graphs. Specifically, in the Step ② and Step ③, we activate different portions of experts in GAME model, despite the fact that both steps are performed in the same epoch. The code is provided this anonymous link 1 . In addition, the suggested GAME is scalable to graphs of varying sizes. We use mini-batch training via neighborhood sampling with batchsize 1k to train GAME on grb-reddit and grb-aminer datasets. In the future, we will transfer the framework of GAME to other GNN models. 

C STATISTICS OF GRAPH ROBUST BENCHMARK DATASETS

We evaluate our proposed GAME as well as adversarial learning framework on five real-world GRB datasets (Zheng et al., 2021) , spanning from small-to large-scales. The data statistics are displayed in Table 5 . To utilize grb-cora, grb-citeseer, grb-flickr, grb-reddit, grb-aminer, we apply the tool provided by Graph Robustness Benchmarkfoot_1 .

D GRADIV'S EFFECTIVENESS ON DIVERSITY OF NODE REPRESENTATIONS

To discuss GRADIV's impact on nodes' representation diversity, on grb-cora dataset, we visualize the node representations generated by GAME without GRADIV and GAME, shown in Figure 8 . We observe that the node representations generated by GAME without GRADIV are more entangled and intertwined with each other then GAME. This phenomenon demonstrates that GRADIV is able to learn more distinguishable node representations.

E ADVERSARIAL GRAPHS GENERATED BY GAME ON GRB-FLICKR

We observe that the adversarial graphs generated from GCN are similarly distributed to the clean graph in Figure 9 (i.e.,the red and black dots in the lower left corner of the GCN figure overlap significantly, but in the GAME figure the overlap is minor). On the contrary, GAME generates adversarial



https://tinyurl.com/game23code https://github.com/thudm/grb



Figure1: The distributions of node representations generated by two GNNs trained over clean and adversarial graphs. In (a), these two distributions are extremely similar. In (b) and (c), as the model gets deeper, a progressively larger differentiation between the two distributions is observed.

Figure 2: The illustration of the GAME framework: In (1) GAME layer, partial experts (in blue shaded region) are activated to compute adversarial gradient just for adversarial graph construction without weight updates (left part). Then, (2) DECOG decouples expert weights used for adversarial graph generation and model update, where the gate module reactivates other partial experts to fit created adversarial graphs in each GAME layer (right part). Finally, (3) GRADIV regularizes multiple experts to learn knowledge with more diversity. Note that in the left part, the model maximizes adversarial loss to obtain the adversarial gradient only for graph perturbations. Next, in the right part, the model minimizes overall loss on previously generated adversarial graph for weight updates.

Figure 4: The distributions of node representations on clean and adversarial graphs (upper: adversarial trained GCN; lower: GAME model) at the input layer (left), after the first batch normalization (BN) layer (middle), and after the last batch normalization (BN) layer (right). Each point indicates the mean and variance of a single channel in the BN layer.

Figure 5: Distributions for clean and adversarial graphs created by GAME (b) and an adversarially trained GCN (a). For fair comparisons, we explore the same setting for both models.

Figure 6: t-SNE (Van der Maaten & Hinton, 2008) for the representations of nodes in the test set in grb-cora. Nodes are colored according to their class labels.

Ablation studies for GAME on graphs with varying scales. Each row includes the model variant (mean result ± standard deviation) without one of the components in our all-around design. GAME 75.80 ± 0.99 75.69 ± 1.24 76.96 ± 1.04 74.86 ± 0.73 74.85 ± 0.71 77.86 ± 0.22 76.00 ± 0.82 w/o GAME layer 73.96 ± 0.60 73.95 ± 0.34 74.21 ± 0.60 74.01 ± 0.52 73.89 ± 0.24 75.28 ± 0.09 74.22 ± 0.39

.

Hyper-parameters of GAME for grb-cora, grb-citeseer, grb-flickr, grb-reddit and grbaminer datasets. The n and k represent the number of total experts and activated experts in each layer, respectively. Note that during generating adversarial samples, we activate all experts. Noisy rate controls the randomness when the gate module activates the partial experts during minimizing the loss.

Hyper-parameters of adversarial training in DECOG for grb-cora, grb-citeseer, grb-flickr, grb-reddit and grb-aminer datasets. Noisy rate controls the randomness when the gate module maximizes the loss and activates the partial experts. Nodes represents the number of injected nodes, and Edges indicates the number of added edges. Model Datasets Attack Step size Iter. Nodes Edges Feature range Noisy rate

Statistics of five GRB datasets covering from small-to large-scale graphs.

ACKNOWLEDGMENTS

This work is partially supported by the NSF under grants IIS-2209814, IIS-2203262, IIS-2214376, IIS-2217239, OAC-2218762, CNS-2203261, CNS-2122631, CMMI-2146076, and the NIJ 2018-75-CX-0032. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.

annex

 GCN (a) . For fair comparisons, we explore the same setting for both models on grb-flickr. graphs whose distribution is statistically distinct from that of the clean graphs, further demonstrating the effectiveness of DECOG and GRADIV. Diverse experts enable GAME to learn distinguishable node representations for robust performance, which significantly mitigates the GNN's training difficulties on the graphs with distinct distributions.

F EVALUATION ON EDGE MODIFICATION ATTACK

We apply the proposed GAME framework into Soft-Medoid/Soft-Median GDCs (Geisler et al., 2020; 2021) . Note that the edge Modification attack (Geisler et al., 2020; 2021) are also crucial for assessing GNN's robustness. Therefore, we present experimental results in Table 6 , which are based on three random splits for rigorous comparisons. We observe that GAME enhances their accuracy.Table 6 : The robust accuracy of Soft-Median GDC and Soft-Medoid GDC without or with our GAME framework on the Cora dataset with the global attacks (GR-BCD & PR-BCD, ϵ = 0.1) proposed by Soft-Medoid GDC (Geisler et al., 2021) . We set the number of experts to 10 and the hidden units of each expert to 32. We run them on three random splits and report the mean and standard error results. GR-BCD (Geisler et al., 2021 ) PR-BCD (Geisler et al., 2021) GCN (Kipf & Welling, 2017) 0.622 ± 0.003 0.645 ± 0.002 GDC (Gasteiger et al., 2019b) 0.677 ± 0.005 0.674 ± 0.004 PPRGo (Bojchevski et al., 2020) 0.726 ± 0.002 0.700 ± 0.002 SVD GCN (Entezari et al., 2020) 0.755 ± 0.006 0.724 ± 0.006 Jaccard GCN (Wu et al., 2019b) 0.664 ± 0.001 0.667 ± 0.003 RGCN (Zhu et al., 2019) 0.665 ± 0.005 0.664 ± 0.004 Soft-Median GDC (Geisler et al., 2020) 0.765 ± 0.001 0.752 ± 0.002 Soft-Medoid GDC (Geisler et al., 2021) 0.775 ± 0.003 0.761 ± 0.003 Soft-Median GDC (+GAME) 0.772 ± 0.005 0.759 ± 0.005 Soft-Medoid GDC (+GAME) 0.780 ± 0.007 0.772 ± 0.006

