COLLECTIVE ROBUSTNESS CERTIFICATES: EXPLOITING INTERDEPENDENCE IN GRAPH NEURAL NETWORKS

Abstract

In tasks like node classification, image segmentation, and named-entity recognition we have a classifier that simultaneously outputs multiple predictions (a vector of labels) based on a single input, i.e. a single graph, image, or document respectively. Existing adversarial robustness certificates consider each prediction independently and are thus overly pessimistic for such tasks. They implicitly assume that an adversary can use different perturbed inputs to attack different predictions, ignoring the fact that we have a single shared input. We propose the first collective robustness certificate which computes the number of predictions that are simultaneously guaranteed to remain stable under perturbation, i.e. cannot be attacked. We focus on Graph Neural Networks and leverage their locality property -perturbations only affect the predictions in a close neighborhood -to fuse multiple single-node certificates into a drastically stronger collective certificate. For example, on the Citeseer dataset our collective certificate for node classification increases the average number of certifiable feature perturbations from 7 to 351.

1. INTRODUCTION

Most classifiers are vulnerable to adversarial attacks (Akhtar & Mian, 2018; Hao-Chen et al., 2020) . Slight perturbations of the data are often sufficient to manipulate their predictions. Even in scenarios where attackers are not present it is critical to ensure that models are robust since data can be noisy, incomplete, or anomalous. We study classifiers that collectively output many predictions based on a single input. This includes node classification, link prediction, molecular property prediction, image segmentation, part-of-speech tagging, named-entity recognition, and many other tasks. Various techniques have been proposed to improve the adversarial robustness of such models. One example is adversarial training (Goodfellow et al., 2015) , which has been applied to part-of-speech tagging (Han et al., 2020) , semantic segmentation (Xu et al., 2020b) and node classification (Feng et al., 2019) . Graph-related tasks in particular have spawned a rich assortment of techniques. These include Bayesian models (Feng et al., 2020) , data-augmentation methods (Entezari et al., 2020) and various robust network architectures (Zhu et al., 2019; Geisler et al., 2020) . There are also robust loss functions which either explicitly model an adversary trying to cause misclassifications (Zhou & Vorobeychik, 2020) or use regularization terms derived from robustness certificates (Zügner & Günnemann, 2019) . Other methods try to detect adversarially perturbed graphs (Zhang et al., 2019; Xu et al., 2020a) or directly correct perturbations using generative models (Zhang & Ma, 2020) . However, none of these techniques provide guarantees and they can only be evaluated based on their ability to defend against known adversarial attacks. Once a technique is established, it may subsequently be defeated using novel attacks (Carlini & Wagner, 2017) . We are therefore interested in deriving adversarial robustness certificates which provably guarantee that a model is robust. In this work we focus on node classification. 1 Here, the goal is to assign a label to each node in a single (attributed) graph. Node classification can be the target of either local or global adversarial attacks. Local attacks, such as Nettack (Zügner et al., 2018; Zügner et al., 2020) (left) . This is impossible in practice due to mutually exclusive perturbations. Our collective certificate enforces a single perturbed graph (center). It aggregates the amount of perturbation within each receptive field and then evaluates a single-node certificate to determine whether the corresponding prediction is robust (right). prediction of a particular node in the graph. Global attacks, as proposed by Zügner & Günnemann (2019) , attempt to alter the predictions of many nodes at once. With global attacks, the attacker is constrained by the fact that all predictions are based on a single shared input. To successfully attack some nodes the attacker might need to insert certain edges in the graph, while for another set of nodes the same edges must not be inserted. With such mutually exclusive adversarial perturbations, the attacker is forced to make a choice and can attack only one subset of nodes (see Fig. 1 ). Existing certificates (Zügner & Günnemann, 2019; Bojchevski & Günnemann, 2019; Bojchevski et al., 2020) are designed for local attacks, i.e. to certify the predictions of individual nodes. So far, there is no dedicated certificate for global attacks, i.e. to certify the predictions of many nodes at oncefoot_1 . A naïve certificate for global attacks can be constructed from existing single-node certificates as follows: One simply certifies each node's prediction independently and counts how many are guaranteed to be robust. This, however, implicitly assumes that an adversary can use different perturbed inputs to attack different predictions, ignoring the fact that we have a single shared input. We propose a collective robustness certificate for global attacks that directly computes the number of simultaneously certifiable nodes for which we can guarantee that their predictions will not change. This certificate explicitly models that the attacker is limited to a single shared input and thus accounts for the resulting mutual exclusivity of certain attacks. Specifically, we fuse multiple single-node certificates, which we refer to as base certificates, into a drastically (and provably) stronger collective one. Our approach is independent of how the base certificates are derived, and any improvements to the base certificates directly translate to improvements to the collective certificate. The key property which we exploit is locality. For example, in a k-layer message-passing graph neural network (Gilmer et al., 2017) the prediction for any given node depends only on the nodes in its k-hop neighborhood. Similarly, the predicted segment for any pixel depends only on the pixels in its receptive field, and the named-entity assigned to any word only depends on words in its surrounding. For classifiers that satisfy locality, perturbations to one part of the graph do not affect all nodes. Adversaries are thus faced with a budget allocation problem: It might be possible to attack different subsets of nodes via perturbations to different subgraphs, but performing all perturbations at once could exceed their adversarial budget. The naïve approach discussed above ignores this, overestimating how many nodes can be attacked. We design a simple (mixed-integer) linear program (LP) that enforces a single perturbed graph. It leverages locality by only considering the amount of perturbation within each receptive field when evaluating the single-node certificates (see Fig. 1 ). We evaluate our approach on different datasets and with different base certificates. We show that incorporating locality alone is sufficient to obtain significantly better results. Our proposed certificate: • Is the first collective certificate that explicitly models simultaneous attacks on multiple outputs. • Fuses individual certificates into a provably stronger certificate by explicitly modeling locality. • Is the first node classification certificate that can model not only global and local budgets, but also the number of adversary-controlled nodes, regardless of whether the base certificates support this.

2. PRELIMINARIES

Data and models. We define our unperturbed data as an attributed graph G = (X, A) ∈ G with G = {0, 1} N ×D ×{0, 1} N ×N , consisting of N D-dimensional feature vectors and a directed N ×N adjacency matrix. Each vertex is assigned one out of C classes by a multi-output classifier f : G → {1, . . . , C} N . In the following, f n (G) = f n (X, A) = f n refers to the prediction for node n. Collective threat model. Unlike previous certificates, we model an adversary that aims to change multiple predictions at once. Let B G ⊆ G be a set of admissible perturbed graphs. Given a clean graph G, the adversary tries to find a G ∈ B G that maximizes the number of misclassified nodes, i.e. n∈T 1 fn(G) =fn(G ) , for some set of target nodes T ⊆ {1, . . . , N }. Following prior work (Zügner & Günnemann, 2019) , we constrain the set of admissible perturbed graphs B G through global and (optionally) local constraints on the number of changed bits. Our global constraints are parameterized by scalars r X add , r X del , r A add , r A del ∈ N 0 . They are an upper limit on how many bits can be added (0 → 1) or deleted (1 → 0) when perturbing X and A. Our local constraints are parameterized by vectors r X add,loc , r X del,loc , r A add,loc , r A del,loc ∈ N N 0 . They are an upper limit on how many bits can be added or deleted per row of X and A, i.e. how much the attributes of a particular node can change and how many incident edges can be perturbed. Often, it is reasonable to assume that no adversary has direct control over the entire graph. Instead, a realistic attacker should only be able to perturb a small (adaptively chosen) subset of nodes. To model this, we introduce an additional parameter σ ∈ N. For all (X , A ) ∈ B G , there must be a set of node indices S ⊆ {1, . . . , N } with |S| ≤ σ such that for all d ∈ {1, . . . , D} and n, m ∈ {1, . . . , N }: X n,d = X n,d =⇒ n ∈ S ∧ A n,m = A n,m =⇒ n ∈ S ∨ m ∈ S . The set S is not fixed, but chosen by the adversary. 3 The resulting set B G is formally defined in Section B. If the global budget parameters are variable and the remaining parameters are clear from the context, we treat B G as a function B G : N 4 0 → P(G) that, given global budget parameters, returns the set of all perturbed graphs fulfilling the constraints. Local predictions. Our certificate exploits the locality of predictions, i.e. the fact that predictions are only based on a subset of the input data. We characterize the receptive field of f n via an indicator vector ψ (n) ∈ {0, 1} N corresponding to rows in attribute matrix X and an indicator matrix Ψ (n) ∈ {0, 1} N ×N corresponding to entries in adjacency matrix A. For all (X , A ), (X , A ) ∈ B G : N m=1 D d=1 ψ (n) m 1 X m,d =X m,d + N i=1 N j=1 Ψ (n) i,j 1 A i,j =A i,j = 0 =⇒ f n (X , A ) = f n (X , A ). (2) Eq. 2 enforces that as long as all nodes and edges for which ψ (n) = 1 or Ψ (n) = 1 remain unperturbed, the prediction f n does not change. Put differently, changes to the rest of the data do not affect the prediction. Note that the adversary can alter receptive fields, e.g. add edges to enlarge them. To capture all potential alterations, ψ (n) and Ψ (n) correspond to all data points that influence f n under some graph in B G , i.e. the union of all receptive fields achievable under the threat model.

3. COMPACT REPRESENTATION OF BASE CERTIFICATES

Before deriving our collective certificate, we define a representation that allows us to efficiently evaluate base certificates. A base certificate is any procedure that can provably guarantee that the prediction f n for a specific node n cannot be changed by any perturbed graph in an admissible set, such as sparsity-aware smoothing (Bojchevski et al., 2020) . As you shall see in the next section, our collective certificate requires evaluating base certificates for varying adversarial budgets within . . . , k}) , the set of vectors that do not exceed the collective global budget. L = [r X add ] × [r X del ] × [r A add ] × [r A del ] (with [k] = {0, A base certificate implicitly partitions L into a set of budgets K (n) ⊆ L for which the prediction f n is certifiably robust and its complement K (n) = L \ K (n) with K (n) ⊆ ρ ∈ N 4 0 | ρ ∈ L ∧ ∀(X , A ) ∈ B G (ρ) : f n (X, A) = f n (X , A ) . Note the subset relation. The set K (n) does not have to contain all budgets for which f n is robust. Conversely, a certain budget vector ρ not being part of K (n) does not necessarily mean that f n can be attacked under threat model B G (ρ) -its robustness is merely unknown. We now make the following natural assumption about base certificates: If a classifier is certifiably robust to perturbations with a large global budget, it should also be certifiably robust to perturbations with a smaller global budget. ∀ρ ∈ K (n) , ρ ∈ L : [∀d ∈ {1, 2, 3, 4} : ρ d ≤ ρ d ] =⇒ ρ ∈ K (n) . From a geometric point of view, Eq. 4 means that the budgets K for which the prediction f n is certifiably robust form a singular enclosed volume around (0 0 0 0) T within the larger volume L. Determining whether a classifier is robust to perturbations in B G (ρ) is equivalent to determining which side of the surface enclosing the volume K the budget vector ρ lies on. This can be be done by evaluating linear inequalities, as shown in the following. First, let us assume that all but one of the budgets are zero, e.g. L = [r X add ] × [0] × [0] × [0] , with r X add > 0. Due to Eq. 4 there must be a distinct value p n ∈ N 0 (the smallest uncertifiable budget) with ∀ρ ∈ L : ρ ∈ K (n) ⇐⇒ ρ 1 ≥ p n . Evaluating the base certificate can thus be performed by evaluating a single inequality. This approach can be generalized to arbitrary types of perturbations. Instead of using a single scalar p n , we characterize the volume of budgets K (n) via the pareto front of points on its enclosing surface: P (n) = ρ ∈ K (n) | ¬∃ρ ∈ K (n) : ρ = ρ ∧ ∀d ∈ {1, 2, 3, 4} : ρ d ≤ ρ d . (5) These points fulfill ∀ρ ∈ L ρ ∈ K (n) ⇐⇒ ∃p ∈ P (n) , ∀d ∈ {1, 2, 3, 4} : ρ d ≥ p d . Here, evaluating the base certificate can be performed by evaluating 4|P| inequalities. In the following, we assume that we are directly given this pareto front (or the smallest uncertifiable budget). Finding the pareto front can be easily implemented via a flood-fill algorithm that identifies the surface of volume K (n) , followed by a thinning operation (for more details, see Section D).

4. COLLECTIVE CERTIFICATE

To improve clarity in this section, we only discuss the global budget constraints. All remaining constraints from the threat model can be easily modelled as linear constraints. You can find the certificate for the full threat model in Section C. We first formalize the naïve collective certificate described in the introduction, which implicitly allows the adversary to use different graphs to attack different predictions. We then derive the proposed collective certificate, first focusing on attribute additions before extending it to arbitrary perturbations. We relax the certificate to a linear program to enable fast computation and show the certificate's tightness when using a randomized smoothing base certificate. We conclude by discussing the certificate's time complexity and limitations. Naïve collective certificate. Assume we are given a clean input (X, A), a multi-output classifier f , a set T of target nodes and a set of admissible perturbed graphs B G fulfilling collective global budget constraints given by r X add , r X del , r A add , r A del . Let L = [r X add ] × [r X del ] × [r A add ] × [r A del ] be the set of all vectors that that do not exceed the collective global budget. Further assume that the base certificate guarantees that each classifier f n is certifiable robust to perturbations within a set of budgets K (n) (see Eq. 3). As discussed, the naïve certificate simply counts the predictions whose robustness to perturbations from B G is guaranteed by the base certificate. Using the representation of base certificates introduced in Section 3, this can be expressed as n∈T 1 K (n) = L . From the definition of K (n) in Eq. 3, we can directly see that this is a lower bound on the optimal value of n∈T min (X ,A )∈B G 1 [f n (X, A) = f n (X , A )], i.e. the number of predictions guaranteed to be stable under attack. Note that each summand involves a different minimization problem, meaning the adversary may use a different graph to attack each of the nodes. Collective certificate for attribute additions. To improve upon the naïve certificate, we want to determine the number of predictions that are simultaneously robust to attacks with a single graph: min (X ,A )∈B G n∈T 1 [f n (X, A) = f n (X , A )] . Solving this problem is usually not tractable. For simplicity, let us assume that the adversary is only allowed to perform attribute additions. As before, we can lower-bound the indicator functions using the base certificates: min (X ,A )∈B G n∈T 1 (b 0 0 0) T ∈ K (n) (7) where b = (n,d):X n,d =0 X n,d is the number of attribute additions for a given perturbed graph. Since this certificate only depends on the number of perturbations, it is sufficient to optimize over the number of attribute additions while enforcing the global budget constraint: min b∈N0 n∈T 1 (b 0 0 0) T ∈ K (n) s.t. b ≤ r X add . There are two limitations: (1) The certificate does not account for locality, but simply considers the number of perturbations in the entire graph. In this regard, it is no different from the naïve collective certificate; (2) Evaluating the indicator functions, i.e. certifying the individual nodes, might involve complex optimization problems that are difficult to optimize through. We tackle (1) by evaluating the base certificates locally. Lemma 1 Assume multi-output classifier f , corresponding receptive field indicators ψ (n) ∈ {0, 1} N and Ψ (n) ∈ {0, 1} N ×N , and a clean graph (X, A). Let K (n) be the set of certifiable global budgets of prediction f n , as defined in Eq. 3. Let (X , A ) be a perturbed graph. Define X ∈ {0, 1} N ×D and A ∈ {0, 1} N ×N as follows: X i,d = ψ (n) i X i,d + (1 -ψ (n) i )X i,d , A i,j = Ψ (n) i,j A i,j + (1 -Ψ (n) i,j )A i,j , i.e. use values from the clean graph for bits that are not in f n 's receptive field. If there exists a vector of budgets ρ ∈ N 4 0 such that (X , A ) ∈ B G (ρ) and ρ ∈ K (n) , then f n (X, A) = f n (X , A ). See proof in Section B. Due to Lemma 1 we can ignore all perturbations outside f n 's receptive field when evaluating its base certificate. We can thus replace (b 0 0 0) T in Eq. 8 with b T ψ (n) 0 0 0 T , where the vector b ∈ N N 0 indicates the number of attribute additions at each node. Optimizing over b yields a collective certificate that accounts for locality: min b∈N0 n∈T 1 b T ψ (n) 0 0 0 T ∈ K (n) s.t. ||b|| 1 ≤ r X add . We now tackle issue (2) by employing the compact representation of base certificates defined in Section 3. Since we are only allowing one type of perturbation, the base certificate of each classifier f n is characterized by the smallest uncertifiable radius p n (see Section 3). To evaluate the indicator function in Eq. 11 we simply have to compare the number of perturbations in f n 's receptive field to p n , which can be implemented via the following MILP: min b∈N N 0 ,t∈{0,1} N |T| - n∈T t n (12) s.t. b T ψ (n) ≥ p n t n ∀n ∈ {1, . . . , N } (13) ||b|| 1 ≤ r X add . Eq. 14 ensures that the number of perturbations fulfills the global budget constraint. Eq. 13 ensures that the indicator t n can only be set to 1 if the local perturbation on the l.h.s. exceeds or matches p n , i.e. f n is not robustly certified by the base certificate. The adversary tries to minimize the number of robustly certified predictions in T (see Eq. 8), which is equivalent to Eq. 12. Collective certificate for arbitrary perturbations. Lemma 1 holds for arbitrary perturbations. We only have to consider the perturbations within a prediction's receptive field when evaluating its base certificate. However, when multiple perturbation types are allowed, the base certificate of a prediction f n is not characterized by a scalar p n , but by its pareto front P (n) (see Eq. 5). Let P (n) ∈ N |P (n) |×4 0 be a matrix encoding of the set P (n) . To determine if f n is robust we can check whether Published as a conference paper at ICLR 2021 there is some pareto-optimal point P (n) i,: such that the amount of perturbation in f n 's receptive field matches or exceeds P (n) i,: in all four dimensions. This can again be expressed as a MILP (see Eq. 15 to Eq. 23 below). As before, we use a vector t with t n = 1 indicating that f n is not certified by the base certificate. The adversary tries to find a budget allocation (parameterized by b X add , b X del and B A ) that minimizes the number of robustly certified predictions in T (see Eq. 15). Eq. 20 and Eq. 21 ensure that the budget allocation is consistent with the global budget parameters characterizing B G . The value of t n is determined by the following constraints: First, Eq. 17 to Eq. 19 ensure that Q (n) p,d is only set to 1 if the local perturbation matches or exceeds the pareto-optimal point corresponding to row p of P (n) in dimension d. The constraints in Eq. 16 implement logic operations on Q (n) : Indicator s (n) p can only be set to 1 if ∀d ∈ {1, 2, 3, 4} : Q (n) p,d = 1. Indicator t n can only be set to 1 if ∃p ∈ {1, . . . , |P (n) |} : s (n) p = 1. Combined, these constraints enforce that if t n = 1, there must be some point in P (n) that is exceeded or matched by the amount of perturbation in all four dimensions. min (Q (n) ,s (n) ) N n=1 ,b X add ,b X del ,B A ,t |T| - n∈T t n (15) s.t. ||s (n) || 1 ≥ t n , Q (n) p,d ≥ s (n) p , (b X add ) T ψ (n) ≥ Q (n) i,1 P (n) p,1 , (b X del ) T ψ (n) ≥ Q (n) p,2 P (n) p,2 , m,m ≤N (1 -A m,m )(Ψ (n) B A ) m,m ≥ Q (n) p,3 P (n) p,3 , m,m ≤N A m,m (Ψ (n) B A ) m,m ≥ Q (n) p,4 P (n) p,4 , ||b X add || 1 ≤ r X add , ||b X add || 1 ≤ r X del , (i,j):Ai,j =0 B Ai,j ≤ r A add , (i,j):Ai,j =1 B Ai,j ≤ r A del , s (n) ∈ {0, 1} |P (n) | , Q (n) ∈ {0, 1} |P (n) |×4 t ∈ {0, 1} N (22) b X add , b X del ∈ N N 0 , B A ∈ {0, 1} N ×N . LP-relaxation. For large graphs, finding an optimum to the mixed-integer problem is prohibitively expensive. In practice, we relax integer variables to reals and binary variables to [0, 1]. Semantically, the relaxation means that bits can be partially perturbed, nodes can be partially controlled by the attacker and classifiers can be partially uncertified (i.e. 1 > t n > 0). The relaxation yields a linear program, which can be solved much faster. Tightness for randomized smoothing. One recent method for robustness certification is randomized smoothing (Cohen et al., 2019) . In randomized smoothing, a (potentially non-deterministic) base classifier h : X → Y that maps from some input space X to a set of labels Y is transformed into a smoothed classifier g(x) with g(x) = argmax y∈Y Pr [h(φ(x) = y], where φ(x) is some randomization scheme parameterized by input x. For the smoothed g(x) we can then derive probabilistic robustness certificates. Randomized smoothing is a black-box method that only depends on h's expected output behavior under φ(x) and does not require any further assumptions. Building on prior randomized smoothing work for discrete data by Lee et al. (2019) , Bojchevski et al. (2020) propose a smoothing distribution and corresponding certificate for graphs. Using their method as a base certificate to our collective certificate, the resulting (non-relaxed) certificate is tight. That is, our mixed-integer collective certificate is the best certificate we can obtain for the specified threat model, if we do not use any information other than the classifier's expected predictions and their locality. Detailed explanation and proof in Section E. Time complexity. For our method we need to construct the pareto fronts corresponding to each prediction's base certificate. This has to be performed only once and the results can then be reused in evaluating the collective certificate with varying parameters. We discuss the details of this preprocessing in Section D. The complexity of the collective certificate is based on the number of constraints and variables of the underlying (MI)LP. In total, we have 13 N n=1 |P (n) | + 8N + 2e + 5 constraints and 5 N n=1 |P (n) |+4N +e variables, where e are the number of edges in the unperturbed graph (we disallow edge additions). For single-type perturbations we have O(N + e) terms, linear in the number of nodes and edges. The relaxed LP takes at most a few seconds to certify robustness for single-type perturbations and a few minutes for multiple types of perturbations (see Section 5). Limitations. The proposed approach is designed to exploit locality. Without locality, it is equivalent to a naïve combination of base certificates that sums over perturbations in the entire graph. A nonobvious limitation is that our notion of locality breaks down if the receptive fields are data-dependent and can be arbitrarily extended by the adversary. Recall how we specified locality in Eq. 2: The indicators ψ (n) and Ψ (n) correspond to the union of all achievable receptive fields. Take for example a two-layer message-passing neural networks and an adversary that can add new edges. Each node is classified based on its 2-hop neighborhood. For any two nodes n, m, the adversary can construct a graph such that m is in f n receptive field. We thus have to treat the f n as global, even if for any single graph they might only process some subgraph. Nonetheless, our method still yields significant improvements for edge deletions and arbitrary attribute perturbations. As discussed in prior work (Zügner & Günnemann, 2020) edge addition is inherently harder and less relevant in practice.

5. EXPERIMENTAL EVALUATION

Experimental setup. We evaluate the proposed approach by certifying node classifiers on multiple graphs and with different base certificates. We use 20 nodes per class to construct a train and a validation set. We certify all remaining nodes. We repeat each experiment five times with different random initializations and data splits. Unless otherwise specified, we do not impose any local budget constraints or constraints on the number of attacker-controlled nodes. We compare the proposed method with the naïve collective certificate, which simply counts the number of predictions that are certified to be robust by the base certificate. All experiments are based on the relaxed linear programming version of the certificate. We assess the integrality gap to the mixed-integer version in Section A. The code is publicly available under https://www.daml.in.tum.de/ collective-robustness/. We also uploaded the implementation as supplementary material. Datasets, models and base certificates. We train and certify models on the following datasets: Cora-ML (McCallum et al. (2000) ; Bojchevski & Günnemann (2018) ; N = 2810, 7981 edges, 7 classes), Citeseer (Sen et al. (2008) ; N = 2110, 3668 edges, 6 classes), PubMed (Namata et al. (2012) ; N = 19717, 44324 edges, 3 classes), Reuters-21578foot_3 (N = 862, 2586 edges, 4 classes) and WebKB (Craven et al. (1998) ; N = 877, 2631 edges, 5 classes). The graphs for the natural language corpora Reuters and WebKB are constructed using the procedure described in Zhou & Vorobeychik (2020) , resulting in 3-regular graphs. We use five types of classifiers: Graph convolution networks (GCN) (Kipf & Welling, 2017) , graph attention networks (GAT) (Veličković et al., 2018) , APPNP (Gasteiger et al., 2019) , robust graph convolution networks (RGCN) (Zhu et al., 2019) and soft medoid aggregation networks (SMA) (Geisler et al., 2020) . All classifiers are configured to have two layers, i.e. each node's classifier is dependent on its two-hop neighborhood. We use two types of base certificates: (Bojchevski et al., 2020) (randomized smoothing, arbitrary perturbations) and Zügner & Günnemann (2019) (convex relaxations of network nonlinearities, attribute perturbations). We provide a summary of all hyperparameters in Section F. Evaluation metrics. We report the certified ratio on the test set, i.e. the percentage of nodes that are certifiably robust under a given threat model, averaged over all data splits. We further calculate the standard sample deviation in certified ratio (visualized as shaded areas in plots) and the average wallclock time per collective certificate. For experiments in which only one global budget parameter is altered, we report the average certifiable radius, i.e. r = ( ∞ r=0 ω(r) * r/ ∞ r=0 ω(r)), where ω(r) is the certified ratio for value r of the global budget parameter, averaged over all splits. Attribute perturbations. We first evaluate the certificate for a single perturbation type. Using randomized smoothing as the base certificate, we evaluate the certified ratio of GCN classifiers for varying global attribute deletion budgets r X del on the citation graphs Cora, Citeseer and PubMed (for Reuters and WebKB, see Section A). The remaining global budget parameters are set to 0. Fig. 2 shows that for all datasets, the proposed method yields significantly larger certified ratios than the naïve certificate and can certify robustness for much larger r X del . The average certifiable radius r We compare the proposed certificate (solid lines) to the naïve certificate (dotted lines). Our method certifies orders of magnitude larger radii (note the logarithmic x-axis). on Citeseer increases from 7.18 to 351.73. The results demonstrate the benefit of using the collective certificate, which explicitly models simultaneous attacks on all predictions. The average wall-clock time per certificate on Cora, Citeseer and PubMed is 2.0 s, 0.29 s and 336.41 s. Interestingly, the base certificate yields the highest certifiable ratios on Cora, while the collective certificate yields the highest certifiable ratios on PubMed. We attribute this to differences in graph structure, which are explicitly taken into account by the proposed certification procedure. Simultaneous attribute and graph perturbations. To evaluate the multi-dimensional version of the certificate, we visualize the certified ratio of randomly smoothed GCN classifiers for different combinations of r X del and r A del on Cora-ML. For an additional experiment on simultaneous attribute additions and deletions, see Section A. Fig. 3 shows that we achieve high certified ratios even when the attacker is allowed to perturb both the attributes and the structure. Comparing the contour lines at 50 % the naïve certiface can only certify much smaller radii, e.g. at most 6 attribute deletions compared to 39 for our approach. The average wall-clock time per certificate is 106.90 s. Different classifiers. Our method is agnostic towards classifier architectures, as long as they are compatible with the base certificate and their receptive fields can be determined. In Fig. 4 we compare the certified collective robustness of GAT, GCN, and APPNP, using the sparse smoothing certificate on Cora-ML. 5 Better base certificates translate into better collective certificates. For an additional experiment on the benefits of robust classifier architectures RGCN and SMA, see Section A. Figure 7 : Certified ratios for smoothed GCN on Cora-ML. We vary r X del and σ. The certified ratios remain constant and non-zero for large r A del . Different base certificates. Our method is also agnostic to the base certificate type. We show that it works equally well with base certificates other than randomized smoothing. Specifically, we use the method from Zügner & Günnemann (2019) . We certify a GCN model for varying r X add on Citeseer. Unlike randomized smoothing, this base certificate models local budget constraints. Using the default in the base certificate's reference implementation, we limit the number of attribute additions per node to 0.01D = 21 for both the base and the collective certificate. Fig. 5 shows that the proposed collective certificate is again significantly stronger. The average certified radius r increases from 17.12 to 971.36. The average wall-clock time per certificate is 0.39 s. Local constraints. We evaluate the effect of additional constraints in our threat model. We can enforce local budget constraints and limit the number of attacker-controlled nodes, even if they are not explicitly modeled by the base certificate. In Fig. 6 , we use a smoothed GCN on Cora-ML and vary both the global budget for edge deletions, r A del , and the local budgets r A del,loc . Even though the base certificate does not support local budget constraints, reducing the number of admissible deletions per node increases the certified ratio as expected. For example, limiting the adversary to one deletion per node more than doubles the certified ratio at r A del = 1000. In Fig. 7 , we fix a relatively large local budget of 16 edge deletions per node (only ∼ 5% of nodes on Cora-ML have a degree > 16) and vary the number of attacker-controlled nodes. We see that for any given number of attacker nodes, there is some point r A del after which the certified ratio curve becomes constant. This constant value is an an upper limit on the number of classifiers that can be attacked with a given local budget and number of attacker-controlled nodes. It is independent of the global budget.

6. CONCLUSION

We propose the first collective robustness certificate. Assuming predictions based on a single shared input, we leverage the fact that an adversary must use a single adversarial example to attack all predictions. We focus on Graph Neural Networks, whose locality guarantees that perturbations to the input graph only affect predictions in a close neighborhood. The proposed method combines many weak base certificates into a provably stronger collective certificate. It is agnostic towards network architectures and base certification procedures. We evaluate it on multiple semi-supervised node classification datasets with different classifier architectures and base certificates. Our empirical results show that the proposed collective approach yields much stronger certificates than existing methods, which assume that an adversary can attack predictions independently with different graphs.

A ADDITIONAL EXPERIMENTS

Robust architectures. Our comparison of different standard classifier architectures demonstrated that the proposed collective certificate is architecture-agnostic and that better base certificates translate into better collective certificates. In Fig. 8 we assess the benefit of using SMA and RGCN, both of which are robust architectures meant to improve adversarial robustness. We use GCN as a baseline for comparison and evaluate the respective certified ratios for varying attribute deletion budgets on Cora-ML. While RGCN is supposed to be more robust to adversarial attacks, it has a lower certified ratio than GCN. Soft medoid aggregation on the other hand has a significantly higher certified ratio. Its base certificate is almost as strong as the collective certificate of RGCN. Its collective certified ratio at r X del = 21 is 88.3%, compared to the 76.9% and 74% of GCN and RGCN. Attribute perturbations on additional datasets. In addition to citation graphs, we also use graphs constructed from the Reuters-21578 and WebKB natural language corpora to evaluate the proposed certificate. As in the main experiments section, we use randomized smoothing as a base certificate for GCN classifiers and assess the certified ratio for varying global attribute deletion budgets. Fig. 9 shows that the certified ratio increases and that much larger r X del (up to approximately 10 3 ) can be certified when using the collective approach. The average certifiable radius for Reuters and WebKB increases from 6.54 and 8.08 to 265.62 and 309.32, respectively. With less than 900 nodes each, both datasets are smaller than our three citation graphs. This leads to even shorter average wallclock times per certificate: 0.105 s and 0.116 s. Simultaneous attribute deletions and additions. In the main experiments section, we applied our collective certificate to simultaneous certification of attribute and adjacency deletions. Here, we assess how it performs for simultaneous deletions and additions of attributes. We again use a randomly smoothed GCN classifier on Cora-ML, and perform collective certification for different combinations of r X add and r X del on Cora-ML. As shown in Fig. 3 , the collective certificate is again much stronger than the naïve collective certificate. For example, we obtain certified ratios between 30% and 60% at radii for which the naïve collective certificate cannot certify any robustness at all. The average wall-clock time per certificate is 40.51 s. Integrality gap. For all previous experiments, we used the relaxed linear programming version of the certificate to reduce the compute time. To assess the integrality gap (i.e. the difference between the mixed-integer linear programming and the linear programming based certificates), we apply both versions of the certificate to a single smoothed GCN on Cora. We certify both robustness to attribute deletions (Fig. 11a ) and edge deletions (Fig. 11b ). The wall-clock time per certificate for the MILP increased from 0.24 s to 64 h with increasing edge deletion budget (0.35 s to 94 h for attribute deletions). Due to the exploding runtime for the MILP, we cannot compute the integrality gap for radii larger than 8 and 12, respectively. 6 The integrality gap is small (at most 4% for attribute deletions, 4.3% for edge deletions), relative to the certified ratio, and appears to be slightly increasing with increasing global budget. 10a ) to the naïve collective certificate (Fig. 10b ) for certification of smoothed GCN on Cora-ML, under varying r X add and r X del . Our method achieves much larger certified ratios for all combinations of attack radii. Figure 11 : Certified ratios for smoothed GCN on Cora, under varying r X del (Fig. 11a ) and r A del (F ig. 11b), using the mixed-integer collective certificate (blue line) and the relaxed linear programming certificate (orange line). The integrality gap is small, relative to the certified ratio.

B FORMAL DEFINITION OF THREAT MODEL PARAMETERS AND PROOFS

Here we define formally the set of admissible perturbed graphs B G described in Section 2. Recall that we have an unperturbed graph (X, A) ∈ G, global budget parameters r X add , r X del , r A add , r A del ∈ N 0 , local budget parameters r X add,loc , r X del,loc , r A add,loc , r A del,loc ∈ N N 0 and at most σ adversarycontrolled nodes. Given these parameters, the set of admissible perturbed graphs B G is defined as follows: (X , A ) ∈ B G =⇒ (n, d) : X n,d = 0 = X n,d ≤ r X add ∧ (n, d) : X n,d = 1 = X n,d ≤ r X del ∧ (n, m) : A n,m = 0 = A n,m ≤ r A add ∧ (n, m) : A n,m = 1 = A n,m ≤ r A del ∧ ∀n ∈ {1, . . . , N } : d : X n,d = 0 = X n,d ≤ r X add,loc n ∧ d : X n,d = 1 = X n,d ≤ r X del,loc n ∧ m : A n,m = 0 = X n,m ≤ r A add,loc n ∧ m : A n,m = 1 = X n,m ≤ r A del,loc n ∧ ∃S ⊆ {1, . . . , N } : |S| ≤ σ ∧ ∀d ∈ {1, . . . , D}, n, m ∈ {1, . . . , N } : X n,d = X n,d =⇒ n ∈ S ∧ A n,m = A n,m =⇒ n ∈ S ∨ m ∈ S . Next, we show the proof of Lemma 1 delegated from the main paper. Proof: By the definition of receptive fields (Eq. 2) changes outside the receptive field do not influence the prediction, i.e. f n (X , A ) = f n (X , A ). Since (X , A ) ∈ B G (ρ) and ρ ∈ K (n) , we know that f n (X, A) = f n (X , A ). By transitivity f n (X, A) = f n (X , A ) .

C FULL COLLECTIVE CERTIFICATE

Here we discuss how to incorporate local budget constraints and constraints on the number of attacker-controlled nodes into the collective certificate for global budget constraints (see Eq. 15). We also discuss how to adapt it to undirected adjacency matrices (see Section C.1) As before, we lower-bound the true objective min (X ,A )∈B G n∈T 1 [f n (X, A) = f n (X , A )] . by replacing the indicator functions with evaluations of the corresponding base certificates and optimizing over the number of perturbations per node / the perturbed edges. The only difference is that more constraints are imposed on B G . The derivation proceeds as before: Lemma 1 still holds, meaning we only have to consider perturbations within f n 's receptive field in evaluating whether its robustness is guaranteed by the base certificate. The base certificates can be efficiently evaluated by comparing the perturbation within f n 's receptive field to all points in the pareto front P (n) characterizing the volume of budgets K (n) . After encoding P (n) as a matrix P (n) ∈ N |P (n) ×4 , we can solve the following optimization problem to obtain a lower bound on Eq. 25: min (Q (n) ,s (n) ) N n=1 ,b X add ,b X del ,B A ,t |T| - n∈T t n (26) s.t. ||s (n) || 1 ≥ t n , Q (n) p,d ≥ s (n) p , (b X add ) T ψ (n) ≥ Q (n) i,1 P (n) p,1 , (b X del ) T ψ (n) ≥ Q (n) p,2 P (n) p,2 , m,m ≤N (1 -A m,m )(Ψ (n) B A ) m,m ≥ Q (n) p,3 P (n) p,3 , m,m ≤N A m,m (Ψ (n) B A ) m,m ≥ Q (n) p,4 P (n) p,4 , ||b X add || 1 ≤ r X add , ||b X add || 1 ≤ r X del , (i,j):Ai,j =0 B Ai,j ≤ r A add , (i,j):Ai,j =1 B Ai,j ≤ r A del , b X add n ≤ a n r X add,loc n , b X del n ≤ a n r X del,loc n , m:An,m=0 B An,m + B Am,n ≤ r A add,loc n , m:An,m=1 B An,m + B Am,n ≤ r A del,loc n (34) B i,j ≤ a i + a j ∀i, j ∈ {1, . . . , N }, (35) ||a|| 1 ≤ σ, s (n) ∈ {0, 1} |P (n) | , Q (n) ∈ {0, 1} |P (n) |×4 , t ∈ {0, 1} N , ( ) a ∈ {0, 1} N , b X add , b X del ∈ N N 0 , B A ∈ {0, 1} N ×N , ∀n ∈ {1, . . . , N }, p ∈ {1, . . . , |P (n) |}, d ∈ {1, . . . , 4}. The constraints from Eq. 27 to Eq. 30 are identical to constraints Eq. 16 to Eq. 19 of our collective certificate for global budget constraints. They simply implement boolean logic to determine whether there is some pareto-optimal p ∈ P(n) such that the perturbation in f n 's receptive field matches or exceeds p in all four dimensions. If this is the case, the base certificate cannot certify the robustness of f n and t n can be set to 1. Eq. 31 and Eq. 32 enforce the global budget constraints. The difference to the global budget certificate lies in Eq. 33 to Eq. 36. We introduce an additional variable vector a ∈ {0, 1} N that indicates which nodes are attacker controlled. Eq. 33 enforces that the attributes of node n remain unperturbed, unless a n = 1. If a n = 1, the adversary can add or delete at most r X del,loc n or r X del,loc n attribute bits. With edge perturbations, it is sufficient for either incident node to be attacker-controlled. This is expressed via Eq. 35. The number of added or deleted edges incident to node n is constrained via Eq. 34. Finally, Eq. 36 ensures that at most σ nodes are attacker-controlled.

C.1 UNDIRECTED ADJACENCY MATRIX

To adapt our certificate to undirected graphs, we simply change the interpretation of the indicator matrix B A . Now, setting either B Ai,j or B Aj,i to 1 corresponds to perturbing the undirected edge {i, j}. An edge should not be perturbed twice, which we express through an additional constraint: B Ai,j + B Aj,i ≤ 1 ∀i, j ∈ {1, . . . , N }. (39) We further combine Eq. 34, which enforced that at least one of the incident nodes of a perturbed edge has to be attacker controlled, and Eq. 35, which enforced the local budgets for edge perturbations, into following constraints: m:An,m=0 B An,m ≤ a n r A add,loc n (40) m:An,m=1 B An,m ≤ a n r A del,loc n (41) These changes do not affect the optimal value of the mixed-integer linear program. But they are more effective than Eq. 34 and Eq. 35 when solving the relaxed linear program and the nodes' local budgets are small relative to their degree.

D DETERMINING THE PARETO FRONT OF BASE CERTIFICATES

For our collective certificate, we assume that base certificates directly yield the pareto front P (n) of points enclosing the volume of budgets K (n) for which the prediction f n is certifiably robust: In practice, finding this representation requires some additional processing which we shall discuss in this section. P (n) = ρ ∈ K (n) | ¬∃ρ ∈ K (n) : ρ = ρ ∧ Existing certificates for graph-structured data are methods that determine for a specific budget ρ ∈ L whether a classifier f n is robust to perturbations in B G (ρ). In other words: They can only test the membership relation ρ ∈ K (n) . One possible way of finding the pareto front is through the following three-step process: 1. Use a flood-fill algorithm starting at (0 0 0 0) T to determine K (n) . 2. Identify all points in K (n) = L \ K (n) that enclose the volume K (n) . 3. Remove all enclosing points that are not pareto-optimal. A pseudo-code implementation is provided in algorithm 1. It has a running time in O c K (n) , where c is the worst-case case of performing a membership test ρ ∈ K (n) .



While we focus on node classification, our approach can easily be applied to other multi-output classifiers. Chiang et al. (2020) certify multi-object detection, but they still treat each detected object independently. Note that the adversary only needs to control one of the nodes incident to an edge in order to perturb it. We do not associate different costs for perturbing different nodes or edges, but such an extension is straightforward. Distribution 1.0, available from http://www.daviddlewis.com/resources/testcollections/reuters21578 Our method can certify larger radii, r ≥ 10 3 (see Fig.2). Here we show r ≤ 20 to highlight the difference. As discussed in the main paper the relaxed LP is fast and efficient to solve, even for radii larger than 1000.



Figure 2: Certified ratios for smoothed GCN on Cora, Citeseer and PubMed, under varying r X del .We compare the proposed certificate (solid lines) to the naïve certificate (dotted lines). Our method certifies orders of magnitude larger radii (note the logarithmic x-axis).

Figure3: Two-dimensional colective certificate for smoothed GCN on Cora-ML under varying r X del and r A del . The solid and dotted contour lines show ratios ≥ 0.5 and ≥ 0.7 for our vs. the naïve certificate respectively. Our method achieves much larger certified ratios and radii.

Figure 6: Certified ratios for smoothed GCN on Cora-ML, under varying r A del and r A del,loc . Stricter local budgets yield larger certified ratios.

Figure8: Certified ratios for smoothed GCN, RGCN and soft medoid aggregation on Cora-ML under varying r X del for our (solid lines) and the naïve (dotted lines) collective certificate.

Figure10: Comparison of the proposed collective certificate (Fig.10a) to the naïve collective certificate (Fig.10b) for certification of smoothed GCN on Cora-ML, under varying r X add and r X del . Our method achieves much larger certified ratios for all combinations of attack radii.

∀d ∈ {1, 2, 3, 4} :ρ d ≤ ρ d (42) with K (n) ⊆ ρ ∈ N 4 0 | ρ ∈ L ∧ ∀(X , A ) ∈ B G (ρ) : f n (X, A) = f n (X , A ) ,(43)K (n) = L\K (n) and L = [r X add ]×[r X del ]×[r A add ]×[r A del ] (with [k] ={0, . . . , k} (see Section 3).

, attempt to alter the Previous certificates consider each node independently. Most nodes cannot be certified since the adversary can choose a different perturbed graph per node

7. ACKNOWLEDGEMENTS

This research was supported by the German Research Foundation, Emmy Noether grant GU 1409/2-1, the German Federal Ministry of Education and Research (BMBF), grant no. 01IS18036B, and the TUM International Graduate School of Science and Engineering (IGSSE), GSC 81.

annex

Algorithm 1: Determining the pareto front of base certificates Result: Set P (n) of pareto-optimal points enclosing the volume of budgets K (n) (n) then pareto optimal ← false ; //Pareto-optimality does not allow decreasing values while staying in K (n) break; end end if pareto optimal then P (n) ← P (n) ∪ {ρ} end end

E TIGHTNESS FOR RANDOMIZED SMOOTHING

In this section we prove that if we use the randomized smoothing based certificate from Bojchevski et al. (2020) as our base certificate, then our collective certificate is tight: If we do not make any further assumptions, outside each smoothed classifier's receptive field and its expected output behavior under the smoothing distribution, we cannot obtain a better collective certificate. We define the base certificate and then provide a constructive proof of the resulting collective certificate's tightness.E.1 RANDOMIZED SMOOTHING FOR SPARSE DATA Bojchevski et al. (2020) provide a robustness certificate for classification of arbitrary sparse binary data. Applied to node classification, it can be summarized as follows:Assume we are given a multi-output classifier h : G → {1, . . . , C} N . Define a smoothed classifier f : G → {1, . . . , C} N withwhere φ attr and φ adj are two independent randomization schemes that assign probability mass to the set of attribute matrices {0, 1} N ×D and adjacency matrices {0, 1} N ×N , respectively. The ran-Published as a conference paper at ICLR 2021 domization schemes are defined as follows:Each bit's probability of being flipped is dependent on its current value, but independent of the other bits.An adversarially perturbed graph (X , A ) is successful in changing prediction y n = f n (X, A), if Optimizing over the set of all possible classifiers might appear hard. We can however use the approach of Lee et al. (2019) to find an optimum. Let X , A be a graph that results from b X add attribute additions, b X del attribute deletions, b A add edge additions, and b A del edge additions applied to (X, A). We can partition the set of all graphs G into (bregions that have a constant likelihood ratio under our smoothing distribution:where the η •,• ∈ R + are constants. The regions have a particular semantic meaning, which will be important for our later proof: Any (X , A ) ∈ J q X ,q A has q X attribute bits and q A adjacency bits that have the same value in (X, A), and a different value in (X , A ):As proven by Lee et al. (2019) , we can find an optimal solution to Eq. 48 by optimizing over the expected output of h within each region of constant likelihood ratio. This can be implemented via the following linear program:Any optimal solution H(n) corresponds to a single-output classifier hn that, given an input graph (X , A ), simply counts the number of attribute bits q X and adjacency bits q A that have the same value in (X, A) and a different value in (X , A ) and then assigns a probability of H(n) q X ,q A to class f n and 1 -H (n) q X ,q A to the remaining classes. The optimal value of Eq. 53 being larger than 0.5 for a fixed perturbed graph (X , A ) only proofs that this particular graph is not a successful attack on f n . For a robustness certificate, we want to know the result for a worst-case graph. However, the result is only dependent on the number of perturbations b X add , b X del , b A add and b A del , and not on which specific bits are perturbed. Therefore, we can solve the problem for an arbitrary fixed perturbed graph with the given number of perturbations, and obtain a valid robustness certificate.For use in our collective certificate, we define the set of budgets K (n) for which prediction f n is certifiably robust aswhere Λ n is defined as in Eq. 53 andis the set of vectors that do not exceed the available collective budget (see Section 3).

E.2 TIGHTNESS PROOF

With the definition of our base certificate in place, we can now formalize and prove that the resulting collective certificate is tight. Recall that randomized smoothing is a black-box method. The classifier that is being smoothed is treated as unknown. A robustness certificate based on randomized smoothing has to account for the worst-case (i.e. least robust under the given threat model) classifier. Our collective certificate lower-bounds the number of predictions that are guaranteed to be simultaneously robust. We show that with the randomized smoothing base certificate from the previous section, it actually yields the exact number of robust predictions, assuming the worst-case unsmoothed classifier.Theorem 1 Let (X, A) be an unperturbed graph. Let h : G → {1, . . . , C} N be a (potentially non-deterministic) multi-output classifier. Let f : G → {1, . . . , C} be the corresponding smoothed classifier with(59) and randomization schemes φ attr (X), φ adj (A) defined as in Eq. 45 and Eq. 46. Let ψ ∈ {0, 1} N , Ψ ∈ {0, 1} N ×N be receptive field indicators corresponding to f n (see Eq. 2). Let B G be a set of admissible perturbed graphs, constrained by parameters r X add , r X del , r A add , r A del , r X add,loc , r X del,loc , r A add,loc , r A del,loc , σ, as defined in Section 2. Let T be the indices of nodes targeted by an adversary. Under the given parameters, let o * be the optimal value of the optimization problem defined in Section C.Then there are a perturbed graph (X , A ), a non-deterministic multi-output classifier h and a corresponding smoothed multi-output classifier f with fn (X, A) = argmax c∈{1,...,C} Pr hn (φ attr (X), φ adj (A)) = c ∀n ∈ {1, . . . , N } (60)Pr hn (φ attr (X),and each fn is only dependent on nodes and edges for which ψ (n) and Ψ (n) have value 1.

Proof:

The optimization problem from Section C has three parameters b X add , b X del , B A , which specify the budget allocation of the adversary. Let b * X add , b * X del , B * A be their value in the optimum. We can construct a perturbed graph (X , A ) from the clean graph (X, A) as follows: For every node n, set the first b * X add n zero-valued bits to one and the first b X del * n non-zero bits to zero. Then, flip any entry (n, m) of A n,m for which B * An,m = 1. The parameters b * X add , b * X del , B * A are part of a feasible solution to the optimization problem. In particular, they must fulfill constraints Eq. 31 to Eq. 36, which guarantee that the constructed graph is in B G .Given the perturbed graph (X , A ), we can calculate the amount of perturbation in the receptive field of each prediction f n :We can now specify the unsmoothed multi-output classifier h. Recall that in the collective certificate's optimization problem, each f n is associated with a binary variable t n . In the optimum, n) , i.e. f n 's robustness is guaranteed by the base certificate, andTrivially, constraint Eq. 62 is fulfilled since fn is only dependent on nodes and edges for which ψ (n) and Ψ (n) have value 1. Whether f is adversarially attacked or not does not influence Eq. 61, as n / ∈ T. Case 2: n ∈ T and t * n = 0. Choose hn = h n . Again, constraint Eq. 62 is fulfilled since fn is only dependent on nodes and edges for which ψ (n) and Ψ (n) have value 1. Since t * n = 0, we know that u n) , i.e. fn (X , A ) = y n . Case 3: n ∈ T and t * n = 1. Since t * n = 1, we know that f n is not certified by the base certificate:be the optimum of the linear program underlying the base certificate (see Eq. 53 to Eq. 55). Define hn to have the following non-deterministic output behavior:for some y n = y n and with)As discussed at the end of Section E.1, this classifier simply counts the number of bits in (X , A ) that are within f n 's receptive field and have the same value in the clean graph (X, A) and a different value in the perturbed graph (X , A ). Since H * (n) is a valid solution to the linear program underlying the base certificate, we know that Eq. 62 is fulfilled, as it is equivalent to Eq. 54 from the base certificate. Since u n) , we know that Pr hn (φ attr (X ), φ adj (A )) = y n ≥ 0.5 (see Eq. 56) , i.e. fn is successfully attacked, fn (X , A ) = y n = fn (X, A).By construction, we have exactly o * nodes for which fn (X , A ) = y n and the remaining constraints are fulfilled as well.

F HYPERPARAMETERS

Training schedule for smoothed classifiers. Training is performed in a semi-supervised fashion with 20 nodes per class as a train set. Another 20 nodes per class serve as a validation set. Models are trained with Adam (learning rate = 0.001 [0.01 for SMA], β 1 = 0.9, β 2 = 0.999, = 10 -8 , weight decay = 0.001) for 3000 epochs, using the average cross-entropy loss across all training set nodes, with a batch size of 1. We employ early stopping, if the validation loss does not decrease for 50 epochs (300 epochs for SMA). In each epoch, a different graph is sampled from the smoothing distribution. We do not use the KL-divergence based regularization loss proposed for RGCN, as we found it to decrease the certifiable robustness of the model.Training schedule for non-smoothed GCN. Training is performed in a semi-supervised fashion with 20 nodes per class as a train set. Another 20 nodes per class serve as a validation set. For the first 100 of 1000 epochs, models are trained with Adam (learning rate = 0.01, β 1 = 0.9, β 2 = 0.999, = 10 -8 , weight decay = 10 -5), using the average cross-entropy loss across all training set nodes, with a batch size of 8. After 100 episodes, we add the robust loss proposed in (Zügner & Günnemann, 2019) (local budget q = 21, global budget Q = 12, training node classification margin = log(90/10), unlabeled node classification margin = log(60/40)). The gradient for the robust loss term is accumulated over 5 epochs before each weight update in order to simulate larger batch sizes.Network parameters. In all models, hidden linear and convolutional layers are followed by a ReLU nonlinearity. During training, each ReLU nonlinearity is followed by 50% dropout. For GCN and GAT, we use two convolution layers with 64 hidden activations. The number of attention heads for GAT is set to 8 for the first layer and 1 for the second layer. RGCN uses independent gaussians as its internal representation (i.e. each feature dimension has a mean and a variance). For RGCN, we use one linear layer, followed by two convolutional layers. We set the number of hidden activations to 32 for the means and 32 for the variances. Dropout is applied to both the means and variances.For APPNP, we use two linear layers with 64 hidden activations, followed by a propagation layer based on approximate personalized pagerank (teleport probablity = 0.15, iterations = 10). To ensure locality, we set all but the top 64 of each row in the approximate pagerank matrix to 0. For SMA, we first transform each node's features using a linear layer with 64 hidden activations. We then apply soft medoid aggregation (k = 64, T = 10) based on the approximate personalized pagerank matrix (teleport probablity = 0.15, iterations = 2). Note that we use the alternative parameterization from appendix B.5 of (Geisler et al., 2020) that is designed to improve robustness to attribute perturbations. After the aggregation, we apply a ReLU nonlinearity. Finally, we apply a second linear layer to each node independently.Randomized smoothing. Randomized smoothing introduces four additional hyperparameters θ X add , θ X add , θ A add , θ A del , which control the probability of flipping bits in the attribute and adjacency matrix under the smoothing distribution. If we only certify attribute perturbations, we set θ A add = θ A del = 0, θ X add = 0.002 and θ X del = 0.6. If we only certify adjacency perturbations, we set θ X add = θ X del = 0, θ A add = 0 and θ A del = 0.4. If we jointly certify attribute and adjacency perturbations, we set θ X add = 0.002, θ X del = 0.6, θ A add = 0 and θ A del = 0.4. Exactly evaluating smoothed classifiers is not possible, they have to be approximated via sampling. We use 1000 samples to determine a classifier's majority class (y n ), followed by 10 6 samples to estimate the probability of the majority class (p n ) via a Clopper-Pearson confidence interval. Applying Bonferroni correction, the confidence level for each confidence interval is set to 1 -0.01/N to obtain an overall confidence level of 99% for all certificates.

