TIGHT SECOND-ORDER CERTIFICATES FOR RANDOMIZED SMOOTHING

Abstract

Randomized smoothing is a popular way of providing robustness guarantees against adversarial attacks: randomly-smoothed functions have a universal Lipschitz-like bound, allowing for robustness certificates to be easily computed. In this work, we show that there also exists a universal curvature-like bound for Gaussian random smoothing: given the exact value and gradient of a smoothed function, we compute a lower bound on the distance of a point to its closest adversarial example, called the Second-order Smoothing (SoS) robustness certificate. In addition to proving the correctness of this novel certificate, we show that SoS certificates are realizable and therefore tight. Interestingly, we show that the maximum achievable benefits, in terms of certified robustness, from using the additional information of the gradient norm are relatively small: because our bounds are tight, this is a fundamental negative result. The gain of SoS certificates further diminishes if we consider the estimation error of the gradient norms, for which we have developed an estimator. We therefore additionally develop a variant of Gaussian smoothing, called Gaussian dipole smoothing, which provides similar bounds to randomized smoothing with gradient information, but with much-improved sample efficiency. This allows us to achieve (marginally) improved robustness certificates on high-dimensional datasets such as CIFAR-10 and ImageNet. Code is available at https://github.com/alevine0/smoothing_second_ order.

1. INTRODUCTION

A topic of much recent interest in machine learning has been the design of deep classifiers with provable robustness guarantees. In particular, for an m-class classifier h : R d → [m], the L 2 certification problem for an input x is to find a radius ρ such that, for all δ with δ 2 < ρ, h(x) = h(x + δ). This robustness certificate serves as a lower bound on the magnitude of any adversarial perturbation of the input that can change the classification: therefore, the certificate is a security guarantee against adversarial attacks. There are many approaches to the certification problem, including exact methods, which compute the precise norm to the decision boundary (Tjeng et al., 2019; Carlini et al., 2017; Huang et al., 2017) as well as methods for which the certificate ρ is merely a lower bound on the distance to the decision boundary (Wong & Kolter, 2018; Gowal et al., 2018; Raghunathan et al., 2018) . One approach that belongs to the latter category is Lipschitz function approximation. Recall that a function f : R d → R is L-Lipschitz if, for all x, x , |f (x) -f (x )| ≤ L x -x 2 . If a classifier is known to be a Lipschitz function, this immediately implies a robustness certificate. In particular, consider a binary classification for simplicity, where we use an L-Lipschitz function f as a classifier, using the sign of f (x) as the classification. Then for any input x, we are assured that the classification (i.e, the sign) will remain constant for all x within a radius |f (x)|/L of x. Numerous methods for training Lipschitz neural networks with small, known Lipschitz constants have been proposed. (Fazlyab et al., 2019; Zhang et al., 2019; Anil et al., 2019; Li et al., 2019b) It is desirable that the network be as expressive as possible, while still maintaining the desired Lipschitz property. Anil et al. (2019) in particular demonstrates that their proposed method can universally approximate Lipschitz functions, given sufficient network complexity. However, in practice, for the robust certification problem on large-scale input, randomized smoothing (Cohen et al., 2019) is the  f (x + ) is (1/σ)-Lipschitz, where N (0, σ 2 I) is a d-dimensional isometric Gaussian distribution with variance σ 2 and Φ -1 is the inverse normal CDF function. As a result, given the smoothed classifier value p a (x) at x, one can calculate the certified radius ρ(x) = σΦ -1 (p a (x)) in which p a (x) ≥ 0.5 (i.e., Φ -1 (p a (x)) ≥ 0). This means that we can use p a (x) ∈ R d → [0, 1] as a robust binary classifier (with one class assignment if p a (x) ≥ 0.5, and the other if p a (x) < 0.5). Cohen et al. (2019) shows that this is a tight certificate result for a classifier smoothed with Gaussian noise: given the value of p a (x), there exists a base classifier function f such that, if p a is the Gaussian-smoothed version of f , then there exists an x with xx 2 = ρ such that p a (x ) = 0.5. In other words, the certificate provided by (Cohen et al., 2019) is the largest possible certificate for Gaussian smoothing, given only the value of p a (x). Previous results (Li et al., 2019a; Lecuyer et al., 2019) provided looser bounds for Gaussian smoothing. Singla & Feizi (2020) have recently shown, for shallow neural networks, that, rather than globally bounding the (first-order) Lipschitz constant of the network, it is possible to achieve larger robustness certificates by instead globally bounding the Lipschitz constant of the gradient of the network. This second-order, curvature-based method takes advantage of the fact that the gradient at x can be computed easily via back-propagation, so certificates can make use of both f (x) and ∇ x f (x). This leads to a question: can we also use the gradient of a smoothed classifier ∇ x p a (x) to improve smoothing-based certificates? In this work, we show that there is a universal curvature-like bound for all randomly-smoothed classifiers. Therefore, given p a (x) and ∇ x p a (x), we can compute larger certificates than is possible using the value of p a (x) alone. Moreover, our bound is tight in that, given only the pair (p a (x), ∇ x p a (x)), the certificate we provide is the largest possible certificate for Gaussian smoothing. We call our certificates "Second-order Smoothing" (SoS) certificates. As shown in Figure 1 , the smoothing-based certificates which we can achieve using second-order smoothing represent relatively modest improvements compared to the first-order bounds. This is a meaningful negative result, given the tightness of our bounds, and is therefore useful in guiding (or limiting) future research into higher-order smoothing certificates. Additionally, this result shows that



Figure 1: (a) Tight lower bound on the value of a smoothed function at x (i.e. p a (x )) as a function of xx 2 . In this example, p a (x) = 0.8 and the smoothing standard deviation σ = 1. The red line shows the lower bound for the function, with no information about the gradient given.The blue line incorporates the additional information that ∇ x p a (x) 2 = 0. Note that the axis at Φ -1 (p a (x)) = 0 corresponds to p a (x) = 0.5, the decision boundary for a binary classifier. (b) Tight robustness certificates for a randomized-smoothed classifier, given the top-class value p a (x) and the gradient norm ∇ x p a (x) 2 . The dashed lines show the certificates given p a (x) alone. Note that the maximum possible gradient for a smoothed classifier depends on p a (x) (see Equation1).

