GENERALIZED UNIVERSAL APPROXIMATION FOR CERTIFIED NETWORKS

Abstract

To certify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using interval bound propagation. To understand the power of interval bounds, we present the abstract universal approximation (AUA) theorem, a generalization of the recent result by Baader et al. (2020) for ReLU networks to a large class of neural networks. The AUA theorem states that for any continuous function f , there exists a neural network that (1) approximates f (universal approximation) and ( 2) whose interval bounds are an arbitrarily close approximation of the set semantics of f . The network may be constructed using any activation function from a rich class of functions-sigmoid, tanh, ReLU, ELU, etc.-making our result quite general. The key implication of the AUA theorem is that there always exists certifiably robust neural networks, which can be constructed using a wide range of activation functions.

1. INTRODUCTION

With wide adoption of neural networks, new safety and security concerns arose. The most prominent property of study has been robustness (Goodfellow et al., 2015) : small perturbations to the input of a network should not change the prediction. For example, a small change to an image of a stop sign should not cause a classifier to think it is a speed-limit sign. A number of researchers have proposed the use of abstract interpretation (Cousot & Cousot, 1977) techniques to prove robustness of neural networks (Gehr et al., 2018; Wang et al., 2018; Anderson et al., 2019) and to train robust models (Mirman et al., 2018; Gowal et al., 2018; Huang et al., 2019; Wong & Kolter, 2018; Wong et al., 2018; Balunovic & Vechev, 2020) . Suppose we want to verify robustness of a neural network to small changes in the brightness of an image. We can represent a large set of images, with varying brightness, as an element of some abstract domain, and propagate it through the network, effectively executing the network on an intractably large number of images. If all images lead to the same prediction, then we have a proof that the network is robust on the original image. The simplest abstract interpretation technique that leads to practical verification results is interval analysis-also referred to as interval bound propagation. In our example, if each pixel in a monochrome image is a real number r, then the pixel can be represented as an interval [r -, r + ], where denotes the range of brightness we wish to be robust on. Then, the box representing the interval of each pixel is propagated through the network using interval arithmetic operations. The interval domain has been successfully used for certifying properties of neural networks in vision (Gehr et al., 2018; Gowal et al., 2018 ), NLP (Huang et al., 2019) , as well as cyber-physical systems (Wang et al., 2018) . Why does the interval domain work for certifying neural networks? To begin understanding this question, Baader et al. ( 2020) demonstrated a surprising connection between the universal approximation theorem and neural-network certification using interval bounds. Their theorem states that not only can neural networks approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using rectified linear unit (ReLU) activation functions, whose interval bounds are an arbitrarily close approximation of the set semantics of f , i.e., the result of applying f to a set of inputs (e.g., set of similar images). AUA theorem (semi-formally): For a continuous function f : R m → R that we wish to approximate and error δ > 0, there is a neural network N that has the following behavior: Let B ⊂ R m be a box. The red interval (top) is the tightest interval that contains all outputs of f when applied to x ∈ B. If we propagate box B through N using interval bounds, we may get the black interval (bottom) N # (B), whose lower/upper bounds are up to δ away from the red interval. The theorem of Baader et al. ( 2020) is restricted to networks that use rectified linear units (ReLU). δ δ minx∈B f (x) maxx∈B f (x) N # (B) In this work, we present a general universal approximation result for certified networks using a rich class of well-behaved activation functions. Specifically, we make the following contributions. Abstract universal approximation (AUA) theorem. We prove what we call the abstract universal approximation theorem, or AUA theorem for short: Let f be the function we wish to approximate, and let δ > 0 be the tolerated error. Then, there exists a neural network N , built using any well-behaved activation function, that has the following behavior: For any box of inputs B, we can certify, using interval bounds, that the range of outputs of N is δ close to the range outputs of f . If the box B of inputs is a single point in Euclidean space, the AUA theorem reduces to the universal approximation theorem; thus, AUA generalizes universal approximation. Fig. 1 further illustrates the AUA theorem. Existence of robust classifiers. While the AUA theorem is purely theoretical, it sheds light on the existence of certifiable neural networks. Suppose there is some ideal robust image classifier f using the ∞ norm, which is typically used to define a set of images in the neighborhood of a given image. The classical universal approximation theorem tells us that, for any desired precision, there is a neural network that can approximate f . We prove that the AUA theorem implies us that there exists a neural network for which we can automatically certify robustness using interval bounds while controlling approximation error. In addition, this neural network can be built using almost any activation function in the literature, and more. Squashable functions. We define a rich class of activation functions, which we call squashable functions, for which our abstract universal approximation theorem holds. This class expands the functions defined by Hornik et al. (1989) for universal approximation and includes popular activation functions, like ReLU, sigmoid, tanh, ELU, and other activations that have been shown to be useful for training robust neural networks (Xie et al., 2020) . The key feature of squashable activation functions is that they have left and right limits (or we can use them to construct functions with limits). We exploit limits to approximate step (sign) functions, and therefore construct step-like approximations of f , while controlling approximation error δ. Proof of AUA theorem. We present a constructive proof of the AUA theorem. 2020) is to construct an indicator function for box-shaped regions. We observe that squashable functions can approximate the sign function, and therefore approximate such indicator functions, while carefully controlling precision of abstract interpretation. Our proof uses a simpler indicator construction compared to Baader et al. ( 2020), and as a result its analysis is also simpler.

2. RELATED WORK

The classical universal approximation (UA) theorem has been established for decades. In contrast to AUA, UA states that a neural network with one single hidden layer can approximate any continuous function on a compact domain. One of the first versions goes back to Cybenko (1989); Hornik et al. (1989) , who showed that the standard feed-forward neural network with sigmoidal or squashing activations is a universal approximator. The most general version of UA was discovered by Leshno et al. (1993) , who showed that the feed-forward neural network is a universal approximator if and only if the activation function is non-polynomial. Because AUA implies UA, this means AUA cannot



Figure 1: Semi-formal illustration of AUA theorem. (Right is adapted from Baader et al. (2020).)

Our construction is inspired by and synthesizes a range of results: (1) the work of Hornik et al. (1989) on squashing functions for universal approximation, (2) the work of Csáji (2001) for using the sign (step) function to construct Haar (wavelet) functions, and (3) the work of Baader et al. (2020) on the specialized AUA theorem for ReLUs. The key idea of Baader et al. (

