GENERALIZED UNIVERSAL APPROXIMATION FOR CERTIFIED NETWORKS

Abstract

To certify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using interval bound propagation. To understand the power of interval bounds, we present the abstract universal approximation (AUA) theorem, a generalization of the recent result by Baader et al. (2020) for ReLU networks to a large class of neural networks. The AUA theorem states that for any continuous function f , there exists a neural network that (1) approximates f (universal approximation) and ( 2) whose interval bounds are an arbitrarily close approximation of the set semantics of f . The network may be constructed using any activation function from a rich class of functions-sigmoid, tanh, ReLU, ELU, etc.-making our result quite general. The key implication of the AUA theorem is that there always exists certifiably robust neural networks, which can be constructed using a wide range of activation functions.

1. INTRODUCTION

With wide adoption of neural networks, new safety and security concerns arose. The most prominent property of study has been robustness (Goodfellow et al., 2015) : small perturbations to the input of a network should not change the prediction. For example, a small change to an image of a stop sign should not cause a classifier to think it is a speed-limit sign. A number of researchers have proposed the use of abstract interpretation (Cousot & Cousot, 1977) techniques to prove robustness of neural networks (Gehr et al., 2018; Wang et al., 2018; Anderson et al., 2019) and to train robust models (Mirman et al., 2018; Gowal et al., 2018; Huang et al., 2019; Wong & Kolter, 2018; Wong et al., 2018; Balunovic & Vechev, 2020) . Suppose we want to verify robustness of a neural network to small changes in the brightness of an image. We can represent a large set of images, with varying brightness, as an element of some abstract domain, and propagate it through the network, effectively executing the network on an intractably large number of images. If all images lead to the same prediction, then we have a proof that the network is robust on the original image. The simplest abstract interpretation technique that leads to practical verification results is interval analysis-also referred to as interval bound propagation. In our example, if each pixel in a monochrome image is a real number r, then the pixel can be represented as an interval [r -, r + ], where denotes the range of brightness we wish to be robust on. Then, the box representing the interval of each pixel is propagated through the network using interval arithmetic operations. The interval domain has been successfully used for certifying properties of neural networks in vision (Gehr et al., 2018; Gowal et al., 2018 ), NLP (Huang et al., 2019) , as well as cyber-physical systems (Wang et al., 2018) . Why does the interval domain work for certifying neural networks? To begin understanding this question, Baader et al. (2020) demonstrated a surprising connection between the universal approximation theorem and neural-network certification using interval bounds. Their theorem states that not only can neural networks approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using rectified linear unit (ReLU) activation functions, whose interval bounds are an arbitrarily close approximation of the set semantics of f , i.e., the result of applying f to a set of inputs (e.g., set of similar images).

