EFFICIENT CERTIFIED TRAINING AND ROBUSTNESS VERIFICATION OF NEURAL ODES

Abstract

Neural Ordinary Differential Equations (NODEs) are a novel neural architecture, built around initial value problems with learned dynamics which are solved during inference. Thought to be inherently more robust against adversarial perturbations, they were recently shown to be vulnerable to strong adversarial attacks, highlighting the need for formal guarantees. However, despite significant progress in robustness verification for standard feed-forward architectures, the verification of high dimensional NODEs remains an open problem. In this work, we address this challenge and propose GAINS, an analysis framework for NODEs combining three key ideas: (i) a novel class of ODE solvers, based on variable but discrete time steps, (ii) an efficient graph representation of solver trajectories, and (iii) a novel abstraction algorithm operating on this graph representation. Together, these advances enable the efficient analysis and certified training of high-dimensional NODEs, by reducing the runtime from an intractable O(exp(d) + exp(T )) to O(d + T 2 log 2 T ) in the dimensionality d and integration time T . In an extensive evaluation on computer vision (MNIST and FMNIST) and time-series forecasting (PHYSIO-NET) problems, we demonstrate the effectiveness of both our certified training and verification methods.

1. INTRODUCTION

As deep learning-enabled systems are increasingly deployed in safety-critical domains, developing neural architectures and specialized training methods that increase their robustness against adversarial examples (Szegedy et al., 2014; Biggio et al., 2013) -imperceptible input perturbations, causing model failures -is more important than ever. As standard neural networks suffer from severely reduced accuracies when trained for robustness, novel architectures with inherent robustness properties have recently received increasing attention (Winston & Kolter, 2020; Müller et al., 2021) . Neural Ordinary Differential Equations One particularly interesting such architecture are neural ODEs (NODEs) (Chen et al., 2018) . Built around solving initial value problems with learned dynamics, they are uniquely suited to time-series-based problems (Rubanova et al., 2019; Brouwer et al., 2019) but have also been successfully applied to image classification (Chen et al., 2018) . More importantly, NODEs have been observed to exhibit inherent robustness properties against adversarial attacks (Yan et al., 2020; Kang et al., 2021; Rodriguez et al., 2022; Zakwan et al., 2022 ). However, recently Huang et al. (2020) found that this robustness is greatly diminished against stronger attacks. They suggest that adaptive ODE solvers, used to solve the underlying initial value problems, cause gradient obfuscation (Athalye et al., 2018) , which, in turn, causes weaker adversarial attacks to fail. This highlights the need for formal robustness guarantees to rigorously evaluate the true robustness of a model or architecture. Robustness Verification For standard neural networks, many robustness verification methods have been proposed (Katz et al., 2017; Tjeng et al., 2019; Singh et al., 2018b; Raghunathan et al., 2018; Wang et al., 2021; Ferrari et al., 2022) . One particularly successful class of such methods (Gehr et al., 2018; Singh et al., 2019a; Zhang et al., 2018) Inference Guarantees Figure 1 : Overview of NODE inference in both the standard and certified (our) setting. In both settings ODE solvers are used to evaluate learned dynamics. We introduce controlled adaptive solvers (CAS) as a modification of adaptive solvers, which reduce the reachable time/step-size pairs from a continuous area to discrete points. This enables GAINS to compute worst-case bounds on NODE outputs given small input ranges, allowing both inference with guarantees and provable training. For example, in the time-series forecasting setting shown on the right, standard NODE inference predicts the blue points given the concrete red inputs. In contrast GAINS computes all possible outputs (blue error bars), for inputs in the red input ranges. whether a given robustness property holds. Unfortunately, none of these methods can be applied to NODEs as the underlying adaptive solvers yield a continuous range of possible step-sizes (illustrated in the top panel of Fig. 1 ), which existing analysis techniques can not handle. First works towards NODE verification (Lopez et al., 2022) have avoided this issue by disregarding the solver behavior and analyzing only the underlying NODE dynamics in extremely low dimensional settings. However, both scaling to high-dimensional NODE architectures and taking the effect of ODE solvers into account remain open problems preventing NODE robustness verification.

This Work

We tackle both of these problems, thereby enabling the systematic verification and study of NODE robustness (illustrated in Fig. 1 ) as follows: (i) We introduce a novel class of ODE solvers, based on the key insight that we can restrict step-sizes to an exponentially spaced grid with minimal impact on solver efficiency, while obtaining a finite number of time/step-size trajectories from the initial to final state (see the second column in Fig. 1 ). We call these Controlled Adaptive ODE Solvers (CAS). Unfortunately, CAS solvers still yield exponentially many trajectories in the integration time. (ii) We, therefore, introduce an efficient graph representation, allowing trajectories to be merged, reducing their number to quadratically many. (iii) We develop a novel algorithm for the popular DEEPPOLY convex relaxation (Singh et al., 2019a) , effective for standard neural network verification, that enables the efficient application of DEEPPOLY to the trajectory graph by handling trajectory splitting in linear instead of exponential time. Combining these core ideas, we propose GAINSfoot_0 , a novel framework for certified training and verification of NODEs that leverages key algorithmic advances to achieve polynomial runtimes and allows us to faithfully assess the robustness of NODEs. Main Contributions Our main contributions are: • A novel class of ODE solvers, CAS solvers, retaining the efficiency of adaptive step size solvers while enabling verification ( §4). • An efficient linear bound propagation based framework, GAINS, which leverages CAS to enable the efficient training and verification of NODEs ( §5). • An extensive empirical evaluation demonstrating the effectiveness of our method in ablation studies and on image classification and time-series problems ( §6).



Graph based Abstract Interpretation for NODEs



propagates convex shapes through the neural network that capture the reachable sets of every neuron's values and uses them to check

