FOR LONG TIME SERIES VIA THE LOG-ODE METHOD

Abstract

Neural Controlled Differential Equations (Neural CDEs) are the continuous-time analogue of an RNN, just as Neural ODEs are analogous to ResNets. However just like RNNs, training Neural CDEs can be difficult for long time series. Here, we propose to apply a technique drawn from stochastic analysis, namely the log-ODE method. Instead of using the original input sequence, our procedure summarises the information over local time intervals via the log-signature map, and uses the resulting shorter stream of log-signatures as the new input. This represents a length/channel trade-off. In doing so we demonstrate efficacy on problems of length up to 17k observations and observe significant training speed-ups, improvements in model performance, and reduced memory requirements compared to the existing algorithm.

1. INTRODUCTION

Neural controlled differential equations (Neural CDEs) (Kidger et al., 2020) are the continuous-time analogue to a recurrent neural network (RNN), and provide a natural method for modelling temporal dynamics with neural networks. Neural CDEs are similar to neural ordinary differential equations (Neural ODEs), as popularised by Chen et al. (2018) . A Neural ODE is determined by its initial condition, without a direct way to modify the trajectory given subsequent observations. In contrast the vector field of a Neural CDE depends upon the time-varying data, so that the trajectory of the system is driven by a sequence of observations.

1.1. CONTROLLED DIFFERENTIAL EQUATIONS

We begin by stating the definition of a CDE. Let a, b ∈ R with a < b, and let v, w ∈ N. Let ξ ∈ R w . Let X : [a, b] → R v be a continuous function of bounded variation (which is for example implied by it being Lipschitz), and let f : R w → R w×v be continuous. Then we may define Z : [a, b] → R w as the unique solution of the controlled differential equation Z a = ξ, Z t = Z a + t a f (Z s )dX s for t ∈ (a, b], The notation "f (Z s )dX s " denotes a matrix-vector product, and if X is differentiable then t a f (Z s )dX s = t a f (Z s ) dX ds (s)ds. If in equation (1), dX s was replaced with ds, then the equation would just be an ODE. Using dX s causes the solution to depend continuously on the evolution of X. We say that the solution is "driven by the control, X". 



1.2 NEURAL CONTROLLED DIFFERENTIAL EQUATIONSWe recall the definition of a Neural CDE as introduced in Kidger et al.(2020).1

