LATENT LINEAR ODES WITH NEURAL KALMAN FIL-TERING FOR IRREGULAR TIME SERIES FORECASTING

Abstract

Over the past four years, models based on Neural Ordinary Differential Equations have become state of the art in the forecasting of irregularly sampled time series. Describing the data-generating process as a dynamical system in continuous time allows predictions at arbitrary time points. However, the numerical integration of Neural ODEs typically comes with a high computational burden or may even fail completely. We propose a novel Neural ODE model that embeds the observations into a latent space with dynamics governed by a linear ODE. Consequently, we do not require any specialized numerical integrator but only an implementation of the matrix exponential readily available in many numerical linear algebra libraries. We also introduce a novel state update component inspired by the classical Kalman filter, which, to our knowledge, makes our model the first Neural ODE variant to explicitly satisfy a specific self-consistency property. It allows forecasting irregularly sampled time series with missing values and comes with some numerical stability guarantees. We evaluate the performance on medical and climate benchmark datasets, where the model outperforms the state of the art by margins up to 30%.

1. INTRODUCTION

Continuous dynamical systems described by ordinary differential equations (ODE) propagate a given state into any time in the future. Hence ODE based models are natural candidates for the task of forecasting irregularly sampled time series. Furthermore many real world systems are well described by ODEs. Since the seminal paper by Chen et al. (2018) Neural ODEs have become building blocks of state of the art models for irregularly sampled time series forecasting. To predict a future state an ODE model would need an estimation of the present state and then propagate the state by solving an initial value problem. The present work proposes a model that introduces novel ideas both with respect to the state estimation and to the propagation. One serious issue with Neural ODEs is the cost and possible failure of the numerical integration. There exist many numerical schemes for this purpose, but in any case the cost of the integration for a required accuracy depends on the analytical properties of the right hand side and can become arbitrarily large or lead to failure. This is a serious problem for Neural ODEs, which has been tackled by different types of regularizations (Finlay et al., 2020; Ghosh et al., 2020; Kelly et al., 2020) . We propose a model where the observations are nonlinearly mapped into a latent space and a linear ODE with constant coefficients describes the latent dynamics. Solving the initial value problem simplifies to taking the matrix exponential, for which efficient and stable numerical implementations are available. According to Koopman operator theory (Brunton et al., 2022) such linear ODEs are expressive enough to approximate nonlinear ODEs. Furthermore, such linear dynamics are well understood and can be analyzed and modified using tools from linear algebra. For the state estimation we propose a filter inspired by the classical Kalman filter that updates the state given a new observation. However, it does not operate in the linear latent domain, but in the observation domain, and it is not probabilistic. The filter is designed to deal in a natural way with missing values and satisfies a self-consistency condition, such that the model state will only change at an observation if it differs from the model prediction. To the best of our knowledge our model is the first model that gives provable guarantees of forward stability at intialization.

