LEARNING CONTINUOUS-TIME DYNAMICS BY STOCHASTIC DIFFERENTIAL NETWORKS

Abstract

Learning continuous-time stochastic dynamics is a fundamental and essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension. For a given system whose latent states and observed data are multivariate, it is generally impossible to derive a precise continuous-time stochastic process to describe the system behaviors. To solve the above problem, we apply Variational Bayesian method and propose a flexible continuous-time stochastic recurrent neural network named Variational Stochastic Differential Networks (VSDN), which embeds the complicated dynamics of the sporadic time series by neural Stochastic Differential Equations (SDE). VSDNs capture the stochastic dependency among latent states and observations by deep neural networks. We also incorporate two differential Evidence Lower Bounds to efficiently train the models. Through comprehensive experiments, we show that VSDNs outperform state-of-the-art continuous-time deep learning models and achieve remarkable performance on prediction and interpolation tasks for sporadic time series.

1. INTRODUCTION AND RELATED WORKS

Many real-world systems experience complicated stochastic dynamics over a continuous time period. The challenges on modeling the stochastic dynamics mainly come from two sources. First, the underlying state transitions of many systems are often uncertain, as they are placed in unpredictable environment with their states continuously affected by unknown disturbances. Second, the monitoring data collected may be sparse and at irregular intervals as a result of the sampling strategy or data corruption. The sporadic data sequence loses a large amount of information and system behaviors hidden behind the intervals of the observed data. In order to accurately model and analyze dynamics of these systems, it is important to reliably and efficiently represent the continuous-time stochastic process based on the discrete-time observations. In some domains, the derivation of the continuous-time stochastic model relies heavily on human knowledge and many studies focus on its inference problem (Ryder et al., 2018; Särkkä et al., 2015) . But in more domains (e.g., video analysis (Vondrick et al., 2016) and human activity detection (Rubanova et al., 2019)), it is difficult and sometimes intractable to derive an accurate model to capture the underlying temporal evolution from the collected sequence of data. Although some studies have been made on approximating the stochastic process from the data collected, the majority of these methods define the system dynamics with a linear model (Macke et al., 2011; Yu et al., 2009b; a) , which can not well represent high-dimensional data with nonlinear relationship. Recently, the Neural Ordinary Differential Equation (ODE) studies (Chen et al., 2018; Rubanova et al., 2019; Jia & Benson, 2019; De Brouwer et al., 2019; Yildiz et al., 2019; Kidger et al., 2020) introduce deep learning models to learn an ODE and apply it to approximate continuous-time dynamics. Nevertheless, these methods generally neglect the randomness of the latent state trajectories and posit simplified assumptions on the data distribution (e.g. Gaussian), which strongly limits their capability of modeling complicated continuous-time stochastic processes. Compared to ODE, Stochastic Differential Equation (SDE) (Jørgensen et al., 2020 ) is a more practical solution in modeling the continuous-time stochastic process. Recently there have been some studies on bridging the gap between deep neural networks and SDEs (Ha et al., 2018) . In (Hegde et al., 2019; Liu et al., 2020; Peluchetti & Favaro, 2020; Wang et al., 2019; Kong et al., 2020) , SDEs

