SEQUENTIAL LATENT VARIABLE MODELS FOR FEW-SHOT HIGH-DIMENSIONAL TIME-SERIES FORECASTING

Abstract

Modern applications increasingly require learning and forecasting latent dynamics from high-dimensional time-series. Compared to univariate time-series forecasting, this adds a new challenge of reasoning about the latent dynamics of an unobserved abstract state. Sequential latent variable models (SLVMs) present an attractive solution, although existing works either struggle with long-term forecasting or have difficulty learning across diverse dynamics. In this paper, we first present a conceptual framework of SLVMs to unify existing works, contrast their fundamental limitations, and identify an intuitive solution to long-term forecasting for diverse dynamics via meta-learning. We then present a few-shot forecasting framework for high-dimensional time-series: instead of learning a single dynamic function, we leverage data of diverse dynamics and learn to adapt latent dynamic functions to few-shot support series. This is realized via Bayesian meta-learning underpinned by: 1) a latent dynamic function conditioned on knowledge derived from few-shot support series, and 2) a meta-model that learns to extract such dynamic-specific knowledge via feed-forward embedding of support set. We compared the presented framework with a comprehensive set of baseline models 1) trained globally on the large meta-training set with diverse dynamics, 2) trained individually on single dynamics with and without fine-tuning to k-shot support series, and 3) extended to few-shot meta-formulations. We demonstrated that the presented framework is agnostic to the latent dynamic function of choice and, at meta-test time, is able to forecast for new dynamics given variable-shot of support series.

1. INTRODUCTION

In many applications, an ultimate goal is to forecast the future states or trajectories of a dynamic system from its high-dimensional observations such as series of images. Compared to the relatively well-studied univariate time-series forecasting (Makridakis et al., 2018; Oreshkina et al., 2020; Salinas et al., 2020) , high-dimensional time-series forecasting raises new challenges: it requires the extraction of the dynamics of an abstract latent state that is not directly observed (Botev et al., 2021) . Sequential latent variable models (SLVMs) provide an attractive solution that, unlike autoregressive models, abstracts a latent dynamic function z i = f (z <i ; θ z ) with state z i and parameter θ z , along with z i 's emission to observations x i = g(z i ) (Chung et al., 2015) . This pair of learned models can support long-term forecasting given only initial frames of observations, as well as controlled generation of new dynamics. Critical bottlenecks however remain in reaching these goals. The earlier formulation of SLVMs relies on a natural extension of the static LVMs: as illustrated in Fig. 1A , the latent state z i is modeled as the latent variable for the generation of x i , and a sequential encoder is used to facilitate the inference of z i from current and past observations x ≤i (Chung et al., 2015; Krishnan et al., 2017) . Recent works have argued to instead model and infer the parameter

