KOOPMAN NEURAL FORECASTER FOR TIME SERIES WITH TEMPORAL DISTRIBUTION SHIFTS

Abstract

Temporal distributional shifts, with underlying dynamics changing over time, frequently occur in real-world time series, and pose a fundamental challenge for deep neural networks (DNNs). In this paper, we propose a novel deep sequence model based on the Koopman theory for time series forecasting: Koopman Neural Forecaster (KNF) that leverages DNNs to learn the linear Koopman space and the coefficients of chosen measurement functions. KNF imposes appropriate inductive biases for improved robustness against distributional shifts, employing both a global operator to learn shared characteristics, and a local operator to capture changing dynamics, as well as a specially-designed feedback loop to continuously update the learnt operators over time for rapidly varying behaviors. We demonstrate that KNF achieves the superior performance compared to the alternatives, on multiple time series datasets that are shown to suffer from distribution shifts.

1. INTRODUCTION

Temporal distribution shifts frequently occur in real-world time-series applications, from forecasting stock prices to detecting and monitoring sensory measures, to predicting fashion trend based sales. Such distribution shifts over time may due to the data being generated in a highly-dynamic and non-stationary environment, abrupt changes that are difficult to predict, or constantly evolving trends in the underlying data distribution (Gama et al., 2014) . Temporal distribution shifts pose a fundamental challenge for time-series forecasting (Kuznetsov & Mohri, 2020) . There are two scenarios of distribution shifts. When the distribution shifts only occur between the training and test domains, meta learning and transfer learning approaches (Jin et al., 2021; Oreshkin et al., 2021) have been developed. The other scenario is much more challenging: distribution shifts occurring continuously over time. This scenario is closely related to "concept drift" (Lu et al., 2018) and non-stationary processes (Dahlhaus, 1997) but has received less attention from the deep learning community. In this work, we focus on the second scenario. To tackle temporal distribution shifts, various statistical estimation methods have been studied, including spectral density analysis (Dahlhaus, 1997), sample reweighting (Bennett & Clarkson, 2022; McCarthy & Jensen, 2016) and Bayesian state-space models (West & Harrison, 2006) . However, these methods are limited to low capacity auto-regressive models and are typically designed for short-horizon forecasting. For large-scale complex time series data, deep learning models (Oreshkin et al., 2021; Woo et al., 2022; Tonekaboni et al., 2022; Zhou et al., 2022) now increasingly outperform traditional statistical methods. Yet, most deep learning approaches are designed for stationary timeseries data (with i.i.d. assumption), such as electricity usage, sales and air quality, that have clear seasonal and trend patterns. For distribution shifts, DNNs have been shown to be problematic in forecasting on data with varying distributions (Kouw & Loog, 2018; Wang et al., 2021) . DNNs are black-box models and often require a large number of samples to learn. For time series with continuous distribution shifts, the number of samples from a given distribution is small, thus DNNs would struggle to adapt to the changing distribution. Furthermore, the non-linear dependencies in a DNN are difficult to interpret or manipulate. Directly modifying the parameters based on the change in dynamics may lead to undesirable effects (Vlachas et al., 2020) . Therefore, if we can

availability

//github.com/google-research/ google-research/tree/

