A TEMPORAL KERNEL APPROACH FOR DEEP LEARN-ING WITH CONTINUOUS-TIME INFORMATION

Abstract

Sequential deep learning models such as RNN, causal CNN and attention mechanism do not readily consume continuous-time information. Discretizing the temporal data, as we show, causes inconsistency even for simple continuous-time processes. Current approaches often handle time in a heuristic manner to be consistent with the existing deep learning architectures and implementations. In this paper, we provide a principled way to characterize continuous-time systems using deep learning tools. Notably, the proposed approach applies to all the major deep learning architectures and requires little modifications to the implementation. The critical insight is to represent the continuous-time system by composing neural networks with a temporal kernel, where we gain our intuition from the recent advancements in understanding deep learning with Gaussian process and neural tangent kernel. To represent the temporal kernel, we introduce the random feature approach and convert the kernel learning problem to spectral density estimation under reparameterization. We further prove the convergence and consistency results even when the temporal kernel is non-stationary, and the spectral density is misspecified. The simulations and real-data experiments demonstrate the empirical effectiveness of our temporal kernel approach in a broad range of settings.

1. INTRODUCTION

Deep learning models have achieved remarkable performances in sequence learning tasks leveraging the powerful building blocks from recurrent neural networks (RNN) (Mikolov et al., 2010) , longshort term memory (LSTM) (Hochreiter & Schmidhuber, 1997) , causal convolution neural network (CausalCNN/WaveNet) (Oord et al., 2016) and attention mechanism (Bahdanau et al., 2014; Vaswani et al., 2017) . Their applicability to the continuous-time data, on the other hand, is less explored due to the complication of incorporating time when the sequence is irregularly sampled (spaced). The widely-adopted workaround is to study the discretized counterpart instead, e.g. the temporal data is aggregated into bins and then treated as equally-spaced, with the hope to approximate the temporal signal using the sequence information. It is perhaps without surprise, as we show in Claim 1, that even for regular temporal sequence the discretization modifies the spectral structure. The gap can only be amplified for irregular data, so discretizing the temporal information will almost always introduce intractable noise and perturbations, which emphasizes the importance to characterize the continuous-time information directly. Previous efforts to incorporate temporal information for deep learning include concatenating the time or timespan to feature vector (Choi et al., 2016; Lipton et al., 2016; Li et al., 2017b) , learning the generative model of time series as missing data problems (Soleimani et al., 2017; Futoma et al., 2017) , characterizing the representation of time (Xu et al., 2019; 2020; Du et al., 2016) and using neural point process (Mei & Eisner, 2017; Li et al., 2018) . While they provide different tools to expand neural networks to coupe with time, the underlying continuous-time system and process are involved explicitly or implicitly. As a consequence, it remains unknown in what way and to what extend are the continuous-time signals interacting with the original deep learning model. Explicitly characterizing the continuous-time system (via differential equations), on the other hand, is the major pursuit of classical signal processing methods such as smoothing and filtering (Doucet & Johansen, 2009; Särkkä, 2013) . The lack of connections is partly due to the compatibility issues between the signal processing methods and the auto-differential gradient computation framework of modern deep learning. Generally speaking, for continuous-time systems, model learning and parameter estimation often rely on the more complicated differential equation solvers (Raissi & Karniadakis, 2018; Raissi et al., 2018a) . Although the intersection of neural network and differential equations is gaining popularity in recent years, the combined neural differential methods often require involved modifications to both the modelling part and implementation detail (Chen et al., 2018; Baydin et al., 2017) . Inspired by the recent advancement in understanding neural network with Gaussian process and the neural tangent kernel (Yang, 2019; Jacot et al., 2018) , we discover a natural connection between the continuous-time system and the neural Gaussian process after composing with a temporal kernel. The significance of the temporal kernel is that it fills in the gap between signal processing and deep learning: we can explicitly characterize the continuous-time systems while maintaining the usual deep learning architectures and optimization procedures. While the kernel composition is also known for integrating signals from various domains (Shawe-Taylor et al., 2004) , we face the additional complication of characterizing and learning the unknown temporal kernel in a data-adaptive fashion. Unlike the existing kernel learning methods where at least the parametric form of the kernel is given (Wilson et al., 2016) , we have little context on the temporal kernel, and aggressively assuming the parametric form will risk altering the temporal structures implicitly just like discretization. Instead, we leverage the Bochner's theorem and its extension (Bochner, 1948; Yaglom, 1987) to first covert the kernel learning problem to the more reasonable spectral domain where we can direct characterize the spectral properties with random (Fourier) features. Representing the temporal kernel by random features is favorable as we show they preserve the existing Gaussian process and NTK properties of neural networks. This is desired from the deep learning's perspective since our approach will not violate the current understandings of deep learning. Then we apply the reparametrization trick (Kingma & Welling, 2013) , which is a standard tool for generative models and Bayesian deep learning, to jointly optimize the spectral density estimator. Furthermore, we provide theoretical guarantees for the random-feature-based kernel learning approach when the temporal kernel is non-stationary, and the spectral density estimator is misspecified. These two scenarios are essential for practical usage but have not been studied in the previous literature. Finally, we conduct simulations and experiments on real-world continuous-time sequence data to show the effectiveness of the temporal kernel approach, which significantly improves the performance of both standard neural architectures and complicated domain-specific models. We summarize our contributions as follow. • We study a novel connection between the continuous-time system and neural network via the composition with a temporal kernel. • We propose an efficient kernel learning method based on random feature representation, spectral density estimation and reparameterization, and provide strong theoretical guarantees when the kernel is nonstationary and the spectral density is misspeficied. • We analyze the empirical performance of our temporal kernel approach for both the standard and domain-specific deep learning models through real-data simulation and experiments.

2. NOTATIONS AND BACKGROUND

We use bold-font letters to denote vectors and matrices. We use x t and (x, t) interchangeably to denote a time-sensitive event occurred at time t, with t ∈ T ≡ [0, t max ]. Neural networks are denoted

