HYPERTIME: IMPLICIT NEURAL REPRESENTATIONS FOR TIME SERIES GENERATION

Abstract

Implicit neural representations (INRs) have recently emerged as a powerful tool that provides an accurate and resolution-independent encoding of data. Their robustness as general approximators has been shown in a wide variety of data sources, with applications on image, sound, and 3D scene representation. However, little attention has been given to leveraging these architectures for the representation and analysis of time series data. In this paper, we propose a new INR architecture for time series (iSIREN) designed to perform an accurate reconstruction of univariate and multivariate data, while also providing an interpretable encoding of the signal. We compare our architecture against SIREN and INRs with different activations, in terms of training convergence, and the reconstruction accuracy of both the signal and its spectral distribution. To achieve generalization, we propose a hypernetwork architecture (HyperTime) that leverages iSIRENs to learn a latent representation of an entire time series dataset. In addition to the traditional reconstruction loss, we introduce an FFT-based loss that guides the training by enforcing a good match of the ground truth spectral distribution. We show how these architectures can be used for time series generation, and evaluate our method through fidelity metrics, presenting results that exceed the performance of state-of-the-art techniques. Finally, we propose an alternative hypernetwork architecture (iHyperTime) that incorporates interpretability into the latent representation, enabling the introduction of prior knowledge by imposing constraints into the generation process.

1. INTRODUCTION

Modeling time series data has been a key topic of research for many years, constituting a crucial component in a wide variety of areas such as climate modeling, medicine, biology, retail and finance (Lim & Zohren, 2021) . Traditional methods for time series modeling have relied on parametric models informed by expert knowledge. However, the development of modern machine learning methods has provided purely data-driven techniques to learn temporal relationships. In particular, neural network-based methods have gained popularity in recent times, with applications to a wide range of tasks, such as time series classification (Ismail Fawaz et al., 2020 ), clustering (Ma et al., 2019; Alqahtani et al., 2021) , segmentation (Perslev et al., 2019; Zeng et al., 2022) , anomaly detection (Choi et al., 2021; Xu et al., 2018; Hundman et al., 2018 ), upsampling (Oh et al., 2020; Bellos et al., 2019) , imputation (Liu, 2018; Luo et al., 2018; Cao et al., 2018) , forecasting (Lim & Zohren, 2021; Torres et al., 2021) and synthesis (Alaa et al., 2021; Yoon et al., 2019b; Jordon et al., 2019) . In particular, generation of synthetic time series has recently gained attention due to the large number of potential applications in medical and financial fields, where data cannot be shared, either due to privacy reasons or proprietary restrictions (Jordon et al., 2021; 2019; Assefa et al., 2020) . Moreover, synthetic time series can be used to augment training datasets to improve model generalization on downstream tasks, such as classification (Fons et al., 2021) , forecasting and anomaly detection. In recent years, implicit neural representations (INRs) have gained popularity as an accurate and flexible method to parameterize signals from diverse sources, such as images, video, audio and 3D scene data (Sitzmann et al., 2020b; Mildenhall et al., 2020) . Conventional methods for data encoding often rely on discrete representations, such as data grids, which are limited by their spatial resolution and present inherent discretization artifacts. In contrast, INRs encode data in terms of continuous functional relationships between signals, and thus are uncoupled to spatial resolution. In practical terms, INRs provide a new data representation framework that is resolution-independent, with many potential applications to time series, where missing values and irregularly sampled data are common occurrences (Fang & Wang, 2020). While there have been a few recent works exploring the application of INRs to time series data, there is virtually no work on leveraging these architectures for generating synthetic time series or producing interpretable time series representations (Jeong & Shin, 2022; Woo et al., 2022) . Applications with regulatory focus such as finance often require transparency and interpretability of proposed machine learning solutions as well as injection of expert knowledge as constraints into the training process to guide learning. For instance, explainable construction of trading agents is preferred when actions of trading agents need to be explicitly attributed to market signals (Vyetrenko & Xu, 2019) . In this paper, we propose a novel methodology that utilizes INRs to encode and generate time series data based on interpretable latent representations. To the best of our knowledge, we are the first to incorporate an interpretable decomposition into the generation of time-series. Our contributions are as follows: Representation and Generation of time-series using INRs: We introduce iSIREN, an INR architecture for multivariate time-series representation which provides an interpretable trend-seasonality decomposition of the data. We show that interpretability does not lead to a loss of reconstruction accuracy, and in some cases increases the spectral reconstruction quality. Moreover, we leverage a hypernetwork for time-series generation via interpolation of learned embeddings. Spectral Loss: To improve the training of the hypernetwork, we introduce a novel spectral loss that enforces the correct reconstruction of the signal's spectral distribution. We show that for some datasets this loss plays a crucial role in the learning process.

Interpretable time-series generation:

We propose iHyperTime, a hypernetwork architecture for time-series generation that learns a disentangled seasonal-trend representation of time series, enabling the introduction of expert knowledge into the synthesis process. We compare iHyperTime against current state-of-the-art methods for time-series generation, showing improved results in terms of standard fidelity metrics.

2. RELATED WORK

Implicit Neural Representations Implicit Neural Representations (INRs) provide a continuous representation of multidimensional data, by encoding a functional relationship between input coordinates and signal values, avoiding possible discretization artifacts. They have recently gained popularity in visual computing (Mescheder et al., 2019; Mildenhall et al., 2020) due to the key development of positional encodings (Tancik et al., 2020) and periodic activations (SIREN (Sitzmann et al., 2020b) ), which have proven to be critical for the learning of high-frequency details. Whilst INRs have been shown to produce accurate reconstructions in a wide variety of data sources, such as video, images and audio (Sitzmann et al., 2020b; Chen et al., 2021; Rott Shaham et al., 2021) , few works have leveraged them for time series representation (Jeong & Shin, 2022; Woo et al., 2022) , and none have focused on interpretability and generation. Hypernetworks Hypernetworks are neural network architectures that are trained to predict the parameters of secondary networks, referred to as Hyponetworks (Ha et al., 2017; Sitzmann et al., 2020a) . In the last few years, some works have leveraged different hypernetwork architectures for the prediction of INR weights, in order to learn priors over image data (Skorokhodov et al., 2021) and 3D scene data (Littwin & Wolf, 2019; Sitzmann et al., 2019; Sztrajman et al., 2021) . Sitzmann et al. (2020b) leverage a set encoder and a hypernetwork decoder to learn a prior over SIRENs encoding image data, and apply it for image in-painting. Our HyperTime and iHyperTime architectures detailed in Section 3 use a similar encoder-decoder structure, however they learn a latent representation over our interpretable SIREN INRs (iSIREN) , which encode time series data. Moreover, we apply these architectures for time series generation via interpolation of learned embeddings. 



Synthesis of time series data using deep generative models has been previously studied in the literature. Examples include the TimeGAN architecture (Yoon et al., 2019a), as well as QuantGAN (Wiese et al., 2020). More recently, Desai et al. (2021) proposed TimeVAE as a variational autoencoder alternative to GAN-based time series generation. Alaa et al. (2021) introduced Fourier Flows, a normalizing flow model for time series data that leverages the frequency domain representation, which is currently considered together with TimeGAN as state-

