LEAVES: LEARNING VIEWS FOR TIME-SERIES DATA IN CONTRASTIVE LEARNING

Abstract

Contrastive learning, a self-supervised learning method that can learn representations from unlabeled data, has been developed promisingly. Many methods of contrastive learning depend on data augmentation techniques, which generate different views from the original signal. However, tuning policies and hyper-parameters for more effective data augmentation methods in contrastive learning is often time and resource-consuming. Researchers have designed approaches to automatically generate new views for some input signals, especially on the image data. But the view-learning method is not well developed for time-series data. In this work, we propose a simple but effective module for automating view generation for time-series data in contrastive learning, named learning views for time-series data (LEAVES). The proposed module learns the hyper-parameters for augmentations using adversarial training in contrastive learning. We validate the effectiveness of the proposed method using multiple time-series datasets. The experiments demonstrate that the proposed method is more effective in finding reasonable views and performs downstream tasks better than the baselines, including manually tuned augmentation-based contrastive learning methods and SOTA methods.

1. INTRODUCTION

Contrastive learning has been widely applied to improve the robustness of the model for various downstream tasks such as images (Chen et al., 2020; Grill et al., 2020; Wang & Qi, 2022) and timeseries data (Mohsenvand et al., 2020; Mehari & Strodthoff, 2022) . Among the developed contrastive learning methods, data augmentation plays an essential role in generating different corrupted transformations, as views of the original input for the pretext task. For example, Chen et al. ( 2020) proposed a SimCLR method to maximize the agreements of augmented views from the same sample to pre-train the model, which significantly outperformed the previous state-of-the-art method in image classification with far less labeled data. However, the data augmentation methods selection is usually empirical, and tuning a set of optimized data augmentation methods can cost thousands of GPU hours even with the automating searching algorithms (Cubuk et al., 2019) . Therefore, it remains an open question how to effectively generate views for a new dataset. Instead of using artificially generated views, researchers have been putting efforts into training deep learning methods to generate optimized views for the input samples (Tamkin et al., 2020; Rusak et al., 2020) . These methods generate reasonably-corrupted views for the image datasets and result in satisfactory results. For example, Tamkin et al. (2020) proposed the ViewMaker, an adversarially trained convolutional module in contrastive learning, to generate augmentation for images. Nevertheless, the aforementioned method, such as the ViewMaker, might not be acclimatized when forthrightly utilized on the time-series data. The main challenge is that, for the time-series signal, we need to not only disturb the magnitudes (spatial) but also distort the temporal dimension Um et al. (2017); Mehari & Strodthoff (2022) . While the image-based methods can only disturb the spatial domain by adding reasonable noise to the input data. In this work, we propose LEAVES, which is a lightweight module for learning views on time-series data in contrastive learning. The LEAVES is optimized adversarially against the contrastive loss to generate challenging views for the encoder in learning representations. In addition, to introduce smooth temporal perturbations to the generated views, we propose a differentiable data augmentation technique for time-series data, named TimeDistort. Figure 1 shows the examples of the gener-

