SIMPER: SIMPLE SELF-SUPERVISED LEARNING OF PERIODIC TARGETS

Abstract

From human physiology to environmental evolution, important processes in nature often exhibit meaningful and strong periodic or quasi-periodic changes. Due to their inherent label scarcity, learning useful representations for periodic tasks with limited or no supervision is of great benefit. Yet, existing self-supervised learning (SSL) methods overlook the intrinsic periodicity in data, and fail to learn representations that capture periodic or frequency attributes. In this paper, we present SimPer, a simple contrastive SSL regime for learning periodic information in data. To exploit the periodic inductive bias, SimPer introduces customized augmentations, feature similarity measures, and a generalized contrastive loss for learning efficient and robust periodic representations. Extensive experiments on common real-world tasks in human behavior analysis, environmental sensing, and healthcare domains verify the superior performance of SimPer compared to state-of-the-art SSL methods, highlighting its intriguing properties including better data efficiency, robustness to spurious correlations, and generalization to distribution shifts.



While learning periodic targets is important, labeling such data is typically challenging and resource intensive. For example, if designing a method to measure heart rate, collecting videos with highly synchronized gold-standard signals from a medical sensor is time consuming, labor intensive, and requires storing privacy sensitive bio-metric data. Fortunately, given the large amount of unlabeled data, self-supervised learning that captures the underlying periodicity in data would be promising.



representations of different methods on a periodic learning dataset, RotatingDigits (details in Section 4). Existing self-supervised learning schemes fail to capture the underlying periodic or frequency information in data. In contrast, SimPer learns robust periodic representations with high frequency resolution.1 INTRODUCTIONPractical and important applications of machine learning in the real world, from monitoring the earth from space using satellite imagery(Espeholt et al., 2021)  to detecting physiological vital signs in a human being(Luo et al., 2019), often involve recovering periodic changes. In the health domain, learning from video measurement has shown to extract (quasi-)periodic vital signs including atrial fibrillation(Yan et al., 2018), sleep apnea episodes (Amelard et al., 2018) and blood pressure(Luo  et al., 2019). In the environmental remote sensing domain, periodic learning is often needed to enable nowcasting of environmental changes such as precipitation patterns or land surface temperature(Sønderby et al., 2020). In the human behavior analysis domain, recovering the frequency of changes or the underlying temporal morphology in human motions (e.g., gait or hand motions) is crucial for those rehabilitating from surgery(Gu et al., 2019), or for detecting the onset or progression of neurological conditions such as Parkinson's disease(Liu et al., 2022; Yang et al., 2022b).

availability

Code and data are available at: https://github.com/YyzHarry/SimPer.

