IMPOSING CONSERVATION PROPERTIES IN DEEP DY-NAMICS MODELING VIA CONTRASTIVE LEARNING

Abstract

Deep neural networks (DNN) has shown great capacity of modeling a dynamical system, but these DNN-based dynamical models usually do not obey conservation laws. To impose the learned DNN dynamical models with key physical properties such as conservation laws, this paper proposes a two-step approach to endow the invariant priors into the simulations. We first establish a contrastive learning framework to capture the system invariants along the trajectory observations. During the dynamics modeling, we design a projection layer of DNNs to preserve the system invariance. Through experiments, we show our method consistently outperforms the baseline in both coordinate error and conservation metrics and can be further extended to complex and large dynamics by leveraging autoencoder. Notably, a byproduct of our framework is the automated conservation law discovery for dynamical systems with single conservation property.

1. INTRODUCTION

With the quick growth of computational resources and massive prediction power of neural networks, recent times have seen great success of artificial intelligence in a wide range of applications such as image classification (He et al., 2016) , natural language processing (Vaswani et al., 2017; Devlin et al., 2018) and reinforcement learning (Mnih et al., 2013) . Despite the scalability and diversity of modern machine learning tasks, extracting the underlying latent mechanism from training data and deploying the knowledge toward the new occurrence have always been the heart of artificial intelligence. The idea to work with a compressed representation or prior information has been historically entangled with the development of machine learning, from earlier tools like clustering and principal component analysis (PCA) to more contemporary autoencoders or embeddings. In recent years, there is a surge in interest to discover knowledge from larger or even different domains leveraging techniques like representation learning (Bengio et al., 2013; Chen & He, 2021) and transfer learning (Weiss et al., 2016) to more general meta-learning (Rusu et al., 2018; Santoro et al., 2016) and foundation models (Bommasani et al., 2021) which are capable of handling a wide range of tasks. Many critical discoveries in the world of physics were driven by distilling the invariants from observations. For instance, the Kepler laws were found by analyzing and fitting parameters for the astronomical observations, and the mass conservation law was first carried out by a series of experiments. However, such discovery usually requires extensive human insights and customized strategies for specific problems. This naturally raises a question, can we learn certain conservation laws from real-world data in an automated fashion? On the other hand, data-driven dynamical modeling is prone to violation of physics laws or instability issues (Greydanus et al., 2019; Kolter & Manek, 2019) , since the model only statistically learns the data or system state function without knowing physics prior. In this paper, we provide a novel contrastive perspective to find one or more distinguishing features (i.e. conservation values) of physics-based system trajectories. By comparing the latent space distance of the system state observations, we aim to learn a low-dimensional representation potentially serving as the invariant term for the system. With such inspiration, we propose ConCerNet consisting of two neural networks. The first network contrastively learns the trajectory invariants, the second network captures the nominal system dynamical behavior which will be corrected by the first network to preserve certain properties of the simulation in the long term prediction. The correction is implemented by projecting the dynamical neural network output on the learned conservation man-

