IMPOSING CONSERVATION PROPERTIES IN DEEP DY-NAMICS MODELING VIA CONTRASTIVE LEARNING

Abstract

Deep neural networks (DNN) has shown great capacity of modeling a dynamical system, but these DNN-based dynamical models usually do not obey conservation laws. To impose the learned DNN dynamical models with key physical properties such as conservation laws, this paper proposes a two-step approach to endow the invariant priors into the simulations. We first establish a contrastive learning framework to capture the system invariants along the trajectory observations. During the dynamics modeling, we design a projection layer of DNNs to preserve the system invariance. Through experiments, we show our method consistently outperforms the baseline in both coordinate error and conservation metrics and can be further extended to complex and large dynamics by leveraging autoencoder. Notably, a byproduct of our framework is the automated conservation law discovery for dynamical systems with single conservation property.

1. INTRODUCTION

With the quick growth of computational resources and massive prediction power of neural networks, recent times have seen great success of artificial intelligence in a wide range of applications such as image classification (He et al., 2016) , natural language processing (Vaswani et al., 2017; Devlin et al., 2018) and reinforcement learning (Mnih et al., 2013) . Despite the scalability and diversity of modern machine learning tasks, extracting the underlying latent mechanism from training data and deploying the knowledge toward the new occurrence have always been the heart of artificial intelligence. The idea to work with a compressed representation or prior information has been historically entangled with the development of machine learning, from earlier tools like clustering and principal component analysis (PCA) to more contemporary autoencoders or embeddings. In recent years, there is a surge in interest to discover knowledge from larger or even different domains leveraging techniques like representation learning (Bengio et al., 2013; Chen & He, 2021) and transfer learning (Weiss et al., 2016) to more general meta-learning (Rusu et al., 2018; Santoro et al., 2016) and foundation models (Bommasani et al., 2021) which are capable of handling a wide range of tasks. Many critical discoveries in the world of physics were driven by distilling the invariants from observations. For instance, the Kepler laws were found by analyzing and fitting parameters for the astronomical observations, and the mass conservation law was first carried out by a series of experiments. However, such discovery usually requires extensive human insights and customized strategies for specific problems. This naturally raises a question, can we learn certain conservation laws from real-world data in an automated fashion? On the other hand, data-driven dynamical modeling is prone to violation of physics laws or instability issues (Greydanus et al., 2019; Kolter & Manek, 2019) , since the model only statistically learns the data or system state function without knowing physics prior. In this paper, we provide a novel contrastive perspective to find one or more distinguishing features (i.e. conservation values) of physics-based system trajectories. By comparing the latent space distance of the system state observations, we aim to learn a low-dimensional representation potentially serving as the invariant term for the system. With such inspiration, we propose ConCerNet consisting of two neural networks. The first network contrastively learns the trajectory invariants, the second network captures the nominal system dynamical behavior which will be corrected by the first network to preserve certain properties of the simulation in the long term prediction. The correction is implemented by projecting the dynamical neural network output on the learned conservation man-Figure 1 : Pipeline to learn the dynamical system conservation and enforce it in simulation. A contrastive learning framework is proposed to extract the invariants across trajectory observations, then the dynamical model is projected to the invariant manifold to guarantee the conservation property. ifold learned by the first module, and therefore enforcing the trajectory conservation for the learned invariants. We summarize our main contributions as follows: • We provide a novel contrastive learning perspective of dynamical system trajectory data to capture the invariants of dynamical systems. One byproduct of this method is that the learned invariant functions discover physical conservation laws in certain cases. To the best of the authors' knowledge, this is the first work that studies the discovery of conservation laws for general dynamical systems through contrastive learning. • We propose a projection layer to impose any invariant function for dynamical system trajectory prediction, guaranteeing the conservation property during simulation. • Based on the above two components, we establish a generic learning framework for dynamical system modeling named ConCerNet (CONtrastive ConsERved Network) which provides robustness in prediction outcomes and flexibility to be applied to a wide range of dynamical systems that mandate conservation properties. We conducted extensive experiments to demonstrate the efficacy of ConCerNet, especially its remarkable improvement over a generic neural network in prediction error and conservation violation metrics. • We draw inferences on the relationship between the contrastively learned function, the exact conservation law, and the logistics of contrastive invariant learning. Further, potential improvements to the proposed method are illustrated that can improve the automated scientific discovery process.

2. BACKGROUND AND RELATED WORK

2.1 CONTRASTIVE LEARNING Unlike discriminative models that explicitly learn the data mappings, contrastive learning aims to extract the data representation implicitly by comparing among examples. The early idea dates back to the 1990s (Bromley et al., 1993) and has been widely adopted in many areas. One related field to

