FINDE: NEURAL DIFFERENTIAL EQUATIONS FOR FINDING AND PRESERVING INVARIANT QUANTITIES

Abstract

Many real-world dynamical systems are associated with first integrals (a.k.a. invariant quantities), which are quantities that remain unchanged over time. The discovery and understanding of first integrals are fundamental and important topics both in the natural sciences and in industrial applications. First integrals arise from the conservation laws of system energy, momentum, and mass, and from constraints on states; these are typically related to specific geometric structures of the governing equations. Existing neural networks designed to ensure such first integrals have shown excellent accuracy in modeling from data. However, these models incorporate the underlying structures, and in most situations where neural networks learn unknown systems, these structures are also unknown. This limitation needs to be overcome for scientific discovery and modeling of unknown systems. To this end, we propose first integral-preserving neural differential equation (FINDE). By leveraging the projection method and the discrete gradient method, FINDE finds and preserves first integrals from data, even in the absence of prior knowledge about underlying structures. Experimental results demonstrate that FINDE can predict future states of target systems much longer and find various quantities consistent with well-known first integrals in a unified manner.

1. INTRODUCTION

Modeling and predicting real-world systems are fundamental aspects of understanding the world in natural science and improving computer simulations in industry. Target systems include chemical dynamics for discovering new drugs (Raff et al., 2012) , climate dynamics for climate change prediction and weather forecasting (Rasp et al., 2020; Trigo & Palutikof, 1999) , and physical dynamics of vehicles and robots for optimal control (Nelles, 2001) . In addition to image processing and natural language processing (Devlin et al., 2018; He et al., 2016) , neural networks have been actively studied for modeling dynamical systems (Nelles, 2001) . Their history dates back to at least the 1990s (see Chen et al. (1990); Clouse et al. (1997) ; Levin & Narendra (1995) ; Narendra & Parthasarathy (1990); Sjöberg et al. (1994) ; Wang & Lin (1998) for examples). Recently, two notable but distinct families have been proposed. Physics-informed neural networks (PINNs) directly solve partial differential equations (PDEs) given as symbolic equations (Raissi et al., 2019) . Neural ordinary differential equations (NODEs) learn ordinary differential equations (ODEs) from observed data and solve them using numerical integrators (Chen et al., 2018) . Our focus this time is on NODEs. Most real-world systems are associated with first integrals (a.k.a. invariant quantities), which are quantities that remain unchanged over time (Hairer et al., 2006) . First integrals arise from intrinsic geometric structures of systems and are sometimes more important than superficial dynamics in understanding systems (see Appendix A for details). Many previous studies have extended NODEs by incorporating prior knowledge about first integrals and attempted to accurately learn a target system. Greydanus et al. (2019) proposed the Hamiltonian neural network (HNN), which employs a neural network to approximate Hamilton's equation, thereby conserving the system energy called the Hamiltonian. Finzi et al. (2020a) proposed neural network architectures that conserve linear and angular momenta by utilizing the graph structure. Finzi et al. (2020b) also extended an HNN to a system with holonomic constraints, which led to first integrals such as a pendulum length. 2020) proposed a model that preserves the total mass of a discretized PDE. These studies have demonstrated that the more prior knowledge a neural network has about first integrals, the more accurate their dynamics prediction. See Table 1 for comparisons. Previous studies have mainly attempted to preserve known first integrals for better computer simulations. However, in situations where a neural network learns a target system, it is naturally expected that first integrals associated with the target system are unknown, and it is not clear which of the above methods are available. Therefore, this study proposes first integral-preserving neural differential equation (FINDE) to find and preserve unknown first integrals from data in a unified manner. FINDE has two versions for continuous and discrete time; these have the following advantages. Finding First Integrals Many studies have designed architectures or operations of neural networks to model continuous-time dynamics with known types of first integrals. However, the underlying geometric structures of a target system are generally unknown in practice. In contrast, FINDE finds various types of first integrals from data in a unified manner and preserves them in predictions. For example, from an energy-dissipating system, FINDE can find first integrals other than energy. FINDE can find not only known first integrals, but also unknown ones. Hence, FINDE can lead to scientific discoveries. Combination with Known First Integrals FINDE can be combined with previously proposed neural networks designed to preserve known first integrals, such as HNNs. In addition, when some first integrals are known in advance, they can also be incorporated into FINDE to avoid rediscovery. Therefore, FINDE is available in various situations. Exact Preservation of First Integrals The first integral associated with a continuous-time system is destroyed after the dynamics is temporally discretized for computer simulations. By leveraging the discrete gradient, the discrete-time version of FINDE preserves first integrals exactly (up to rounding errors) in discrete time and further improves the prediction performance.



Comparison of Related Studies on Preservation of First Integrals.

2. BACKGROUND AND RELATED WORK

First Integrals Let us consider a time-invariant differential system d dt u = f (u) on an Ndimensional manifold M, where u denotes the system state and f : M → T u M represents a vector field on M. For simplicity, we suppose the manifold M to be a Euclidean space R N . Definition 1 (first integral). A quantity V : M → R is referred to as a first integral of a system(1) The tangent space T u M ′ ⊂ T u M of the submanifold M ′ ⊂ M at a point u is the orthogonal complement to the space spanned by the gradients ∇V k (u) of the first integrals V k for k = 1, . . . , K;(2) Conversely, if the time-derivative f at point u is on the tangent space T u M ′ for certain functions V k 's, the quantities V k 's are first integrals of the system d dt u = f (u); it holds that d dt V k (u) = ∇V k (u) ⊤ d dt u = ∇V k (u) ⊤ f (u) = 0.

