COMPETITIVE PHYSICS INFORMED NETWORKS

Abstract

Neural networks can be trained to solve partial differential equations (PDEs) by using the PDE residual as the loss function. This strategy is called "physicsinformed neural networks" (PINNs), but it currently cannot produce high-accuracy solutions, typically attaining about 0.1% relative error. We present an adversarial approach that overcomes this limitation, which we call competitive PINNs (CPINNs). CPINNs train a discriminator that is rewarded for predicting mistakes the PINN makes. The discriminator and PINN participate in a zero-sum game with the exact PDE solution as an optimal strategy. This approach avoids squaring the large condition numbers of PDE discretizations, which is the likely reason for failures of previous attempts to decrease PINN errors even on benign problems. Numerical experiments on a Poisson problem show that CPINNs achieve errors four orders of magnitude smaller than the best-performing PINN. We observe relative errors on the order of single-precision accuracy, consistently decreasing with each epoch. To the authors' knowledge, this is the first time this level of accuracy and convergence behavior has been achieved. Additional experiments on the nonlinear Schrödinger, Burgers', and Allen-Cahn equation show that the benefits of CPINNs are not limited to linear problems.

1. INTRODUCTION

PDE-constrained deep learning. Partial differential equations (PDEs) model physical phenomena like fluid dynamics, heat transfer, electromagnetism, and more. The rising interest in scientific machine learning motivates the study of PDE-constrained neural network training (Lavin et al., 2021) . Such methods can exploit physical structure for learning or serve as PDE solvers in their own right. Physics informed networks. Lagaris et al. (1998) represent PDE solutions as neural networks by including the square of the PDE residual in the loss function, resulting in a neural network-based PDE solver. Raissi et al. (2019) recently refined this approach further and called it "physics informed neural networks (PINNs)," initiating a flurry of follow-up work. PINNs are far less efficient than classical methods for solving most PDEs but are promising tools for high-dimensional or parametric PDEs (Xue et al., 2020) and data assimilation problems. The training of PINNs also serves as a model problem for the general challenge of imposing physical constraints on neural networks, an area of fervent and increasing interest (Wang et al., 2021b; Li et al., 2021; Donti et al., 2021) . Training pathologies in PINNs. PINNs can, in principle, be applied to all PDEs, but their numerous failure modes are well-documented (Wang et al., 2021a; Liu et al., 2021; Krishnapriyan et al., 2021) . For example, they are often unable to achieve high-accuracy solutions. The first works on PINNs reported relative L 2 errors of about 10 -3 (Raissi et al., 2019) . The authors are unaware of PINNs achieving errors below 10 -5 , even in carefully crafted, favorable settings. Higher accuracy is required in many applications. Existing remedies. A vast and growing body of work aims to improve the training of PINNs, often using problem-specific insights. For example, curriculum learning can exploit causality in timedependent PDEs (Krishnapriyan et al., 2021; Wang et al., 2022a; Wight & Zhao, 2020) . Krishnapriyan et al. ( 2021) also design curricula by embedding the PDE in a parametric family of problems of varying difficulty. Other works propose adaptive methods for selecting the PINN collocation points (Lu et al., 2021; Nabian et al., 2021; Daw et al., 2022) . Adaptive algorithms for weighing components of the PINN loss function have also been proposed (McClenny & Braga-Neto, 2020; Wang et al., 2022b; van der Meer et al., 2022) . Despite these improvements, the squared residual penalty method used by such PINNs imposes a fundamental limitation associated with conditioning, which is discussed next. The key problem: Squared residuals. Virtually all PINN-variants use the squared PDE residual as loss functions. For a linear PDE of order s, this is no different than solving an equation of order 2s, akin to using normal equations in linear algebra. The condition number κ of the resulting problem is thus the square of the condition number of the original one. Solving discretized PDEs is an ill-conditioned problem, inhibiting the convergence of iterative solvers and explaining the low accuracy of most previous PINNs. It is tempting to address this problem using penalties derived from p-norms with p = 2. However, choosing p > 2 leads to worse condition numbers, whereas p < 2 sacrifices the smoothness of the objective. The convergence rates of gradient descent on (non-)smooth convex problems suggest that this trade is unfavorable (Bubeck et al., 2015) . Weak formulations. Integration by parts allows the derivation of a weak form of a PDE, which for some PDEs can be turned into a minimization formulation that does not square the condition number. This procedure has been successfully applied by E & Yu (2017) to solve PDEs with neural networks (Deep Ritz). However, the derivation of such minimization principles is problem-dependent, limiting the generality of the formulation. Deep Ritz also employs penalty methods to enforce boundary values, though these preclude the minimization problem's solution from being the PDE's exact solution. Liao & Ming (2019) proposed a partial solution to this problem. The work most closely related to ours is by Zang et al. (2020) , who proposes a game formulation based on the weak form. Competitive PINNs. We propose Competitive Physics Informed Neural Networks (CPINNs) to address the above problems. CPINNs are trained using a minimax game between the PINN and a discriminator network. The discriminator learns to predict mistakes of the PINN and is rewarded for correct predictions, whereas the PINN is penalized. We train both players simultaneously on the resulting zero-sum game to reach a Nash equilibrium that matches the exact solution of the PDE.

Summary of contributions.

A novel variant of PINNs, called CPINNs, is introduced, replacing the penalty employed by PINNs with a primal-dual approach. This simultaneously optimizes the PINN and a discriminator network that learns to identify PDE violations and boundary constraint violations. We optimize PINNs with competitive gradient (CGD) (Schäfer & Anandkumar, 2019) and compare their performance to regular PINNs trained with Adam. On a two-dimensional Poisson problem, CPINNs achieve a relative accuracy of almost 10 -8 , improving over PINNs by four orders of magnitude. To the best of our knowledge, this is the first time a PINN-like network was trained to this level of accuracy. We compare PINNs with CPINNs on a nonlinear Schrödinger equation, a viscous Burgers' equation, and an Allen-Cahn equation. In all but the last case, CPINNs improve over PINNs trained with a comparable computational budget.

2. COMPETITIVE PINN FORMULATION

We formulate CPINNs for a PDE of the general form A[u] = f, in Ω (1) u = g, on ∂Ω, where A[•] is a (possibly nonlinear) differential operator and Ω is a domain in R d with boundary ∂Ω. To simplify notation, we assume that f , g, and u are real-valued functions on Ω, ∂Ω, and Ω ∪ ∂Ω, respectively. One can extend both PINNs and CPINNs to vector-valued such functions if needed. 



PHYSICS INFORMED NEURAL NETWORKS (PINNS)PINNs approximate the PDE solution u by a neural network P mapping d-variate inputs to real numbers. The weights are chosen such as to satisfy equation 1 and equation 2 on the points x ⊂ Ω and x ⊂ ∂Ω. The loss function used to train P has the form L PINN (P, x, x) = L PINN Ω (P, x Ω ) + L PINN ∂Ω (P, x),

