MITIGATING PROPAGATION FAILURES IN PINNS USING EVOLUTIONARY SAMPLING

Abstract

Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), it is known that PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing and mitigating the "failure modes" of PINNs. While most of these studies have focused on balancing loss functions or adaptively tuning PDE coefficients, what is missing is a thorough understanding of the connection between failure modes of PINNs and sampling strategies used for training PINNs. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that the training of PINNs rely on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures. We additionally demonstrate that propagation failures are characterized by highly imbalanced PDE residual fields where very high residuals are observed over very narrow regions. To mitigate propagation failures, we propose a novel evolutionary sampling (Evo) method that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of Evo to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of Evo and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.

1. INTRODUCTION

Physics-informed neural networks (PINNs) (Raissi et al., 2019) represent a seminal line of work in deep learning for solving partial differential equations (PDEs), which appear naturally in a number of domains. The basic idea of PINNs for solving a PDE is to train a neural network to minimize errors w.r.t. the solution provided at initial/boundary points of a spatio-temporal domain, as well as the PDE residuals observed over a sample of interior points, referred to as collocation points. Despite the success of PINNs, it is known that PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs, as reflected in several recent studies on characterizing the "failure modes" of PINNs (Wang et al., 2020; 2022c; Krishnapriyan et al., 2021) . Many of these failure modes are related to the susceptibility of PINNs in getting stuck at trivial solutions acting as poor local minima, due to the unique optimization challenges of PINNs. In particular, note that training PINNs is different from conventional deep learning problems as we only have access to the correct solution on the initial and/or boundary points, while for all interior points in the domain, we can only compute PDE residuals. Also note that minimizing PDE residuals does not guarantee convergence to a correct solution since there are many trivial solutions of commonly observed PDEs that show 0 residuals. While previous studies on understanding and preventing failure modes of PINNs have mainly focused on modifying network architectures or balancing loss functions during PINN training, the effect of sampling collocation points on avoiding failure modes of PINNs has been largely overlooked. Although some previous approaches have explored the effect of sampling strategies on PINN training (Wang et al., 2022a; Lu et al., 2021) , they either suffer from large computation costs or fail to converge to correct solutions, empirically demonstrated in our results. In this work, we present a novel perspective of failure modes of PINNs by postulating the propagation hypothesis: "in order for PINNs to avoid converging to trivial solutions at interior points, the correct solution must be propagated from the initial/boundary points to the interior points." When this propagation is hindered, PINNs can get stuck at trivial solutions that are difficult to escape, referred to as the propagation failure mode. This hypothesis is motivated from a similar behavior observed in numerical methods where the solution of the PDE at initial/boundary points are iteratively propagated to interior points using finite differencing schemes (LeVeque, 2007) . We show that propagation failures in PINNs are characterized by highly imbalanced PDE residual fields, where very high residuals are observed in narrow regions of the domain. Such high residual regions are not adequately sampled in the set of collocation points (which generally is kept fixed across all training iterations), making it difficult to overcome the propagation failure mode. This motivates us to develop sampling strategies that focus on selecting more collocation points from high residual regions. This is related to the idea of local-adaptive mesh refinement used in FEM (Zienkiewicz et al., 2005) to selectively refine the computational mesh in regions with higher errors. We propose a novel evolutionary sampling (Evo) strategy that can accumulate collocation points in high PDE residual regions, thereby dynamically emphasizing on these skewed regions as we progress in training iterations. We also provide a causal extension of our proposed Evo algorithm (Causal Evo) that can explicitly encode the strong inductive bias of causality in propagating the solution from initial points to interior points over training iterations, when solving time-dependent PDEs. We theoretically prove the adaptive quality of Evo to accumulate points from high residual regions that persist over iterations. We empirically demonstrate the efficacy and efficiency of our proposed sampling methods in a variety of benchmark PDE problems previously studied in the PINN literature. We show the Evo and Causal Evo are able to mitigate propagation failure modes and converge to the correct solution with significantly smaller sample sizes as compared to baseline methods, while incurring negligible computational overhead. We also demonstrate the ability of Evo to solve a particularly hard PDE problem-solving 2D Eikonal equations for complex arbitrary surface geometries. The novel contributions of our work are as follows: (1) We provide a novel perspective for characterizing failure modes in PINNs by postulating the "Propagation Hypothesis." (2) We propose a novel evolutionary algorithm Evo to adaptively sample collocation points in PINNs that shows superior prediction performance empirically with little to no computational overhead compared to existing methods for adaptive sampling. (3) We theoretically show that Evo can accumulate points from high residual regions if they persist over iterations and release points if they have been resolved by PINN training, while maintaining non-zero representation of points from other regions.

2. BACKGROUND AND RELATED WORK

Physics-Informed Neural Networks (PINNs). The basic formulation of PINN (Raissi et al., 2017) is to use a neural network f θ (x, t) to infer the forward solution u of a non-linear PDE: u t + N x [u] = 0, x ∈ X , t ∈ [0, T ]; u(x, 0) = h(x), x ∈ X ; u(x, t) = g(x, t), t ∈ [0, T ], x ∈ ∂X where x and t are the space and time coordinates, respectively, X is the spatial domain, ∂X is the boundary of spatial domain, and T is the time horizon. The PDE is enforced on the entire spatiotemporal domain (Ω = X × [0, T ]) on a set of collocation points {x r i = (x i r , t i r )} Nr i=1 by computing the PDE residual (R(x, t)) and the corresponding PDE Loss (L r ) as follows: R θ (x, t) = ∂ ∂t f θ (x, t) -N x [f θ (x, t)] (1) L r (θ) = E xr∼U (Ω) [R θ (x r ) 2 ] ≈ 1 N r Nr i=1 [R θ (x i r , t i r )] 2 (2) where L r is the expectation of the squared PDE Residuals over collocation points sampled from a uniform distribution U. PINNs approximate the solution of the PDE by optimizing the following overall loss function L = λ r L r (θ) + λ bc L bc (θ) + λ ic L ic (θ), where L ic and L bc are the mean squared loss on the initial and boundary data respectively, and λ r , λ ic , λ bc are hyperparameters that control the interplay between the different loss terms. Although PINNs can be applied to inverse problems, i.e., to estimate PDE parameters from observations, we only focus on forward problems in this paper. 



Work on Characterizing Failure Modes of PINNs. Despite the popularity of PINNs in approximating PDEs, several works have emphasized the presence of failure modes while training PINNs. One early work (Wang et al., 2020) demonstrated that imbalance in the gradients of multiple loss terms could lead to poor convergence of PINNs, motivating the development of Adaptive PINNs. Another recent development (Wang et al., 2022c) made use of the Neural Tangent Kernel (NTK) theory to indicate that the different convergence rates of the loss terms can lead to training instabilities.

