FOURIER PINNS: FROM STRONG BOUNDARY CONDI-TIONS TO ADAPTIVE FOURIER BASES

Abstract

Interest in Physics-Informed Neural Networks (PINNs) is rising as a mesh-free alternative to traditional numerical solvers for partial differential equations (PDEs). While successful, PINNs often struggle to learn high-frequency and multi-scale target solutions-which, according to prior analysis, might arise from competition during optimization between the weakly enforced boundary loss and residual loss terms. By creatively modifying the neural network architecture, some simple boundary conditions (BCs) can be satisfied exactly without jointly optimizing an additional loss term, thus avoiding the aforementioned competition. Motivated by this analysis, we first study a strong BC version of PINNs for Dirichlet BCs and observe a consistent improvement compared to the standard PINNs. We conducted a Fourier analysis and found that strong BC PINNs can better learn the amplitudes of high-frequency components of the target solutions. While BC PINNs provide improvement, constructing such architectures is an intricate process made difficult (if not impossible) by certain BCs and domain geometries. Enlightened by our analysis, we propose Fourier PINNs -a simple, general, yet powerful method that augments PINNs with pre-specified, dense Fourier bases. Our proposed architecture likewise better learns high-frequency components but places no restrictions on the particular BCs. We developed an adaptive learning and basis selection algorithm based on alternating NN basis optimization, Fourier and NN basis coefficient estimation, and coefficient truncation. This scheme can flexibly identify the significant frequencies while weakening the nominal to better capture the target solution's power spectrum. We show the advantage of our approach in a set of systematic experiments.

1. Introduction

Physics-informed neural networks (PINNs) (Raissi et al., 2019a) are emergent mesh-free approaches to solving partial differential equations (PDE)s. They have shown successful in many scientific and engineering problems, such as bio-engineering (Sahli Costabal et al., 2020; Kissas et al., 2020) , fluids mechanics (Raissi et al., 2019b; Sun et al., 2020; Raissi et al., 2020) , fractional PDEs (Pang et al., 2019b; 2020) , and material design (Fang & Zhan, 2019; Liu & Wang, 2019) . The PINN framework uses neural networks (NNs) to estimate PDE solutions, in light of the universal approximation ability of the NNs. Specifically, consider a PDE of the following general form, F[u](x) = f (x) (x ∈ Ω), u(x) = g(x) (x ∈ ∂Ω) (1) where F is the differential operator for the PDE, Ω is the domain, ∂Ω is the boundary of the domain. To solve the PDE, the PINN uses a deep neural network u θ (x) to represent the solution u, samples N collocation points {x i c } N i=1 from Ω and M points {x i b } M i=1 from ∂Ω, and minimizes the loss, θ * = argmin θ L b (θ) + L r (θ) where L b (θ) = 1 M M j=1 u θ (x j b ) -g(x j b ) 2 is the boundary loss to fit the boundary condition, and L r (θ) = 1 N N j=1 F[ u θ ](x j c ) -f (x j c ) 2 is the residual loss to fit the equation. Despite their success, the training of PINNs is often unstable, and the performance can be poor from time to time, especially when solutions includes high-frequency and multi-scale components. From the optimization perspective, Wang et al. (2020a) pointed out that due to the imbalance of the gradient magnitudes of the boundary loss and residual loss (the latter is often much larger), the training can be dominated by the residual loss and hence underfits the boundary condition. Wang et al. (2020c) confirmed this conclusion from a neural tangent kernel (NTK) analysis on the training behaviors of PINNs with wide networks. They found the eigenvalues of the residual kernel matrix are often dominant, which can cause the training to mainly fit the residual loss. On the other hand, Rahaman et al. ( 2019) found the "spectrum bias" in learning standard NNs, namely, the low frequency information can be easily learned from data but grasping the high frequencies is much slower and harder. To alleviate this issue, Tancik et al. (2020) proposed to randomly sample a set of high frequencies from a large-variance Gaussian distribution, and use these frequencies to construct random Fourier features as the input to the subsequent NN. Wang et al. ( 2021) gave a justification of this approach via NTK analysis, and used multiple Gaussian variances to sample the frequencies and construct Fourier features so as to capture multi-scale information in the PINN framework. While effective, the performance of this method is sensitive to the number and scales of the Gaussian variances, which are often difficult to choose because the solution is unknown apriori. The contributions of our work are as follows: • Motivated by the prior analysis from the optimization perspective, we investigated a strong boundary condition (BC) version of PINNs for simple Dirichlet BC's. This PINN variant satisfies the boundary conditions exactly (through specific NN architecture construction) and does not require a boundary loss term during optimization-thus avoiding the competition with the residual loss term. We observed significant improvement upon the standard PINN, especially for higher frequency problems. • Different from the previous investigation, we conducted a Fourier analysis on our strong BC PINNs using a specific boundary function-the function that constrains the NN output to satisfying the boundary conditions exactly. Through Fourier series and convolution theory, we found that, interestingly, multiplying the NN with the boundary function enables faster and more accurate learning of the coefficients of higher frequencies in the target solution. By contrast, standard PINNs exhibit hardship in capturing correct coefficients in the high frequency domain. • Enlightened by our analysis, we developed Fourier PINNs-a simple, general, yet powerful extension of PINNs, independent of any particular boundary condition (unlike Strong BC PINNs). The solution is modeled as an NN plus the linear combination of a set of dense Fourier bases, where the frequencies are evenly sampled from a large range. We developed an adaptive learning and basis selection algorithm, which alternately optimize the NN basis parameters and the coefficients of the NN and Fourier bases, meanwhile pruning useless or insignificant bases. In this way, our method can quickly identify important frequencies, supplement frequencies missed by the NN, and improve the amplitude estimation. We only need to specify a large enough range and small enough spacing for the Fourier bases, without the need for worrying about the actual number and scales of the frequencies in the true solution as in previous methods. All these can be automatically inferred during training. • We evaluated FourierPINNs in several benchmark PDEs with high-frequency and multifrequency solutions and solutions that couple high-frequency components with plain functions. In all the cases, Fourier PINNs consistently achieve reasonable and good solution errors, e.g., ∼ 10 -3 or ∼ 10 -2 . As a comparison, the standard PINNs always failed, while the strong BC PINNs were much worse than our method (though better than the standard PINNs). The PINNs with random Fourier features often failed under a variety of choices of the Gaussian variance number and scales. The performance is highly sensitive to this choice. We also tested PINNs with large boundary loss weights (Wight & Zhao, 2020), and with an adaptive activation function (Jagtap et al., 2020) . FourierPINNs consistently outperformed both methods.

2. Strong Boundary Condition PINNs

According to prior analysis (Wang et al., 2020a; c) , the instability of PINNs is likely from the competition between the weakly-enforced boundary loss and the residual loss during optimization. To sidestep this issue, a natural idea is to design a surrogate model that satisfies the boundary condition, and hence the training no longer needs a weakly-forced boundary loss. To this end, we consider a

