SELF-PACED LEARNING ENHANCED PHYSICS-INFORMED NEURAL NETWORKS FOR SOLVING PAR-TIAL DIFFERENTIAL EQUATIONS

Abstract

There is a hit discussion on solving partial differential equation by neural network. The famous PINN (physics-informed neural networks) has drawn worldwide attention since it was put forward. Despite its success in solving nonlinear partial differential equation, the difficulty in converging and the inefficiency in training process are definitely huge concerns. Normally, data for PINN is randomly chosen for a given distribution. Additionally, it's fitted to a model in a meaningless way. Curriculum Learning is a learning strategy that trains a model from easy samples to hard ones, which represents the meaningful human learning order. Self-paced Learning (SPL) is one of the significant branches of Automatic Curriculum Learning, which takes example-wise the training loss as Difficulty Measurer. SPL is an efficient strategy in enhancing the convergence rate of numerous models. In this paper, we propose a novel SPL-PINN learning framework, with SPL to accelerate the convergence progress of PINN. We demonstrate the effectiveness of SPL-PINN in a typical parabolic equation and Burgers equation.

1. INTRODUCTION

There is an increasing number of methods for solving partial differential equation. Weinan & Yu (2018) proposed a deep learning-based method, the Deep Rize Method, for solving variational problems, which are derived form partial differential equations. Sirignano & Spiliopoulos (2018) proposed a mesh-free method, the Deep Galerkin Method, to approximate the solution of a partial differential equation by a neural network instead of a linear combination of bias functions. (RAI, 2019) aimed to approximate the solution by neural network as well as the inner physical information. There is much profound research related to PINN, the research topics can be filed to three fields: the application of PINN to a specific problem, the parallel PINN and the acceleration of PINN. Papers related to PINN abound, suggesting that a great burst of creativity has occured since the emergence of PINN. In this paper, we concentrate on how to accelerate the converging rate in the training process. 

2.1. PHYSICS-INFORMED NEURAL NETWORK

Physics-informed neural network(RAI, 2019) is a successful mesh-free method for solving partial differential equations. Let's start by focusing on a general partial differential equation with both initial-value condition and boundary-value condition. ∂ t u(t, x) + L x u(t, x) = f (t, x) (x, t) ∈ Ω × (0, T ) u(0, x) = g(x) x ∈ Ω u(t, x) = h(t, x) (x, t) ∈ ∂Ω × (0, T ) (1) Here u(x, t) is the solution of this initial/boundary problem. L is the differential operator on the variable x. Ω is a subset of R n . PINN approximates u(x, t) by neural network with the physical information instead of traditional finite differential method and finite element method. The loss function can be divided into several parts. L = L F + L I + L B where L F = 1 n F n F i=1 |∂ t u net (t F i , x F i ) + L x u net (t F i , x F i ) -f (t F i , x F i )| 2 L I = 1 n I n I j=1 |u net (0, x I j ) -g(x I j )| 2 L B = 1 n B n B k=1 |u net (t B k , x B k ) -h(t B k , x B k ) In PINN, (t F i , x F i ), x I j and (t B k , x B k ) are randomly sampled from a given distribution. Specially, u(t, x) is derived from a back-propagation neural network by minimizing the proposed loss function 2 θ = argmin θ L (4) 2.2 SELF-PACED LEARNING Self-paced Learning(Kumar et al., 2010) is one of the significant branches of Automatic Curriculum Learning (Bengio et al., 2009) , which devotes a readily computable example-wise the training loss as Difficulty Measurer, alleviating the lack of how to efficiently identify easy samples. Let D = {x i , y i } N i=1 be the training dataset where x i ∈ X represents the i th input variable and y i ∈ Y represents the correspondingly i th output variable. Given the training data, the parameters θ of a special model are trained by minimizing the defined loss function L. Traditionally, the parameters are trained as follows: 



Xiang et al. (2021) proposed a self-adaptive loss function which automatically assigns the weights of different objectives. LIU (2021) is also proposed to automatically adjust the weights of different objectives. Thanasutives et al. (2021) treated it as a multi-task learning problem, they proposed adversarial learning to generate high-loss samples, similarly distributed to the original training distribution. Kim et al. (2020) proposed a dynamic pulling method, enabling PINN to learn dynamics of the governing equations. Yu et al. (2022) assumed that the solution of the given partial differential equation is smooth, so they took the gradient of the PDE residual as part of the loss function.Chiu et al. (2021) proposed a CAN-PINN framework, taking the advantage of automatic differentiation and numerical differentiation, increase the accuracy of PINN. Bischof & Kraus (2021) treated it as a multi-objective problem, proposed a novel self-adaptive loss balancing of PINN with random lookback. Generally, the methods aiming at the inefficiency in training process are treating the weights of different losses. In this paper, we concentrate on the training pattern, fit model form easy samples to hard samples.

i , y i ; θ) + R(θ)} (5)where R(•) is a Regularizer. SPL introduces SPL-Regularizer to the traditional learning pattern, thus parameters are trained in a meaningful way.θ = min θ { N i=1 v i L(x i , y i ; θ) + g(v i , λ)

