HOMOTOPY-BASED TRAINING OF NEURALODES FOR ACCURATE DYNAMICS DISCOVERY

Abstract

Conceptually, Neural Ordinary Differential Equations (NeuralODEs) pose an attractive way to extract dynamical laws from time series data, as they are natural extensions of the traditional differential equation-based modeling paradigm of the physical sciences. In practice, NeuralODEs display long training times and suboptimal results, especially for longer duration data where they may fail to fit the data altogether. While methods have been proposed to stabilize NeuralODE training, many of these involve placing a strong constraint on the functional form the trained NeuralODE can take that the actual underlying governing equation does not guarantee satisfaction. In this work, we present a novel NeuralODE training algorithm that leverages tools from the chaos and mathematical optimization communities -synchronization and homotopy optimization -for a breakthrough in tackling the NeuralODE training obstacle. We demonstrate architectural changes are unnecessary for effective NeuralODE training. Compared to the conventional training methods, our algorithm achieves drastically lower loss values without any changes to the model architectures. Experiments on both simulated and real systems with complex temporal behaviors demonstrate NeuralODEs trained with our algorithm are able to accurately capture true long term behaviors and correctly extrapolate into the future.

1. INTRODUCTION

Predicting the evolution of a time varying system and discovering mathematical models that govern it is paramount to both deeper scientific understanding and potential engineering applications. The centuries-old paradigm to tackle this problem was to either ingeniously deduce empirical rules from experimental data, or mathematically derive physical laws from first principles. However, the complexities of the systems of interest have grown so much that these traditional approaches are now often insufficient. This has led to a growing interest in using machine learning methods to infer dynamical laws from data. One school of thought, such as the seminal work of Schmidt & Lipson (2009) or Brunton et al. (2016) , focuses on deducing the exact symbolic form of the governing equations from data using techniques such as genetic algorithm or sparse regression. While these methods are powerful in that they output mathematical equations that are directly human-interpretable, they require prior information on the possible terms that may enter the underlying equation. This hinders the application of symbolic approaches to scenarios where there is insufficient prior information on the possible candidate terms, or complex, nonlinear systems whose governing equations involve non-elementary functions. On the other hand, neural network-based methods, such as Raissi et al. (2018) , leverage the universal approximation capabilities of neural networks to model the underlying dynamics of the system without explicitly involving mathematical formulae. Of the various architectual designs in literature, Neural Ordinary Differential Equations(NeuralODEs) Chen et al. (2018) stand out in particular because these seamlessly incorporate neural networks inside ordinary differential equations (ODES), thus bridging the expressibility and flexibility of neural networks with the de facto mathematical language of the physical sciences. Subsequent works have expanded on this idea, including blending NeuralODEs with partial information on the form of the governing equation to produce "grey-box"

