We introduce Hidden Physics Models, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expressed by time dependent and nonlinear partial differential equations, to extract patterns from high-dimensional data generated from experiments. The proposed methodology may be applied to the problem of learning, system identification, or data-driven discovery of partial differential equations. Our framework relies on Gaussian Processes, a powerful tool for probabilistic inference over functions, that enables us to strike a balance between model complexity and data fitting. The effectiveness of the proposed approach is demonstrated through a variety of canonical problems, spanning a number of scientific domains, including the Navier-Stokes, Schrödinger, Kuramoto-Sivashinsky, and time dependent linear fractional equations. The methodology provides a promising new direction for harnessing the long-standing developments of classical methods in applied mathematics and mathematical physics to design learning machines with the ability to operate in complex domains without requiring large quantities of data.
Let us consider parametrized and nonlinear partial differential equations of the general form
where denotes the latent (hidden) solution and is a nonlinear operator parametrized by . Given noisy measurements of the system, one is typically interested in the solution of two distinct problems.
The first problem is that of inference or filtering and smoothing, which states: given fixed model parameters what can be said about the unknown hidden state of the system? This question is the topic Numerical Gaussian Processes in which we address the problem of inferring solutions of time dependent and nonlinear partial differential equations using noisy observations.
The second problem is that of learning, system identification, or data driven discovery of partial differential equations stating: what are the parameters that best describe the observed data?
Here, we assume that all we observe are two snapshots of the system at times and that are apart. The main assumption is that is small enough so that we can employ the backward Euler time stepping scheme and obtain the discretized equation
Here, is the hidden state of the system at time . Approximating the nonlinear operator on the left-hand-side of this equation by a linear one we obtain
Similar to the ideas presented here and here, we build upon the analytical property of Gaussian processes that the output of a linear system whose input is Gaussian distributed is again Gaussian. Specifically, we proceed by placing a Gaussian process prior over the latent function ; i.e.,
Here, denotes the hyper-parameters of the covariance function . This enables us to capture the entire structure of the operator in the resulting multi-output Gaussian process
We call the resulting multi-output Gaussian process a Hidden Physics Model, because its matrix of covariance functions explicitly encodes the underlying laws of physics expressed by the corresponding partial differential equation.
Given noisy data and on the latent solution at times and , respectively, the hyper-parameters of the covariance functions and more importantly the parameters of the operators and can be learned by employing a Quasi-Newton optimizer L-BFGS to minimize the negative log marginal likelihood
where , , and is given by
Here, is the total number of data points in . Moreover, is included to capture the noise in the data and is also learned by minimizing the negative log marginal likelihood.
The Kuramoto-Sivashinsky equation is a canonical model of a pattern forming system with spatio-temporal chaotic behavior. The sign of the second derivative term is such that it acts as an energy source and thus has a destabilizing effect. The nonlinear term, however, transfers energy from low to high wave numbers where the stabilizing fourth derivative term dominates.
Navier-Stokes equations describe the physics of many phenomena of scientific and engineering interest. They may be used to model the weather, ocean currents, water flow in a pipe and air flow around a wing. The Navier-Stokes equations in their full and simplified forms help with the design of aircraft and cars, the study of blood flow, the design of power stations, the analysis of the dispersion of pollutants, and many other applications.
Fractional operators often arise in modeling anomalous diffusion processes and other non-local interactions. Integer values can model classical advection and diffusion phenomena. However, under the fractional calculus setting, the fractional order can assume real values and thus continuously interpolate between inherently different model behaviors. The proposed framework allows the fractional order to be directly inferred from noisy data, and opens the path to a flexible formalism for model discovery and calibration.
We have introduced a structured learning machine which is explicitly informed by the underlying physics that possibly generated the observed data. Exploiting this structure is critical for constructing data-efficient learning algorithms that can effectively distill information in the data-scarce scenarios appearing routinely when we study complex physical systems. We applied the proposed framework to the problem of identifying general parametric nonlinear partial differential equations from noisy data. This generality was demonstrated using various benchmark problems with different attributes. This work should be considered a direct follow up on Numerical Gaussian Processes in which a similar methodology is employed to infer solutions to time-dependent and nonlinear partial differential equations, and effectively quantify and propagate uncertainty due to noisy initial or boundary data. The ideas introduced in these two papers provide a natural platform for learning from noisy data and computing under uncertainty.
This work received support by the DARPA EQUiPS grant N66001-15-2-4055, the MURI/ARO grant W911NF-15-1-0562, and the AFOSR grant FA9550-17-1-0013. All data and codes are publicly available on GitHub.