Regularized dynamical parametric approximation for stiff evolution problems

Abstract : Evolutionary deep neural networks have emerged as a rapidly growing field of research. This talk
discusses numerical integrators for such and other classes of nonlinear parametrizations u(t) = Φ(θ(t)) where
the evolving parameters θ(t) are to be computed. The primary focus is on tackling the challenges posed
by the combination of stiff evolution problems and irregular parametrizations, which typically arise with
neural networks, tensor networks, flocks of evolving Gaussians, and in further cases of overparametrization.
Regularized parametric versions of classical time stepping schemes for the time integration of the parameters
in nonlinear approximations to evolutionary partial differential equations are presented. At each time step,
an ill-conditioned nonlinear optimization problem is solved approximately with a few regularized Gauß–
Newton iterations. Error bounds for the resulting parametric integrator are shown. Numerical experiments
that are designed to show key properties of the proposed parametric integrators are discussed.