Decision & Control Seminar: Prof. Rainer Engelken
- Sponsor
- Decision and Control Laboratory, Coordinated Science Laboratory
- Speaker
- Prof. Rainer Engelken, Assisant Professor, Electrical & Computer Engineering
- Contact
- Daniel Liberzon
- liberzon@illinois.edu
- Views
- 19
- Originating Calendar
- CSL Decision and Control Group
Title: Gradient Flossing: Regularizing Lyapunov Exponents to Stabilize RNN Training
Abstract: Exploding and vanishing gradients in recurrent neural networks arise from the repeated multiplication of state-dependent Jacobians along a trajectory. From a dynamical systems perspective, these long Jacobian products amplify or attenuate perturbations at exponential rates that are captured by the (finite-time) Lyapunov spectrum.
In this talk, I will present Gradient Flossing, a regularization method that stabilizes learning by directly shaping the leading Lyapunov exponents of the network dynamics. The key idea is to add a training objective that discourages excessive expansion or contraction in the linearized dynamics. By keeping the singular values of long Jacobian products near unity, we explicitly optimize the condition number of the backward pass, acting as a dynamics-based stabilizer for backpropagation through time.
I will demonstrate that this approach enables standard RNNs to learn dependencies across substantially longer time horizons than conventional training. Finally, I will discuss how this method bridges the gap between the stability of nonlinear dynamical systems and the optimization geometry of recurrent models.
Location & Time: CSL B02, January 28, 3-4PM
Reception in CSL154 at 2:30PM