Beyond Backpropagation: Bilevel Optimization Through Implicit Differentiation and Equilibrium Propagation
METADATA ONLY
Loading...
Author / Producer
Date
2022-12
Publication Type
Journal Article
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
This review examines gradient-based techniques to solve bilevel optimization problems. Bilevel optimization extends the loss minimization framework underlying statistical learning to systems that are implicitly defined through a quantity they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers, and even physical systems and allows for greater modeling flexibility compared to the usual explicit definition of such systems. We focus on solving learning problems of this kind through gradient descent, leveraging the toolbox of implicit differentiation and, for the first time applied to this setting, the equilibrium propagation theorem. We present the mathematical foun-dations behind such methods, introduce the gradient estimation algorithms in detail, and compare the competitive advantages of the different approaches.
Permanent link
Publication status
published
External links
Editor
Book title
Journal / series
Volume
34 (12)
Pages / Article No.
2309 - 2346
Publisher
MIT Press
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Organisational unit
03672 - Steger, Angelika (emeritus) / Steger, Angelika (emeritus)
09479 - Grewe, Benjamin / Grewe, Benjamin