Deterministic equivalent and error universality of deep random features learning
Metadata only
Date
2023Type
- Conference Paper
ETH Bibliography
yes
Altmetrics
Abstract
This manuscript considers the problem of learning a random Gaussian network function using a fully connected network with frozen intermediate layers and trainable readout layer. This problem can be seen as a natural generalization of the widely studied random features model to deeper architectures. First, we prove Gaussian universality of the test error in a ridge regression setting where the learner and target networks share the same intermediate layers, and provide a sharp asymptotic formula for it. Establishing this result requires proving a deterministic equivalent for traces of the deep random features sample covariance matrices which can be of independent interest. Second, we conjecture the asymptotic Gaussian universality of the test error in the more general setting of arbitrary convex losses and generic learner/target architectures. We provide extensive numerical evidence for this conjecture, which requires the derivation of closed-form expressions for the layer-wise post-activation population covariances. In light of our results, we investigate the interplay between architecture design and implicit regularization. Show more
Publication status
publishedExternal links
Editor
Book title
Proceedings of the 40th International Conference on Machine LearningJournal / series
Proceedings of Machine Learning ResearchVolume
Pages / Article No.
Publisher
PMLREvent
Organisational unit
02219 - ETH AI Center / ETH AI Center09679 - Bandeira, Afonso / Bandeira, Afonso
09652 - Yang, Fan / Yang, Fan
09652 - Yang, Fan / Yang, Fan
Funding
209089 - Random matrix universality in data science and theoretical physics (SNF)
More
Show all metadata
ETH Bibliography
yes
Altmetrics