Metadata only
Autor(in)
Alle anzeigen
Datum
2021Typ
- Conference Paper
ETH Bibliographie
yes
Altmetrics
Abstract
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting and both compression and improved training stability in the generative adversarial training setting. Mehr anzeigen
Publikationsstatus
publishedExterne Links
Buchtitel
Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS 2021)Zeitschrift / Serie
Proceedings of Machine Learning ResearchBand
Seiten / Artikelnummer
Verlag
PMLRKonferenz
Zugehörige Publikationen und Daten
Is part of: https://doi.org/10.3929/ethz-b-000578378
ETH Bibliographie
yes
Altmetrics