Show simple item record

dc.contributor.author
Kobayashi, Seijin
dc.contributor.author
Vilimelis Aceituno, Pau
dc.contributor.author
von Oswald, Johannes
dc.contributor.editor
Koyejo, Sanmi
dc.contributor.editor
Mohamed, Shakir
dc.contributor.editor
Agarwal, Alekh
dc.contributor.editor
Belgrave, Danielle
dc.contributor.editor
Cho, Kyunghyun
dc.contributor.editor
Oh, Alice
dc.date.accessioned
2023-04-05T06:24:45Z
dc.date.available
2023-01-17T16:24:48Z
dc.date.available
2023-02-15T10:50:17Z
dc.date.available
2023-04-05T06:24:45Z
dc.date.issued
2022
dc.identifier.isbn
978-1-7138-7108-8
en_US
dc.identifier.uri
http://hdl.handle.net/20.500.11850/593154
dc.description.abstract
Identifying unfamiliar inputs, also known as out-of-distribution (OOD) detection, is a crucial property of any decision making process. A simple and empirically validated technique is based on deep ensembles where the variance of predictions over different neural networks acts as a substitute for input uncertainty. Nevertheless, a theoretical understanding of the inductive biases leading to the performance of deep ensemble's uncertainty estimation is missing. To improve our description of their behavior, we study deep ensembles with large layer widths operating in simplified linear training regimes, in which the functions trained with gradient descent can be described by the neural tangent kernel. We identify two sources of noise, each inducing a distinct inductive bias in the predictive variance at initialization. We further show theoretically and empirically that both noise sources affect the predictive variance of non-linear deep ensembles in toy models and realistic settings after training. Finally, we propose practical ways to eliminate part of these noise sources leading to significant changes and improved OOD detection in trained deep ensembles.
en_US
dc.language.iso
en
en_US
dc.publisher
Curran
en_US
dc.title
Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
en_US
dc.type
Conference Paper
ethz.book.title
Advances in Neural Information Processing Systems 35
en_US
ethz.pages.start
25335
en_US
ethz.pages.end
25348
en_US
ethz.event
36th Annual Conference on Neural Information Processing Systems (NeurIPS 2022)
en_US
ethz.event.location
New Orleans, LA, USA
en_US
ethz.event.date
November 28 - December 9, 2022
en_US
ethz.notes
Poster presentation on November 29, 2022.
en_US
ethz.grant
Temporal Information Integration in Neural Networks
en_US
ethz.publication.place
Red Hook, NY
en_US
ethz.publication.status
published
en_US
ethz.leitzahl
ETH Zürich::00002 - ETH Zürich::00012 - Lehre und Forschung::00007 - Departemente::02150 - Dep. Informatik / Dep. of Computer Science::02643 - Institut für Theoretische Informatik / Inst. Theoretical Computer Science::03672 - Steger, Angelika / Steger, Angelika
en_US
ethz.leitzahl.certified
ETH Zürich::00002 - ETH Zürich::00012 - Lehre und Forschung::00007 - Departemente::02150 - Dep. Informatik / Dep. of Computer Science::02643 - Institut für Theoretische Informatik / Inst. Theoretical Computer Science::03672 - Steger, Angelika / Steger, Angelika
en_US
ethz.identifier.url
https://proceedings.neurips.cc/paper_files/paper/2022/hash/a205fda871b0f6c1e18a7ad7325eb6cf-Abstract-Conference.html
ethz.identifier.url
https://nips.cc/virtual/2022/poster/53309
ethz.grant.agreementno
173721
ethz.grant.fundername
SNF
ethz.grant.funderDoi
10.13039/501100001711
ethz.grant.program
Sinergia
ethz.date.deposited
2023-01-17T16:24:49Z
ethz.source
FORM
ethz.eth
yes
en_US
ethz.availability
Metadata only
en_US
ethz.rosetta.installDate
2023-04-05T06:24:46Z
ethz.rosetta.lastUpdated
2023-04-05T06:24:46Z
ethz.rosetta.versionExported
true
ethz.COinS
ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.atitle=Disentangling%20the%20Predictive%20Variance%20of%20Deep%20Ensembles%20through%20the%20Neural%20Tangent%20Kernel&rft.date=2022&rft.spage=25335&rft.epage=25348&rft.au=Kobayashi,%20Seijin&Vilimelis%20Aceituno,%20Pau&von%20Oswald,%20Johannes&rft.isbn=978-1-7138-7108-8&rft.genre=proceeding&rft.btitle=Advances%20in%20Neural%20Information%20Processing%20Systems%2035
 Search print copy at ETH Library

Files in this item

FilesSizeFormatOpen in viewer

There are no files associated with this item.

Publication type

Show simple item record