Show simple item record

dc.contributor.author
Zucchet, Nicolas
dc.contributor.author
Meier, Robert
dc.contributor.author
Schug, Simon
dc.contributor.author
Mujika, Asier
dc.contributor.author
Sacramento, João
dc.contributor.editor
Oh, Alice
dc.contributor.editor
Naumann, Tristan
dc.contributor.editor
Globerson, Amir
dc.contributor.editor
Saenko, Kate
dc.contributor.editor
Hardt, Moritz
dc.contributor.editor
Levine, Sergey
dc.date.accessioned
2024-07-24T12:17:56Z
dc.date.available
2024-01-19T14:35:40Z
dc.date.available
2024-02-05T12:46:52Z
dc.date.available
2024-07-24T12:17:56Z
dc.date.issued
2024-07
dc.identifier.isbn
978-1-7138-9992-1
en_US
dc.identifier.uri
http://hdl.handle.net/20.500.11850/654083
dc.description.abstract
Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.
en_US
dc.language.iso
en
en_US
dc.publisher
Curran
en_US
dc.subject
Machine Learning (cs.LG)
en_US
dc.subject
Neural and Evolutionary Computing (cs.NE)
en_US
dc.subject
FOS: Computer and information sciences
en_US
dc.title
Online learning of long-range dependencies
en_US
dc.type
Conference Paper
ethz.book.title
Advances in Neural Information Processing Systems 36
en_US
ethz.pages.start
10477
en_US
ethz.pages.end
10493
en_US
ethz.event
37th Annual Conference on Neural Information Processing Systems (NeurIPS 2023)
en_US
ethz.event.location
New Orleans, LA, USA
en_US
ethz.event.date
December 10-16, 2023
en_US
ethz.notes
Poster presented on December 12, 2023.
en_US
ethz.identifier.wos
ethz.publication.place
Red Hook, NY
en_US
ethz.publication.status
published
en_US
ethz.leitzahl
ETH Zürich::00002 - ETH Zürich::00012 - Lehre und Forschung::00007 - Departemente::02150 - Dep. Informatik / Dep. of Computer Science::02643 - Institut für Theoretische Informatik / Inst. Theoretical Computer Science::03672 - Steger, Angelika / Steger, Angelika
en_US
ethz.leitzahl.certified
ETH Zürich::00002 - ETH Zürich::00012 - Lehre und Forschung::00007 - Departemente::02150 - Dep. Informatik / Dep. of Computer Science::02643 - Institut für Theoretische Informatik / Inst. Theoretical Computer Science::03672 - Steger, Angelika / Steger, Angelika
en_US
ethz.identifier.url
https://neurips.cc/virtual/2023/poster/71404
ethz.identifier.url
https://papers.nips.cc/paper_files/paper/2023/hash/2184d8450c8a641f9a10c49279087c97-Abstract-Conference.html
ethz.relation.isNewVersionOf
https://openreview.net/forum?id=Wa1GGPqjUn
ethz.date.deposited
2024-01-19T14:35:40Z
ethz.source
FORM
ethz.eth
yes
en_US
ethz.availability
Metadata only
en_US
ethz.rosetta.installDate
2024-07-24T12:17:59Z
ethz.rosetta.lastUpdated
2024-07-24T12:17:59Z
ethz.rosetta.exportRequired
true
ethz.rosetta.versionExported
true
ethz.COinS
ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.atitle=Online%20learning%20of%20long-range%20dependencies&rft.date=2024-07&rft.spage=10477&rft.epage=10493&rft.au=Zucchet,%20Nicolas&Meier,%20Robert&Schug,%20Simon&Mujika,%20Asier&Sacramento,%20Jo%C3%A3o&rft.isbn=978-1-7138-9992-1&rft.genre=proceeding&rft.btitle=Advances%20in%20Neural%20Information%20Processing%20Systems%2036
 Search print copy at ETH Library

Files in this item

FilesSizeFormatOpen in viewer

There are no files associated with this item.

Publication type

Show simple item record