Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Open access
Datum
2023-12Typ
- Conference Paper
ETH Bibliographie
yes
Altmetrics
Abstract
Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations. These intensive resource demands limit the ability of researchers to train new models and use existing models as developmentally plausible cognitive models. The BabyLM Challenge is a communal effort in which participants compete to optimize language model training on a fixed data budget. Submissions are compared on various evaluation tasks targeting grammatical ability, downstream task performance, and generalization. Participants can submit to up to three tracks with progressively looser data restrictions. From over 30 submissions, we extract concrete recommendations on how best to train data-efficient language models, and on where future efforts should (and perhaps should not) focus. The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words. Other submissions achieved strong results through training on shorter input sequences or training a student model on a pretrained teacher. Curriculum learning attempts, which accounted for a large number of submissions, were largely unsuccessful, though some showed modest improvements. Mehr anzeigen
Persistenter Link
https://doi.org/10.3929/ethz-b-000650680Publikationsstatus
publishedExterne Links
Buchtitel
Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language LearningSeiten / Artikelnummer
Verlag
Association for Computational LinguisticsKonferenz
Organisationseinheit
09682 - Cotterell, Ryan / Cotterell, Ryan
Zugehörige Publikationen und Daten
Is part of: http://hdl.handle.net/20.500.11850/651188
ETH Bibliographie
yes
Altmetrics