Benchmarking Music Generation Models and Metrics via Human Preference Studies
OPEN ACCESS
Loading...
Author / Producer
Date
2024-12-14
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Recent advancements have brought generated music closer to human-created compositions, yet evaluating these models remains challenging. While human preference is the gold standard for assessing quality, translating these subjective judgments into objective metrics, particularly for text-audio alignment and music quality, has proven difficult. In this work, we generate 6k songs using 12 state-of-the-art models and conduct a survey of 15k pairwise audio comparisons with 2.5k human participants to evaluate the correlation between human preferences and widely used metrics. To the best of our knowledge, this work is the first to rank current state-of-the-art music generation models and metrics based on human preference. To further the field of subjective metric evaluation, we provide open access to our dataset of generated music and human evaluations.
Permanent link
Publication status
published
Editor
Book title
Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation
Journal / series
Volume
Pages / Article No.
Publisher
OpenReview
Event
Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Organisational unit
03604 - Wattenhofer, Roger / Wattenhofer, Roger
Notes
Poster presentation on December 14, 2024.