- Conference Paper
The development workflow for today's AI applications has grown far beyond the standard model training task. This workflow typically consists of various data and model management tasks. It includes a "data cycle"aimed at producing high-quality training data, and a "model cycle"aimed at managing trained models on their way to production. This broadened workflow has opened a space for already emerging tools and systems for AI development. However, as a research community, we are still missing standardized ways to evaluate these tools and systems. In a humble effort to get this wheel turning, we developed dcbench, a benchmark for evaluating systems for data-centric AI development. In this report, we present the main ideas behind dcbench, some benchmark tasks that we included in the initial release, and a short summary of its implementation. Show more
Book titleProceedings of the 6th Workshop on Data Management for End-To-End Machine Learning (DEEM 2022)
Pages / Article No.
PublisherAssociation for Computing Machinery
SubjectBenchmarks; data-centric AI; AI development
NotesHeld in conjunction with the 2022 ACM SIGMOD/PODS Conference
MoreShow all metadata