Open access
Date
2024-02-29Type
- Other Conference Item
ETH Bibliography
yes
Altmetrics
Abstract
Thorough validation and benchmarking against the state-of-the-art are critical components in the development of novel algorithms and tools. Nevertheless, the comprehensive performance comparison between different solutions to the same problem is still sparse in the literature, mostly relegated to dedicated review studies rather than a standard practice. This is especially noticeable in the field of uncertainty quantification, where algorithm performance is also strongly affected by repeatability, finite computational resources, and the wide range of possible application fields.
We introduce the in-development benchmark module for the UQLab platform, a novel toolbox that enables thorough, standardized benchmarking facilities to ease the cumbersome process of placing the performance of a new algorithm in the rich landscape of possible alternatives. To achieve this, we propose a general framework for the benchmarking of new algorithms, that includes three main ingredients:
* Case studies, consisting either of datasets or of computational models with uncertain parameters
* Competitors, which include different sets of algorithms to be compared against one another
* Performance measures, which allow users to compare and rank the performance of each competitor quantitatively.
Additionally, the benchmark module comes with an extensive pre-calculated library of benchmarks and case studies based on current literature and our own experience. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000662725Publication status
publishedPublisher
ETH Zurich, Risk, Safety and Uncertainty QuantificationEvent
Subject
UQLab; Benchmarking; Surrogate models; Uncertainty QuantificationOrganisational unit
03962 - Sudret, Bruno / Sudret, Bruno
Notes
Presentation held on February 29, 2024;
Funded by “Open Research Data (ORD) - Open Platform for Benchmarking Uncertainty Quantification Algorithms”More
Show all metadata
ETH Bibliography
yes
Altmetrics