Open access
Date
2023Type
- Conference Paper
ETH Bibliography
yes
Altmetrics
Abstract
Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000637152Publication status
acceptedEvent
Subject
Machine learning; information theoryOrganisational unit
02219 - ETH AI Center / ETH AI Center03790 - Beerenwinkel, Niko / Beerenwinkel, Niko
09670 - Vogt, Julia / Vogt, Julia
More
Show all metadata
ETH Bibliography
yes
Altmetrics