The Mixtures and the Neural Critics: On the Pointwise Mutual Information Profiles of Fine Distributions
OPEN ACCESS
Author / Producer
Date
2023-10-16
Publication Type
Working Paper
ETH Bibliography
yes
Citations
Altmetric
OPEN ACCESS
Data
Rights / License
Abstract
Mutual information quantifies the dependence between two random variables and remains invariant under diffeomorphisms. In this paper, we explore the pointwise mutual information profile, an extension of mutual information that maintains this invariance. We analytically describe the profiles of multivariate normal distributions and introduce the family of fine distributions, for which the profile can be accurately approximated using Monte Carlo methods. We then show how fine distributions can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how fine distributions can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.
Permanent link
Publication status
published
Editor
Book title
Journal / series
Volume
Pages / Article No.
2310.1024
Publisher
Cornell University
Event
Edition / version
v1
Methods
Software
Geographic location
Date collected
Date created
Subject
Machine Learning (stat.ML); Information Theory (cs.IT); Machine Learning (cs.LG); FOS: Computer and information sciences
Organisational unit
03790 - Beerenwinkel, Niko / Beerenwinkel, Niko
09670 - Vogt, Julia / Vogt, Julia
02219 - ETH AI Center / ETH AI Center
Notes
Funding
Related publications and datasets
Is original form of: https://doi.org/10.3929/ethz-b-000723924