Morpho-Mnist: Quantitative assessment and diagnostics for representation learning

Abstract

Revealing latent structure in data is an active field of research, having introduced excitingtechnologies such as variational autoencoders and adversarial networks, and is essentialto push machine learning towards unsupervised knowledge discovery. However, a majorchallenge is the lack of suitable benchmarks for an objective and quantitative evaluation oflearned representations. To address this issue we introduce Morpho-MNIST, a frameworkthat aims to answer: “to what extent has my model learned to represent specific factors ofvariation in the data?” We extend the popular MNIST dataset by adding a morphometricanalysis enabling quantitative comparison of trained models, identification of the rolesof latent variables, and characterisation of sample diversity. We further propose a setof quantifiable perturbations to assess the performance of unsupervised and supervisedmethods on challenging tasks such as outlier detection and domain adaptation. Data andcode are available athttps://github.com/dccastro/Morpho-MNIST.