MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer

Abstract

We introduce MIDI-VAE, a neural network model basedon Variational Autoencoders that is capable of handlingpolyphonic music with multiple instrument tracks, as wellas modeling the dynamics of music by incorporating notedurations and velocities. We show that MIDI-VAE can per-form style transfer on symbolic music by automaticallychanging pitches, dynamics and instruments of a musicpiece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separatestyle validation classifiers. Our model can also interpolatebetween short pieces of music, produce medleys and cre-ate mixtures of entire songs. The interpolations smoothlychange pitches, dynamics and instrumentation to create aharmonic bridge between two music pieces. To the best ofour knowledge, this work represents the first successful at-tempt at applying neural style transfer to complete musicalcompositions.