System-Aware Algorithms For Machine Learning

Abstract

The design of machine learning algorithms is often conducted in a system-agnostic manner. As a consequence, established methods may not be well aligned with the particularities of the systems on which they are deployed. In this thesis we demonstrate that there is a huge potential for improving the performance and efficiency of machine learning applications by incorporating system characteristics into the algorithm design. We develop new principled tools and methods for training machine learning models that are theoretically sound and enable the systematic utilization of individual hardware resources available in heterogeneous systems. In particular, we focus on lowering the impact of slow interconnects on distributed training and exploiting hierarchical memory structures, compute parallelism and accelerator units.