Why Dataset Properties Bound the Scalability of Parallel Machine Learning Training Algorithms
- Journal Article
As the training dataset size and the model size of machine learning increase rapidly, more computing resources are consumed to speedup the training process. However, the scalability and performance reproducibility of parallel machine learning training, which mainly uses stochastic optimization algorithms, are limited. In this paper, we demonstrate that the sample difference in the dataset plays a prominent role in the scalability of parallel machine learning algorithms. We propose to use statistical properties of dataset to measure sample differences. These properties include the variance of sample features, sample sparsity, sample diversity, and similarity in sampling sequences. We choose four types of parallel training algorithms as our research objects: (1) the asynchronous parallel SGD algorithm (Hogwild! algorithm), (2) the parallel model average SGD algorithm (minibatch SGD algorithm), (3) the decentralization optimization algorithm, and (4) the dual coordinate optimization (DADM algorithm). Our results show that the statistical properties of training datasets determine the scalability upper bound of these parallel training algorithms. © 1990-2012 IEEE. Show more
Journal / seriesIEEE Transactions on Parallel and Distributed Systems
Pages / Article No.
PublisherInstitute of Electrical and Electronics Engineers
SubjectParallel training algorithms; training dataset; scalability; stochastic optimization methods
MoreShow all metadata