Training Quantized Nets: A Deeper Understanding
METADATA ONLY
Loading...
Date
2018
Publication Type
Conference Paper
ETH Bibliography
no
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.
Permanent link
Publication status
published
Book title
Advances in Neural Information Processing Systems 30
Journal / series
Volume
9
Pages / Article No.
5812
Publisher
Curran
Event
31st Annual Conference on Neural Information Processing Systems (NIPS 2017)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Organisational unit
09695 - Studer, Christoph / Studer, Christoph