Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks
METADATA ONLY
Loading...
Author / Producer
Date
2022
Publication Type
Conference Paper
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Neural Architecture Search (NAS) is increasingly popular to automatically explore the accuracy versus computational complexity trade-off of Deep Learning (DL) architectures. When targeting tiny edge devices, the main challenge for DL deployment is matching the tight memory constraints, hence most NAS algorithms consider model size as the complexity metric. Other methods reduce the energy or latency of DL models by trading off accuracy and number of inference operations. Energy and memory are rarely considered simultaneously, in particular by low-search-cost Differentiable NAS (DNAS) solutions. We overcome this limitation proposing the first DNAS that directly addresses the most realistic scenario from a designer's perspective: the co-optimization of accuracy and energy (or latency) under a memory constraint, determined by the target HW. We do so by combining two complexity-dependent loss functions during training, with independent strength. Testing on three edge-relevant tasks from the MLPerf Tiny benchmark suite, we obtain rich Pareto sets of architectures in the energy vs. accuracy space, with memory footprints constraints spanning from 75% to 6.25% of the baseline networks. When deployed on a commercial edge device, the STM NUCLEO-H743ZI2, our networks span a range of 2.18x in energy consumption and 4.04% in accuracy for the same memory constraint, and reduce energy by up to 2.2 with negligible accuracy drop with respect to the baseline.
Permanent link
Publication status
published
External links
Book title
ISLPED '22: Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design
Journal / series
Volume
Pages / Article No.
28
Publisher
Association for Computing Machinery
Event
Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED '22)
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Deep Learning; TinyML; Energy-efficiency; NAS
Organisational unit
03996 - Benini, Luca / Benini, Luca