The AI Driving Olympics: An Accessible Robot Learning Benchmark

Abstract

Despite recent breakthroughs, the ability of deep learning and reinforcement learn- ing to outperform traditional approaches to control physically embodied robotic agents remains largely unproven. To help bridge this gap, we have developed the “AI Driving Olympics” (AI-DO), a competition with the objective of evaluating the state-of-the-art in machine learning and artificial intelligence for mobile robotics. Based on the simple and well specified autonomous driving and navigation en- vironment called “Duckietown,” AI-DO includes a series of tasks of increasing complexity—from simple lane-following to fleet management. For each task, we provide tools for competitors to use in the form of simulators, data logs, code templates, baseline implementations, and low-cost access to robotic hardware. We evaluate submissions in simulation online, on standardized hardware environments, and finally at the competition events. We have held successful AI-DO competitions at NeurIPS 2018 and ICRA 2019, and will be holding AI-DO 3 at NeurIPS 2020. Together, these competitions highlight the need for better benchmarks, which are lacking in robotics, as well as improved mechanisms to bridge the gap between simulation and reality.