Reinforcement Learning-Based Bucket Filling for Autonomous Excavation

Abstract

This article presents a bucket-filling controller for autonomous excavation. The key innovation of this controller is that it can react to the encountered soil conditions and adapt the excavation behavior online without the explicit knowledge of soil properties while respecting machine limitations to avoid stalling. At the same time, the controller takes into account the current terrain elevation and adheres to a maximum-depth constraint to achieve a desired design. The controller is trained entirely in simulation with reinforcement learning (RL). A simple analytical soil model based on the fundamental equation of Earth moving (FEE) is used to simulate ground interactions. To learn an appropriate excavation strategy for a wide variety of scenarios, soil parameters, as well as other properties of the environment, are randomized extensively during training. We test and evaluate the controller on a 12-ton excavator with a conventional two-stage hydraulic system in a wide range of different soil conditions. In addition, we show the excavation of a complete trench by integrating the controller into an autonomous excavation planning system. The experiments demonstrate that the controller can robustly adapt the excavation trajectory based on the encountered conditions and shows competitive performance compared to a professional machine operator.