Trust Region Policy Optimization with Optimal Transport Discrepancies: Duality and Algorithm for Continuous Actions
Metadata only
Date
2022-10-20Type
- Working Paper
ETH Bibliography
yes
Altmetrics
Abstract
Policy Optimization (PO) algorithms have been proven particularly suited to handle the high-dimensionality of real-world continuous control tasks. In this context, Trust Region Policy Optimization methods represent a popular approach to stabilize the policy updates. These usually rely on the Kullback-Leibler (KL) divergence to limit the change in the policy. The Wasserstein distance represents a natural alternative, in place of the KL divergence, to define trust regions or to regularize the objective function. However, state-of-the-art works either resort to its approximations or do not provide an algorithm for continuous state-action spaces, reducing the applicability of the method. In this paper, we explore optimal transport discrepancies (which include the Wasserstein distance) to define trust regions, and we propose a novel algorithm - Optimal Transport Trust Region Policy Optimization (OT-TRPO) - for continuous state-action spaces. We circumvent the infinite-dimensional optimization problem for PO by providing a one-dimensional dual reformulation for which strong duality holds. We then analytically derive the optimal policy update given the solution of the dual problem. This way, we bypass the computation of optimal transport costs and of optimal transport maps, which we implicitly characterize by solving the dual formulation. Finally, we provide an experimental evaluation of our approach across various control tasks. Our results show that optimal transport discrepancies can offer an advantage over state-of-the-art approaches. Show more
Publication status
publishedJournal / series
arXivPages / Article No.
Publisher
Cornell UniversityEdition / version
v1Subject
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Systems and Control (eess.SY); FOS: Computer and information sciences; FOS: Electrical engineering, electronic engineering, information engineeringOrganisational unit
09478 - Dörfler, Florian / Dörfler, Florian
09729 - He, Niao / He, Niao
Related publications and datasets
Is previous version of: https://doi.org/10.3929/ethz-b-000584154
More
Show all metadata
ETH Bibliography
yes
Altmetrics