Kavli Affiliate: Yi Zhou
| First 5 Authors: Shaocong Ma, Shaocong Ma, , ,
| Summary:
The goal of robust constrained reinforcement learning (RL) is to optimize an
agent’s performance under the worst-case model uncertainty while satisfying
safety or resource constraints. In this paper, we demonstrate that strong
duality does not generally hold in robust constrained RL, indicating that
traditional primal-dual methods may fail to find optimal feasible policies. To
overcome this limitation, we propose a novel primal-only algorithm called
Rectified Robust Policy Optimization (RRPO), which operates directly on the
primal problem without relying on dual formulations. We provide theoretical
convergence guarantees under mild regularity assumptions, showing convergence
to an approximately optimal feasible policy with iteration complexity matching
the best-known lower bound when the uncertainty set diameter is controlled in a
specific level. Empirical results in a grid-world environment validate the
effectiveness of our approach, demonstrating that RRPO achieves robust and safe
performance under model uncertainties while the non-robust method can violate
the worst-case safety constraints.
| Search Query: ArXiv Query: search_query=au:”Yi Zhou”&id_list=&start=0&max_results=3