Jorge Cortés
Professor
Cymer Corporation Endowed Chair
Back to base: towards hands-off learning via safe
resets with reach-avoid safety filters
A. Begzadić, N. Uday Shinde, S. Tonkens, D. Hirsch, K. Ugalde, M. C. Yip, J. Cortés, S. Herbert
Conference on Learning for Dynamics and
Control, Ann Arbor, Michigan, 2025, to appear
Abstract
Designing controllers to accomplish a task while
guaranteeing constraints on safety remains a
significant challenge. We often want an agent to
perform well in a nominal task, such as environment
exploration, while ensuring it can avoid unsafe
states and return to a desired target by a specific
time. In particular we are motivated by the setting
of safe, efficient, hands-off training for
reinforcement learning in the real world. By
enabling a robot to safely and autonomously reset to
a desired region (e.g., charging stations) without
human intervention, we can enhance efficiency and
facilitate training. Safety filters, such as those
based on control barrier functions, enable
decoupling safety from nominal control objectives
and rigorously guaranteeing safety. Despite their
success, constructing these functions for general
nonlinear systems with control constraints and
system uncertainties remains an open problem. This
paper introduces a safety filter obtained from the
value function associated with the reach-avoid
problem. The proposed safety filter minimally
modifies the nominal controller while avoiding
unsafe regions and guiding the system back to the
desired target set. By preserving policy performance
while allowing safe resetting, we enable efficient
hands-off reinforcement learning and advance the
feasibility of safe training for real world robots.
We demonstrate our approach using a modified version
of soft actor-critic to safely train a swing-up task
on a modified cartpole stabilization problem.
pdf
Mechanical and Aerospace Engineering,
University of California, San Diego
9500 Gilman Dr,
La Jolla, California, 92093-0411
Ph: 1-858-822-7930
Fax: 1-858-822-3107
cortes at ucsd.edu
Skype id:
jorgilliyo