{"type":"ScholarlyArticle","title":"SPoRt - Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL","authors":[{"name":"Jacques Cloete","slug":"jacques-cloete","given":"Jacques","family":"Cloete","affiliation":""},{"name":"Nikolaus Vertovec","slug":"nikolaus-vertovec","given":"Nikolaus","family":"Vertovec","affiliation":""},{"name":"Alessandro Abate","slug":"alessandro-abate","given":"Alessandro","family":"Abate","affiliation":""}],"abstract":"To apply reinforcement learning to safety-critical applications, we ought to provide safety guarantees during both policy training and deployment. In this work we present novel theoretical results that provide a bound on the probability of violating a safety property for a new task-specific policy in a model-free, episodic setup: the bound, based on a 'maximum policy ratio' that is computed with respect to a 'safe' base policy, can also be more generally applied to temporally-extended properties (beyond safety) and to robust control problems. We thus present SPoRt, which also provides a data-driven approach for obtaining such a bound for the base policy, based on scenario theory, and which includes Projected PPO, a new projection-based approach for training the task-specific policy while maintaining a user-specified bound on property violation. Hence, SPoRt enables the user to trade off safety guarantees in exchange for task-specific performance. Accordingly, we present experimental results demonstrating this trade-off, as well as a comparison of the theoretical bound to posterior bounds based on empirical violation rates.","doi":"10.24963/ijcai.2025/554","url":"https://ijcai.org/proceedings/2025/554","pdf":"https://ijcai.org/proceedings/2025/0554.pdf","markdown":"https://ijcai.org/proceedings/2025/554.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/554","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":554,"pages":{"first":4976,"last":4984},"keywords":["Machine Learning: ML: Robustness","Machine Learning: ML: Reinforcement learning","Constraint Satisfaction and Optimization: CSO: Constraint satisfaction"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}