{"type":"ScholarlyArticle","title":"Combining MORL with Restraining Bolts to Learn Normative Behaviour","authors":[{"name":"Emery A. Neufeld","slug":"emery-a-neufeld","given":"Emery A.","family":"Neufeld","affiliation":"TU Wien"},{"name":"Agata Ciabattoni","slug":"agata-ciabattoni","given":"Agata","family":"Ciabattoni","affiliation":"TU Wien"},{"name":"Radu Florin Tulcan","slug":"radu-florin-tulcan","given":"Radu Florin","family":"Tulcan","affiliation":"TU Wien"}],"abstract":"Normative Restraining Bolts (NRBs) adapt the restraining bolt technique (originally developed for safe reinforcement learning) to ensure compliance with social, legal, and ethical norms. While effective, NRBs rely on trial-and-error weight tuning, which hinders their ability to enforce hierarchical norms; moreover, norm updates require retraining. In this paper, we reformulate learning with NRBs as a multi-objective reinforcement learning (MORL) problem, where each norm is treated as a distinct objective. This enables the introduction of Ordered Normative Restraining Bolts (ONRBs), which support algorithmic weight selection, prioritized norms, norm updates, and provide formal guarantees on minimizing norm violations. Case studies show that ONRBs offer a robust and principled foundation for RL-agents to comply with a wide range of norms while achieving their goals.","doi":"10.24963/ijcai.2025/514","url":"https://ijcai.org/proceedings/2025/514","pdf":"https://ijcai.org/proceedings/2025/0514.pdf","markdown":"https://ijcai.org/proceedings/2025/514.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/514","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":514,"pages":{"first":4615,"last":4623},"keywords":["Knowledge Representation and Reasoning: KRR: Learning and reasoning","Machine Learning: ML: Reinforcement learning","AI Ethics, Trust, Fairness: ETF: Values","Agent-based and Multi-agent Systems: MAS: Normative systems"],"videos":[],"references":[{"position":1,"raw":"[Abel et al., 2016] David Abel, James MacGlashan, and Michael L Littman. Reinforcement learning as a frame- work for ethical decision making. In AAAI Workshop: AI, Ethics, and Society, volume 16, 2016.","title":"Reinforcement learning as a framework for ethical decision making","authors":"David Abel, James Macglashan, Michael L Littman","year":"2016","venue":"AAAI Workshop: AI, Ethics, and Society","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alechina et al., 2018] Natasha Alechina, Mehdi Dastani, and Brian Logan. Norm specification and verification in multi-agent systems. Journal of Applied Logics, 5(2):457, 2018.","title":"Norm specification and verification in multi-agent systems","authors":"Natasha Alechina, Mehdi Dastani, Brian Logan","year":"2018","venue":"Journal of Applied Logics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alshiekh et al., 2018] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proc. AAAI, pages 2669-2678, 2018.","title":"Safe reinforcement learning via shielding","authors":"Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu","year":"2018","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anderson, 1958] Alan Ross Anderson. A reduction of de- ontic logic to alethic modal logic. Mind, 67(265):100-103, 1958.","title":"A reduction of deontic logic to alethic modal logic","authors":"Alan Ross, Anderson","year":"1958","venue":"Mind","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bacchus et al., 1996] Fahiem Bacchus, Craig Boutilier, and Adam Grove. Rewarding behaviors. In Proceedings of the National Conference on Artificial Intelligence, pages 1160-1167, 1996.","title":"Rewarding behaviors","authors":"Bacchus","year":"1996","venue":"Proceedings of the National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barrett and Narayanan, 2008] Leon Barrett and Srini Narayanan. Learning all optimal policies with multi- ple criteria. In Proceedings of the 25th international conference on Machine learning, pages 41-47, 2008.","title":"Learning all optimal policies with multiple criteria","authors":"Narayanan Barrett, Leon Barrett, Srini Narayanan","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brafman et al., 2018] Ronen Brafman, Giuseppe De Gia- como, and Fabio Patrizi. Ltlf/ldlf non-markovian rewards. In Proceedings of the AAAI conference on artificial intel- ligence, volume 32, 2018.","title":"Ltlf/ldlf non-markovian rewards","authors":"Ronen Brafman, Giuseppe De, Giacomo, Fabio Patrizi","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[De Giacomo and Favorito, 2021] Giuseppe De Giacomo and Marco Favorito. Compositional approach to translate ltlf/ldlf into deterministic finite automata. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 31, pages 122-130, 2021.","title":"Compositional approach to translate ltlf/ldlf into deterministic finite automata","authors":"De Giacomo, Giuseppe Favorito, Giacomo De, Marco Favorito","year":"2021","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[De Giacomo et al., 2013] Giuseppe De Giacomo, Moshe Y Vardi, et al. Linear temporal logic and linear dynamic logic on finite traces. In IJCAI, volume 13, pages 854- 860, 2013.","title":"Linear temporal logic and linear dynamic logic on finite traces","authors":"Giuseppe De, Giacomo, Moshe Y Vardi","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[De Giacomo et al., 2015] Giuseppe De Giacomo, Moshe Y Vardi, et al. Synthesis for ltl and ldl on finite traces. In Pro- ceedings of the Twenty-Fourth International Joint Confer- ence on Artificial Intelligence, IJCAI 2015, pages 1558- 1564. AAAI Press, 2015.","title":"Synthesis for ltl and ldl on finite traces","authors":"Giuseppe De, Giacomo, Moshe Y Vardi","year":"2015","venue":"Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[De Giacomo et al., 2019] Giuseppe De Giacomo, Luca Ioc- chi, Marco Favorito, and Fabio Patrizi. Foundations for restraining bolts: Reinforcement learning with ltlf/ldlf re- straining specifications. In Proceedings of the interna- tional conference on automated planning and scheduling, volume 29, pages 128-136, 2019.","title":"Foundations for restraining bolts: Reinforcement learning with ltlf/ldlf restraining specifications","authors":"Giuseppe De, Giacomo, Luca Iocchi, Marco Favorito, Fabio Patrizi","year":"2019","venue":"Proceedings of the international conference on automated planning and scheduling","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[De Giacomo et al., 2023] Giuseppe De Giacomo, Marco Favorito, Francesco Leotta, Massimo Mecella, Flavia Monti, and Luciana Silo. Aida: a tool for resiliency in smart manufacturing. In International Conference on Ad- vanced Information Systems Engineering, pages 112-120. Springer, 2023.","title":"Aida: a tool for resiliency in smart manufacturing","authors":"Giacomo De","year":"2023","venue":"International Conference on Advanced Information Systems Engineering","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[De Giacomo et al., 2024] Giuseppe De Giacomo, Marco Favorito, and Luciana Silo. Composition of stochastic ser- vices for ltl goal specifications. In International Sympo- sium on Foundations of Information and Knowledge Sys- tems, pages 298-316. Springer, 2024.","title":"Composition of stochastic services for ltl goal specifications","authors":"Giacomo De","year":"2024","venue":"International Symposium on Foundations of Information and Knowledge Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ding et al., 2011a] Xu Chu Ding, Stephen L Smith, Calin Belta, and Daniela Rus. Mdp optimal control under tem- poral logic constraints. In 2011 50th IEEE Conference on Decision and Control and European Control Conference, pages 532-538. IEEE, 2011.","title":"Mdp optimal control under temporal logic constraints","authors":"Ding","year":"2011","venue":"50th IEEE Conference on Decision and Control and European Control Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ding et al., 2011b] Xu Chu Dennis Ding, Stephen L Smith, Calin Belta, and Daniela Rus. Ltl control in uncertain envi- ronments with probabilistic satisfaction guarantees. IFAC Proceedings Volumes, 44(1):3515-3520, 2011.","title":"Ltl control in uncertain environments with probabilistic satisfaction guarantees","authors":"Ding","year":"2011","venue":"IFAC Proceedings Volumes","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ecoffet and Lehman, 2021] Adrien Ecoffet and Joel Lehman. Reinforcement learning under moral uncertainty. In International Conference on Machine Learning, pages 2926-2936. PMLR, 2021.","title":"Adrien Ecoffet and Joel Lehman. Reinforcement learning under moral uncertainty","authors":"Lehman Ecoffet","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Governatori et al., 2007] Guido Governatori, Joris Hulstijn, Régis Riveret, and Antonino Rotolo. Characterising dead- lines in temporal modal defeasible logic. In Proc. of AU- SAI, LNCS, pages 486-496, 2007.","title":"Characterising deadlines in temporal modal defeasible logic","authors":"Guido Governatori, Joris Hulstijn, Régis Riveret, Antonino Rotolo","year":"2007","venue":"Proc. of AU-SAI, LNCS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Governatori, 2015] Guido Governatori. Thou shalt is not you will. In Proc. of ICAIL, pages 63-68, 2015.","title":"Thou shalt is not you will","authors":"Guido Governatori","year":"2015","venue":"Proc. of ICAIL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hahn et al., 2021] Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Model-free reinforcement learning for lexico- graphic omega-regular objectives. In International sympo- sium on formal methods, pages 142-159. Springer, 2021.","title":"Model-free reinforcement learning for lexicographic omega-regular objectives","authors":"Hahn","year":"2021","venue":"International symposium on formal methods","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hahn et al., 2023] Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, and Dominik Wojtczak. Multi-objective ω-regular reinforcement learn- ing. Formal Aspects of Computing, 35(2):1-24, 2023.","title":"Multi-objective ω-regular reinforcement learning","authors":"Hahn","year":"2023","venue":"Formal Aspects of Computing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jansen et al., 2020] Nils Jansen, Bettina Könighofer, Sebas- tian Junges, Alex Serban, and Roderick Bloem. Safe Re- inforcement Learning Using Probabilistic Shields (Invited Paper). In 31st International Conference on Concurrency Theory (CONCUR 2020), volume 171 of Leibniz Inter- national Proceedings in Informatics (LIPIcs), pages 3:1- 3:16, 2020.","title":"Safe Reinforcement Learning Using Probabilistic Shields","authors":"Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, Roderick Bloem","year":"2020","venue":"31st International Conference on Concurrency Theory (CONCUR 2020)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kasenberg and Scheutz, 2018] Daniel Kasenberg and Matthias Scheutz. Norm conflict resolution in stochastic domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.","title":"Norm conflict resolution in stochastic domains","authors":"Scheutz Kasenberg, Daniel Kasenberg, Matthias Scheutz","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kasenberg et al., 2020] Daniel Kasenberg, Ravenna Thiel- strom, and Matthias Scheutz. Generating explanations for temporal logic planner decisions. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 449-458, 2020.","title":"Generating explanations for temporal logic planner decisions","authors":"Daniel Kasenberg, Ravenna Thielstrom, Matthias Scheutz","year":"2020","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Klee and Minty, 1972] Victor Klee and George J Minty. How good is the simplex algorithm. Inequalities, 3(3):159-175, 1972.","title":"How good is the simplex algorithm","authors":"Klee, Minty, Victor Klee, George J Minty","year":"1972","venue":"Inequalities","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2023] Lening Li, Hazhar Rahmani, and Jie Fu. Probabilistic planning with prioritized preferences over temporal logic objectives. In Proceedings of the Thirty- Second International Joint Conference on Artificial Intel- ligence, pages 189-198, 2023.","title":"Probabilistic planning with prioritized preferences over temporal logic objectives","authors":"Lening Li, Hazhar Rahmani, Jie Fu","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/22"},{"position":26,"raw":"[Neufeld et al., 2022] Emery Neufeld, Ezio Bartocci, and Agata Ciabattoni. On normative reinforcement learning via safe reinforcement learning. In PRIMA 2022, 2022.","title":"On normative reinforcement learning via safe reinforcement learning","authors":"Emery Neufeld, Ezio Bartocci, Agata Ciabattoni","year":"2022","venue":"PRIMA 2022","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Neufeld et al., 2024] Emery A Neufeld, Agata Ciabattoni, and Radu Florin Tulcan. Norm compliance in reinforce- ment learning agents via restraining bolts. In Legal Knowl- edge and Information Systems, pages 119-130. IOS Press, 2024.","title":"Norm compliance in reinforcement learning agents via restraining bolts","authors":"Agata Emery A Neufeld, Radu Ciabattoni, Florin Tulcan","year":"2024","venue":"Legal Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Neufeld, 2024] Emery A Neufeld. Learning normative behaviour through automated theorem proving. KI- Künstliche Intelligenz, pages 1-19, 2024.","title":"Learning normative behaviour through automated theorem proving","authors":"A Emery, Neufeld","year":"2024","venue":"KI-Künstliche Intelligenz","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pnueli, 1977] Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science, pages 46-57. IEEE, 1977.","title":"The temporal logic of programs","authors":"Amir Pnueli, Pnueli","year":"1977","venue":"18th Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rodriguez-Soto et al., 2021] Manel Rodriguez-Soto, Maite Lopez-Sanchez, and Juan A Rodriguez-Aguilar. Multi- objective reinforcement learning for designing ethical en- vironments. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 1-7, 2021.","title":"Multiobjective reinforcement learning for designing ethical environments","authors":"Manel Rodriguez-Soto, Maite Lopez-Sanchez, Juan A Rodriguez-Aguilar","year":"2021","venue":"Proceedings of the 30th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rodriguez-Soto et al., 2022] Manel Rodriguez-Soto, Marc Serramia, Maite Lopez-Sanchez, and Juan Antonio Rodriguez-Aguilar. Instilling moral value alignment by means of multi-objective reinforcement learning. Ethics and Information Technology, 24(1):1-17, 2022.","title":"Instilling moral value alignment by means of multi-objective reinforcement learning","authors":"Manel Rodriguez-Soto, Marc Serramia, Maite Lopez-Sanchez, Juan Antonio Rodriguez-Aguilar","year":"2022","venue":"Ethics and Information Technology","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Rodriguez-Soto et al., 2023] Manel Rodriguez-Soto, Rox- ana Rȃdulescu, Juan A Rodriguez-Aguilar, Maite Lopez- Sanchez, and Ann Nowé. Multi-objective reinforcement learning for guaranteeing alignment with multiple values. In 2023 Adaptive and Learning Agents Workshop at AA- MAS, 2023.","title":"Multi-objective reinforcement learning for guaranteeing alignment with multiple values","authors":"Manel Rodriguez-Soto, Roxana Rȃdulescu, Juan A Rodriguez-Aguilar, Maite Lopez-Sanchez, Ann Nowé","year":"2023","venue":"2023 Adaptive and Learning Agents Workshop at AA-MAS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Roijers et al., 2013a] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Ar- tificial Intelligence Research, 48:67-113, 2013.","title":"A survey of multi-objective sequential decision-making","authors":"Roijers","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Roijers et al., 2013b] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Ar- tificial Intelligence Research, 48:67-113, 2013.","title":"A survey of multi-objective sequential decision-making","authors":"Roijers","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Spielman and Teng, 2004] Daniel A Spielman and Shang- Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM), 51(3):385-463, 2004.","title":"Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time","authors":"A Daniel, Shang-Hua Spielman, Teng","year":"2004","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning -an introduction. Adaptive computation and machine learning. MIT Press, 1998.","title":"Reinforcement learning -an introduction. Adaptive computation and machine learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vamplew et al., 2011] Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Em- pirical evaluation methods for multiobjective reinforce- ment learning algorithms. Machine learning, 84(1):51-80, 2011.","title":"Empirical evaluation methods for multiobjective reinforcement learning algorithms","authors":"Vamplew","year":"2011","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[van Rysewyk and Pontier, 2015] Simon van Rysewyk and Matthijs Pontier. A hybrid bottom-up and top-down ap- proach to machine medical ethics: Theory and data. 2015.","title":"A hybrid bottom-up and top-down approach to machine medical ethics: Theory and data","authors":"Van Rysewyk, Matthijs Pontier ; Simon Van Rysewyk, Pontier","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Varricchione et al., 2024] Giovanni Varricchione, Natasha Alechina, Mehdi Dastani, Giuseppe De Giacomo, Brian Logan, and Giuseppe Perelli. Pure-past action masking. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 38, pages 21646-21655, 2024.","title":"Pure-past action masking","authors":"Giovanni Varricchione, Natasha Alechina, Mehdi Dastani, Giuseppe De Giacomo, Brian Logan, Giuseppe Perelli","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}