{"type":"ScholarlyArticle","title":"A Multi-Objective Approach to Mitigate Negative Side Effects","authors":[{"name":"Sandhya Saisubramanian","slug":"sandhya-saisubramanian","given":"Sandhya","family":"Saisubramanian","affiliation":"University of Massachusetts , Amherst , MA"},{"name":"Ece Kamar","slug":"ece-kamar","given":"Ece","family":"Kamar","affiliation":"Microsoft Research , Redmond , WA"},{"name":"Shlomo Zilberstein","slug":"shlomo-zilberstein","given":"Shlomo","family":"Zilberstein","affiliation":"University of Massachusetts , Amherst , MA"}],"abstract":"Agents operating in unstructured environments often create negative side effects (NSE) that may not be easy to identify at design time. We examine how various forms of human feedback or autonomous exploration can be used to learn a penalty function associated with NSE during system deployment.  We formulate the problem of mitigating the impact of NSE as a multi-objective Markov decision process with lexicographic reward preferences and slack. The slack denotes the maximum deviation from an optimal policy with respect to the agent's primary objective allowed in order to mitigate NSE as a secondary objective. Empirical evaluation of our approach shows that the proposed framework can successfully mitigate NSE and that different feedback mechanisms introduce different biases, which influence the identification of NSE.","doi":"10.24963/ijcai.2020/50","url":"https://ijcai.org/proceedings/2020/50","pdf":"https://ijcai.org/proceedings/2020/0050.pdf","markdown":"https://ijcai.org/proceedings/2020/50.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/50","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":50,"pages":{"first":354,"last":361},"keywords":["Humans and AI: Human-AI Collaboration","Agent-based and Multi-agent Systems: Human-Agent Interaction","Planning and Scheduling: Markov Decisions Processes"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24079/file_24079.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete prob- lems in AI safety. arXiv preprint arXiv:1606.06565, 2016.","title":"Concrete problems in AI safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dietterich, 2017] Thomas G. Dietterich. Steps toward robust arti- ficial intelligence. AI Magazine, 38(3):3-24, 2017.","title":"Steps toward robust artificial intelligence","authors":"G Thomas, Dietterich","year":"2017","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hadfield-Menell et al., 2017] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, 2017.","title":"Inverse reward design","authors":"Hadfield-Menell","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hansen and Zilberstein, 2001] Eric A. Hansen and Shlomo Zilber- stein. LAO*: A heuristic search algorithm that finds solutions with loops. Artificial Intelligence, 129:35-62, 2001.","title":"LAO*: A heuristic search algorithm that finds solutions with loops","authors":"Eric A Hansen, Shlomo Zilberstein","year":"2001","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Krakovna et al., 2019] Victoria Krakovna, Laurent Orseau, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability. In IJCAI AI Safety Workshop, 2019.","title":"Penalizing side effects using stepwise relative reachability","authors":"Krakovna","year":"2019","venue":"IJCAI AI Safety Workshop","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ramakrishnan et al., 2019] Ramya Ramakrishnan, Ece Kamar, Besmira Nushi, Debadeepta Dey, Julie Shah, and Eric Horvitz. Overcoming blind spots in the real world: Leveraging comple- mentary abilities for joint execution. In Proceedings of the 33rd Conference on Artificial Intelligence (AAAI), 2019.","title":"Overcoming blind spots in the real world: Leveraging complementary abilities for joint execution","authors":"Ramya Ramakrishnan, Ece Kamar, Besmira Nushi, Debadeepta Dey, Julie Shah, Eric Horvitz","year":"2019","venue":"Proceedings of the 33rd Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Roijers et al., 2013] Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective se- quential decision-making. Journal of Artificial Intelligence Re- search, 48:67-113, 2013.","title":"A survey of multi-objective sequential decision-making","authors":"M Diederik, Peter Roijers, Shimon Vamplew, Richard Whiteson, Dazeley","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Saisubramanian et al., 2019] Sandhya Saisubramanian, Kyle Wray, Luis Pineda, and Shlomo Zilberstein. Planning in stochas- tic environments with goal uncertainty. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.","title":"Planning in stochastic environments with goal uncertainty","authors":"Sandhya Saisubramanian, Kyle Wray, Luis Pineda, Shlomo Zilberstein","year":"2019","venue":"Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Saisubramanian et al., 2020] Sandhya Saisubramanian, Ece Ka- mar, and Shlomo Zilberstein. Mitigating the negative side ef- fects of reasoning with imperfect models: A multi-objective ap- proach. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.","title":"Mitigating the negative side effects of reasoning with imperfect models: A multi-objective approach","authors":"Sandhya Saisubramanian, Ece Kamar, Shlomo Zilberstein","year":"2020","venue":"Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Seuken and Zilberstein, 2007] Sven Seuken and Shlomo Zilber- stein. Improved memory-bounded dynamic programming for de- centralized POMDPs. In Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI), 2007.","title":"Sven Seuken and Shlomo Zilberstein. Improved memory-bounded dynamic programming for decentralized POMDPs","authors":"Zilberstein Seuken","year":"2007","venue":"Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Wray et al., 2015] Kyle Hollins Wray, Shlomo Zilberstein, and Abdel-Illah Mouaddib. Multi-objective MDPs with conditional lexicographic reward preferences. In Proceedings of the 29th Conference on Artificial Intelligence (AAAI), 2015.","title":"Multi-objective MDPs with conditional lexicographic reward preferences","authors":"Kyle Hollins Wray, Shlomo Zilberstein, Abdel-Illah Mouaddib","year":"2015","venue":"Proceedings of the 29th Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Zhang et al., 2018] Shun Zhang, Edmund H. Durfee, and Satin- der P. Singh. Minimax-regret querying on side effects for safe optimality in factored Markov decision processes. In Proceed- ings of the 27th International Joint Conference on Artificial In- telligence (IJCAI), 2018.","title":"Minimax-regret querying on side effects for safe optimality in factored Markov decision processes","authors":"Shun Zhang, Edmund H Durfee, Satinder P Singh","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/676"},{"position":13,"raw":"[Zilberstein, 2015] Shlomo Zilberstein. Building strong semi- autonomous systems. In Proceedings of the 29th Conference on Artificial Intelligence (AAAI), 2015.","title":"Building strong semiautonomous systems","authors":"Shlomo Zilberstein, Zilberstein","year":"2015","venue":"Proceedings of the 29th Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}