{"type":"ScholarlyArticle","title":"Equi-Reward Utility Maximizing Design in Stochastic Environments","authors":[{"name":"Sarah Keren","slug":"sarah-keren","given":"Sarah","family":"Keren","affiliation":"University of Massachusetts Amherst"},{"name":"Luis Pineda","slug":"luis-pineda","given":"Luis","family":"Pineda","affiliation":"‡ Technion-Israel Institute of Technology College of Information and Computer Sciences ,"},{"name":"Avigdor Gal","slug":"avigdor-gal","given":"Avigdor","family":"Gal","affiliation":"University of Massachusetts Amherst"},{"name":"Erez Karpas","slug":"erez-karpas","given":"Erez","family":"Karpas","affiliation":"University of Massachusetts Amherst"},{"name":"Shlomo Zilberstein","slug":"shlomo-zilberstein","given":"Shlomo","family":"Zilberstein","affiliation":"‡ Technion-Israel Institute of Technology College of Information and Computer Sciences ,"}],"abstract":"We present the Equi Reward Utility Maximizing Design (ER-UMD) problem for redesigning stochastic environments to maximize agent performance. ER-UMD fits well contemporary applications that require offline design of environments where robots and humans act and cooperate. To find an optimal modification sequence we present two novel solution techniques: a compilation that embeds design into a planning problem, allowing use of off-the-shelf solvers to find a solution, and a heuristic search in the modifications space, for which we present an admissible heuristic. Evaluation shows the feasibility of the approach using standard benchmarks from the probabilistic planning competition and a benchmark we created for a vacuum cleaning robot setting.","doi":"10.24963/ijcai.2017/608","url":"https://ijcai.org/proceedings/2017/608","pdf":"https://ijcai.org/proceedings/2017/0608.pdf","markdown":"https://ijcai.org/proceedings/2017/608.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/608","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Planning and Scheduling","track":"Main track","number":608,"pages":{"first":4353,"last":4360},"keywords":["Planning and Scheduling: Planning under Uncertainty","Combinatorial & Heuristic Search: Heuristic Search","Planning and Scheduling: Markov Decisions Processes"],"videos":[],"references":[{"position":1,"raw":"Ex-h 0 Ex-h 0+ Ex-h M inM in DC-h 0 DC-h 0 + DC-h M inM in BFD-h 0 BFD-h 0+ BFD-h M inM in BOX B=1 158.4(8) 159.0(8) 158.9(8) 163.9(8) 70.7(8) 221.4 (8) 157.4(8) 68.2(8) 216.4(8) B=2 264.7(7) 264.9(7) 267.8(7) 270.6(7) 92.1(8) 332.7(7) 260.8(7) 88.0(8) 325.3(7) B=3 238.5(4) 236.5(4) 235.6 (4) 241.5(4) 73.5(4) 271.7(4) 234.3(4) 70.2 (7) 265.94(4) BLOCKS B=1 50.5(6) 50.5(6) 50.8(6) 50.7(6) 41.7(6) 77.1(6) 50.3(6) 41.6(6) 74.4(6) B=2 28.0(4) 28.2(4) 28.0(4) 28.2(4) 17.4(4) 36.4(3) 28.0(4) 17.2(4) 35.4(3) B=3 348.9(2) 347.3(2) 348.2(2) 354.5(2) 194.6(3) 363.5(2) 352.2(2) 118.2(3) 354.8(2) EX. BLOCKS B=1 69.4(9) 70.2(9) 69.9 (9) 68.4(9) 38.7(9) 6.7(10) 69.5(9) 40.3(9) 60.3(9) B=2 161.7(9) 170.9(9) 168.1(9) 153.1(9) 88.2(9) 30.2(10) 153.9(9) 85.6(9) 135.0(9) B=3 250.7 (9) 265.9(9) 292.2(9) 252.5(9) 134.9(9) 88.8(8) 285.9(9) 160.9(9) 237.4(9) TIRE B=1 32.9(9) 32.9(9) 33.1(9) 33.3(9) 30.2(9) 36.8(9) 33.0(9) 29.5 (9) 36.9(9) B=2 55.2(7) 55.4(7) 55.0(7) 55.5(7) 51.1(8) 88.8(8) 55.0(7) 50.9(8) 89.1(8) B=3 270.3(6) 136.5)6 258.4(6) 269.7(6) 136.5(6) 258.4(6) 267.6(6) 188.3(6) 256.2(6) ELEVATOR B=1 300.4(8) 299.6(8) 301.6(8) 301.9(8) 236.2(9) 192.6(9) 302.6(8) 238.3(9) 176.6(9) B=2 361.8(5) 360.9(5) 366.2(5) 363.4(5) 261.0(5) 243.89(5) 360.8(5) 258.6(5) 231.2()","title":"Ex-h 0 Ex-h 0+ Ex-h M inM in DC-h 0 DC-h 0 + DC-h M inM in BFD-h 0 BFD-h 0+ BFD-h M inM in BOX B=1 158","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[B=3 References Bellman, 1957] Richard Bellman. A Markovian decision process. Indiana University Mathematics Journal, 6:679- 684, 1957.","title":"A Markovian decision process","authors":"Richard Bellman","year":"1957","venue":"Indiana University Mathematics Journal","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 1995] Dimitri P. Bertsekas. Dynamic program- ming and optimal control, volume 1. Athena Scientific Belmont, MA, 1995.","title":"Dynamic programming and optimal control","authors":"Dimitri P Bertsekas","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonet and Geffner, 2005] Blai Bonet and Héctor Geffner. mGPT: A probabilistic planner based on heuristic search. Journal of Artificial Intelligence Research, 24:933-944, 2005.","title":"mGPT: A probabilistic planner based on heuristic search","authors":"Geffner ; ] Blai Bonet, Héctor Bonet, Geffner","year":"2005","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dietterich, 2000] Thomas G. Dietterich. Hierarchical re- inforcement learning with the maxq value function de- composition. Journal of Artificial Intelligence Research, 13:227-303, 2000.","title":"Hierarchical reinforcement learning with the maxq value function decomposition","authors":"G Thomas, Dietterich","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Göbelbecker et al., 2010] Moritz Göbelbecker, Thomas Keller, Patrick Eyerich, Michael Brenner, and Bernhard Nebel. Coming up with good excuses: What to do when no plan can be found. Cognitive Robotics, 2010.","title":"Coming up with good excuses: What to do when no plan can be found","authors":"Moritz Göbelbecker, Thomas Keller, Patrick Eyerich, Michael Brenner, Bernhard Nebel","year":"2010","venue":"Cognitive Robotics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hansen and Zilberstein, 1998] Eric A. Hansen and Shlomo Zilberstein. Heuristic search in cyclic AND/OR graphs. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pages 412-418, 1998.","title":"Heuristic search in cyclic AND/OR graphs","authors":"Eric A Hansen, Shlomo Zilberstein","year":"1998","venue":"Proceedings of the Fifteenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Herzig et al., 2014] Andreas Herzig, Viviane Menezes, Leliane Nunes de Barros, and Renata Wassermann. On the revision of planning tasks. In Proceedings of the Twenty-first European Conference on Artificial Intelli- gence (ECAI), pages 435-440, 2014.","title":"On the revision of planning tasks","authors":"Andreas Herzig, Viviane Menezes, Leliane Nunes De Barros, Renata Wassermann","year":"2014","venue":"Proceedings of the Twenty-first European Conference on Artificial Intelligence (ECAI)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hoffmann and Nebel, 2001] Jörg Hoffmann and Bernhard Nebel. The FF planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research, 14:253-302, 2001.","title":"The FF planning system: Fast plan generation through heuristic search","authors":"Nebel Hoffmann, Jörg Hoffmann, Bernhard Nebel","year":"2001","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Holte et al., 1996] Robert C. Holte, M. B. Perez, R. M. Zim- mer, and Alan J. MacDonald. Hierarchical A*: Searching abstraction hierarchies efficiently. In Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI), pages 530-535, 1996.","title":"Hierarchical A*: Searching abstraction hierarchies efficiently","authors":"Robert C Holte, M B Perez, R M Zimmer, Alan J Macdonald","year":"1996","venue":"Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Keren et al., 2014] Sarah Keren, Avigdor Gal, and Erez Karpas. Goal recognition design. In Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling (ICAPS), pages 154-162, 2014.","title":"Goal recognition design","authors":"Sarah Keren, Avigdor Gal, Erez Karpas","year":"2014","venue":"Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keren et al., 2015] Sarah Keren, Avigdor Gal, and Erez Karpas. Goal recognition design for non-optimal agents. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI), pages 3298-3304, 2015.","title":"Goal recognition design for non-optimal agents","authors":"Sarah Keren, Avigdor Gal, Erez Karpas","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mausam and Kolobov, 2012] Mausam and Andrey Kolobov. Planning with Markov decision processes: An AI perspective. Morgan & Claypool Publishers, 2012.","title":"Mausam and Andrey Kolobov. Planning with Markov decision processes: An AI perspective","authors":"Kolobov Mausam","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Menezes et al., 2012] Maria Viviane Menezes, Leliane N. de Barros, and Silvio do Lago Pereira. Planning task val- idation. In ICAPS Workshop on Scheduling and Planning Applications (SPARK), pages 48-55, 2012.","title":"Planning task validation","authors":"Maria Viviane Menezes, Leliane N De Barros, Silvio Do, Lago Pereira","year":"2012","venue":"ICAPS Workshop on Scheduling and Planning Applications (SPARK)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nilsson, 1980] Nils J. Nilsson. Principles of Artificial Intel- ligence. Tioga Publishers, Palo Alto, California, 1980.","title":"Principles of Artificial Intelligence","authors":"Nils J Nilsson","year":"1980","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pineda and Zilberstein, 2014] Luis Pineda and Shlomo Zil- berstein. Planning under uncertainty using reduced mod- els: Revisiting determinization. In Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling (ICAPS), pages 217-225, 2014.","title":"Luis Pineda and Shlomo Zilberstein. Planning under uncertainty using reduced models: Revisiting determinization","authors":"Zilberstein Pineda","year":"2014","venue":"Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pineda and Zilberstein, 2017] Luis Pineda and Shlomo Zil- berstein. Generalizing the role of determinization in prob- abilistic planning. Technical Report UM-CS-2017-006, University of Massachusetts Amherst, 2017.","title":"Luis Pineda and Shlomo Zilberstein. Generalizing the role of determinization in probabilistic planning","authors":"Zilberstein Pineda","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wayllace et al., 2016] Christabel Wayllace, Ping Hou, William Yeoh, and Tran Cao Son. Goal recognition design with stochastic agent action outcomes. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), pages 3279-3285, 2016.","title":"Goal recognition design with stochastic agent action outcomes","authors":"Christabel Wayllace, Ping Hou, William Yeoh, Tran Cao, Son","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2016/464"},{"position":19,"raw":"[Yoon et al., 2007] Sung Wook Yoon, Alan Fern, and Robert Givan. FF-Replan: A baseline for probabilistic planning. In Proceedings of the Seventeenth International Confer- ence on Automated Planning and Scheduling (ICAPS), pages 352-359, 2007.","title":"FF-Replan: A baseline for probabilistic planning","authors":"Sung Wook Yoon, Alan Fern, Robert Givan","year":"2007","venue":"Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Younes and Littman, 2004] Håkan L. S. Younes and Michael L. Littman. PPDDL1.0: The language for the probabilistic part of IPC-4. In Proceedings of the International Planning Competition (IPC), 2004.","title":"PPDDL1.0: The language for the probabilistic part of IPC-4","authors":"L S Håkan, Michael L Younes, Littman","year":"2004","venue":"Proceedings of the International Planning Competition (IPC)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Parkes, 2008] Haoqi Zhang and David Parkes. Value-based policy teaching with active indirect elicita- tion. In Proceedings of the Twenty-Third Conference on Artificial Intelligence (AAAI), pages 208-214, 2008.","title":"Value-based policy teaching with active indirect elicitation","authors":"Parkes Zhang, Haoqi Zhang, David Parkes","year":"2008","venue":"Proceedings of the Twenty-Third Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2009] Haoqi Zhang, Yiling Chen, and David Parkes. A general approach to environment design with one agent. In Proceedings of the Twenty-First Interna- tional Joint Conference on Artifical Intelligence, pages 2002-2008, 2009.","title":"A general approach to environment design with one agent","authors":"Zhang","year":"2009","venue":"Proceedings of the Twenty-First International Joint Conference on Artifical Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2009/330"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}