{"type":"ScholarlyArticle","title":"Importance Sampling for Fair Policy Selection","authors":[{"name":"Shayan Doroudi","slug":"shayan-doroudi","given":"Shayan","family":"Doroudi","affiliation":"Carnegie Mellon University; Stanford University"},{"name":"Philip S. Thomas","slug":"philip-s-thomas","given":"Philip S.","family":"Thomas","affiliation":"University of Massachusetts Amherst"},{"name":"Emma Brunskill","slug":"emma-brunskill","given":"Emma","family":"Brunskill","affiliation":"Stanford University"}],"abstract":"We consider the problem of off-policy policy selection in reinforcement learning: using historical data generated from running one policy to compare two or more policies. We show that approaches based on importance sampling can be unfair---they can select the worse of two policies more often than not. We then give an example that shows importance sampling is systematically unfair in a practically relevant setting; namely, we show that it unreasonably favors shorter trajectory lengths. We then present sufficient conditions to theoretically guarantee fairness. Finally, we provide a practical importance sampling-based estimator to help mitigate the unfairness due to varying trajectory lengths.","doi":"10.24963/ijcai.2018/729","url":"https://ijcai.org/proceedings/2018/729","pdf":"https://ijcai.org/proceedings/2018/0729.pdf","markdown":"https://ijcai.org/proceedings/2018/729.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/729","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":729,"pages":{"first":5239,"last":5243},"keywords":["Uncertainty in AI: Sequential Decision Making","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Doroudi et al., 2017] Shayan Doroudi, Philip S. Thomas, and Emma Brunskill. Importance sampling for fair policy selection. In Uncertainity in Artificial Intelligence. Asso- ciation of Uncertainty in Artificial Intelligence, 2017.","title":"Importance sampling for fair policy selection","authors":"Shayan Doroudi, Philip S Thomas, Emma Brunskill","year":"2017","venue":"Uncertainity in Artificial Intelligence. Association of Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dudík et al., 2011] Miroslav Dudík, John Langford, and Li- hong Li. Doubly robust policy evaluation and learning. In International Conference on Machine Learning, pages 1097-1104. Omnipress, 2011.","title":"Doubly robust policy evaluation and learning","authors":"Dudík","year":"2011","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Jabbari et al., 2017] Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in reinforcement learning. In International Conference on Machine Learning, pages 1617-1626. PMLR, 2017.","title":"Fairness in reinforcement learning","authors":"Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Aaron Roth","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jiang and Li, 2016] Nan Jiang and Lihong Li. Doubly ro- bust off-policy value evaluation for reinforcement learn- ing. In International Conference on Machine Learning, pages 652-661. PMLR, 2016.","title":"Doubly robust off-policy value evaluation for reinforcement learn-ing","authors":"Nan Jiang, Lihong Li","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jie and Abbeel, 2010] Tang Jie and Pieter Abbeel. On a connection between importance sampling and the likeli- hood ratio policy gradient. In Advances in Neural Infor- mation Processing Systems, pages 1000-1008.","title":"On a connection between importance sampling and the likelihood ratio policy gradient","authors":"Abbeel ; ] Jie, Jie Tang, Pieter Abbeel","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Levine and Koltun, 2013] Sergey Levine and Vladlen Koltun. Guided policy search. In International Confer- ence on Machine Learning, pages 1-9. PMLR, 2013.","title":"Guided policy search","authors":"Koltun Levine, Sergey Levine, Vladlen Koltun","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Mandel et al., 2014] Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline pol- icy evaluation across representations with applications to educational games. In Internatonal Conference on Au- tonomous Agents and Multi-Agent Systems, pages 1077- 1084. International Foundation for Autonomous Agents and Multiagent Systems, 2014.","title":"Offline policy evaluation across representations with applications to educational games","authors":"Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, Zoran Popovic","year":"2014","venue":"Internatonal Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Powell and Swann, 1966] Michael J. D. Powell and J. Swann. Weighted uniform sampling-a Monte Carlo technique for reducing variance. IMA Journal of Applied Mathematics, 2(3):228-236, 1966.","title":"Weighted uniform sampling-a Monte Carlo technique for reducing variance","authors":"J D Michael, J Powell, Swann","year":"1966","venue":"IMA Journal of Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Precup et al., 2000] Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off-policy pol- icy evaluation. In International Conference on Machine Learning, pages 759-766.","title":"Eligibility traces for off-policy policy evaluation","authors":"Doina Precup, Richard S Sutton, Satinder Singh","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Morgan Kaufman, 2000.","title":null,"authors":"Morgan Kaufman","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Thomas and Brunskill, 2016] Philip S. Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139-2148. PMLR, 2016.","title":"Data-efficient off-policy policy evaluation for reinforcement learning","authors":"Philip S Thomas, Emma Brunskill","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Thomas and Brunskill, 2017] Philip S. Thomas and Emma Brunskill. Importance sampling with unequal support. In AAAI, pages 2646-2652. Association for the Advancement of Artificial Intelligence, 2017.","title":"Importance sampling with unequal support","authors":"Philip S Thomas, Emma Brunskill","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Thomas et al., 2015a] Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High- confidence off-policy evaluation. In AAAI, pages 3000-3006. Association for the Advancement of Artificial Intelligence, 2015.","title":"Highconfidence off-policy evaluation","authors":"Thomas","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Thomas et al., 2015b] Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Con- ference on Machine Learning, pages 2380-2388.","title":"High confidence policy improvement","authors":"Thomas","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"PMLR, 2015.","title":null,"authors":"Pmlr","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Thomas et al., 2017] Philip S. Thomas, Bruno Castro da Silva, Andrew G Barto, and Emma Brunskill. On en- suring that intelligent machines are well-behaved. arXiv preprint arXiv:1708.05448, 2017.","title":"On ensuring that intelligent machines are well-behaved","authors":"Philip S Thomas, Bruno Castro Da Silva, Andrew G Barto, Emma Brunskill","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2016] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with expe- rience replay. arXiv preprint arXiv:1611.01224, 2016.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}