{"type":"ScholarlyArticle","title":"Planning and Learning with Stochastic Action Sets","authors":[{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":"Google Research"},{"name":"Alon Cohen","slug":"alon-cohen","given":"Alon","family":"Cohen","affiliation":"Google Research"},{"name":"Avinatan Hassidim","slug":"avinatan-hassidim","given":"Avinatan","family":"Hassidim","affiliation":"Google Research"},{"name":"Yishay Mansour","slug":"yishay-mansour","given":"Yishay","family":"Mansour","affiliation":"Google Research"},{"name":"Ofer Meshi","slug":"ofer-meshi","given":"Ofer","family":"Meshi","affiliation":"Google Research"},{"name":"Martin Mladenov","slug":"martin-mladenov","given":"Martin","family":"Mladenov","affiliation":"Google Research"},{"name":"Dale Schuurmans","slug":"dale-schuurmans","given":"Dale","family":"Schuurmans","affiliation":"Google Research"}],"abstract":"In many practical uses of reinforcement learning (RL) the set of actions available at a given state is a random variable, with realizations governed by an exogenous stochastic process. Somewhat surprisingly, the foundations for such sequential decision processes have been unaddressed. In this work, we formalize and investigate MDPs with stochastic action sets (SAS-MDPs) to provide these foundations. We show that optimal policies and value functions in this model have a structure that admits a compact representation. From an RL perspective, we show that Q-learning with sampled action sets is sound. In model-based settings, we consider two important special cases: when individual actions are available with independent probabilities, and a sampling-based model for unknown distributions. We develop polynomial-time value and policy iteration methods for both cases, and provide a polynomial-time linear programming solution for the first case.","doi":"10.24963/ijcai.2018/650","url":"https://ijcai.org/proceedings/2018/650","pdf":"https://ijcai.org/proceedings/2018/0650.pdf","markdown":"https://ijcai.org/proceedings/2018/650.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/650","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":650,"pages":{"first":4674,"last":4682},"keywords":["Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Amin et al., 2012] K. Amin, M. Kearns, P. Key, and A. Schwaighofer. Budget optimization for sponsored search: Censored learning in MDPs. In Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI-12), pp.543-553, Catalina, CA, 2012.","title":"Budget optimization for sponsored search: Censored learning in MDPs","authors":"K Amin, M Kearns, P Key, A Schwaighofer","year":"2012","venue":"Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI-12)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Archak et al., 2010] N. Archak, V. S. Mirrokni, and S. Muthukrishnan. Mining advertiser-specific user behav- ior using adfactors. In Proceedings of the 19th Intl. World Wide Web Conference (WWW 2010), pp.31-40, Raleigh, NC, 2010.","title":"Mining advertiser-specific user behavior using adfactors","authors":"N Archak, V S Mirrokni, S Muthukrishnan","year":"2010","venue":"Proceedings of the 19th Intl. World Wide Web Conference (WWW 2010)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Archak et al., 2012] N. Archak, V. Mirrokni, and S. Muthukrishnan. Budget optimization for online campaigns with positive carryover effects. In Proceed- ings of the 8th Intl. Workshop on Internet and Network Economics (WINE-12), pp.86-99, Liverpool, 2012.","title":"Budget optimization for online campaigns with positive carryover effects","authors":"N Archak, V Mirrokni, S Muthukrishnan","year":"2012","venue":"Proceedings of the 8th Intl. Workshop on Internet and Network Economics (WINE-12)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boutilier et al., 2018] C. Boutilier, A. Cohen, A. Hassidim, Y. Mansour, O. Meshi, M. Mladenov, and D. Schuurmans. Planning and learning in Markov decision processes with stochastic action sets. Technical Report arXiv:1805.02363 [cs.AI], ArXiV, May 2018.","title":"Planning and learning in Markov decision processes with stochastic action sets","authors":"C Boutilier, A Cohen, A Hassidim, Y Mansour, O Meshi, M Mladenov, D Schuurmans","year":"2018","venue":"ArXiV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Charikar et al., 1999] M. Charikar, R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. On targeting Markov segments. In Proceedings of the 31st Annual ACM Sym- posium on Theory of Computing (STOC-99), pp.99-108, Atlanta, 1999.","title":"On targeting Markov segments","authors":"M Charikar, R Kumar, P Raghavan, S Rajagopalan, A Tomkins","year":"1999","venue":"Proceedings of the 31st Annual ACM Symposium on Theory of Computing (STOC-99)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grötschel et al., 1988] M. Grötschel, L. Lovász, and A. Schrijver. Geometric Algorithms and Combinatorial Optimization, Vol. 2 of Algorithms and Combinatorics. Springer, 1988.","title":"Geometric Algorithms and Combinatorial Optimization","authors":"M Grötschel, L Lovász, A Schrijver","year":"1988","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hansen et al., 2013] T. D. Hansen, P. B. Miltersen, and U. Zwick. Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant dis- count factor. Journal of the ACM (JACM), 60(1), 2013. Article 1, 16pp.","title":"Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor","authors":"T D Hansen, P B Miltersen, U Zwick","year":"2013","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kanade et al., 2009] V. Kanade, H. B. McMahan, and B. Bryan. Sleeping experts and bandits with stochastic ac- tion availability and adversarial rewards. In 12th Intl. Con- ference on Artificial Intelligence and Statistics (AIStats- 09), pp.272-279, Clearwater Beach, FL, 2009.","title":"Sleeping experts and bandits with stochastic action availability and adversarial rewards","authors":"V Kanade, H B Mcmahan, B Bryan","year":"2009","venue":"12th Intl. Conference on Artificial Intelligence and Statistics (AIStats-09)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kleinberg et al., 2010] R. Kleinberg, A. Niculescu-Mizil, and Y. Sharma. Regret bounds for sleeping experts and bandits. Machine learning, 80(2-3):245-272, 2010.","title":"Regret bounds for sleeping experts and bandits","authors":"R Kleinberg, A Niculescu-Mizil, Y Sharma","year":"2010","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2009] T. Li, N. Liu, J. Yan, G. Wang, F. Bai, and Z. Chen. A Markov chain model for integrating behav- ioral targeting into contextual advertising. In Proceedings of the 3rd Intl. Workshop on Data Mining and Audience Intelligence for Advertising, pp.1-9, Paris, 2009.","title":"A Markov chain model for integrating behavioral targeting into contextual advertising","authors":"T Li, N Liu, J Yan, G Wang, F Bai, Z Chen","year":"2009","venue":"Proceedings of the 3rd Intl. Workshop on Data Mining and Audience Intelligence for Advertising","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Meister and Holzbaur, 1986] U. Meister and U. Holzbaur. A polynomial time bound for Howard's policy improve- ment algorithm. OR Spektrum, 8:37-40, 1986.","title":"A polynomial time bound for Howard's policy improvement algorithm","authors":"U Meister, U Holzbaur","year":"1986","venue":"OR Spektrum","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mladenov et al., 2017] M. Mladenov, C. Boutilier, D. Schu- urmans, O. Meshi, G. Elidan, and T. Lu. Logistic Markov decision processes. In Proceedings of the 26th Intl. Joint Conference on Artificial Intelligence (IJCAI-17), pp.2486- 2493, Melbourne, 2017.","title":"Logistic Markov decision processes","authors":"M Mladenov, C Boutilier, D Schuurmans, O Meshi, G Elidan, T Lu","year":"2017","venue":"Proceedings of the 26th Intl. Joint Conference on Artificial Intelligence (IJCAI-17)","doi":null,"url":null,"internal_url":"/proceedings/2017/346"},{"position":13,"raw":"[Mnih et al., 2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing Atari with deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, A Graves, I Antonoglou, D Wierstra, M Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] V. Mnih, K. Kavukcuoglu, D. Silver, An- drei A Rusu, J. Veness, Marc G Bellemare, A. Graves, M. Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"V Mnih, K Kavukcuoglu, D Silver, Andrei A Rusu, J Veness, A Marc G Bellemare, M Graves, Andreas K Riedmiller, Georg Fidjeland, Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nikolova and Karger, 2008] E. Nikolova and D. R. Karger. Route planning under uncertainty: The Canadian traveller problem. In Proceedings of the 23rd National Conference on Artificial Intelligence (AAAI-08), pp.969-974, 2008.","title":"Route planning under uncertainty: The Canadian traveller problem","authors":"E Nikolova, D R Karger","year":"2008","venue":"Proceedings of the 23rd National Conference on Artificial Intelligence (AAAI-08)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papadimitriou and Yannakakis, 1991] C. H. Papadimitriou and M. Yannakakis. Shortest paths without a map. Theo- retical Computer Science, 84(1):127 -150, 1991.","title":"Shortest paths without a map","authors":"C H Papadimitriou, M Yannakakis","year":"1991","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Polychronopoulos and Tsitsiklis, 1996] G. H. Poly- chronopoulos and J. N. Tsitsiklis. Stochastic shortest path problems with recourse. Networks, 27(2):133-143, 1996.","title":"Stochastic shortest path problems with recourse","authors":"G H Polychronopoulos, J N Tsitsiklis","year":"1996","venue":"Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Puterman, 1994] M. L. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. Wi- ley, New York, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"M L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2013] D. Silver, L. Newnham, D. Barker, S. Weller, and J. McFall. Concurrent reinforcement learn- ing from customer interactions. In Proceedings of the 30th Intl. Conference on Machine Learning (ICML-13), pp.924-932, Atlanta, 2013.","title":"Concurrent reinforcement learning from customer interactions","authors":"D Silver, L Newnham, D Barker, S Weller, J Mcfall","year":"2013","venue":"Proceedings of the 30th Intl. Conference on Machine Learning (ICML-13)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Theocharous et al., 2015] G. Theocharous, P. S. Thomas, and M. Ghavamzadeh. Personalized ad recommenda- tion systems for life-time value optimization with guaran- tees. In Proceedings of the 24th Intl. Joint Conference on Artificial Intelligence (IJCAI-15), pp.1806-1812, Buenos Aires, 2015.","title":"Personalized ad recommendation systems for life-time value optimization with guarantees","authors":"G Theocharous, P S Thomas, M Ghavamzadeh","year":"2015","venue":"Proceedings of the 24th Intl. Joint Conference on Artificial Intelligence (IJCAI-15)","doi":null,"url":null,"internal_url":"/proceedings/2015/257"},{"position":21,"raw":"[Tseng, 1990] P. Tseng. Solving h-horizon, stationary Markov decision problems in time proportional to log(h). Operations Research Letters, 9(5):287-297, 1990.","title":"Solving h-horizon, stationary Markov decision problems in time proportional to log(h)","authors":"P Tseng","year":"1990","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Watkins and Dayan, 1992] C.J.C.H. Watkins and P. Dayan. Q-learning. Machine Learning, 8:279-292, 1992.","title":"Q-learning","authors":"C J C H Watkins, P Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ye, 2011] Y. Ye. The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate. Mathematics of Operations Re- search, 36(4):593-603, 2011.","title":"The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate","authors":"Y Ye","year":"2011","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}