{"type":"ScholarlyArticle","title":"Scalable Initial State Interdiction for Factored MDPs","authors":[{"name":"Swetasudha Panda","slug":"swetasudha-panda","given":"Swetasudha","family":"Panda","affiliation":"Electrical Engineering and Computer Science , Vanderbilt University , Nashville , TN"},{"name":"Yevgeniy Vorobeychik","slug":"yevgeniy-vorobeychik","given":"Yevgeniy","family":"Vorobeychik","affiliation":"Electrical Engineering and Computer Science , Vanderbilt University , Nashville , TN"}],"abstract":"We propose a novel Stackelberg game model of MDP interdiction in which the defender modifies the initial state of the planner, who then responds\r\nby computing an optimal policy starting with that state. We first develop a novel approach for MDP interdiction in factored state space that allows the defender to modify the initial state. The resulting approach can be computationally expensive for large factored MDPs. To address this, we develop several interdiction algorithms that leverage variations of reinforcement learning using both linear and non-linear function approximation. Finally, we extend the interdiction framework to consider a Bayesian interdiction problem in which the interdictor is uncertain about some of the planner's initial state features.\r\nExtensive experiments demonstrate the effectiveness of our approaches.","doi":"10.24963/ijcai.2018/667","url":"https://ijcai.org/proceedings/2018/667","pdf":"https://ijcai.org/proceedings/2018/0667.pdf","markdown":"https://ijcai.org/proceedings/2018/667.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/667","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":667,"pages":{"first":4801,"last":4807},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty","Multidisciplinary Topics and Applications: Security and Privacy","Agent-based and Multi-agent Systems: Algorithmic Game Theory"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, San- jay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265-283, 2016.","title":"Tensorflow: A system for large-scale machine learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard","year":"2016","venue":"OSDI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ammann et al., 2002] Paul Ammann, Duminda Wijesekera, and Saket Kaushik. Scalable, graph-based network vulnerability analysis. In Proceedings of the 9th ACM Conference on Com- puter and Communications Security, pages 217-224. ACM, 2002.","title":"Scalable, graph-based network vulnerability analysis","authors":"Duminda Paul Ammann, Saket Wijesekera, Kaushik","year":"2002","venue":"Proceedings of the 9th ACM Conference on Computer and Communications Security","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boddy et al., 2005] Mark Boddy, Johnathan Gohde, Tom Haigh, and Steven Harp. Course of action generation for cyber security using classical planning. In International Conference on Auto- mated Planning and Scheduling, pages 12-21, 2005.","title":"Course of action generation for cyber security using classical planning","authors":"Mark Boddy, Johnathan Gohde, Tom Haigh, Steven Harp","year":"2005","venue":"International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guestrin et al., 2003] Carlos Guestrin, Daphne Koller, Ronald Parr, and Shobha Venkataraman. Efficient solution algorithms for factored mdps. Journal of Artificial Intelligence Research, 19:399-468, 2003.","title":"Efficient solution algorithms for factored mdps","authors":"Carlos Guestrin, Daphne Koller, Ronald Parr, Shobha Venkataraman","year":"2003","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jain et al., 2008] Manish Jain, James Pita, Milind Tambe, Fer- nando Ordónez, Praveen Paruchuri, and Sarit Kraus. Bayesian stackelberg games and their application for security at los angeles international airport. ACM SIGecom Exchanges, 7(2):10, 2008.","title":"Bayesian stackelberg games and their application for security at los angeles international airport","authors":"Manish Jain, James Pita, Milind Tambe, Fernando Ordónez, Praveen Paruchuri, Sarit Kraus","year":"2008","venue":"ACM SIGecom Exchanges","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Koller and Parr, 1999] Daphne Koller and Ronald Parr. Computing factored value functions for policies in structured mdps. In IJCAI, volume 99, pages 1332-1339, 1999.","title":"Computing factored value functions for policies in structured mdps","authors":"Parr Koller, Daphne Koller, Ronald Parr","year":"1999","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/1999/189"},{"position":7,"raw":"[Korzhyk et al., 2011] Dmytro Korzhyk, Zhengyu Yin, Christopher Kiekintveld, Vincent Conitzer, and Milind Tambe. Stackelberg vs. nash in security games: An extended investigation of in- terchangeability, equivalence, and uniqueness. J. Artif. Intell. Res.(JAIR), 41:297-327, 2011.","title":"Stackelberg vs. nash in security games: An extended investigation of interchangeability, equivalence, and uniqueness","authors":"Dmytro Korzhyk, Zhengyu Yin, Christopher Kiekintveld, Vincent Conitzer, Milind Tambe","year":"2011","venue":"J. Artif. Intell. Res.(JAIR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krautsevich et al., 2012] Leanid Krautsevich, Fabio Martinelli, and Artsiom Yautsiukhin. Towards modelling adaptive attacker's behaviour. In International Symposium on Foundations and Practice of Security, pages 357-364. Springer, 2012.","title":"Leanid Krautsevich, Fabio Martinelli, and Artsiom Yautsiukhin. Towards modelling adaptive attacker's behaviour","authors":"Krautsevich","year":"2012","venue":"International Symposium on Foundations and Practice of Security","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Letchford and Vorobeychik, 2013] Joshua Letchford and Yev- geniy Vorobeychik. Optimal interdiction of attack plans. In In- ternational Conference on Autonomous Agents and Multi-Agent Systems, pages 199-206, 2013.","title":"Joshua Letchford and Yevgeniy Vorobeychik. Optimal interdiction of attack plans","authors":"Vorobeychik Letchford","year":"2013","venue":"International Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learn- ing. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostro- vski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ng et al., 2010] Brenda Ng, Carol Meyers, Kofi Boakye, and John J Nitao. Towards applying interactive pomdps to real-world adversary modeling. In IAAI, 2010.","title":"Towards applying interactive pomdps to real-world adversary modeling","authors":"Brenda Ng, Carol Meyers, Kofi Boakye, John J Nitao","year":"2010","venue":"IAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Obes et al., 2013] Jorge Lucangeli Obes, Carlos Sarraute, and Ger- ardo Richarte. Attack planning in the real world. arXiv preprint arXiv:1306.4044, 2013.","title":"Attack planning in the real world","authors":"Jorge Lucangeli Obes, Carlos Sarraute, Gerardo Richarte","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[O'Donnell, 2008] Ryan O'Donnell. Some topics in analysis of boolean functions. In Proceedings of the fortieth annual ACM symposium on Theory of computing, pages 569-578. ACM, 2008.","title":"Some topics in analysis of boolean functions","authors":"O' O'donnell ; Ryan, Donnell","year":"2008","venue":"Proceedings of the fortieth annual ACM symposium on Theory of computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Panda and Vorobeychik, 2017] Swetasudha Panda and Yevgeniy Vorobeychik. Near-optimal interdiction of factored mdps. In Conference on Uncertainty in Artificial Intelligence, 2017.","title":"Swetasudha Panda and Yevgeniy Vorobeychik. Near-optimal interdiction of factored mdps","authors":"Vorobeychik Panda","year":"2017","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paruchuri et al., 2008] Praveen Paruchuri, Jonathan P Pearce, Janusz Marecki, Milind Tambe, Fernando Ordonez, and Sarit Kraus. Playing games for security: An efficient exact algorithm for solving bayesian stackelberg games. In Proceedings of the 7th international joint conference on Autonomous agents and multia- gent systems-Volume 2, pages 895-902. International Foundation for Autonomous Agents and Multiagent Systems, 2008.","title":"Playing games for security: An efficient exact algorithm for solving bayesian stackelberg games","authors":"Praveen Paruchuri, Jonathan P Pearce, Janusz Marecki, Milind Tambe, Fernando Ordonez, Sarit Kraus","year":"2008","venue":"Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Puterman, 1994] Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. 1994.","title":"Markov decision processes: Discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}