{"type":"ScholarlyArticle","title":"Formal Explanations of Neural Network Policies for Planning","authors":[{"name":"Renee Selvey","slug":"renee-selvey","given":"Renee","family":"Selvey","affiliation":"School of Computing , The Australian National University"},{"name":"Alban Grastien","slug":"alban-grastien","given":"Alban","family":"Grastien","affiliation":"School of Computing , The Australian National University"},{"name":"Sylvie Thiébaux","slug":"sylvie-thiebaux","given":"Sylvie","family":"Thiébaux","affiliation":"LAAS-CNRS , ANITI , Université de Toulouse; School of Computing , The Australian National University"}],"abstract":"Deep learning is increasingly used to learn policies for planning problems, yet policies represented by neural networks are difficult to interpret, verify and trust. Existing formal approaches to post-hoc explanations provide concise reasons for a single decision made by an ML model. However, understanding planning policies require explaining sequences of decisions. In this paper,  we formulate the problem of finding explanations for the sequence of decisions recommended by a learnt policy in a given state. We show that, under certain assumptions, a minimal explanation for a sequence can be computed by solving a  number of single decision explanation problems which is linear in the length of the sequence. We present experimental results of our implementation of this approach for ASNet policies for classical planning domains.","doi":"10.24963/ijcai.2023/605","url":"https://ijcai.org/proceedings/2023/605","pdf":"https://ijcai.org/proceedings/2023/0605.pdf","markdown":"https://ijcai.org/proceedings/2023/605.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/605","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":605,"pages":{"first":5446,"last":5456},"keywords":["Machine Learning: ML: Explainable/Interpretable machine learning","Planning and Scheduling: PS: Model-based reasoning","Planning and Scheduling: PS: Learning in planning and scheduling"],"videos":[],"references":[{"position":1,"raw":"[Bäckström and Nebel, 1995] Christer Bäckström and Bern- hard Nebel. Complexity results for SAS+ planning. Com- put. Intell., 11:625-656, 1995.","title":"Christer Bäckström and Bernhard Nebel","authors":"Nebel Bäckström","year":"1995","venue":"Comput. Intell","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chakraborti et al., 2017] Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, and Subbarao Kambhampati. Plan explanations as model reconciliation: Moving beyond explanation as soliloquy. In Proc. IJCAI, pages 156-163, 2017.","title":"Plan explanations as model reconciliation: Moving beyond explanation as soliloquy","authors":"Tathagata Chakraborti, Sarath Sreedharan, Yu Zhang, Subbarao Kambhampati","year":"2017","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/23"},{"position":3,"raw":"[Chakraborti et al., 2020] Tathagata Chakraborti, Sarath Sreedharan, and Subbarao Kambhampati. The emerging landscape of explainable automated planning & decision making. In Proc. IJCAI, pages 4803-4811, 2020.","title":"The emerging landscape of explainable automated planning & decision making","authors":"Tathagata Chakraborti, Sarath Sreedharan, Subbarao Kambhampati","year":"2020","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/669"},{"position":4,"raw":"[Darwiche and Hirth, 2020] Adnan Darwiche and Auguste Hirth. On the reasons behind decisions. In Proc. ECAI, pages 712-720, 2020.","title":"On the reasons behind decisions","authors":"Hirth Darwiche, Adnan Darwiche, Auguste Hirth","year":"2020","venue":"Proc. ECAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eifler et al., 2020] Rebecca Eifler, Marcel Steinmetz, Álvaro Torralba, and Jörg Hoffmann. Plan-space expla- nation via plan-property dependencies: Faster algorithms & more powerful properties. In Proc. IJCAI, pages 4091-4097, 2020.","title":"Plan-space explanation via plan-property dependencies: Faster algorithms & more powerful properties","authors":"Rebecca Eifler, Marcel Steinmetz, Álvaro Torralba, Jörg Hoffmann","year":"2020","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/566"},{"position":6,"raw":"[Ferber et al., 2020] Patrick Ferber, Malte Helmert, and Jörg Hoffmann. Neural network heuristics for classical plan- ning: A study of hyperparameter space. In Proc. ECAI, pages 2346-2353, 2020.","title":"Neural network heuristics for classical planning: A study of hyperparameter space","authors":"Patrick Ferber, Malte Helmert, Jörg Hoffmann","year":"2020","venue":"Proc. ECAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ferber et al., 2022] Patrick Ferber, Florian Geißer, Fe- lipe W. Trevizan, Malte Helmert, and Jörg Hoffmann. Neural network heuristic functions for classical planning: Bootstrapping and comparison to other methods. In Proc. ICAPS, pages 583-587, 2022.","title":"Neural network heuristic functions for classical planning: Bootstrapping and comparison to other methods","authors":"Patrick Ferber, Florian Geißer, Felipe W Trevizan, Malte Helmert, Jörg Hoffmann","year":"2022","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fischetti and Jo, 2018] Matteo Fischetti and Jason Jo. Deep neural networks and mixed integer linear optimization. Constraints, 23(3):296-309, 2018.","title":"Deep neural networks and mixed integer linear optimization","authors":"Matteo Fischetti, Jason Jo","year":"2018","venue":"Constraints","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garg et al., 2020] Sankalp Garg, Aniket Bajpai, and Mausam. Symbolic network: Generalized neural policies for relational mdps. In Proc. ICML, pages 3397-3407, 2020.","title":"Symbolic network: Generalized neural policies for relational mdps","authors":"Sankalp Garg, Aniket Bajpai, Mausam","year":"2020","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gehring et al., 2022] Clement Gehring, Masataro Asai, Ro- han Chitnis, Tom Silver, Leslie Pack Kaelbling, Shirin Sohrabi, and Michael Katz. Reinforcement learning for classical planning: Viewing heuristics as dense reward generators. In Proc. ICAPS, pages 588-596, 2022.","title":"Reinforcement learning for classical planning: Viewing heuristics as dense reward generators","authors":"Clement Gehring, Masataro Asai, Rohan Chitnis, Tom Silver, Leslie Pack Kaelbling, Shirin Sohrabi, Michael Katz","year":"2022","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Groshev et al., 2018] Edward Groshev, Maxwell Goldstein, Aviv Tamar, Siddharth Srivastava, and Pieter Abbeel. Learning generalized reactive policies using deep neural networks. In Proc. ICAPS, pages 408-416, 2018.","title":"Learning generalized reactive policies using deep neural networks","authors":"Edward Groshev, Maxwell Goldstein, Aviv Tamar, Siddharth Srivastava, Pieter Abbeel","year":"2018","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gurobi Optimization, LLC, 2023] Gurobi Optimiza- tion, LLC. Gurobi Optimizer Reference Man- ual. https://www.gurobi.com/wp-content/plugins/ hd documentations/documentation/current/refman.pdf, 2023. Accessed: 2023-03-23.","title":"Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual","authors":null,"year":"2023","venue":null,"doi":null,"url":"https://www.gurobi.com/wp-content/plugins/hddocumentations/documentation/current/refman.pdf","internal_url":null},{"position":13,"raw":"[Ignatiev et al., 2019] Alexey Ignatiev, Nina Narodytska, and João Marques-Silva. Abduction-based explanations for machine learning models. In Proc. AAAI, pages 1511- 1519, 2019.","title":"Abduction-based explanations for machine learning models","authors":"Alexey Ignatiev, Nina Narodytska, João Marques-Silva","year":"2019","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karia and Srivastava, 2021] Rushang Karia and Siddharth Srivastava. Learning generalized relational heuristic net- works for model-agnostic planning. In Proc. AAAI, pages 8064-8073, 2021.","title":"Rushang Karia and Siddharth Srivastava. Learning generalized relational heuristic networks for model-agnostic planning","authors":"Srivastava Karia","year":"2021","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karia and Srivastava, 2022] Rushang Karia and Siddharth Srivastava. Relational abstractions for generalized rein- forcement learning on symbolic problems. In Proc. IJCAI, pages 3135-3142, 2022.","title":"Relational abstractions for generalized reinforcement learning on symbolic problems","authors":"Srivastava Karia, Rushang Karia, Siddharth Srivastava","year":"2022","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/435"},{"position":16,"raw":"[Kasenberg et al., 2020] Daniel Kasenberg, Ravenna Thiel- strom, and Matthias Scheutz. Generating explanations for temporal logic planner decisions. In Proc. ICAPS, pages 449-458, 2020.","title":"Generating explanations for temporal logic planner decisions","authors":"Daniel Kasenberg, Ravenna Thielstrom, Matthias Scheutz","year":"2020","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krarup et al., 2021] Benjamin Krarup, Senka Krivic, Daniele Magazzeni, Derek Long, Michael Cashmore, and David E. Smith. Contrastive explanations of plans through model restrictions. J. Artif. Intell. Res., 72:533-612, 2021.","title":"Contrastive explanations of plans through model restrictions","authors":"Benjamin Krarup, Senka Krivic, Daniele Magazzeni, Derek Long, Michael Cashmore, David E Smith","year":"2021","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lundberg and Lee, 2017] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proc. NeurIPS, pages 4765-4774, 2017.","title":"A unified approach to interpreting model predictions","authors":"M Scott, Su-In Lundberg, Lee","year":"2017","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Marques-Silva and Ignatiev, 2022] João Marques-Silva and Alexey Ignatiev. Delivering trustworthy AI through formal XAI. In Proc. AAAI, pages 12342-12350, 2022.","title":"João Marques-Silva and Alexey Ignatiev. Delivering trustworthy AI through formal XAI","authors":"Marques-Silva, Ignatiev","year":"2022","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ribeiro et al., 2016] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. \"why should I trust you?\": Explain- ing the predictions of any classifier. In Proc. KDD, pages 1135-1144, 2016.","title":"why should I trust you?\": Explaining the predictions of any classifier","authors":"Marco Túlio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proc. KDD","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ribeiro et al., 2018] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model- agnostic explanations. In Proc. AAAI, pages 1527-1535, 2018.","title":"Anchors: High-precision modelagnostic explanations","authors":"Marco Túlio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2018","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rintanen, 2017] Jussi Rintanen. Schematic invariants by re- duction to ground invariants. In Proc. AAAI, pages 3644- 3650, 2017.","title":"Jussi Rintanen. Schematic invariants by reduction to ground invariants","authors":"Rintanen","year":"2017","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shen et al., 2020] William Shen, Felipe W. Trevizan, and Sylvie Thiébaux. Learning domain-independent planning heuristics with hypergraph networks. In Proc. ICAPS, pages 574-584, 2020.","title":"Learning domain-independent planning heuristics with hypergraph networks","authors":"William Shen, Felipe W Trevizan, Sylvie Thiébaux","year":"2020","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sreedharan et al., 2021] Sarath Sreedharan, Tathagata Chakraborti, and Subbarao Kambhampati. Foundations of explanations as model reconciliation. Artif. Intell., 301:103558, 2021.","title":"Foundations of explanations as model reconciliation","authors":"Sarath Sreedharan, Tathagata Chakraborti, Subbarao Kambhampati","year":"2021","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Steinmetz et al., 2022] Marcel Steinmetz, Daniel Fiser, Hasan Ferit Eniser, Patrick Ferber, Timo P. Gros, Philippe Heim, Daniel Höller, Xandra Schuler, Valentin Wüstholz, Maria Christakis, and Jörg Hoffmann. Debugging a pol- icy: Automatic action-policy testing in AI planning. In Proc. ICAPS, pages 353-361, 2022.","title":"Debugging a policy: Automatic action-policy testing in AI planning","authors":"Marcel Steinmetz, Daniel Fiser, Ferit Hasan, Patrick Eniser, Ferber, P Timo, Philippe Gros, Daniel Heim, Xandra Höller, Valentin Schuler, Maria Wüstholz, Jörg Christakis, Hoffmann","year":"2022","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thiébaux et al., 2005] Sylvie Thiébaux, Jörg Hoffmann, and Bernhard Nebel. In defense of PDDL axioms. Artif. Intell., 168(1-2):38-69, 2005.","title":"In defense of PDDL axioms","authors":"Sylvie Thiébaux, Jörg Hoffmann, Bernhard Nebel","year":"2005","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Toyer et al., 2018] Sam Toyer, Felipe W. Trevizan, Sylvie Thiébaux, and Lexing Xie. Action schema networks: Gen- eralised policies with deep learning. In Proc. AAAI, pages 6294-6301, 2018.","title":"Action schema networks: Generalised policies with deep learning","authors":"Sam Toyer, Felipe W Trevizan, Sylvie Thiébaux, Lexing Xie","year":"2018","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Toyer et al., 2020] Sam Toyer, Sylvie Thiébaux, Felipe W. Trevizan, and Lexing Xie. Asnets: Deep learning for gen- eralised planning. J. Artif. Intell. Res., 68:1-68, 2020.","title":"Asnets: Deep learning for generalised planning","authors":"Sam Toyer, Sylvie Thiébaux, Felipe W Trevizan, Lexing Xie","year":"2020","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vasileiou et al., 2022] Stylianos Loukas Vasileiou, William Yeoh, Tran Cao Son, Ashwin Kumar, Michael Cashmore, and Daniele Magazzeni. A logic-based explanation gen- eration framework for classical and hybrid planning prob- lems. J. Artif. Intell. Res., 73:1473-1534, 2022.","title":"A logic-based explanation generation framework for classical and hybrid planning problems","authors":"Loukas Stylianos, William Vasileiou, Tran Yeoh, Ashwin Cao Son, Michael Kumar, Daniele Cashmore, Magazzeni","year":"2022","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vinzent et al., 2022] Marcel Vinzent, Marcel Steinmetz, and Jörg Hoffmann. Neural network action policy veri- fication via predicate abstraction. In Proc. ICAPS, pages 371-379, 2022.","title":"Neural network action policy verification via predicate abstraction","authors":"Vinzent","year":"2022","venue":"Proc. ICAPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vinzent et al., 2023] Marcel Vinzent, Min Wu, Haoe Wu, and Jörg Hoffmann. Policy-specific abstraction predicate selection in neural policy safety verification. In Proc. 2nd Workshop on Reliable Data-Driven Planning and Schedul- ing (RDDPS@ICAPS), 2023.","title":"Policy-specific abstraction predicate selection in neural policy safety verification","authors":"Vinzent","year":"2023","venue":"Proc. 2nd Workshop on Reliable Data-Driven Planning and Scheduling (RDDPS@ICAPS)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2022] Min Wu, Haoze Wu, and Clark W. Barrett. Verix: Towards verified explainability of deep neural net- works. CoRR, abs/2212.01051, 2022.","title":"Verix: Towards verified explainability of deep neural networks","authors":"Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Younes and Littman, 2004] Håkan Younes and Michael Littman. PPDDL1.0: an extension to PDDL for express- ing planning domains with probabilistic effects. Technical report, CMU, 2004.","title":"PPDDL1.0: an extension to PDDL for expressing planning domains with probabilistic effects","authors":"Littman Younes, Håkan Younes, Michael Littman","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yu et al., 2023] Jianqiang Yu, Alexey Ignatiev, Peter J. Stuckey, Nina Narodytska, and João Marques-Silva. Elim- inating the impossible, whatever remains must be true. In Proc. AAAI, 2023.","title":"Eliminating the impossible, whatever remains must be true","authors":"Jianqiang Yu, Alexey Ignatiev, Peter J Stuckey, Nina Narodytska, João Marques-Silva","year":"2023","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang and Geißer, 2022] Ziqi Zhang and Florian Geißer. Extending graph neural networks for generalized stochas- tic planning. In Proc. ICAPS-21 Planning and Reinforce- ment Learning Workshop (PRL), 2022.","title":"Extending graph neural networks for generalized stochastic planning","authors":"Geißer Zhang, Ziqi Zhang, Florian Geißer","year":"2022","venue":"Proc. ICAPS-21 Planning and Reinforcement Learning Workshop (PRL)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}