{"type":"ScholarlyArticle","title":"Synthesizing Programmatic Policy for Generalization within Task Domain","authors":[{"name":"Tianyi Wu","slug":"tianyi-wu","given":"Tianyi","family":"Wu","affiliation":"Fudan University"},{"name":"Liwei Shen","slug":"liwei-shen","given":"Liwei","family":"Shen","affiliation":"Fudan University"},{"name":"Zhen Dong","slug":"zhen-dong","given":"Zhen","family":"Dong","affiliation":"Fudan University"},{"name":"Xin Peng","slug":"xin-peng","given":"Xin","family":"Peng","affiliation":"Fudan University"},{"name":"Wenyun Zhao","slug":"wenyun-zhao","given":"Wenyun","family":"Zhao","affiliation":"Fudan University"}],"abstract":"Deep reinforcement learning struggles to generalize across tasks that remain unseen during training. Consider a neural process observed in humans and animals, where they not only learn new solutions but also deduce shared subroutines. These subroutines can be applied to tasks involving similar states to improve efficiency. Inspired by this phenomenon, we consider synthesizing a programmatic policy characterized by a conditional branch structure, which is capable of capturing subroutines and state patterns. This enables the learned policy to generalize to unseen tasks. The architecture of the programmatic policy is synthesized based on a context-free grammar. Such a grammar supports a nested If-Then-Else derivation and the incorporation of Recurrent Neural Network. The programmatic policy is trained across tasks in a domain through a meta-learning algorithm. We evaluate our approach in benchmarks, adapted from PDDLGym for task planning and Pybullet for robotic manipulation. Experimental results showcase the effectiveness of our approach across diverse benchmarks. Moreover, the learned policy demonstrates the ability to generalize to tasks that were not seen during training.","doi":"10.24963/ijcai.2024/577","url":"https://ijcai.org/proceedings/2024/577","pdf":"https://ijcai.org/proceedings/2024/0577.pdf","markdown":"https://ijcai.org/proceedings/2024/577.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/577","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":577,"pages":{"first":5217,"last":5225},"keywords":["Machine Learning: ML: Reinforcement learning","Robotics: ROB: Learning in robotics"],"videos":[],"references":[{"position":1,"raw":"[Akkaya et al., 2019] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.","title":"Solving rubik's cube with a robot hand","authors":"Akkaya","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bastani et al., 2018] Osbert Bastani, Yewen Pu, and Ar- mando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Informa- tion Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, De- cember 3-8, 2018, Montréal, Canada, pages 2499-2509, 2018.","title":"Verifiable reinforcement learning via policy extraction","authors":"Osbert Bastani, Yewen Pu, Armando Solar-Lezama","year":"2018","venue":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cobbe et al., 2019] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying general- ization in reinforcement learning. In International Con- ference on Machine Learning, pages 1282-1289. PMLR, 2019.","title":"Quantifying generalization in reinforcement learning","authors":"Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cui and Zhu, 2021] Guofeng Cui and He Zhu. Differen- tiable synthesis of program architectures. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 11123-11135, 2021.","title":"Differentiable synthesis of program architectures","authors":"Zhu ; ] Guofeng Cui, He Cui, Zhu","year":"2021","venue":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farebrother et al., 2018] Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.","title":"Generalization and regularization in dqn","authors":"Jesse Farebrother, Marlos C Machado, Michael Bowling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gallouédec et al., 2021] Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. panda- gym: Open-Source Goal-Conditioned Environments for Robotic Learning. 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS, 2021.","title":"pandagym: Open-Source Goal-Conditioned Environments for Robotic Learning","authors":"Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, Liming Chen","year":"2021","venue":"4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hansen and Wang, 2021] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In IEEE International Conference on Robotics and Automation, ICRA 2021, Xi'an, China, May 30 -June 5, 2021, pages 13611-13617. IEEE, 2021.","title":"Generalization in reinforcement learning by soft data augmentation","authors":"Wang ; Hansen, Nicklas Hansen, Xiaolong Wang","year":"2021","venue":"IEEE International Conference on Robotics and Automation, ICRA 2021","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Inala et al., 2020] Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthe- sizing programmatic policies that inductively generalize. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.","title":"Synthesizing programmatic policies that inductively generalize","authors":"Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, Armando Solar-Lezama","year":"2020","venue":"8th International Conference on Learning Representations, ICLR 2020","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Landajuela et al., 2021] Mikel Landajuela, Brenden K. Pe- tersen, Sookyung Kim, Cláudio P. Santiago, Ruben Glatt, T. Nathan Mundhenk, Jacob F. Pettit, and Daniel M. Fais- sol. Discovering symbolic policies with deep reinforce- ment learning. In Marina Meila and Tong Zhang, edi- tors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5979-5989. PMLR, 2021.","title":"Discovering symbolic policies with deep reinforcement learning","authors":"Mikel Landajuela, K Brenden, Sookyung Petersen, Cláudio P Kim, Ruben Santiago, T Nathan Glatt, Jacob F Mundhenk, Daniel M Pettit, Faissol","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning, ICML 2021","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laskin et al., 2020] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.","title":"Reinforcement learning with augmented data","authors":"Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, Aravind Srinivas","year":"2020","venue":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2019] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple tech- nique for generalization in deep reinforcement learning. arXiv preprint arXiv:1910.05396, 2019.","title":"Network randomization: A simple technique for generalization in deep reinforcement learning","authors":"Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. DARTS: differentiable architecture search. CoRR, abs/1806.09055, 2018.","title":"DARTS: differentiable architecture search","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization-an empirical study on continuous control. arXiv preprint arXiv:1910.09191, 2019.","title":"Regularization matters in policy optimization-an empirical study on continuous control","authors":"Zhuang Liu, Xuanlin Li, Bingyi Kang, Trevor Darrell","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2023] Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Lee, and Shao-Hua Sun. Hierarchical pro- grammatic reinforcement learning via learning to com- pose programs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Ma- chine Learning Research, pages 21672-21697. PMLR, 2023.","title":"Hierarchical programmatic reinforcement learning via learning to compose programs","authors":"Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Lee, Shao-Hua Sun","year":"2023","venue":"International Conference on Machine Learning, ICML 2023","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2020] Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. arXiv preprint arXiv:2008.00614, 2020.","title":"Dynamics generalization via information bottleneck in deep reinforcement learning","authors":"Xingyu Lu, Kimin Lee, Pieter Abbeel, Stas Tiomkin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nachum et al., 2018] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical rein- forcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Informa- tion Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, De- cember 3-8, 2018, Montréal, Canada, pages 3307-3317, 2018.","title":"Data-efficient hierarchical reinforcement learning","authors":"Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine","year":"2018","venue":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nichol and Schulman, 2018] Alex Nichol and John Schul- man. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2(3):4, 2018.","title":"Reptile: a scalable metalearning algorithm","authors":"Alex Nichol, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Packer et al., 2018] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282, 2018.","title":"Assessing generalization in deep reinforcement learning","authors":"Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qiu and Zhu, 2022] Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"Programmatic reinforcement learning without oracles","authors":"Zhu ; ] Wenjie Qiu, He Qiu, Zhu","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver and Chitnis, 2020] Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems. In International Conference on Automated Planning and Scheduling (ICAPS) PRL Workshop, 2020.","title":"Tom Silver and Rohan Chitnis. Pddlgym: Gym environments from pddl problems","authors":"Chitnis Silver","year":"2020","venue":"International Conference on Automated Planning and Scheduling (ICAPS) PRL Workshop","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tobin et al., 2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural net- works from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017, Vancouver, BC, Canada, Septem- ber 24-28, 2017, pages 23-30. IEEE, 2017.","title":"Domain randomization for transferring deep neural networks from simulation to the real world","authors":"Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel","year":"2017","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2017","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Trivedi et al., 2021] Dweep Trivedi, Jesse Zhang, Shao- Hua Sun, and Joseph J. Lim. Learning to synthe- size programs as interpretable and generalizable policies. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems 34: Annual Conference on Neural Information Pro- cessing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 25146-25163, 2021.","title":"Learning to synthesize programs as interpretable and generalizable policies","authors":"Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, Joseph J Lim","year":"2021","venue":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tzeng et al., 2020] Eric Tzeng, Coline Devin, Judy Hoff- man, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting deep visuomotor representations with weak pairwise constraints. In Al- gorithmic Foundations of Robotics XII: Proceedings of the Twelfth Workshop on the Algorithmic Foundations of Robotics, pages 688-703. Springer, 2020.","title":"Adapting deep visuomotor representations with weak pairwise constraints","authors":"Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, Trevor Darrell","year":"2020","venue":"Algorithmic Foundations of Robotics XII: Proceedings of the Twelfth Workshop on the Algorithmic Foundations of Robotics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Verma et al., 2018] Abhinav Verma, Vijayaraghavan Mu- rali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaud- huri. Programmatically interpretable reinforcement learn- ing. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Ma- chine Learning, ICML 2018, Stockholmsmässan, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Proceed- ings of Machine Learning Research, pages 5052-5061.","title":"Programmatically interpretable reinforcement learning","authors":"Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Verma et al., 2019] Abhinav Verma, Hoang Minh Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Sys- tems 2019, NeurIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pages 15726-15737, 2019.","title":"Imitation-projected programmatic reinforcement learning","authors":"Abhinav Verma, Minh Hoang, Yisong Le, Swarat Yue, Chaudhuri","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Winskel, 1993] Glynn Winskel. The formal semantics of programming languages: an introduction. MIT press, 1993.","title":"The formal semantics of programming languages: an introduction","authors":"Glynn Winskel","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yarats et al., 2021] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regulariz- ing deep reinforcement learning from pixels. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021.","title":"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels","authors":"Denis Yarats, Ilya Kostrikov, Rob Fergus","year":"2021","venue":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}