{"type":"ScholarlyArticle","title":"Integrating Intent Understanding and Optimal Behavior Planning for Behavior Tree Generation from Human Instructions","authors":[{"name":"Xinglin Chen","slug":"xinglin-chen","given":"Xinglin","family":"Chen","affiliation":"College of Computer Science and Technology , National University of Defense Technology"},{"name":"Yishuai Cai","slug":"yishuai-cai","given":"Yishuai","family":"Cai","affiliation":"College of Computer Science and Technology , National University of Defense Technology"},{"name":"Yunxin Mao","slug":"yunxin-mao","given":"Yunxin","family":"Mao","affiliation":"College of Computer Science and Technology , National University of Defense Technology"},{"name":"Minglong Li","slug":"minglong-li","given":"Minglong","family":"Li","affiliation":"College of Computer Science and Technology , National University of Defense Technology"},{"name":"Wenjing Yang","slug":"wenjing-yang","given":"Wenjing","family":"Yang","affiliation":"College of Computer Science and Technology , National University of Defense Technology"},{"name":"Weixia Xu","slug":"weixia-xu","given":"Weixia","family":"Xu","affiliation":"College of Computer Science and Technology , National University of Defense Technology"},{"name":"Ji Wang","slug":"ji-wang","given":"Ji","family":"Wang","affiliation":"College of Computer Science and Technology , National University of Defense Technology"}],"abstract":"Robots executing tasks following human instructions in domestic or industrial environments essentially require both adaptability and reliability. Behavior Tree (BT) emerges as an appropriate control architecture for these scenarios due to its modularity and reactivity. Existing BT generation methods, however, either do not involve interpreting natural language or cannot theoretically guarantee the BTs' success. This paper proposes a two-stage framework for BT generation, which first employs large language models (LLMs) to interpret goals from high-level instructions, then constructs an efficient goal-specific BT through the Optimal Behavior Tree Expansion Algorithm (OBTEA). We represent goals as well-formed formulas in first-order logic, effectively bridging intent understanding and optimal behavior planning. Experiments in the service robot validate the proficiency of LLMs in producing grammatically correct and accurately interpreted goals, demonstrate OBTEA's superiority over the baseline BT Expansion algorithm in various metrics, and finally confirm the practical deployability of our framework. The project website is https://dids-ei.github.io/Project/LLM-OBTEA.","doi":"10.24963/ijcai.2024/755","url":"https://ijcai.org/proceedings/2024/755","pdf":"https://ijcai.org/proceedings/2024/0755.pdf","markdown":"https://ijcai.org/proceedings/2024/755.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/755","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Robotics","track":"Main Track","number":755,"pages":{"first":6832,"last":6840},"keywords":["Planning and Scheduling: PS: Robot planning","Robotics: ROB: Human robot interaction","Robotics: ROB: Behavior and control"],"videos":[],"references":[{"position":1,"raw":"[Ahn et al., 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.","title":"Do as i can, not as i say: Grounding language in robotic affordances","authors":"Anthony Michael Ahn, Noah Brohan, Yevgen Brown, Omar Chebotar, Byron Cortes, Chelsea David, Chuyuan Finn, Keerthana Fu, Karol Gopalakrishnan, Hausman","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Banerjee, 2018] Bikramjit Banerjee. Autonomous Acquisi- tion of Behavior Trees for Robot Control. pages 3460- 3467, 2018.","title":"Autonomous Acquisition of Behavior Trees for Robot Control","authors":"Bikramjit Banerjee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- ter, Christopher Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. 2020.","title":"Language models are fewshot learners","authors":"B Tom, Benjamin Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Sandhini Askell, Ariel Agarwal, Gretchen Herbert-Voss, Tom Krueger, Rewon Henighan, Aditya Child, Daniel M Ramesh, Jeffrey Ziegler, Clemens Wu, Christopher Winter, Mark Hesse, Eric Chen, Mateusz Sigler, Scott Litwin, Benjamin Gray, Jack Chess, Christopher Clark, Sam Berner, Alec Mccandlish, Ilya Radford, Dario Sutskever, Amodei","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2021] Zhongxuan Cai, Minglong Li, Wanrong Huang, and Wenjing Yang. BT Expansion: A Sound and Complete Algorithm for Behavior Planning of Intelligent Robots with Behavior Trees. pages 6058-6065. AAAI Press, 2021.","title":"BT Expansion: A Sound and Complete Algorithm for Behavior Planning of Intelligent Robots with Behavior Trees","authors":"Cai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chaudhuri et al., 2021] Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar- Lezama, Yisong Yue, et al. Neurosymbolic programming. Foundations and Trends® in Programming Languages, 7(3):158-243, 2021.","title":"Neurosymbolic programming","authors":"Chaudhuri","year":"2021","venue":"Foundations and Trends® in Programming Languages","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2023] Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, and He Wang. RoboGPT: An intelligent agent of making embodied long- term decisions for daily instruction tasks. arXiv preprint arXiv:2311.15649, 2023.","title":"RoboGPT: An intelligent agent of making embodied longterm decisions for daily instruction tasks","authors":"Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, He Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Colledanchise and Ögren, 2018] Michele Colledanchise and Petter Ögren. Behavior Trees in Robotics and AI: An Introduction. CRC Press, 2018.","title":"Behavior Trees in Robotics and AI: An Introduction","authors":"Michele Colledanchise, Petter Ögren","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Colledanchise et al., 2019] Michele Colledanchise, Ramviyas Parasuraman, and Petter Ögren. Learning of Behavior Trees for Autonomous Agents. IEEE Transactions on Games, 11(2):183-189, 2019.","title":"Learning of Behavior Trees for Autonomous Agents","authors":"Michele Colledanchise, Ramviyas Parasuraman, Petter Ögren","year":"2019","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Driess et al., 2023] Danny Driess, Fei Xia, Mehdi SM Saj- jadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.","title":"Palm-e: An embodied multimodal language model","authors":"Danny Driess, Fei Xia, S M Mehdi, Corey Sajjadi, Aakanksha Lynch, Brian Chowdhery, Ayzaan Ichter, Jonathan Wahid, Quan Tompson, Tianhe Vuong, Yu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fikes and Nilsson, 1971] Richard E. Fikes and Nils J. Nils- son. Strips: A new approach to the application of theorem proving to problem solving. 2(3-4):189-208, December 1971.","title":"Strips: A new approach to the application of theorem proving to problem solving","authors":"Richard E Fikes, Nils J Nilsson","year":"1971","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/1971/63"},{"position":11,"raw":"[French et al., 2019] Kevin French, Shiyu Wu, Tianyang Pan, Zheming Zhou, and Odest Chadwicke Jenkins. Learning behavior trees from demonstration. In 2019 International Conference on Robotics and Automation (ICRA), pages 7791-7797. IEEE, 2019.","title":"Learning behavior trees from demonstration","authors":"Kevin French, Shiyu Wu, Tianyang Pan, Zheming Zhou, Odest Chadwicke Jenkins","year":"2019","venue":"2019 International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hodel, 2013] Richard E Hodel. An Introduction to Mathe- matical Logic. Courier Corporation, 2013.","title":"An Introduction to Mathematical Logic","authors":"Richard E Hodel","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Shen Li, Daehyung Park, Yoonchang Sung, Julie A Shah, and Nicholas Roy. Reactive task and mo- tion planning under temporal logic specifications. In 2021 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 12618-12624. IEEE, 2021.","title":"Reactive task and motion planning under temporal logic specifications","authors":"Shen Li, Daehyung Park, Yoonchang Sung, Julie A Shah, Nicholas Roy","year":"2021","venue":"2021 IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liang et al., 2022] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. In arXiv Preprint arXiv:2209.07753, 2022.","title":"Code as Policies: Language Model Programs for Embodied Control","authors":"Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liang et al., 2023] Xiwen Liang, Liang Ma, Shanshan Guo, Jianhua Han, Hang Xu, Shikui Ma, and Xiaodan Liang. MO-VLN: A Multi-Task Benchmark for Open-set Zero- Shot Vision-and-Language Navigation. arXiv, 2023.","title":"MO-VLN: A Multi-Task Benchmark for Open-set Zero-Shot Vision-and-Language Navigation","authors":"Xiwen Liang, Liang Ma, Shanshan Guo, Jianhua Han, Hang Xu, Shikui Ma, Xiaodan Liang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lim et al., 2010] Chong-U Lim, Robin Baumgarten, and Simon Colton. Evolving behaviour trees for the com- mercial game DEFCON. In Applications of Evolution- ary Computation: EvoApplicatons 2010: EvoCOMPLEX, EvoGAMES, EvoIASP, EvoINTELLIGENCE, EvoNUM, and EvoSTOC, Istanbul, Turkey, April 7-9, 2010, Proceed- ings, Part I, pages 100-110. Springer, 2010.","title":"Evolving behaviour trees for the commercial game DEFCON","authors":"Chong-U Lim, Robin Baumgarten, Simon Colton","year":"2010","venue":"Applications of Evolutionary Computation: EvoApplicatons 2010","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2023a] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: Empowering Large Language Models with Opti- mal Planning Proficiency. arXiv, 2023.","title":"LLM+P: Empowering Large Language Models with Optimal Planning Proficiency","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023b] Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, et al. Fimo: A challenge formal dataset for automated theorem proving. arXiv preprint arXiv:2309.04295, 2023.","title":"Fimo: A challenge formal dataset for automated theorem proving","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lykov and Tsetserukou, 2023] Artem Lykov and Dzmitry Tsetserukou. LLM-BRAIn: AI-driven Fast Generation of Robot Behaviour Tree based on Large Language Model. arXiv preprint arXiv:2305.19352, 2023.","title":"Artem Lykov and Dzmitry Tsetserukou","authors":"Tsetserukou Lykov","year":"2023","venue":"LLM-BRAIn: AI-driven Fast Generation of Robot Behaviour Tree based on Large Language Model","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lykov et al., 2023] Artem Lykov, Maria Dronova, Niko- lay Naglov, Mikhail Litvinov, Sergei Satsevich, Artem Bazhenov, Vladimir Berman, Aleksei Shcherbak, and Dzmitry Tsetserukou. LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems. arXiv preprint arXiv:2312.09348, 2023.","title":"LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems","authors":"Artem Lykov, Maria Dronova, Nikolay Naglov, Mikhail Litvinov, Sergei Satsevich, Artem Bazhenov, Vladimir Berman, Aleksei Shcherbak, Dzmitry Tsetserukou","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Neupane and Goodrich, 2019] Aadesh Neupane and Michael Goodrich. Learning Swarm Behaviors using Grammatical Evolution and Behavior Trees. pages 513-520. IJCAI Organization, 2019.","title":"Learning Swarm Behaviors using Grammatical Evolution and Behavior Trees","authors":"Goodrich Neupane","year":"2019","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2019/73"},{"position":22,"raw":"[Neupane and Goodrich, 2023] Aadesh Neupane and Michael A Goodrich. Designing Behavior Trees from Goal-Oriented LTLf Formulas. arXiv preprint arXiv:2307.06399, 2023.","title":"Designing Behavior Trees from Goal-Oriented LTLf Formulas","authors":"Aadesh Neupane, Michael A Goodrich","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pereira and Engel, 2015] Renato de Pontes Pereira and Paulo Martins Engel. A framework for constrained and adaptive behavior-based agents. arXiv preprint arXiv:1506.02312, 2015.","title":"Renato de Pontes Pereira and Paulo Martins Engel. A framework for constrained and adaptive behavior-based agents","authors":"Engel Pereira","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Puig et al., 2018] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494- 8502, 2018.","title":"Virtualhome: Simulating household activities via programs","authors":"Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, Antonio Torralba","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Scheide et al., 2021] Emily Scheide, Graeme Best, and Ge- offrey A. Hollinger. Behavior Tree Learning for Robotic Task Planning through Monte Carlo DAG Search over a Formal Grammar. pages 4837-4843, Xi'an, China, May 2021. IEEE.","title":"Behavior Tree Learning for Robotic Task Planning through Monte Carlo DAG Search over a Formal Grammar","authors":"Emily Scheide, Graeme Best, Geoffrey A Hollinger","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shridhar et al., 2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10740-10749, 2020.","title":"Alfred: A benchmark for interpreting grounded instructions for everyday tasks","authors":"Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Singh et al., 2022] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Trem- blay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. September 2022.","title":"ProgPrompt: Generating Situated Robot Task Plans using Large Language Models","authors":"Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Song et al., 2023] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM-Planner: Few-Shot Grounded Planning for Embod- ied Agents with Large Language Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.","title":"LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models","authors":"Hee Chan, Jiaman Song, Clayton Wu, Brian M Washington, Wei-Lun Sadler, Yu Chao, Su","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tadewos et al., 2022] Tadewos G Tadewos, Abdullah Al Redwan Newaz, and Ali Karimoddini. Specification- guided behavior tree synthesis and execution for coordination of autonomous systems. Expert Systems with Applications, 201:117022, 2022.","title":"Specificationguided behavior tree synthesis and execution for coordination of autonomous systems","authors":"Tadewos","year":"2022","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeng et al., 2022] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero- shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.","title":"Socratic models: Composing zeroshot multimodal reasoning with language","authors":"Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2023] Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large Language Models can Learn Rules. arXiv, October 2023.","title":"Large Language Models can Learn Rules","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}