{"type":"ScholarlyArticle","title":"MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models","authors":[{"name":"Kanxue Li","slug":"kanxue-li","given":"Kanxue","family":"Li","affiliation":"Yunnan University; JD Explore Academy; Yunnan Key Laboratory of Media Convergence"},{"name":"Baosheng Yu","slug":"baosheng-yu","given":"Baosheng","family":"Yu","affiliation":"University of Sydney"},{"name":"Qi Zheng","slug":"qi-zheng","given":"Qi","family":"Zheng","affiliation":"Shenzhen University"},{"name":"Yibing Zhan","slug":"yibing-zhan","given":"Yibing","family":"Zhan","affiliation":"JD Explore Academy"},{"name":"Yuhui Zhang","slug":"yuhui-zhang","given":"Yuhui","family":"Zhang","affiliation":"University of Sydney; JD Explore Academy"},{"name":"Tianle Zhang","slug":"tianle-zhang","given":"Tianle","family":"Zhang","affiliation":"JD Explore Academy"},{"name":"Yijun Yang","slug":"yijun-yang","given":"Yijun","family":"Yang","affiliation":"University of Technology Sydney"},{"name":"Yue Chen","slug":"yue-chen","given":"Yue","family":"Chen","affiliation":"JD Explore Academy"},{"name":"Lei Sun","slug":"lei-sun","given":"Lei","family":"Sun","affiliation":"JD Explore Academy"},{"name":"Qiong Cao","slug":"qiong-cao","given":"Qiong","family":"Cao","affiliation":"JD Explore Academy"},{"name":"Li Shen","slug":"li-shen","given":"Li","family":"Shen","affiliation":"JD Explore Academy"},{"name":"Lusong Li","slug":"lusong-li","given":"Lusong","family":"Li","affiliation":"JD Explore Academy"},{"name":"Dapeng Tao","slug":"dapeng-tao","given":"Dapeng","family":"Tao","affiliation":"Yunnan University; Yunnan Key Laboratory of Media Convergence"},{"name":"Xiaodong He","slug":"xiaodong-he","given":"Xiaodong","family":"He","affiliation":"JD Explore Academy"}],"abstract":"Foundation models have demonstrated significant emergent abilities, holding great promise for enhancing embodied agents' reasoning and planning capacities. However, the absence of a comprehensive benchmark for evaluating embodied agents with multimodal observations in complex environments remains a notable gap. In this paper, we present MuEP, a comprehensive Multimodal benchmark for Embodied Planning. MuEP facilitates the evaluation of multimodal and multi-turn interactions of embodied agents in complex scenes, incorporating fine-grained evaluation metrics that provide insights into the performance of embodied agents throughout each task. Furthermore, we evaluate embodied agents with recent state-of-the-art foundation models, including large language models (LLMs) and large multimodal models (LMMs), on the proposed benchmark. Experimental results show that foundation models based on textual representations of environments usually outperform their visual counterparts, suggesting a gap in embodied planning abilities with multimodal observations. We also find that control language generation is an indispensable ability beyond common-sense knowledge for accurate embodied task completion. We hope the proposed MuEP benchmark can contribute to the advancement of embodied AI with foundation models.","doi":"10.24963/ijcai.2024/15","url":"https://ijcai.org/proceedings/2024/15","pdf":"https://ijcai.org/proceedings/2024/0015.pdf","markdown":"https://ijcai.org/proceedings/2024/15.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/15","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":15,"pages":{"first":129,"last":138},"keywords":["Multidisciplinary Topics and Applications: MTA: Databases","Agent-based and Multi-agent Systems: MAS: Agent-based simulation and emergence","Computer Vision: CV: Embodied vision: Active agents, simulation"],"videos":[],"references":[{"position":1,"raw":"[Aeronautiques et al., 1998] Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins SRI, Anthony Bar- rett, Dave Christianson, et al. Pddl: the planning domain defini- tion language. Technical Report, Tech. Rep., 1998.","title":"Pddl: the planning domain definition language","authors":"Aeronautiques","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahn et al., 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affor- dances. arXiv preprint arXiv:2204.01691, 2022.","title":"Do as i can, not as i say: Grounding language in robotic affordances","authors":"Anthony Michael Ahn, Noah Brohan, Yevgen Brown, Omar Chebotar, Byron Cortes, Chelsea David, Chuyuan Finn, Keerthana Fu, Karol Gopalakrishnan, Hausman","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alayrac et al., 2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022.","title":"Flamingo: a visual language model for few-shot learning","authors":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anderson et al., 2018a] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navi- gation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.","title":"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Anderson","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Anderson et al., 2018b] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instruc- tions in real environments. In 2018 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018.","title":"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Anderson","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brohan et al., 2022] Anthony Brohan, Noah Brown, Justice Car- bajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.","title":"Rt-1: Robotics transformer for real-world control at scale","authors":"Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck et al., 2023] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.","title":"Sparks of artificial general intelligence: Early experiments with gpt-4","authors":"Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2019] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019.","title":"Touchdown: Natural language navigation and spatial reasoning in visual street environments","authors":"Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, Yoav Artzi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qim- ing Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.","title":"Evaluating large language models trained on code","authors":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Côté et al., 2019] Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games Workshop, IJCAI, 2019.","title":"Textworld: A learning environment for text-based games","authors":"Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada","year":"2019","venue":"Computer Games Workshop, IJCAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dai et al., 2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.","title":"InstructBLIP: Towards general-purpose visionlanguage models with instruction tuning","authors":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Das et al., 2018a] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In CVPR, 2018.","title":"Embodied question answering","authors":"Das","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Das et al., 2018b] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018, 2018.","title":"Embodied question answering","authors":"Das","year":"2018","venue":"2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Deitke et al., 2022] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. NeurIPS, 35:5982-5994, 2022.","title":"Procthor: Large-scale embodied ai using procedural generation","authors":"Matt Deitke, Eli Vanderbilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, Roozbeh Mottaghi","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dettmers et al., 2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtz- man, and Luke Zettlemoyer. Qlora: Efficient finetuning of quan- tized llms. arXiv preprint arXiv:2305.14314, 2023.","title":"Qlora: Efficient finetuning of quantized llms","authors":"Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Driess et al., 2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.","title":"Palme: An embodied multimodal language model","authors":"Danny Driess, Fei Xia, S M Mehdi, Corey Sajjadi, Aakanksha Lynch, Brian Chowdhery, Ayzaan Ichter, Jonathan Wahid, Quan Tompson, Tianhe Vuong, Yu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ebert et al., 2021] Frederik Ebert, Yanlai Yang, Karl Schmeck- peper, Bernadette Bucher, Georgios Georgakis, Kostas Dani- ilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396, 2021.","title":"Bridge data: Boosting generalization of robotic skills with cross-domain datasets","authors":"Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, Sergey Levine","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gan et al., 2020] Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Fre- itas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04954, 2020.","title":"Threedworld: A platform for interactive multi-modal physical simulation","authors":"Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gan et al., 2021] Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel L. K. Yamins, James J. DiCarlo, Josh H. McDermott, Antonio Tor- ralba, and Joshua B. Tenenbaum. The threedworld transport chal- lenge: A visually guided task-and-motion planning benchmark for physically realistic embodied AI. CoRR, 2021.","title":"The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied AI","authors":"Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, L K Daniel, James J Yamins, Josh H Dicarlo, Antonio Mcdermott, Joshua B Torralba, Tenenbaum","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gao et al., 2023] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xi- angyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023.","title":"Llama-adapter v2: Parameter-efficient visual instruction model","authors":"Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hoffmann and Nebel, 2001] Jörg Hoffmann and Bernhard Nebel. The ff planning system: Fast plan generation through heuristic search. JAIR, 2001.","title":"The ff planning system: Fast plan generation through heuristic search","authors":"Nebel Hoffmann, Jörg Hoffmann, Bernhard Nebel","year":"2001","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Huang et al., 2022] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.","title":"Inner monologue: Embodied reasoning through planning with language models","authors":"Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Huang et al., 2023] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.11176, 2023.","title":"Instruct2act: Mapping multi-modality instructions to robotic actions with large language model","authors":"Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, Hongsheng Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kolve et al., 2017] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.","title":"Ai2-thor: An interactive 3d environment for visual ai","authors":"Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Vanderbilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.","title":"BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models","authors":"Junnan Li, Dongxu Li, Silvio Savarese, Steven C H Hoi","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lin et al., 2023] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From nat- ural language instructions to feasible plans. arXiv preprint arXiv:2303.12153, 2023.","title":"Text2motion: From natural language instructions to feasible plans","authors":"Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, Jeannette Bohg","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.","title":"Visual instruction tuning","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mangrulkar et al., 2022] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods, 2022.","title":"Peft: State-of-the-art parameter-efficient fine-tuning methods","authors":"Mangrulkar","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mu et al., 2023] Yao Mu, Qinglong Zhang, Mengkang Hu, Wen- hai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. arXiv preprint arXiv:2305.15021, 2023.","title":"Embodiedgpt: Vision-language pre-training via embodied chain of thought","authors":"Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, Ping Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Muennighoff et al., 2022] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022. [OpenAI, 2023] OpenAI. Gpt-4 technical report, 2023.","title":"Crosslingual generalization through multitask finetuning","authors":"Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Padmakumar et al., 2022] Aishwarya Padmakumar, Jesse Thoma- son, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In AAAI, 2022.","title":"Teach: Task-driven embodied agents that chat","authors":"Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, Dilek Hakkani-Tur","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ramakrishnan et al., 2020] Santhosh K Ramakrishnan, Ziad Al- Halah, and Kristen Grauman. Occupancy anticipation for effi- cient exploration and navigation. In ECCV. Springer, 2020.","title":"Occupancy anticipation for efficient exploration and navigation","authors":"Ziad Santhosh K Ramakrishnan, Kristen Al-Halah, Grauman","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shinn et al., 2023] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023.","title":"Reflexion: Language agents with verbal reinforcement learning","authors":"Shinn","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Shridhar et al., 2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for inter- preting grounded instructions for everyday tasks. 2020.","title":"ALFRED: A benchmark for interpreting grounded instructions for everyday tasks","authors":"Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, Dieter Fox","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shridhar et al., 2021] Mohit Shridhar, Xingdi Yuan, Marc- Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environ- ments for interactive learning. In ICLR, 2021.","title":"Alfworld: Aligning text and embodied environments for interactive learning","authors":"Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, Matthew J Hausknecht","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.","title":"Llama 2: Open foundation and fine-tuned chat models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vemprala et al., 2023] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design princi- ples and model abilities. Microsoft Auton. Syst. Robot. Res, 2023.","title":"Chatgpt for robotics: Design principles and model abilities","authors":"Sai Vemprala, Rogerio Bonatti, Arthur Bucker, Ashish Kapoor","year":"2023","venue":"Microsoft Auton. Syst. Robot. Res","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Waisberg et al., 2024] Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, Nasif Zaman, Prithul Sarker, Andrew G Lee, and Alireza Tavakkoli. Google's ai chatbot \"bard\": a side-by-side comparison with chatgpt and its utilization in ophthalmology. Eye, 2024.","title":"Google's ai chatbot \"bard\": a side-by-side comparison with chatgpt and its utilization in ophthalmology","authors":"Ethan Waisberg, Joshua Ong, Mouayad Masalkhi, Nasif Zaman, Prithul Sarker, Andrew G Lee, Alireza Tavakkoli","year":"2024","venue":"Eye","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2022] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- naneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.","title":"Self-instruct: Aligning language model with self generated instructions","authors":"Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, Hannaneh Hajishirzi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2023] Yufei Wang, Zhou Xian, Feng Chen, Tsun- Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erick- son, David Held, and Chuang Gan. Robogen: Towards unleash- ing infinite data for automated robot learning via generative sim- ulation. arXiv preprint arXiv:2311.01455, 2023.","title":"Robogen: Towards unleashing infinite data for automated robot learning via generative simulation","authors":"Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, Chuang Gan","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.","title":"Chain-of-thought prompting elicits reasoning in large language models","authors":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, V Quoc, Denny Le, Zhou","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wijmans et al., 2019] Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answer- ing in photorealistic environments with point cloud perception. In CVPR, 2019.","title":"Embodied question answering in photorealistic environments with point cloud perception","authors":"Wijmans","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wu et al., 2023a] Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 2023.","title":"A brief overview of chatgpt: The history, status quo and potential future development","authors":"Wu","year":"2023","venue":"IEEE/CAA Journal of Automatica Sinica","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wu et al., 2023b] Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848, 2023.","title":"Embodied task planning with large language models","authors":"Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Xiang et al., 2023] Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language mod- els meet world models: Embodied experiences enhance language models. arXiv preprint arXiv:2305.10626, 2023.","title":"Language models meet world models: Embodied experiences enhance language models","authors":"Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, Zhiting Hu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Xu et al., 2023] Canwen Xu, Daya Guo, Nan Duan, and Ju- lian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196, 2023.","title":"Baize: An open-source chat model with parameter-efficient tuning on self-chat data","authors":"Canwen Xu, Daya Guo, Nan Duan, Julian Mcauley","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Yang et al., 2023a] Hui Yang, Sifu Yue, and Yunzhong He. Auto- gpt for online decision making: Benchmarks and additional opin- ions. arXiv preprint arXiv:2306.02224, 2023.","title":"Autogpt for online decision making: Benchmarks and additional opinions","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Yang et al., 2023b] Yijun Yang, Tianyi Zhou, Jing Jiang, Guodong Long, and Yuhui Shi. Continual task allocation in meta-policy network via sparse prompting. In International Conference on Machine Learning, pages 39623-39638. PMLR, 2023.","title":"Continual task allocation in meta-policy network via sparse prompting","authors":"Yang","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yang et al., 2023c] Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied multi-modal agent trained by an llm from a par- allel textworld. arXiv preprint arXiv:2311.16714, 2023.","title":"Embodied multi-modal agent trained by an llm from a parallel textworld","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Yao et al., 2023] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023.","title":"React: Synergizing reasoning and acting in language models","authors":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, R Karthik, Yuan Narasimhan, Cao","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhang et al., 2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.","title":"Opt: Open pre-trained transformer language models","authors":"Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhou et al., 2023] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A re- alistic web environment for building autonomous agents. CoRR, 2023.","title":"Webarena: A realistic web environment for building autonomous agents","authors":"Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, Graham Neubig","year":"2023","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.","title":"Minigpt-4: Enhancing visionlanguage understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}