{"type":"ScholarlyArticle","title":"A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings","authors":[{"name":"Vanya Cohen","slug":"vanya-cohen","given":"Vanya","family":"Cohen","affiliation":""},{"name":"Jason Xinyu Liu","slug":"jason-xinyu-liu","given":"Jason Xinyu","family":"Liu","affiliation":"Brown University"},{"name":"Raymond Mooney","slug":"raymond-mooney","given":"Raymond","family":"Mooney","affiliation":""},{"name":"Stefanie Tellex","slug":"stefanie-tellex","given":"Stefanie","family":"Tellex","affiliation":"Brown University; The AI Institute"},{"name":"David Watkins","slug":"david-watkins","given":"David","family":"Watkins","affiliation":"The AI Institute"}],"abstract":"With large language models, robots can understand language more flexibly and more capable than ever before. This survey reviews and situates recent literature into a spectrum with two poles: 1) mapping between language and some manually defined formal representation of meaning, and 2) mapping between language and high-dimensional vector spaces that translate directly to low-level robot policy. Using a formal representation allows the meaning of the language to be precisely represented, limits the size of the learning problem, and leads to a framework for interpretability and formal safety guarantees. Methods that embed language and perceptual data into high-dimensional spaces avoid this manually specified symbolic structure and thus have the potential to be more general when fed enough data but require more data and computing to train. We discuss the benefits and tradeoffs of each approach and finish by providing directions for future work that achieves the best of both worlds.","doi":"10.24963/ijcai.2024/885","url":"https://ijcai.org/proceedings/2024/885","pdf":"https://ijcai.org/proceedings/2024/0885.pdf","markdown":"https://ijcai.org/proceedings/2024/885.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/885","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Survey Track","track":"Survey Track","number":885,"pages":{"first":7999,"last":8009},"keywords":["Machine Learning: ML: Neuro-symbolic methods","Robotics: ROB: Learning in robotics","Natural Language Processing: NLP: Language grounding","Machine Learning: ML: Symbolic methods"],"videos":[],"references":[{"position":1,"raw":"[Ahn et al., 2022] Michael Ahn, Anthony Brohan, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022.","title":"Do as i can, not as i say: Grounding language in robotic affordances","authors":"Anthony Michael Ahn, Brohan","year":"2022","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anjomshoae et al., 2019] Sule Anjomshoae, Amro Najjar, Da- vide Calvaresi, and Kary Främling. Explainable agents and robots: Results from a systematic literature review. In 18th International Conference on Autonomous Agents and Multia- gent Systems, pages 1078-1088. International Foundation for Au- tonomous Agents and Multiagent Systems, 2019.","title":"Explainable agents and robots: Results from a systematic literature review","authors":"Sule Anjomshoae, Amro Najjar, Davide Calvaresi, Kary Främling","year":"2019","venue":"18th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Katoen. Principles of model checking. MIT press, 2008.","title":"Principles of model checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baker et al., 2022] Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (VPT): Learn- ing to act by watching unlabeled online videos, 2022.","title":"Video pretraining (VPT): Learning to act by watching unlabeled online videos","authors":"Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Black et al., 2023] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image- editing diffusion models, 2023.","title":"Zero-shot robotic manipulation with pretrained imageediting diffusion models","authors":"Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, Sergey Levine","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brohan et al., 2023a] Anthony Brohan, Noah Brown, et al. RT-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems, 2023.","title":"RT-1: Robotics transformer for real-world control at scale","authors":"Brohan","year":"2023","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brohan et al., 2023b] Anthony Brohan, Noah Brown, et al. RT- 2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, 2023.","title":"RT-2: Vision-language-action models transfer web knowledge to robotic control","authors":"Brohan","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2023] Yongchao Chen, Rujul Gandhi, Yang Zhang, and Chuchu Fan. NL2TL: Transforming natural languages to temporal logics using large language models. arXiv:2305.07766, 2023.","title":"NL2TL: Transforming natural languages to temporal logics using large language models","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2024] Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. AutoTAMP: Autoregressive task and motion planning with llms as translators and checkers. In IEEE International Conference on Robotics and Automation, 2024.","title":"AutoTAMP: Autoregressive task and motion planning with llms as translators and checkers","authors":"Chen","year":"2024","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chinchali et al., 2012] Sandeep Chinchali, Scott C Livingston, Ufuk Topcu, Joel W Burdick, and Richard M Murray. Towards formal synthesis of reactive controllers for dexterous robotic ma- nipulation. In IEEE International Conference on Robotics and Automation, 2012.","title":"Towards formal synthesis of reactive controllers for dexterous robotic manipulation","authors":"Sandeep Chinchali, Ufuk Scott C Livingston, Joel W Topcu, Richard M Burdick, Murray","year":"2012","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Collins et al., 2022] Katherine M. Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Joshua B. Tenenbaum. Structured, flex- ible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. In Proceedings of the 44th Annual Conference of the Cognitive Science Society, 2022.","title":"Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks","authors":"Katherine M Collins, Catherine Wong, Jiahai Feng, Megan Wei, Joshua B Tenenbaum","year":"2022","venue":"Proceedings of the 44th Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Das et al., 2021] Devleena Das, Siddhartha Banerjee, and Sonia Chernova. Explainable AI for robot failures: Generating explana- tions that improve user assistance in fault recovery. In Proceed- ings of the 2021 ACM/IEEE International Conference on Human- Robot Interaction, pages 351-360, 2021.","title":"Explainable AI for robot failures: Generating explanations that improve user assistance in fault recovery","authors":"Das","year":"2021","venue":"Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dawson et al., 2023] Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and con- trol. IEEE Transactions on Robotics, 39(3):1749-1767, 2023.","title":"Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control","authors":"Charles Dawson, Sicun Gao, Chuchu Fan","year":"2023","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Driess et al., 2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM- E: An embodied multimodal language model. International Con- ference on Machine Learning, 2023.","title":"PaLM-E: An embodied multimodal language model","authors":"Danny Driess, Fei Xia, S M Mehdi, Corey Sajjadi, Aakanksha Lynch, Brian Chowdhery, Ayzaan Ichter, Jonathan Wahid, Quan Tompson, Tianhe Vuong, Yu","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Du et al., 2023] Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning. arXiv:2310.10625, 2023.","title":"Video language planning","authors":"Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Edelkamp and Hoffmann, 2004] Stefan Edelkamp and Jörg Hoff- mann. PDDL2. 2: The language for the classical part of the 4th international planning competition. Technical report, Technical Report 195, University of Freiburg, 2004.","title":"PDDL2. 2: The language for the classical part of the 4th international planning competition","authors":"Hoffmann Edelkamp","year":"2004","venue":"Stefan Edelkamp and Jörg Hoffmann","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Fox and Long, 2003] Maria Fox and Derek Long. PDDL.1: An ex- tension to pddl for expressing temporal planning domains. Jour- nal of artificial intelligence research, 20:61-124, 2003.","title":"PDDL.1: An extension to pddl for expressing temporal planning domains","authors":"Maria Fox, Derek Long","year":"2003","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Fuggitti and Chakraborti, 2023] Francesco Fuggitti and Tathagata Chakraborti. NL2LTL -a python package for converting natural language (NL) instructions to linear temporal logic (LTL) formu- las. AAAI Conference on Artificial Intelligence, 2023. System Demonstration.","title":"Francesco Fuggitti and Tathagata Chakraborti. NL2LTL -a python package for converting natural language (NL) instructions to linear temporal logic (LTL) formulas","authors":"Chakraborti Fuggitti","year":"2023","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gopalan et al., 2018] Nakul Gopalan, Dilip Arumugam, Law- son LS Wong, and Stefanie Tellex. Sequence-to-sequence language grounding of non-markovian task specifications. In Robotics: Science and Systems, volume 2018, 2018.","title":"Sequence-to-sequence language grounding of non-markovian task specifications","authors":"Dilip Nakul Gopalan, Lawson Ls Arumugam, Stefanie Wong, Tellex","year":"2018","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Gopalan et al., 2020] Nakul Gopalan, Eric Rosen, George Konidaris, and Stefanie Tellex. Simultaneously learning transfer- able symbols and language groundings from perceptual data for instruction following. In Robotics: Science and Systems, 2020.","title":"Simultaneously learning transferable symbols and language groundings from perceptual data for instruction following","authors":"Nakul Gopalan, Eric Rosen, George Konidaris, Stefanie Tellex","year":"2020","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Guan et al., 2023] Lin Guan, Karthik Valmeekam, Sarath Sreed- haran, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model- based task planning, 2023.","title":"Leveraging pre-trained large language models to construct and utilize world models for modelbased task planning","authors":"Lin Guan, Karthik Valmeekam, Sarath Sreedharan, Subbarao Kambhampati","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Gunning and Aha, 2019] David Gunning and David Aha. Darpa's explainable artificial intelligence (XAI) program. AI magazine, 40(2):44-58, 2019.","title":"Darpa's explainable artificial intelligence (XAI) program","authors":"David Gunning, David Aha","year":"2019","venue":"AI magazine","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Harnad, 1990] Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1):335-346, 1990.","title":"The symbol grounding problem","authors":"Stevan Harnad","year":"1990","venue":"Physica D: Nonlinear Phenomena","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Harnad, 2007] Stevan Harnad. Symbol grounding problem. Schol- arpedia, 2(7):2373, 2007. revision #73220.","title":"Symbol grounding problem","authors":"Stevan Harnad","year":"2007","venue":"Scholarpedia","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Helmert, 2006] Malte Helmert. The fast downward planning sys- tem. JAIR, 26:191-246, 2006.","title":"The fast downward planning system","authors":"Malte Helmert, Helmert","year":"2006","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Henighan et al., 2020] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam Mc- Candlish. Scaling laws for autoregressive generative modeling. ArXiv, abs/2010.14701, 2020.","title":"Scaling laws for autoregressive generative modeling","authors":"Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M Ziegler, John Schulman, Dario Amodei, Sam Mc-Candlish","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Hoffmann et al., 2022] Jordan Hoffmann, Sebastian Borgeaud, et al. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Bel- grave, and Kyunghyun Cho, editors, Advances in Neural Infor- mation Processing Systems, 2022.","title":"An empirical analysis of compute-optimal large language model training","authors":"Hoffmann","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Hsiung et al., 2022] Eric Hsiung, Hiloni Mehta, Junchi Chu, Xinyu Liu, Roma Patel, Stefanie Tellex, and George Konidaris. Generalizing to new domains by mapping natural language to lifted ltl. In IEEE International Conference on Robotics and Au- tomation, 2022.","title":"Generalizing to new domains by mapping natural language to lifted ltl","authors":"Eric Hsiung, Hiloni Mehta, Junchi Chu, Xinyu Liu, Roma Patel, Stefanie Tellex, George Konidaris","year":"2022","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Hsu et al., 2023] Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum, and Jiajun Wu. What's left? concept grounding with logic- enhanced foundation models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.","title":"What's left? concept grounding with logicenhanced foundation models","authors":"Joy Hsu, Jiayuan Mao, Joshua B Tenenbaum, Jiajun Wu","year":"2023","venue":"Thirty-seventh Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Hu et al., 2023] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gi- anluca Corrado. GAIA-1: A generative world model for au- tonomous driving, 2023.","title":"GAIA-1: A generative world model for autonomous driving","authors":"Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, Gianluca Corrado","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Huang et al., 2022a] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot plan- ners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118- 9147. PMLR, 2022.","title":"Language models as zero-shot planners: Extracting actionable knowledge for embodied agents","authors":"Huang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Huang et al., 2022b] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv:2207.05608, 2022.","title":"Inner monologue: Embodied reasoning through planning with language models","authors":"Huang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Jiang et al., 2023] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, An- ima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General robot manipulation with multimodal prompts. In Fortieth Inter- national Conference on Machine Learning, 2023.","title":"VIMA: General robot manipulation with multimodal prompts","authors":"Jiang","year":"2023","venue":"Fortieth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Konidaris et al., 2018] George Konidaris, Leslie Pack Kaelbling, and Tomas Lozano-Perez. From skills to symbols: Learning sym- bolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215-289, 2018.","title":"From skills to symbols: Learning symbolic representations for abstract high-level planning","authors":"Konidaris","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Konidaris, 2020] GD Konidaris. Simultaneously learning transfer- able symbols and language groundings from perceptual data for instruction following. Robotics: Science and Systems, 2020.","title":"Simultaneously learning transferable symbols and language groundings from perceptual data for instruction following","authors":"; Gd Konidaris, Konidaris","year":"2020","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Kovacs, 2011] Daniel L Kovacs. BNF definition of PDDL 3.1. Un- published manuscript from the IPC-2011 website, 15, 2011.","title":"BNF definition of PDDL 3.1. Unpublished manuscript from the IPC-2011 website","authors":"L Daniel, Kovacs","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Kress-Gazit et al., 2018] Hadas Kress-Gazit, Morteza Lahijanian, and Vasumathi Raman. Synthesis for robots: Guarantees and feedback for robot behavior. Annual Review of Control, Robotics, and Autonomous Systems, 1:211-236, 2018.","title":"Synthesis for robots: Guarantees and feedback for robot behavior","authors":"Kress-Gazit","year":"2018","venue":"Annual Review of Control, Robotics, and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Li et al., 2022] Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mor- datch. Pre-trained language models for interactive decision- making. arXiv:2202.01771, 2022.","title":"Pre-trained language models for interactive decisionmaking","authors":"Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, Igor Mordatch","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Li et al., 2023a] Boyi Li, Philipp Wu, Pieter Abbeel, and Jiten- dra Malik. Interactive task planning with language models. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.","title":"Interactive task planning with language models","authors":"Li","year":"2023","venue":"2nd Workshop on Language and Robot Learning: Language as Grounding","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Li et al., 2023b] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, and Tao Kong. Vision-language foundation models as effective robot imitators. arXiv:2311.01378, 2023.","title":"Vision-language foundation models as effective robot imitators","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Liang et al., 2022] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied con- trol. In arXiv:2209.07753, 2022.","title":"Code as policies: Language model programs for embodied control","authors":"Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Lignos et al., 2015] Constantine Lignos, Vasumathi Raman, Cameron Finucane, Mitchell Marcus, and Hadas Kress-Gazit. Provably correct reactive control from natural language. Autonomous Robots, 38:89-105, 2015.","title":"Provably correct reactive control from natural language","authors":"Constantine Lignos, Vasumathi Raman, Cameron Finucane, Mitchell Marcus, Hadas Kress-Gazit","year":"2015","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Liu et al., 2023a] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: Empowering large language models with optimal planning profi- ciency. arXiv:2304.11477, 2023.","title":"LLM+P: Empowering large language models with optimal planning proficiency","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Liu et al., 2023b] Jason Xinyu Liu, Ziyi Yang, Ifrah Idrees, Sam Liang, Benjamin Schornstein, Stefanie Tellex, and Ankit Shah. Grounding complex natural language commands for temporal tasks in unseen environments. In Conference on Robot Learn- ing, 2023.","title":"Grounding complex natural language commands for temporal tasks in unseen environments","authors":"Liu","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Lucy, 1992] John A Lucy. Language diversity and thought: A re- formulation of the linguistic relativity hypothesis. Cambridge University Press, 1992.","title":"Language diversity and thought: A reformulation of the linguistic relativity hypothesis","authors":"Lucy John","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Maler and Nickovic, 2004] Oded Maler and Dejan Nickovic. Monitoring temporal properties of continuous signals. In Interna- tional Symposium on Formal Techniques in Real-Time and Fault- Tolerant Systems, pages 152-166, 2004.","title":"Monitoring temporal properties of continuous signals","authors":"Nickovic Maler, Oded Maler, Dejan Nickovic","year":"2004","venue":"International Symposium on Formal Techniques in Real-Time and Fault-Tolerant Systems","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[McDermott et al., 1998] Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. PDDL-the planning domain definition language. In Technical Report, Tech. Rep., 1998.","title":"PDDL-the planning domain definition language","authors":"Drew Mcdermott, Malik Ghallab, Adele E Howe, Craig A Knoblock, Ashwin Ram, Manuela M Veloso, Daniel S Weld, David E Wilkins","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Ng and Jordan, 2001] Andrew Ng and Michael Jordan. On dis- criminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information pro- cessing systems, 14, 2001.","title":"On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes","authors":"Andrew Ng, Michael Jordan","year":"2001","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Octo Model Team et al., 2023] Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. https: //octo-models.github.io, 2023.","title":"Octo Model Team","authors":"Octo Model, Team","year":"2023","venue":"Octo: An open-source generalist robot policy","doi":null,"url":"https://octo-models.github.io","internal_url":null},{"position":51,"raw":"[O'Neill et al., 2024] Abby O'Neill, Abdul Rehman, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation, 2024.","title":"Open X-Embodiment: Robotic learning datasets and RT-X models","authors":"O'neill","year":"2024","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Pan et al., 2023] Jiayi Pan, Glen Chou, and Dmitry Berenson. Data-efficient learning of natural language to linear temporal logic translators for robot task specification. In IEEE Interna- tional Conference on Robotics and Automation, 2023.","title":"Data-efficient learning of natural language to linear temporal logic translators for robot task specification","authors":"Jiayi Pan, Glen Chou, Dmitry Berenson","year":"2023","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Patel et al., 2020] Roma Patel, Ellie Pavlick, and Stefanie Tellex. Grounding language to non-markovian tasks with no supervision of task specifications. In Robotics: Science and Systems, 2020.","title":"Grounding language to non-markovian tasks with no supervision of task specifications","authors":"Patel","year":"2020","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Raman et al., 2013] Vasumathi Raman, Constantine Lignos, Cameron Finucane, Kenton CT Lee, Mitchell P Marcus, and Hadas Kress-Gazit. Sorry dave, i'm afraid i can't do that: Explaining unachievable robot tasks using natural language. Robotics: Science and Systems, 2(1):2-1, 2013.","title":"Sorry dave, i'm afraid i can't do that: Explaining unachievable robot tasks using natural language","authors":"Raman","year":"2013","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Raman et al., 2024] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompting. In IEEE International Conference on Robotics and Automation, 2024.","title":"Planning with large language models via corrective re-prompting","authors":"Vanya Shreyas Sundara Raman, Eric Cohen, Ifrah Rosen, David Idrees, Stefanie Paulius, Tellex","year":"2024","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Reed et al., 2022] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Ed- wards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. Featured Certification, Outstanding Certification.","title":"A generalist agent","authors":"Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barthmaron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, Nando De Freitas","year":"2022","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Reid et al., 2024] Machel Reid, Nikolay Savinov, et al. Gemini 1.5: Unlocking multimodal understanding across millions of to- kens of context. arXiv:2403.05530, 2024.","title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","authors":"Reid","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Rodriguez-Sanchez et al., 2023] Rafael Rodriguez-Sanchez, Ben- jamin Adin Spiegel, Jennifer Wang, Roma Patel, Stefanie Tellex, and George Konidaris. RLang: a declarative language for describing partial world knowledge to reinforcement learning agents. In International Conference on Machine Learning, pages 29161-29178. PMLR, 2023.","title":"RLang: a declarative language for describing partial world knowledge to reinforcement learning agents","authors":"Rodriguez-Sanchez","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Shridhar et al., 2023] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic ma- nipulation. In Conference on Robot Learning, 2023.","title":"Perceiver-actor: A multi-task transformer for robotic manipulation","authors":"Mohit Shridhar, Lucas Manuelli, Dieter Fox","year":"2023","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Silva et al., 2023] Andrew Silva, Mariah Schrum, Erin Hedlund- Botti, Nakul Gopalan, and Matthew Gombolay. Explainable ar- tificial intelligence: Evaluating the objective and subjective im- pacts of xai on human-agent interaction. International Journal of Human-Computer Interaction, 39(7):1390-1404, 2023.","title":"Explainable artificial intelligence: Evaluating the objective and subjective impacts of xai on human-agent interaction","authors":"Andrew Silva, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, Matthew Gombolay","year":"2023","venue":"International Journal of Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Silver et al., 2024] Tom Silver, Soham Dan, Kavitha Srinivas, Josh Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in PDDL domains with pretrained large language mod- els. In AAAI Conference on Artificial Intelligence, 2024.","title":"Generalized planning in PDDL domains with pretrained large language models","authors":"Tom Silver, Soham Dan, Kavitha Srinivas, Josh Tenenbaum, Leslie Kaelbling, Michael Katz","year":"2024","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Singh et al., 2023] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating situated robot task plans using large language models. In IEEE Interna- tional Conference on Robotics and Automation, 2023.","title":"ProgPrompt: Generating situated robot task plans using large language models","authors":"Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg","year":"2023","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Sutton, 2019] Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1), 2019.","title":"The bitter lesson. Incomplete Ideas","authors":"Richard Sutton","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Tellex et al., 2020] Stefanie Tellex, Nakul Gopalan, Hadas Kress- Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.","title":"Robots that use language","authors":"Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, Cynthia Matuszek","year":"2020","venue":"Annual Review of Control, Robotics, and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Valmeekam et al., 2023a] Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large lan- guage models on planning and reasoning about change, 2023. [Valmeekam et al., 2023b] Karthik Valmeekam, Matthew Mar- quez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models : A critical investiga- tion, 2023.","title":"PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change","authors":"Valmeekam","year":"2023","venue":"On the planning abilities of large language models : A critical investigation","doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Varley et al., 2024] Jake Varley, Sumeet Singh, Deepali Jain, Krzysztof Choromanski, Andy Zeng, Somnath Basu Roy Chowd- hury, Avinava Dubey, and Vikas Sindhwani. Embodied ai with two arms: Zero-shot learning, safety and modularity. arXiv:2404.03570, 2024.","title":"Embodied ai with two arms: Zero-shot learning, safety and modularity","authors":"Jake Varley, Sumeet Singh, Deepali Jain, Krzysztof Choromanski, Andy Zeng, Somnath Basu, Roy Chowdhury, Avinava Dubey, Vikas Sindhwani","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Wahlström et al., 2015] Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv:1502.02251, 2015.","title":"From pixels to torques: Policy learning with deep dynamical models","authors":"Wahlström","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Wang et al., 2021] Christopher Wang, Candace Ross, Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Learning a natural-language to ltl executable semantic parser for grounded robotics. In Con- ference on Robot Learning, 2021.","title":"Learning a natural-language to ltl executable semantic parser for grounded robotics","authors":"Christopher Wang, Candace Ross, Yen-Ling Kuo, Boris Katz, Andrei Barbu","year":"2021","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Wang et al., 2023] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv:2305.16291, 2023.","title":"Voyager: An open-ended embodied agent with large language models","authors":"Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, Anima Anandkumar","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Wang et al., 2024] Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, et al. Large language models for robotics: Opportunities, challenges, and perspectives. arXiv:2401.04334, 2024.","title":"Large language models for robotics: Opportunities, challenges, and perspectives","authors":"Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Wei et al., 2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raf- fel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.","title":"Emergent abilities of large language models","authors":"Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Whorf, 1956] Benjamin Lee Whorf. Language, thought, and real- ity: Selected writings of Benjamin Lee Whorf. MIT press, 1956.","title":"Language, thought, and reality: Selected writings of Benjamin Lee Whorf","authors":"Benjamin Lee, Whorf","year":"1956","venue":null,"doi":null,"url":null,"internal_url":null},{"position":73,"raw":"[Xie et al., 2023] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to plan- ning goals with large-language models. arXiv:2302.05128, 2023.","title":"Translating natural language to planning goals with large-language models","authors":"Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, Harold Soh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":74,"raw":"[Yan et al., 2023] Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transformers for video gen- eration, 2023.","title":"Temporally consistent transformers for video generation","authors":"Wilson Yan, Danijar Hafner, Stephen James, Pieter Abbeel","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":75,"raw":"[Yang et al., 2023] Mengjiao Yang, Yilun Du, et al. Learning inter- active real-world simulators. arXiv:2310.06114, 2023.","title":"Learning interactive real-world simulators","authors":"Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":76,"raw":"[Yang et al., 2024] Ziyi Yang, Shreyas S Raman, Ankit Shah, and Stefanie Tellex. Plug in the safety chip: Enforcing constraints for LLM-driven robot agents. In IEEE International Conference on Robotics and Automation, 2024.","title":"Plug in the safety chip: Enforcing constraints for LLM-driven robot agents","authors":"Ziyi Yang, Ankit Shreyas S Raman, Stefanie Shah, Tellex","year":"2024","venue":"IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":77,"raw":"[Zeng et al., 2023a] Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sind- hwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. So- cratic models: Composing zero-shot multimodal reasoning with language. In The Eleventh International Conference on Learning Representations, 2023.","title":"Socratic models: Composing zero-shot multimodal reasoning with language","authors":"Zeng","year":"2023","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":78,"raw":"[Zeng et al., 2023b] Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S Yu. Large language models for robotics: A survey. arXiv:2311.07226, 2023.","title":"Large language models for robotics: A survey","authors":"Zeng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":79,"raw":"[Zhang et al., 2023] Ceng Zhang, Junxin Chen, Jiatong Li, Yan- hong Peng, and Zebing Mao. Large language models for human-robot interaction: A review. Biomimetic Intelligence and Robotics, page 100131, 2023.","title":"Large language models for human-robot interaction: A review","authors":"Ceng Zhang, Junxin Chen, Jiatong Li, Yanhong Peng, Zebing Mao","year":"2023","venue":"Biomimetic Intelligence and Robotics","doi":null,"url":null,"internal_url":null},{"position":80,"raw":"[Zhao et al., 2023] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv:2304.13705, 2023.","title":"Learning fine-grained bimanual manipulation with low-cost hardware","authors":"Tony Z Zhao, Vikash Kumar, Sergey Levine, Chelsea Finn","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}