{"type":"ScholarlyArticle","title":"Core Challenges in Embodied Vision-Language Planning (Extended Abstract)","authors":[{"name":"Jonathan Francis","slug":"jonathan-francis","given":"Jonathan","family":"Francis","affiliation":"Bosch Center for Artificial Intelligence; School of Computer Science , Carnegie Mellon University"},{"name":"Nariaki Kitamura","slug":"nariaki-kitamura","given":"Nariaki","family":"Kitamura","affiliation":"Komatsu Limited"},{"name":"Felix Labelle","slug":"felix-labelle","given":"Felix","family":"Labelle","affiliation":"School of Computer Science , Carnegie Mellon University"},{"name":"Xiaopeng Lu","slug":"xiaopeng-lu","given":"Xiaopeng","family":"Lu","affiliation":"School of Computer Science , Carnegie Mellon University"},{"name":"Ingrid Navarro","slug":"ingrid-navarro","given":"Ingrid","family":"Navarro","affiliation":"School of Computer Science , Carnegie Mellon University"},{"name":"Jean Oh","slug":"jean-oh","given":"Jean","family":"Oh","affiliation":"School of Computer Science , Carnegie Mellon University"}],"abstract":"Recent advances in the areas of Multimodal Machine Learning and Artificial Intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Robotics. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly leverage computer vision and natural language for interaction in physical environments. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulators, and datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalisability and furthers real-world deployment.","doi":"10.24963/ijcai.2023/775","url":"https://ijcai.org/proceedings/2023/775","pdf":"https://ijcai.org/proceedings/2023/0775.pdf","markdown":"https://ijcai.org/proceedings/2023/775.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/775","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":775,"pages":{"first":6878,"last":6883},"keywords":["Robotics: ROB: Learning in robotics","Robotics: General","Planning and Scheduling: PS: Applications","Natural Language Processing: NLP: Applications"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3674-3683. IEEE Computer Society, 2018.","title":"Vision-and-language navigation: Interpreting visuallygrounded navigation instructions in real environments","authors":"Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D Reid, Stephen Gould, Anton Van Den, Hengel","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 39-48. IEEE Computer Society, 2016.","title":"Neural module networks","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":"2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Castelfranchi, 1998] Cristiano Castelfranchi. Modelling so- cial action for ai agents. Artificial intelligence, 103(1- 2):157-182, 1998.","title":"Modelling social action for ai agents","authors":"Cristiano Castelfranchi","year":"1998","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Das et al., 2018] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1-10. IEEE Computer Society, 2018.","title":"Embodied question answering","authors":"Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguis- tics.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Francis et al., 2019] Jonathan Francis, Matias Quintana, Nadine von Frankenberg, Sirajum Munir, and Mario Berges. Occutherm: Occupant thermal comfort inference using body shape information. In Proceedings of the 6th International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, BuildSys '19, New York, NY, USA, 2019. ACM.","title":"Occutherm: Occupant thermal comfort inference using body shape information","authors":"Jonathan Francis, Matias Quintana, Nadine Von Frankenberg, Sirajum Munir, Mario Berges","year":"2019","venue":"Proceedings of the 6th International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, BuildSys '19","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Francis et al., 2022a] Jonathan Francis, Bingqing Chen, Siddha Ganju, Sidharth Kathpal, Jyotish Poonganam, Ayush Shivani, Vrushank Vyas, Sahika Genc, Ivan Zhukov, Max Kumskoy, Jean Oh, Eric Nyberg, and Sylvia L. Herbert. Learn-to-race challenge 2022: Bench- marking safe learning and cross-domain generalisation in autonomous racing. In 1st ICML Workshop on Safe Learn- ing for Autonomous Driving, 2022.","title":"Learn-to-race challenge 2022: Benchmarking safe learning and cross-domain generalisation in autonomous racing","authors":"Francis","year":"2022","venue":"1st ICML Workshop on Safe Learning for Autonomous Driving","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Francis et al., 2022b] Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, and Jean Oh. Core challenges in embodied vision-language planning. Journal of Artificial Intelligence Research, 74:459-515, 2022.","title":"Core challenges in embodied vision-language planning","authors":"Francis","year":"2022","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Francis, 2022] Jonathan Francis. Knowledge-enhanced Representation Learning for Multiview Context Under- standing. PhD thesis, Carnegie Mellon University, 2022.","title":"Knowledge-enhanced Representation Learning for Multiview Context Understanding","authors":"Jonathan Francis","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gordon et al., 2018] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: visual question answering in interac- tive environments. In 2018 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 4089-4098. IEEE Computer Society, 2018.","title":"IQA: visual question answering in interactive environments","authors":"Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, Ali Farhadi","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hao et al., 2020] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre- training. In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13134-13143. IEEE, 2020.","title":"Towards learning a generic agent for vision-and-language navigation via pretraining","authors":"Hao","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Soci- ety, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Herman et al., 2021] James Herman, Jonathan Francis, Sid- dha Ganju, Bingqing Chen, Anirudh Koul, Abhinav Gupta, Alexey Skabelkin, Ivan Zhukov, Max Kumskoy, and Eric Nyberg. Learn-to-race: A multimodal control environment for autonomous racing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9793-9802, 2021.","title":"Learn-to-race: A multimodal control environment for autonomous racing","authors":"James Herman, Jonathan Francis, Siddha Ganju, Bingqing Chen, Anirudh Koul, Abhinav Gupta, Alexey Skabelkin, Ivan Zhukov, Max Kumskoy, Eric Nyberg","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krantz et al., 2020] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous en- vironments. CoRR, abs/2004.02857, 2020.","title":"Beyond the navgraph: Vision-and-language navigation in continuous environments","authors":"Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Dongxu Li, Cristian Rodriguez Opazo, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In IEEE Winter Conference on Ap- plications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pages 1448-1458. IEEE, 2020.","title":"Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison","authors":"Li","year":"2020","venue":"IEEE Winter Conference on Applications of Computer Vision, WACV 2020","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2021] Yikang Li, Pulkit Goel, Varsha Kuppur Rajendra, Har Simrat Singh, Jonathan Francis, Kaixin Ma, Eric Nyberg, and Alessandro Oltramari. Lexically- constrained text generation through commonsense knowl- edge extraction and injection. In Common Sense Knowl- edge Graphs at the 35th AAAI Conference on Artificial In- telligence (CSKGs@AAAI-21), 2021.","title":"Lexicallyconstrained text generation through commonsense knowledge extraction and injection","authors":"Yikang Li, Pulkit Goel, Varsha Kuppur Rajendra, Simrat Har, Jonathan Singh, Kaixin Francis, Eric Ma, Alessandro Nyberg, Oltramari","year":"2021","venue":"Common Sense Knowledge Graphs at the 35th AAAI Conference on Artificial Intelligence (CSKGs@AAAI-21)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Gar- nett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13-23, 2019.","title":"Roman Garnett","authors":"M Hanna, Hugo Wallach, Alina Larochelle, Beygelzimer, Emily B Florence D'alché-Buc, Fox","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2019] Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. Towards generalizable neuro-symbolic systems for commonsense question answering. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Process- ing, pages 22-32, Hong Kong, China, 2019. Association for Computational Linguistics.","title":"Towards generalizable neuro-symbolic systems for commonsense question answering","authors":"Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, Alessandro Oltramari","year":"2019","venue":"Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ma et al., 2021] Kaixin Ma, Filip Ilievski, Jonathan Fran- cis, Yonatan Bisk, Eric Nyberg, and Alessandro Oltramari. Knowledge-driven data construction for zero-shot evalu- ation in commonsense question answering. In Proceed- ings of the 35th AAAI Conference on Artificial Intelligence (AAAI-21), 2021.","title":"Knowledge-driven data construction for zero-shot evaluation in commonsense question answering","authors":"Kaixin Ma, Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, Alessandro Oltramari","year":"2021","venue":"Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI-21)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Majumdar et al., 2020] Arjun Majumdar, Ayush Shrivas- tava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. CoRR, abs/2004.14973, 2020.","title":"Improving vision-and-language navigation with image-text pairs from the web","authors":"Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen and Daumé III, 2019] Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity- encouraging imitation learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 684-695, Hong Kong, China, 2019. Association for Computational Linguistics.","title":"Help, anna! visual navigation with natural multimodal assistance via retrospective curiosityencouraging imitation learning","authors":"Daumé Iii ; Khanh Nguyen, Hal Nguyen, Iii Daumé","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Park et al., 2020] Seong Hyeon Park, Gyubok Lee, Manoj Bhat, Jimin Seo, Minseok Kang, Jonathan Francis, Ash- win R Jadhav, Paul Pu Liang, and Louis-Philippe Morency. Diverse and admissible trajectory forecasting through mul- timodal context understanding. In European Conference on Computer Vision (ECCV), 2020.","title":"Diverse and admissible trajectory forecasting through multimodal context understanding","authors":"Gyubok Seong Hyeon Park, Manoj Lee, Jimin Bhat, Minseok Seo, Jonathan Kang, Ashwin R Francis, Paul Pu Jadhav, Louis-Philippe Liang, Morency","year":"2020","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Talmor et al., 2019] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.","title":"CommonsenseQA: A question answering challenge targeting commonsense knowledge","authors":"Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tatiya et al., 2022] Gyan Tatiya, Jonathan Francis, Luca Bondi, Ingrid Navarro, Eric Nyberg, Jivko Sinapov, and Jean Oh. Knowledge-driven scene priors for seman- tic audio-visual embodied navigation. arXiv preprint arXiv:2212.11345, 2022.","title":"Knowledge-driven scene priors for semantic audio-visual embodied navigation","authors":"Gyan Tatiya, Jonathan Francis, Luca Bondi, Ingrid Navarro, Eric Nyberg, Jivko Sinapov, Jean Oh","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tatiya et al., 2023a] Gyan Tatiya, Jonathan Francis, and Jivko Sinapov. Cross-tool and cross-behavior perceptual knowledge transfer for grounded object recognition. arXiv preprint arXiv:2303.04023, 2023.","title":"Cross-tool and cross-behavior perceptual knowledge transfer for grounded object recognition","authors":"Tatiya","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tatiya et al., 2023b] Gyan Tatiya, Jonathan Francis, and Jivko Sinapov. Transferring implicit knowledge of non- visual object properties across heterogeneous robot mor- phologies. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2023.","title":"Transferring implicit knowledge of nonvisual object properties across heterogeneous robot morphologies","authors":"Tatiya","year":"2023","venue":"Proceedings of the International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thomason et al., 2019] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. CoRR, abs/1907.04957, 2019.","title":"Vision-and-dialog navigation","authors":"Jesse Thomason, Michael Murray, Maya Cakmak, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wooldridge and Jennings, 1995] Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and prac- tice. The knowledge engineering review, 10(2):115-152, 1995.","title":"Intelligent agents: Theory and practice","authors":"Michael Wooldridge, Nicholas R Jennings","year":"1995","venue":"The knowledge engineering review","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2019a] Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.","title":"Visual semantic navigation using scene priors","authors":"Yang","year":"2019","venue":"7th International Conference on Learning Representations, ICLR 2019","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2019b] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764, 2019.","title":"Xlnet: Generalized autoregressive pretraining for language understanding","authors":"Yang","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}