{"type":"ScholarlyArticle","title":"Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Prompts","authors":[{"name":"Haodong Hong","slug":"haodong-hong","given":"Haodong","family":"Hong","affiliation":"The University of Queensland; CSIRO Data61"},{"name":"Sen Wang","slug":"sen-wang","given":"Sen","family":"Wang","affiliation":"The University of Queensland"},{"name":"Zi Huang","slug":"zi-huang","given":"Zi","family":"Huang","affiliation":"The University of Queensland"},{"name":"Qi Wu","slug":"qi-wu","given":"Qi","family":"Wu","affiliation":"The University of Adelaide"},{"name":"Jiajun Liu","slug":"jiajun-liu","given":"Jiajun","family":"Liu","affiliation":"CSIRO Data61; The University of Queensland"}],"abstract":"Current Vision-and-Language Navigation (VLN) tasks mainly employ textual instructions to guide agents. However, being inherently abstract, the same textual instruction can be associated with different visual signals, causing severe ambiguity and limiting the transfer of prior knowledge in the vision domain from the user to the agent. To fill this gap, we propose Vision-and-Language Navigation with Multi-modal Prompts (VLN-MP), a novel task augmenting traditional VLN by integrating both natural language and images in instructions. VLN-MP not only maintains backward compatibility by effectively handling text-only prompts but also consistently shows advantages with different quantities and relevance of visual prompts. Possible forms of visual prompts include both exact and similar object images, providing adaptability and versatility in diverse navigation scenarios. To evaluate VLN-MP under a unified framework, we implement a new benchmark that offers: (1) a training-free pipeline to transform textual instructions into multi-modal forms with landmark images; (2) diverse datasets with multi-modal instructions for different downstream tasks; (3) a novel module designed to process various image prompts for seamless integration with state-of-the-art VLN models. Extensive experiments on four VLN benchmarks (R2R, RxR, REVERIE, CVDN) show that incorporating visual prompts would significantly boost navigation performance. While maintaining efficiency with text-only prompts, VLN-MP enables agents to navigate in the pre-explore setting and outperform text-based models, showing its broader applicability. Code is available at https://github.com/honghd16/VLN-MP.","doi":"10.24963/ijcai.2024/93","url":"https://ijcai.org/proceedings/2024/93","pdf":"https://ijcai.org/proceedings/2024/0093.pdf","markdown":"https://ijcai.org/proceedings/2024/93.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/93","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":93,"pages":{"first":839,"last":847},"keywords":["Computer Vision: CV: Multimodal learning","Computer Vision: CV: Vision, language and reasoning","Machine Learning: ML: Multi-modal learning"],"videos":[],"references":[{"position":1,"raw":"[Alayrac et al., 2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35:23716-23736, 2022.","title":"Flamingo: a visual language model for few-shot learning","authors":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018a] Peter Anderson, Angel Chang, De- vendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evalu- ation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018.","title":"On evaluation of embodied navigation agents","authors":"Anderson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anderson et al., 2018b] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, pages 3674-3683, 2018.","title":"Visionand-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Anderson","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berg et al., 2020] Matthew Berg, Deniz Bayazit, Rebecca Mathew, Ariel Rotter-Aboyoun, Ellie Pavlick, and Ste- fanie Tellex. Grounding language to landmarks in arbitrary outdoor environments. In ICRA, pages 208-215, 2020.","title":"Grounding language to landmarks in arbitrary outdoor environments","authors":"Matthew Berg, Deniz Bayazit, Rebecca Mathew, Ariel Rotter-Aboyoun, Ellie Pavlick, Stefanie Tellex","year":"2020","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang et al., 2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Mat- terport3d: Learning from rgb-d data in indoor environ- ments. In 3DV, pages 667-676, 2017.","title":"Matterport3d: Learning from rgb-d data in indoor environments","authors":"Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, Yinda Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. History aware mul- timodal transformer for vision-and-language navigation. NeurIPS, 34:5834-5847, 2021.","title":"History aware multimodal transformer for vision-and-language navigation","authors":"Chen","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2022] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. In CVPR, pages 16537-16547, 2022.","title":"Think global, act local: Dual-scale graph transformer for vision-and-language navigation","authors":"Chen","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cui et al., 2023] Yibo Cui, Liang Xie, Yakun Zhang, Meis- han Zhang, Ye Yan, and Erwei Yin. Grounded entity- landmark adaptive pre-training for vision-and-language navigation. In ICCV, pages 12043-12053, 2023.","title":"Grounded entitylandmark adaptive pre-training for vision-and-language navigation","authors":"Yibo Cui, Liang Xie, Yakun Zhang, Meishan Zhang, Ye Yan, Erwei Yin","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2022] Geonmo Gu, Byungsoo Ko, SeoungHyun Go, Sung-Hyun Lee, Jingeun Lee, and Minchul Shin. To- wards light-weight and real-time line segment detection. In AAAI, volume 36, pages 726-734, 2022.","title":"Towards light-weight and real-time line segment detection","authors":"Gu","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2021] Keji He, Yan Huang, Qi Wu, Jianhua Yang, Dong An, Shuanglin Sima, and Liang Wang. Landmark- rxr: Solving vision-and-language navigation with fine- grained alignment supervision. NeurIPS, 34:652-663, 2021.","title":"Landmarkrxr: Solving vision-and-language navigation with finegrained alignment supervision","authors":"Keji He, Yan Huang, Qi Wu, Jianhua Yang, Dong An, Shuanglin Sima, Liang Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hong et al., 2020] Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. Language and visual entity relationship graph for agent navigation. NeurIPS, 33:7685-7696, 2020.","title":"Language and visual entity relationship graph for agent navigation","authors":"Hong","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2022] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, pages 9118-9147, 2022.","title":"Language models as zero-shot planners: Extracting actionable knowledge for embodied agents","authors":"Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ilharco et al., 2019] Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General evaluation for instruction conditioned navigation using dynamic time warping. arXiv preprint arXiv:1907.05446, 2019.","title":"General evaluation for instruction conditioned navigation using dynamic time warping","authors":"Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, Jason Baldridge","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jain et al., 2019] Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-language navigation. In ACL, pages 1862-1872, 2019.","title":"Stay on the path: Instruction fidelity in vision-and-language navigation","authors":"Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, Jason Baldridge","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jiang et al., 2022] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General robot manipulation with multimodal prompts. In NeurIPS 2022 Foundation Models for Deci- sion Making Workshop, 2022.","title":"VIMA: General robot manipulation with multimodal prompts","authors":"Jiang","year":"2022","venue":"NeurIPS 2022 Foundation Models for Decision Making Workshop","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Khattak et al., 2023] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pages 19113-19122, 2023.","title":"Maple: Multi-modal prompt learning","authors":"Khattak","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krantz et al., 2020] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous en- vironments. In ECCV, pages 104-120, 2020.","title":"Beyond the navgraph: Vision-and-language navigation in continuous environments","authors":"Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krantz et al., 2023] Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, and Jesse Thomason. Iterative vision-and-language navigation. In CVPR, pages 14921-14930, 2023.","title":"Iterative vision-and-language navigation","authors":"Jacob Krantz, Shurjo Banerjee, Wang Zhu, Jason Corso, Peter Anderson, Stefan Lee, Jesse Thomason","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ku et al., 2020] Alexander Ku, Peter Anderson, Roma Pa- tel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In EMNLP, pages 4392-4412, 2020.","title":"Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding","authors":"Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, Jason Baldridge","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2022a] Jialu Li, Hao Tan, and Mohit Bansal. Clear: Improving vision-language navigation with cross-lingual, environment-agnostic representations. In NAACL, pages 633-649, 2022.","title":"Clear: Improving vision-language navigation with cross-lingual, environment-agnostic representations","authors":"Li","year":"2022","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2022b] Jialu Li, Hao Tan, and Mohit Bansal. Envedit: Environment editing for vision-and-language navigation. In CVPR, pages 15407-15417, 2022.","title":"Envedit: Environment editing for vision-and-language navigation","authors":"Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2022c] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, pages 10965-10975, 2022.","title":"Grounded language-image pre-training","authors":"Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In ICML, pages 19730-19742, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2021] Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, and Yi-Dong Shen. Vision- language navigation with random environmental mixup. In ICCV, pages 1644-1654, 2021.","title":"Visionlanguage navigation with random environmental mixup","authors":"Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, Yi-Dong Shen","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2023a] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Mar- rying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.","title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection","authors":"Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Liu et al., 2023b] Shubo Liu, Hongsheng Zhang, Yuankai Qi, Peng Wang, Yanning Zhang, and Qi Wu. Aerialvln: Vision-and-language navigation for uavs. In ICCV, pages 15384-15394, 2023.","title":"Aerialvln: Vision-and-language navigation for uavs","authors":"Liu","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Majumdar et al., 2020] Arjun Majumdar, Ayush Shrivas- tava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving vision-and-language navigation with image-text pairs from the web. In ECCV, pages 259-274, 2020. [","title":"Improving vision-and-language navigation with image-text pairs from the web","authors":"Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[OpenAI, 2023] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.","title":null,"authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Qi et al., 2020] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual re- ferring expression in real indoor environments. In CVPR, pages 9982-9991, 2020.","title":"Reverie: Remote embodied visual referring expression in real indoor environments","authors":"Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton Van Den, Hengel","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Qiao et al., 2023a] Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History- enhanced and order-aware pre-training for vision-and- language navigation. TPAMI, 2023.","title":"Hop+: Historyenhanced and order-aware pre-training for vision-andlanguage navigation","authors":"Qiao","year":"2023","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Qiao et al., 2023b] Yanyuan Qiao, Zheng Yu, and Qi Wu. Vln-petl: Parameter-efficient transfer learning for vision- and-language navigation. In ICCV, pages 15443-15452, 2023.","title":"Vln-petl: Parameter-efficient transfer learning for visionand-language navigation","authors":"Qiao","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shen et al., 2022] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision- and-language tasks? ICLR, 2022.","title":"How much can clip benefit visionand-language tasks? ICLR","authors":"Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, Kurt Keutzer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Thomason et al., 2020] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In CoRL, pages 394-406, 2020.","title":"Vision-and-dialog navigation","authors":"Thomason","year":"2020","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard","authors":"Touvron","year":"2023","venue":"Llama: Open and efficient foundation language models","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2022] Su Wang, Ceslee Montgomery, Jordi Or- bay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, and Pe- ter Anderson. Less is more: Generating grounded naviga- tion instructions from landmarks. In CVPR, pages 15428- 15438, 2022.","title":"Less is more: Generating grounded navigation instructions from landmarks","authors":"Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, Peter Anderson","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wang et al., 2023a] Liuyi Wang, Zongtao He, Jiagui Tang, Ronghao Dang, Naijia Wang, Chengju Liu, and Qijun Chen. A dual semantic-aware recurrent global-adaptive network for vision-and-language navigation. In IJCAI, pages 1479-1487, 2023.","title":"A dual semantic-aware recurrent global-adaptive network for vision-and-language navigation","authors":"Wang","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/164"},{"position":42,"raw":"[Wang et al., 2023b] Zun Wang, Jialu Li, Yicong Hong, Yi Wang, Qi Wu, Mohit Bansal, Stephen Gould, Hao Tan, and Yu Qiao. Scaling data generation in vision-and- language navigation. In ICCV, pages 12009-12020, 2023.","title":"Scaling data generation in vision-andlanguage navigation","authors":"Wang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yesiltepe et al., 2021] Demet Yesiltepe, Ruth Conroy Dal- ton, and Ayse Ozbil Torun. Landmarks in wayfinding: a review of the existing literature. Cognitive processing, 22:369-410, 2021.","title":"Landmarks in wayfinding: a review of the existing literature","authors":"Demet Yesiltepe, Ruth Conroy Dalton, Ayse Ozbil Torun","year":"2021","venue":"Cognitive processing","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2021] Yubo Zhang, Hao Tan, and Mohit Bansal. Diagnosing the environment bias in vision-and- language navigation. In IJCAI, pages 890-897, 2021.","title":"Diagnosing the environment bias in vision-andlanguage navigation","authors":"Yubo Zhang, Hao Tan, Mohit Bansal","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhang et al., 2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. In ICCV, pages 3836-3847, 2023.","title":"Adding conditional control to text-to-image diffusion models","authors":"Zhang","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhou et al., 2024] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navi- gation with large language models. In AAAI, pages 7641- 7649, 2024.","title":"Navgpt: Explicit reasoning in vision-and-language navigation with large language models","authors":"Zhou","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhu et al., 2020] Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In CVPR, pages 10012-10022, 2020.","title":"Vision-language navigation with selfsupervised auxiliary reasoning tasks","authors":"Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhu et al., 2021] Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, and Xiaodan Liang. Soon: Scenario oriented object navigation with graph-based exploration. In CVPR, pages 12689-12699, 2021.","title":"Soon: Scenario oriented object navigation with graph-based exploration","authors":"Fengda Zhu, Xiwen Liang, Yi Zhu, Qizhi Yu, Xiaojun Chang, Xiaodan Liang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhu et al., 2023] Fengda Zhu, Vincent CS Lee, Xiaojun Chang, and Xiaodan Liang. Vision language navigation with knowledge-driven environmental dreamer. In IJCAI, pages 1840-1848, 2023.","title":"Vision language navigation with knowledge-driven environmental dreamer","authors":"Fengda Zhu, C S Vincent, Xiaojun Lee, Xiaodan Chang, Liang","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/204"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}