{"type":"ScholarlyArticle","title":"Diagnosing the Environment Bias in Vision-and-Language Navigation","authors":[{"name":"Yubo Zhang","slug":"yubo-zhang","given":"Yubo","family":"Zhang","affiliation":"UNC Chapel Hill"},{"name":"Hao Tan","slug":"hao-tan","given":"Hao","family":"Tan","affiliation":"UNC Chapel Hill"},{"name":"Mohit Bansal","slug":"mohit-bansal","given":"Mohit","family":"Bansal","affiliation":"UNC Chapel Hill"}],"abstract":"Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are crucial when the agent is navigating new environments about which it has no prior knowledge. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations that contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gaps between seen and unseen on multiple datasets (i.e. R2R, R4R, and CVDN) and achieve competitive unseen results to previous state-of-the-art models.","doi":"10.24963/ijcai.2020/124","url":"https://ijcai.org/proceedings/2020/124","pdf":"https://ijcai.org/proceedings/2020/0124.pdf","markdown":"https://ijcai.org/proceedings/2020/124.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/124","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":124,"pages":{"first":890,"last":897},"keywords":["Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25261/file_25261.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018a] P. Anderson, X. He, et al. Bottom- up and top-down attention for image captioning and visual question answering. In CVPR, 2018.","title":"Bottomup and top-down attention for image captioning and visual question answering","authors":"Anderson","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018b] P. Anderson, Q. Wu, et al. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.","title":"Visionand-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Anderson","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anderson et al., 2019] P. Anderson, A. Shrivastava, et al. Chasing ghosts: Instruction following as bayesian state tracking. In NeurIPS, 2019.","title":"Chasing ghosts: Instruction following as bayesian state tracking","authors":"P Anderson, A Shrivastava","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bahdanau et al., 2015] D. Bahdanau, K. Cho, and Y. Ben- gio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"D Bahdanau, K Cho, Y Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blanchard et al., 2011] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS, 2011.","title":"Generalizing from several related classification tasks to a new unlabeled sample","authors":"G Blanchard, G Lee, C Scott","year":"2011","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blukis et al., 2018] V. Blukis, D. Misra, et al. Mapping navigation instructions to continuous control actions with position-visitation prediction. In CoRL, 2018.","title":"Mapping navigation instructions to continuous control actions with position-visitation prediction","authors":"V Blukis, D Misra","year":"2018","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Carlucci et al., 2019] F. M. Carlucci, A. D'Innocente, et al. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.","title":"Domain generalization by solving jigsaw puzzles","authors":"F M Carlucci, A D'innocente","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chang et al., 2017] A. Chang, A. Dai, et al. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, 2017.","title":"Matterport3d: Learning from rgb-d data in indoor environments","authors":"A Chang, A Dai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2019a] C. Chen, Z. Chen, et al. Joint domain alignment and discriminative feature learning for unsuper- vised deep domain adaptation. In AAAI, 2019.","title":"Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation","authors":"Chen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2019b] H. Chen, A. Suhr, et al. Touchdown: Natural language navigation and spatial reasoning in vi- sual street environments. In CVPR, 2019.","title":"Touchdown: Natural language navigation and spatial reasoning in visual street environments","authors":"Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Das et al., 2018] A. Das, S. Datta, et al. Embodied question answering. In CVPR, 2018.","title":"Embodied question answering","authors":"A Das, S Datta","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fried et al., 2018] D. Fried, R. Hu, et al. Speaker-follower models for vision-and-language navigation. In NeurIPS, 2018.","title":"Speaker-follower models for vision-and-language navigation","authors":"D Fried, R Hu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goodfellow et al., 2014] I.","title":null,"authors":"Goodfellow","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Goodfellow, J. Pouget-Abadie, et al. Generative adversarial nets. In NeurIPS, 2014.","title":"Generative adversarial nets","authors":"J Goodfellow, Pouget-Abadie","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2016] K. He, X. Zhang, et al. Deep residual learn- ing for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2019] R. Hu, D. Fried, et al. Are you looking? grounding to multiple modalities in vision-and-language navigation. In ACL, 2019.","title":"Are you looking? grounding to multiple modalities in vision-and-language navigation","authors":"R Hu, D Fried","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2019] H. Huang, V. Jain, et al. Transferable representation learning in vision-and-language navigation. In ICCV, 2019.","title":"Transferable representation learning in vision-and-language navigation","authors":"H Huang, V Jain","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jain et al., 2019] V. Jain, G. Magalhaes, et al. Stay on the path: Instruction fidelity in vision-and-language naviga- tion. In ACL, 2019.","title":"Stay on the path: Instruction fidelity in vision-and-language navigation","authors":"V Jain, G Magalhaes","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ke et al., 2019] L. Ke, X. Li, et al. Tactical rewind: Self- correction via backtracking in vision-and-language navi- gation. In CVPR, 2019.","title":"Tactical rewind: Selfcorrection via backtracking in vision-and-language navigation","authors":"L Ke, X Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krishna et al., 2017] R. Krishna, Y. Zhu, et al. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. IJCV, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"R Krishna, Y Zhu","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin, 2004] C. Lin. Rouge: A package for automatic evalu- ation of summaries. In Text summarization branches out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"C Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Long et al., 2018] M. Long, Z. Cao, et al. Conditional ad- versarial domain adaptation. In NeurIPS, 2018.","title":"Conditional adversarial domain adaptation","authors":"M Long, Z Cao","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ma et al., 2019a] C. Ma, J. Lu, et al. Self-monitoring nav- igation agent via auxiliary progress estimation. In ICLR, 2019.","title":"Self-monitoring navigation agent via auxiliary progress estimation","authors":"Ma","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ma et al., 2019b] C. Ma, Z. Wu, et al. The regretful agent: Heuristic-aided navigation through progress estimation. In CVPR, 2019.","title":"The regretful agent: Heuristic-aided navigation through progress estimation","authors":"Ma","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[MacMahon et al., 2006] M. MacMahon, B. Stankiewicz, and B. Kuipers. Walk the talk: connecting language, knowledge, and action in route instructions. In AAAI, 2006.","title":"Walk the talk: connecting language, knowledge, and action in route instructions","authors":"M Macmahon, B Stankiewicz, B Kuipers","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mirowski et al., 2018] P. Mirowski, M. Grimes, et al. Learning to navigate in cities without a map. In NeurIPS, 2018.","title":"Learning to navigate in cities without a map","authors":"P Mirowski, M Grimes","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Muandet et al., 2013] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013.","title":"Domain generalization via invariant feature representation","authors":"K Muandet, D Balduzzi, B Schölkopf","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Papineni et al., 2002] K. Papineni, S. Roukos, et al. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"K Papineni, S Roukos","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ren et al., 2015] S. Ren, K. He, et al. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"S Ren, K He","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rozantsev et al., 2018] A. Rozantsev, M. Salzmann, and P. Fua. Residual parameter transfer for deep domain adap- tation. In CVPR, 2018.","title":"Residual parameter transfer for deep domain adaptation","authors":"A Rozantsev, M Salzmann, P Fua","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Russakovsky et al., 2015] O. Russakovsky, J. Deng, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"O Russakovsky, J Deng","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tan et al., 2019] H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In NACCL, 2019.","title":"Learning to navigate unseen environments: Back translation with environmental dropout","authors":"H Tan, L Yu, M Bansal","year":"2019","venue":"NACCL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Thomason et al., 2019a] J. Thomason, D. Gordan, and Y. Bisk. Shifting the baseline: Single modality perfor- mance on visual navigation & qa. In NACCL, 2019.","title":"Shifting the baseline: Single modality performance on visual navigation & qa","authors":"Thomason","year":"2019","venue":"NACCL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Thomason et al., 2019b] J. Thomason, M. Murray, et al. Vision-and-dialog navigation. In CoRL, 2019.","title":"Vision-and-dialog navigation","authors":"Thomason","year":"2019","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2018] X. Wang, W. Xiong, et al. Look before you leap: Bridging model-free and model-based reinforce- ment learning for planned-ahead vision-and-language nav- igation. In ECCV, 2018.","title":"Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation","authors":"Wang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2019] X. Wang, Q. Huang, et al. Reinforced cross-modal matching and self-supervised imitation learn- ing for vision-language navigation. In CVPR, 2019.","title":"Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yu et al., 2019] L. Yu, X. Chen, et al. Multi-target embod- ied question answering. In CVPR, 2019.","title":"Multi-target embodied question answering","authors":"L Yu, X Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}