{"type":"ScholarlyArticle","title":"Embodied Multimodal Multitask Learning","authors":[{"name":"Devendra Singh Chaplot","slug":"devendra-singh-chaplot","given":"Devendra Singh","family":"Chaplot","affiliation":"Carnegie Mellon University"},{"name":"Lisa Lee","slug":"lisa-lee","given":"Lisa","family":"Lee","affiliation":"Carnegie Mellon University"},{"name":"Ruslan Salakhutdinov","slug":"ruslan-salakhutdinov","given":"Ruslan","family":"Salakhutdinov","affiliation":"Carnegie Mellon University"},{"name":"Devi Parikh","slug":"devi-parikh","given":"Devi","family":"Parikh","affiliation":"Facebook AI Research; Georgia Institute of Technology"},{"name":"Dhruv Batra","slug":"dhruv-batra","given":"Dhruv","family":"Batra","affiliation":"Facebook AI Research; Georgia Institute of Technology"}],"abstract":"Visually-grounded embodied language learning models have recently shown to be effective at learning multiple multimodal tasks such as following navigational instructions and answering questions. In this paper, we address two key limitations of these models, (a) the inability to transfer the grounded knowledge across different tasks and (b) the inability to transfer to new words and concepts not seen during training using only a few examples. We propose a multitask model which facilitates knowledge transfer across tasks by disentangling the knowledge of words and visual attributes in the intermediate representations. We create scenarios and datasets to quantify cross-task knowledge transfer and show that the proposed model outperforms a range of baselines in simulated 3D environments. We also show that this disentanglement of representations makes our model modular and interpretable which allows for transfer to instructions containing new concepts.","doi":"10.24963/ijcai.2020/338","url":"https://ijcai.org/proceedings/2020/338","pdf":"https://ijcai.org/proceedings/2020/0338.pdf","markdown":"https://ijcai.org/proceedings/2020/338.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/338","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":338,"pages":{"first":2442,"last":2448},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25550/file_25550.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real en- vironments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683, 2018.","title":"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton Van Den, Hengel","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48, 2016.","title":"Neural module networks","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Artzi and Zettlemoyer, 2013] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping in- structions to actions. Transactions of the Association for Compu- tational Linguistics, 1:49-62, 2013.","title":"Weakly supervised learning of semantic parsers for mapping instructions to actions","authors":"Zettlemoyer Artzi, Yoav Artzi, Luke Zettlemoyer","year":"2013","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barsalou, 2008] Lawrence W Barsalou. Grounded cognition. Annu. Rev. Psychol., 59:617-645, 2008.","title":"Grounded cognition","authors":"Lawrence W Barsalou","year":"2008","venue":"Annu. Rev. Psychol","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blukis et al., 2018] Valts Blukis, Dipendra Misra, Ross A Knepper, and Yoav Artzi. Mapping navigation instructions to continuous control actions with position-visitation prediction. Proceedings of The 2nd Conference on Robot Learning, 2018.","title":"Mapping navigation instructions to continuous control actions with position-visitation prediction","authors":"Valts Blukis, Dipendra Misra, Ross A Knepper, Yoav Artzi","year":"2018","venue":"Proceedings of The 2nd Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaplot et al., 2018] Devendra Singh Chaplot, Kan- thashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In AAAI, 2018.","title":"Gated-attention architectures for task-oriented language grounding","authors":"Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, Ruslan Salakhutdinov","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and Mooney, 2011] David L Chen and Raymond J Mooney. Learning to interpret natural language navigation instructions from observations. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.","title":"Learning to interpret natural language navigation instructions from observations","authors":"L David, Raymond J Chen, Mooney","year":"2011","venue":"Twenty-Fifth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2019] Howard Chen, Alane Shur, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019.","title":"Touchdown: Natural language navigation and spatial reasoning in visual street environments","authors":"Howard Chen, Alane Shur, Dipendra Misra, Noah Snavely, Yoav Artzi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural ma- chine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Das et al., 2018] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In CVPR, 2018.","title":"Embodied question answering","authors":"Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[de Vries et al., 2018] Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv:1807.03367, 2018.","title":"Harm de Vries","authors":"De Vries","year":"2018","venue":"Talk the walk: Navigating new york city through grounded dialogue","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gordon et al., 2018] Daniel Gordon, Aniruddha Kembhavi, Mo- hammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4089-4098, 2018.","title":"Iqa: Visual question answering in interactive environments","authors":"Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, Ali Farhadi","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hemachandra et al., 2015] Sachithra Hemachandra, Felix Duval- let, Thomas M Howard, Nicholas Roy, Anthony Stentz, and Matthew R Walter. Learning models for following natural lan- guage directions in unknown environments. In 2015 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 5608-5615. IEEE, 2015.","title":"Learning models for following natural language directions in unknown environments","authors":"Hemachandra","year":"2015","venue":"2015 IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hermann et al., 2017] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepes- vari, Wojtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017.","title":"Grounded language learning in a simulated 3d world","authors":"Hermann","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kempka et al., 2016] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom- based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8. IEEE, 2016.","title":"Vizdoom: A doombased ai research platform for visual reinforcement learning","authors":"Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, Wojciech Jaśkowski","year":"2016","venue":"IEEE Conference on Computational Intelligence and Games (CIG)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1995] Yann LeCun, Yoshua Bengio, et al. Convolu- tional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.","title":"Convolutional networks for images, speech, and time series","authors":"Yann Lecun, Yoshua Bengio","year":"1995","venue":"The handbook of brain theory and neural networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[MacMahon et al., 2006] Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowl- edge, and action in route instructions. Def, 2(6):4, 2006.","title":"Walk the talk: Connecting language, knowledge, and action in route instructions","authors":"Matt Macmahon, Brian Stankiewicz, Benjamin Kuipers","year":"2006","venue":"Def","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Matuszek et al., 2012] Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. A joint model of language and perception for grounded attribute learning. arXiv preprint arXiv:1206.6423, 2012.","title":"A joint model of language and perception for grounded attribute learning","authors":"Cynthia Matuszek, Nicholas Fitzgerald, Luke Zettlemoyer, Liefeng Bo, Dieter Fox","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mei et al., 2016] Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Listen, attend, and walk: Neural mapping of navigational instructions to action sequences","authors":"Hongyuan Mei, Mohit Bansal, Matthew R Walter","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Misra et al., 2016] Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281-300, 2016.","title":"Tell me dave: Context-sensitive grounding of natural language to manipulation instructions","authors":"Jaeyong Dipendra K Misra, Kevin Sung, Ashutosh Lee, Saxena","year":"2016","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Misra et al., 2017] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with re- inforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1004-1015, 2017.","title":"Mapping instructions and visual observations to actions with reinforcement learning","authors":"Dipendra Misra, John Langford, Yoav Artzi","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Misra et al., 2018] Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3d environments with visual goal predic- tion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2667-2678, 2018.","title":"Mapping instructions to actions in 3d environments with visual goal prediction","authors":"Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, Yoav Artzi","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Oh et al., 2017] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In ICML, 2017.","title":"Zero-shot task generalization with multi-task deep reinforcement learning","authors":"Junhyuk Oh, Satinder Singh, Honglak Lee, Pushmeet Kohli","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Perez et al., 2018] Ethan Perez, Florian Strub, Harm De Vries, Vin- cent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.","title":"Film: Visual reasoning with a general conditioning layer","authors":"Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, Aaron Courville","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Smith and Gasser, 2005] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-29, 2005.","title":"The development of embodied cognition: Six lessons from babies","authors":"Gasser ; Smith, Linda Smith, Michael Gasser","year":"2005","venue":"Artificial life","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tellex et al., 2011] Stefanie A Tellex, Thomas Fleming Kollar, Steven R Dickerson, Matthew R Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language com- mands for robotic navigation and mobile manipulation. In AAAI, 2011.","title":"Understanding natural language commands for robotic navigation and mobile manipulation","authors":"Stefanie A Tellex, Thomas Fleming Kollar, Matthew R Steven R Dickerson, Ashis Walter, Seth Banerjee, Nicholas Teller, Roy","year":"2011","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2018] Haonan Yu, Xiaochen Lian, Haichao Zhang, and Wei Xu. Guided feature transformation (gft): A neural language grounding module for embodied agents. In Conference on Robot Learning, pages 81-98, 2018.","title":"Guided feature transformation (gft): A neural language grounding module for embodied agents","authors":"Yu","year":"2018","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}