{"type":"ScholarlyArticle","title":"Dynamic Language Binding in Relational Visual Reasoning","authors":[{"name":"Thao Minh Le","slug":"thao-minh-le","given":"Thao","family":"Minh Le","affiliation":"Applied Artificial Intelligence Institute , Deakin University , Australia"},{"name":"Vuong Le","slug":"vuong-le","given":"Vuong","family":"Le","affiliation":"Applied Artificial Intelligence Institute , Deakin University , Australia"},{"name":"Svetha Venkatesh","slug":"svetha-venkatesh","given":"Svetha","family":"Venkatesh","affiliation":"Applied Artificial Intelligence Institute , Deakin University , Australia"},{"name":"Truyen Tran","slug":"truyen-tran","given":"Truyen","family":"Tran","affiliation":"Applied Artificial Intelligence Institute , Deakin University , Australia"}],"abstract":"We present Language-binding Object Graph Network, the first neural reasoning method with dynamic relational structures across both visual and textual domains with applications in visual question answering. Relaxing the common assumption made by current models that the object predicates pre-exist and stay static, passive to the reasoning process, we propose that these dynamic predicates expand across the domain borders to include pair-wise visual-linguistic object binding. In our method, these contextualized object links are actively found within each recurrent reasoning step without relying on external predicative priors. These dynamic structures reflect the conditional dual-domain object dependency given the evolving context of the reasoning through co-attention. Such discovered dynamic graphs facilitate multi-step knowledge combination and refinements that iteratively deduce the compact representation of the final answer. The effectiveness of this model is demonstrated on image question answering demonstrating favorable performance on major VQA datasets. Our method outperforms other methods in sophisticated question-answering tasks wherein multiple object relations are involved. The graph structure effectively assists the progress of training, and therefore the network learns efficiently compared to other reasoning models.","doi":"10.24963/ijcai.2020/114","url":"https://ijcai.org/proceedings/2020/114","pdf":"https://ijcai.org/proceedings/2020/0114.pdf","markdown":"https://ijcai.org/proceedings/2020/114.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/114","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":114,"pages":{"first":818,"last":824},"keywords":["Computer Vision: Language and Vision","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24223/file_24223.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, X. He, C. Buehler, D. Teney, M. Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"X Peter Anderson, C He, D Buehler, M Teney, Stephen Johnson, Lei Gould, Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baradel et al., 2018] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, pages 105-121, 2018.","title":"Object level visual reasoning in videos","authors":"Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, Greg Mori","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cadene et al., 2019] Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome. Murel: Multimodal re- lational reasoning for visual question answering. In CVPR, pages 1989-1998, 2019.","title":"Murel: Multimodal relational reasoning for visual question answering","authors":"Cadene","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Desta et al., 2018] Mikyas Desta, L. Chen, and T. Kornuta. Object-based reasoning in VQA. In WACV, pages 1814- 1823, 2018.","title":"Object-based reasoning in VQA","authors":"L Mikyas Desta, T Chen, Kornuta","year":"2018","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goyal et al., 2017] Yash Goyal, T. Khot, D. Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR, pages 6904-6913, 2017.","title":"Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering","authors":"Yash Goyal, T Khot, D Summers-Stay, Dhruv Batra, Devi Parikh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2017] Ronghang Hu, J. Andreas, M. Rohrbach, T. Darrell, and Kate Saenko. Learning to reason: End-to- end module networks for visual question answering. In ICCV, pages 804-813, 2017.","title":"Learning to reason: End-toend module networks for visual question answering","authors":"Ronghang Hu, J Andreas, M Rohrbach, T Darrell, Kate Saenko","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2019] Ronghang Hu, Anna Rohrbach, Trevor Dar- rell, and Kate Saenko. Language-conditioned graph net- works for relational reasoning. In ICCV, pages 10294- 10303, 2019.","title":"Language-conditioned graph networks for relational reasoning","authors":"Hu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2019] Pingping Huang, Jianhui Huang, Yuqing Guo, Min Qiao, and Yong Zhu. Multi-grained attention with object-level grounding for visual question answering. In ACL, pages 3595-3600, 2019.","title":"Multi-grained attention with object-level grounding for visual question answering","authors":"Pingping Huang, Jianhui Huang, Yuqing Guo, Min Qiao, Yong Zhu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hudson and Manning, 2018] Drew A Hudson and Christo- pher D Manning. Compositional attention networks for machine reasoning. ICLR, 2018.","title":"Compositional attention networks for machine reasoning","authors":"A Drew, Christopher D Hudson, Manning","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hudson and Manning, 2019a] Drew Hudson and Christo- pher D Manning. Learning by abstraction: The neural state machine. In NeurIPS, pages 5901-5914, 2019.","title":"Learning by abstraction: The neural state machine","authors":"Manning Hudson, Drew Hudson, Christopher D Manning","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hudson and Manning, 2019b] Drew A Hudson and Christo- pher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional qa. In CVPR, pages 6700- 6709, 2019.","title":"Gqa: A new dataset for real-world visual reasoning and compositional qa","authors":"Manning ; Hudson, A Drew, Christopher D Hudson, Manning","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2017a] Justin Johnson, B. Hariharan, L. van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901-2910, 2017.","title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Johnson","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johnson et al., 2017b] Justin Johnson, Bharath Hariharan, L. van der Maaten, Judy Hoffman, Li Fei-Fei, C.L. Zitnick, and Ross B Girshick. Inferring and executing programs for visual reasoning. In ICCV, pages 2989-2998, 2017.","title":"Inferring and executing programs for visual reasoning","authors":"Johnson","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2018] Seung Wook Kim, Makarand Tapaswi, and Sanja Fidler. Visual reasoning by progressive module net- works. ICLR, 2018.","title":"Visual reasoning by progressive module networks","authors":"Kim","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kipf and Welling, 2017] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lake et al., 2017] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. BBS, 2017.","title":"Building machines that learn and think like people","authors":"Brenden M Lake, Joshua B Tomer D Ullman, Samuel J Tenenbaum, Gershman","year":"2017","venue":"BBS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2019] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. ICCV, pages 10313-10322, 2019.","title":"Relation-aware graph attention network for visual question answering","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nagaraja et al., 2016] Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. In ECCV, pages 792-807, 2016.","title":"Modeling context between objects for referring expression understanding","authors":"Vlad I Varun K Nagaraja, Larry S Morariu, Davis","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Norcliffe-Brown et al., 2018] Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. In NeurIPS, pages 8334-8343, 2018.","title":"Learning conditioned graph structures for interpretable visual question answering","authors":"Norcliffe-Brown","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Palm et al., 2018] Rasmus Palm, U. Paquet, and O. Winther. Recurrent relational networks. In NeurIPS, 2018.","title":"Recurrent relational networks","authors":"U Rasmus Palm, O Paquet, Winther","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Perez et al., 2018] Ethan Perez, F. Strub, H. De Vries, V. Du- moulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.","title":"Film: Visual reasoning with a general conditioning layer","authors":"Ethan Perez, F Strub, H De Vries, V Dumoulin, A Courville","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ren et al., 2015] Shaoqing Ren, K. He, R. Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS, pages 91-99, 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"K Shaoqing Ren, R He, Jian Girshick, Sun","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Santoro et al., 2017] Adam Santoro, David Raposo, D.G. Barrett, M. Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In NIPS, pages 4967-4976, 2017.","title":"A simple neural network module for relational reasoning","authors":"Adam Santoro, David Raposo, D G Barrett, M Malinowski, Razvan Pascanu, Peter Battaglia, Tim Lillicrap","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shi et al., 2019] Jiaxin Shi, Hanwang Zhang, and Juanzi Li. Explainable and explicit visual reasoning over scene graphs. In CVPR, pages 8376-8384, 2019.","title":"Explainable and explicit visual reasoning over scene graphs","authors":"Jiaxin Shi, Hanwang Zhang, Juanzi Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shrestha et al., 2019] Robik Shrestha, Kushal Kafle, and Christopher Kanan. Answer them all! toward universal visual question answering models. In CVPR, pages 10472- 10481, 2019.","title":"Answer them all! toward universal visual question answering models","authors":"Shrestha","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Veličković et al., 2018] Petar Veličković, G. Cucurull, A. Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Veličković, G Cucurull, A Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2017] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 5410-5419, 2017.","title":"Scene graph generation by iterative message passing","authors":"Danfei Xu, Yuke Zhu, Christopher B Choy, Li Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2020] Keyulu Xu, Jingling Li, Mozhi Zhang, Si- mon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. What can neural networks reason about? ICLR, 2020.","title":"What can neural networks reason about? ICLR","authors":"Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S Du, Ken-Ichi Kawarabayashi, Stefanie Jegelka","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2019] Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression compre- hension. In ICCV, pages 4644-4653, 2019.","title":"Dynamic graph attention for referring expression comprehension","authors":"Sibei Yang, Guanbin Li, Yizhou Yu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2017] Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speaker-listener-reinforcer model for referring expressions. CVPR, pages 7282-7290, 2017.","title":"A joint speaker-listener-reinforcer model for referring expressions","authors":"Licheng Yu, Hao Tan, Mohit Bansal, Tamara L Berg","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}