{"type":"ScholarlyArticle","title":"Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation","authors":[{"name":"Tao He","slug":"tao-he","given":"Tao","family":"He","affiliation":"Faculty of Information Technology , Monash University , Australia; Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , China"},{"name":"Lianli Gao","slug":"lianli-gao","given":"Lianli","family":"Gao","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , China"},{"name":"Jingkuan Song","slug":"jingkuan-song","given":"Jingkuan","family":"Song","affiliation":"Center for Future Media and School of Computer Science and Engineering , University of Electronic Science and Technology of China , China"},{"name":"Jianfei Cai","slug":"jianfei-cai","given":"Jianfei","family":"Cai","affiliation":"Faculty of Information Technology , Monash University , Australia"},{"name":"Yuan-Fang Li","slug":"yuan-fang-li","given":"Yuan-Fang","family":"Li","affiliation":"Faculty of Information Technology , Monash University , Australia"}],"abstract":"Despite the huge progress in scene graph generation in recent years, its long-tail distribution in object relationships remains a challenging and pestering issue. Existing methods largely rely on either external knowledge or statistical bias information to alleviate this problem. In this paper, we tackle this issue from another two aspects: (1) scene-object interaction aiming at learning specific knowledge from a scene via an additive attention mechanism; and (2) long-tail knowledge transfer which tries to transfer the rich knowledge learned from the head into the tail. Extensive experiments on the benchmark dataset Visual Genome on three tasks demonstrate that our method outperforms current state-of-the-art competitors. Our source code is available at https://github.com/htlsn/issg.","doi":"10.24963/ijcai.2020/82","url":"https://ijcai.org/proceedings/2020/82","pdf":"https://ijcai.org/proceedings/2020/0082.pdf","markdown":"https://ijcai.org/proceedings/2020/82.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/82","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":82,"pages":{"first":587,"last":593},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24220/file_24220.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Britz et al., 2017] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. Massive exploration of neu- ral machine translation architectures. arXiv preprint arXiv:1703.03906, 2017.","title":"Massive exploration of neural machine translation architectures","authors":"Denny Britz, Anna Goldie, Minh-Thang Luong, Quoc Le","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing net- work for scene graph generation. In CVPR, pages 6163- 6171, 2019.","title":"Knowledge-embedded routing network for scene graph generation","authors":"Chen","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Bo Dai, Yuqi Zhang, and Dahua Lin. De- tecting visual relationships with deep relational networks. In CVPR, pages 3076-3086, 2017.","title":"Detecting visual relationships with deep relational networks","authors":"Bo Dai, Yuqi Zhang, Dahua Lin","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2014] Jia Deng, Nan Ding, Yangqing Jia, and Andrea Frome. Large-scale object classification using la- bel relation graphs. In ECCV, pages 48-64. Springer, 2014.","title":"Large-scale object classification using label relation graphs","authors":"Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dornadula et al., 2019] Apoorva Dornadula, Austin Nar- comey, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationships as functions: Enabling few-shot scene graph prediction. arXiv preprint arXiv:1906.04876, 2019.","title":"Visual relationships as functions: Enabling few-shot scene graph prediction","authors":"Apoorva Dornadula, Austin Narcomey, Ranjay Krishna, Michael Bernstein, Li Fei-Fei","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In CVPR, pages 1440-1448, 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2019] Jiuxiang Gu, Handong Zhao, Zhe Lin, and Sheng Li. Scene graph generation with external knowl- edge and image reconstruction. In CVPR, pages 1969- 1978, 2019.","title":"Scene graph generation with external knowledge and image reconstruction","authors":"Gu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In CVPR, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hsu et al., 2017] Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across do- mains and tasks. arXiv preprint arXiv:1711.10125, 2017.","title":"Learning to cluster in order to transfer across domains and tasks","authors":"Yen-Chang Hsu, Zhaoyang Lv, Zsolt Kira","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2015] Justin Johnson, Ranjay Krishna, and Michael Stark. Image retrieval using scene graphs. In CVPR, pages 3668-3678, 2015.","title":"Image retrieval using scene graphs","authors":"Justin Johnson, Ranjay Krishna, Michael Stark","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, and Joshua Kravitz. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32- 73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2016] Cewu Lu, Ranjay Krishna, Michael Bern- stein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, pages 852-869. Springer, 2016.","title":"Visual relationship detection with language priors","authors":"Lu","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2019] Mengshi Qi, Weijian Li, Zhengyuan Yang, and Yunhong Wang. Attentive relational networks for mapping images to scene graphs. In CVPR, pages 3957- 3966, 2019.","title":"Attentive relational networks for mapping images to scene graphs","authors":"Qi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Savinov et al., 2018] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, and Pollefeys. Episodic curiosity through reachability. ICLR, 2018.","title":"Episodic curiosity through reachability","authors":"Savinov","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shi et al., 2019] Jiaxin Shi, Hanwang Zhang, and Juanzi Li. Explainable and explicit visual reasoning over scene graphs. In CVPR, pages 8376-8384, 2019.","title":"Explainable and explicit visual reasoning over scene graphs","authors":"Jiaxin Shi, Hanwang Zhang, Juanzi Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Speer et al., 2017] Robert Speer, Chin, and Joshua. Con- ceptnet 5.5: An open multilingual graph of general knowl- edge. In AAAI, 2017.","title":"Conceptnet 5.5: An open multilingual graph of general knowledge","authors":"Robert Speer, Chin, Joshua","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2019] Wenbin Wang, Ruiping Wang, and Shiguang Shan. Exploring context and visual pattern of relationship for scene graph generation. In CVPR, pages 8188-8197, 2019.","title":"Exploring context and visual pattern of relationship for scene graph generation","authors":"Wenbin Wang, Ruiping Wang, Shiguang Shan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Woo et al., 2018] Sanghyun Woo, Dahun Kim, and Donghyeon Cho. Linknet: Relational embedding for scene graph. In NIPS, pages 560-570, 2018.","title":"Linknet: Relational embedding for scene graph","authors":"Sanghyun Woo, Dahun Kim, Donghyeon Cho","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xie et al., 2018] Shaoan Xie, Zibin Zheng, and Liang Chen. Learning semantic representations for unsupervised do- main adaptation. In ICML, pages 5419-5428, 2018.","title":"Learning semantic representations for unsupervised domain adaptation","authors":"Xie","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2017] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In CVPR, pages 5410-5419, 2017.","title":"Scene graph generation by iterative message passing","authors":"Danfei Xu, Yuke Zhu, Christopher B Choy, Li Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2018] Jianwei Yang, Jiasen Lu, and Stefan Lee. Graph r-cnn for scene graph generation. In ECCV, pages 670-685, 2018.","title":"Graph r-cnn for scene graph generation","authors":"Jianwei Yang, Jiasen Lu, Stefan Lee","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2019] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In CVPR, pages 10685-10694, 2019.","title":"Auto-encoding scene graphs for image captioning","authors":"Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zellers and Yatskar, 2018] Rowan Zellers and Mark Yatskar. Neural motifs: Scene graph parsing with global context. In CVPR, pages 5831-5840, 2018.","title":"Neural motifs: Scene graph parsing with global context","authors":"Yatskar Zellers, Rowan Zellers, Mark Yatskar","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zellers et al., 2019] Rowan Zellers, Yonatan Bisk, and Ali Farhadi. From recognition to cognition: Visual common- sense reasoning. In CVPR, pages 6720-6731, 2019.","title":"From recognition to cognition: Visual commonsense reasoning","authors":"Rowan Zellers, Yonatan Bisk, Ali Farhadi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019a] Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via saliency-guided hal- lucination of samples. In CVPR, June 2019.","title":"Few-shot learning via saliency-guided hallucination of samples","authors":"Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2019b] Ji Zhang, Yannis Kalantidis, and Mar- cus Rohrbach. Large-scale visual relationship understand- ing. In AAAI, volume 33, pages 9185-9194, 2019.","title":"Large-scale visual relationship understanding","authors":"Zhang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2019c] Weihe Zhang, Yali Wang, and Yu Qiao. Metacleaner: Learning to hallucinate clean representations for noisy-labeled visual recognition. In CVPR, June 2019.","title":"Metacleaner: Learning to hallucinate clean representations for noisy-labeled visual recognition","authors":"Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2014] Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In CVPR, pages 915-922, 2014.","title":"Capturing long-tail distributions of object subcategories","authors":"Zhu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhuang et al., 2017] Bohan Zhuang, Lingqiao Liu, and Chunhua Shen. Towards context-aware interaction recog- nition for visual relationship detection. In CVPR, pages 589-598, 2017.","title":"Towards context-aware interaction recognition for visual relationship detection","authors":"Bohan Zhuang, Lingqiao Liu, Chunhua Shen","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}