{"type":"ScholarlyArticle","title":"Cross-modal Representation Learning and Relation Reasoning for Bidirectional Adaptive Manipulation","authors":[{"name":"Lei Li","slug":"lei-li","given":"Lei","family":"Li","affiliation":"Department of Computer Science and Technology , Tsinghua University; Tsinghua Shenzhen International Graduate School , Peng Cheng Lab"},{"name":"Kai Fan","slug":"kai-fan","given":"Kai","family":"Fan","affiliation":"Alibaba DAMO Academy , Alibaba Group Inc."},{"name":"Chun Yuan","slug":"chun-yuan","given":"Chun","family":"Yuan","affiliation":"Tsinghua Shenzhen International Graduate School , Peng Cheng Lab"}],"abstract":"Since single-modal controllable manipulation typically requires supervision of information from other modalities or cooperation with complex software and experts, this paper addresses the problem of cross-modal adaptive manipulation (CAM). The novel task performs cross-modal semantic alignment from mutual supervision and implements bidirectional exchange of attributes, relations, or objects in parallel, benefiting both modalities while significantly reducing manual effort. We introduce a robust solution for CAM, which includes two essential modules, namely Heterogeneous Representation Learning (HRL) and Cross-modal Relation Reasoning (CRR). The former is designed to perform representation learning for cross-modal semantic alignment on heterogeneous graph nodes. The latter is adopted to identify and exchange the focused attributes, relations, or objects in both modalities. Our method produces pleasing cross-modal outputs on CUB and Visual Genome.","doi":"10.24963/ijcai.2022/447","url":"https://ijcai.org/proceedings/2022/447","pdf":"https://ijcai.org/proceedings/2022/0447.pdf","markdown":"https://ijcai.org/proceedings/2022/447.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/447","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":447,"pages":{"first":3222,"last":3228},"keywords":["Computer Vision: Vision and language","Machine Learning: Sequence and Graph Learning","Machine Learning: Multi-modal learning","Machine Learning: Representation learning","Machine Learning: Relational Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987022","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985165","length_sec":662,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cornia et al., 2019] Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Show, control and tell: A framework for generating controllable and grounded captions. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8307-8316, 2019.","title":"Show, control and tell: A framework for generating controllable and grounded captions","authors":"Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dhamo et al., 2020] Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D Hager, Federico Tombari, and Christian Rupprecht. Semantic image manipulation using scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5213-5222, 2020.","title":"Semantic image manipulation using scene graphs","authors":"Dhamo","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2017] Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. Semantic image synthesis via adversarial learn- ing. In Proceedings of the IEEE International Conference on Computer Vision, pages 5706-5714, 2017.","title":"Semantic image synthesis via adversarial learning","authors":"Hao Dong, Simiao Yu, Chao Wu, Yike Guo","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2020] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.","title":"Generative adversarial networks","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2020","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Johnson et al., 2018] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 1219-1228, 2018.","title":"Image generation from scene graphs","authors":"Justin Johnson, Agrim Gupta, Li Fei-Fei","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kampffmeyer et al., 2019] Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero- shot learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 11487-11496, 2019.","title":"Rethinking knowledge graph propagation for zeroshot learning","authors":"Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, Eric P Xing","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national journal of computer vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2019] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Controllable text- to-image generation. arXiv preprint arXiv:1909.07083, 2019.","title":"Controllable textto-image generation","authors":"Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip Hs Torr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020a] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Manigan: Text- guided image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7880-7889, 2020.","title":"Manigan: Textguided image manipulation","authors":"Li","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020b] Bowen Li, Xiaojuan Qi, Philip HS Torr, and Thomas Lukasiewicz. Lightweight generative adver- sarial networks for text-guided image manipulation. arXiv preprint arXiv:2010.12136, 2020.","title":"Lightweight generative adversarial networks for text-guided image manipulation","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2019] Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. Fine-grained fact verification with kernel graph attention network. arXiv preprint arXiv:1910.09796, 2019.","title":"Fine-grained fact verification with kernel graph attention network","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mihalcea and Tarau, 2004] Rada Mihalcea and Paul Tarau. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural lan- guage processing, pages 404-411, 2004.","title":"Textrank: Bringing order into text","authors":"Tarau Mihalcea, Rada Mihalcea, Paul Tarau","year":"2004","venue":"Proceedings of the 2004 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nam et al., 2018] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. Text-adaptive generative adversarial net- works: Manipulating images with natural language. arXiv preprint arXiv:1810.11919, 2018.","title":"Text-adaptive generative adversarial networks: Manipulating images with natural language","authors":"Nam","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. California Institute of Tech- nology, 2011.","title":"The caltechucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2018] Zhouxia Wang, Tianshui Chen, Jimmy Ren, Weihao Yu, Hui Cheng, and Liang Lin. Deep rea- soning with knowledge graph for social relationship un- derstanding. arXiv preprint arXiv:1807.00504, 2018.","title":"Deep reasoning with knowledge graph for social relationship understanding","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/142"},{"position":18,"raw":"[Xu et al., 2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xi- aodong He. Attngan: Fine-grained text to image gener- ation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316-1324, 2018.","title":"Attngan: Fine-grained text to image generation with attentional generative adversarial networks","authors":"Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yao et al., 2019] Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intel- ligence, volume 33, pages 7370-7377, 2019.","title":"Graph convolutional networks for text classification","authors":"Liang Yao, Chengsheng Mao, Yuan Luo","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yujian and Bo, 2007] Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pat- tern analysis and machine intelligence, 29(6):1091-1095, 2007.","title":"A normalized levenshtein distance metric","authors":"Li Yujian, Liu Bo","year":"2007","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}