{"type":"ScholarlyArticle","title":"Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation","authors":[{"name":"Shaofei Huang","slug":"shaofei-huang","given":"Shaofei","family":"Huang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Han Li","slug":"han-li","given":"Han","family":"Li","affiliation":"School of Computer Science and Engineering , Beihang University"},{"name":"Yuqing Wang","slug":"yuqing-wang","given":"Yuqing","family":"Wang","affiliation":"Alibaba Group"},{"name":"Hongji Zhu","slug":"hongji-zhu","given":"Hongji","family":"Zhu","affiliation":"Alibaba Group"},{"name":"Jiao Dai","slug":"jiao-dai","given":"Jiao","family":"Dai","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Jizhong Han","slug":"jizhong-han","given":"Jizhong","family":"Han","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Wenge Rong","slug":"wenge-rong","given":"Wenge","family":"Rong","affiliation":"School of Computer Science and Engineering , Beihang University"},{"name":"Si Liu","slug":"si-liu","given":"Si","family":"Liu","affiliation":"Institute of Artificial Intelligence , Beihang University; Hangzhou Innovation Institute , Beihang University"}],"abstract":"Audio visual segmentation (AVS) aims to segment the sounding objects for each frame of a given video. To distinguish the sounding objects from silent ones, both audio-visual semantic correspondence and temporal interaction are required. The previous method applies multi-frame cross-modal attention to conduct pixel-level interactions between audio features and visual features of multiple frames simultaneously, which is both redundant and implicit. In this paper, we propose an Audio-Queried Transformer architecture, AQFormer, where we define a set of object queries conditioned on audio information and associate each of them to particular sounding objects. Explicit object-level semantic correspondence between audio and visual modalities is established by gathering object information from visual features with predefined audio queries. Besides, an Audio-Bridged Temporal Interaction module is proposed to exchange sounding object-relevant information among multiple frames with the bridge of audio features. Extensive experiments are conducted on two AVS benchmarks to show that our method achieves state-of-the-art performances, especially 7.1% M_J and 7.6% M_F gains on the MS3 setting.","doi":"10.24963/ijcai.2023/97","url":"https://ijcai.org/proceedings/2023/97","pdf":"https://ijcai.org/proceedings/2023/0097.pdf","markdown":"https://ijcai.org/proceedings/2023/97.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/97","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":97,"pages":{"first":875,"last":883},"keywords":["Computer Vision: CV: Segmentation","Computer Vision: CV: Video analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Alwassel et al., 2020] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio- video clustering. NeurIPS, 2020.","title":"Self-supervised learning by cross-modal audiovideo clustering","authors":"Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, Du Tran","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arandjelovic and Zisserman, 2017] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.","title":"Look, listen and learn","authors":"Relja Arandjelovic, Andrew Zisserman","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arandjelovic and Zisserman, 2018] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018.","title":"Objects that sound","authors":"Relja Arandjelovic, Andrew Zisserman","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Botach et al., 2022] Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object seg- mentation with multimodal transformers. In CVPR, 2022.","title":"End-to-end referring video object segmentation with multimodal transformers","authors":"Botach","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cheng et al., 2021] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021.","title":"Per-pixel classification is not all you need for semantic segmentation","authors":"Bowen Cheng, Alex Schwing, Alexander Kirillov","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cheng et al., 2022] Bowen Cheng, Ishan Misra, Alexan- der G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.","title":"Masked-attention mask transformer for universal image segmentation","authors":"Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, Rohit Girdhar","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding et al., 2022] Zihan Ding, Tianrui Hui, Junshi Huang, Xiaoming Wei, Jizhong Han, and Si Liu. Language- bridged spatial-temporal interaction for referring video ob- ject segmentation. In CVPR, 2022.","title":"Languagebridged spatial-temporal interaction for referring video object segmentation","authors":"Zihan Ding, Tianrui Hui, Junshi Huang, Xiaoming Wei, Jizhong Han, Si Liu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Duke et al., 2021] Brendan Duke, Abdalla Ahmed, Chris- tian Wolf, Parham Aarabi, and Graham W Taylor. Sstvos: Sparse spatiotemporal transformers for video object seg- mentation. In CVPR, 2021.","title":"Sstvos: Sparse spatiotemporal transformers for video object segmentation","authors":"Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, Graham W Taylor","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fayyaz et al., 2016] Mohsen Fayyaz, Mohammad Ha- jizadeh Saffar, Mohammad Sabokrou, Mahmood Fathy, Reinhard Klette, and Fay Huang. Stfcn: spatio-temporal fcn for semantic video segmentation. arXiv preprint arXiv:1608.05971, 2016.","title":"Stfcn: spatio-temporal fcn for semantic video segmentation","authors":"Mohsen Fayyaz, Mohammad Hajizadeh Saffar, Mohammad Sabokrou, Mahmood Fathy, Reinhard Klette, Fay Huang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fu et al., 2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual at- tention network for scene segmentation. In CVPR, 2019.","title":"Dual attention network for scene segmentation","authors":"Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, Hanqing Lu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hershey et al., 2017] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale au- dio classification. In ICASSP, 2017.","title":"Cnn architectures for large-scale audio classification","authors":"Shawn Hershey, Sourish Chaudhuri, P W Daniel, Ellis, Aren Jort F Gemmeke, R Channing Jansen, Manoj Moore, Devin Plakal, Rif A Platt, Bryan Saurous, Seybold","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hu et al., 2019] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learn- ing. In CVPR, 2019.","title":"Deep multimodal clustering for unsupervised audiovisual learning","authors":"Di Hu, Feiping Nie, Xuelong Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Milletari et al., 2016] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neu- ral networks for volumetric medical image segmentation. 3DV, 2016.","title":"V-net: Fully convolutional neural networks for volumetric medical image segmentation","authors":"Fausto Milletari, Nassir Navab, Seyed-Ahmad Ahmadi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nilsson and Sminchisescu, 2018] David Nilsson and Cris- tian Sminchisescu. Semantic video segmentation by gated recurrent flow propagation. In CVPR, 2018.","title":"Semantic video segmentation by gated recurrent flow propagation","authors":"David Nilsson, Cristian Sminchisescu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Oh et al., 2019] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, 2019.","title":"Video object segmentation using space-time memory networks","authors":"Oh","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Perazzi et al., 2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, pages 724-732, 2016.","title":"A benchmark dataset and evaluation methodology for video object segmentation","authors":"Federico Perazzi, Jordi Pont-Tuset, Brian Mcwilliams, Luc Van Gool, Markus Gross, Alexander Sorkine-Hornung","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qian et al., 2020] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In ECCV, 2020.","title":"Multiple sound sources localization from coarse to fine","authors":"Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, Weiyao Lin","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rouditchenko et al., 2019] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Tor- ralba. Self-supervised audio-visual co-segmentation. In ICASSP, 2019.","title":"Self-supervised audio-visual co-segmentation","authors":"Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh Mcdermott, Antonio Torralba","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Seo et al., 2020] Seonguk Seo, Joon-Young Lee, and Bo- hyung Han. Urvos: Unified referring video object segmen- tation network with a large-scale benchmark. In ECCV, 2020.","title":"Urvos: Unified referring video object segmentation network with a large-scale benchmark","authors":"Seo","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Strudel et al., 2021] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.","title":"Segmenter: Transformer for semantic segmentation","authors":"Strudel","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022.","title":"Pvt v2: Improved baselines with pyramid vision transformer","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2022","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2022a] Ho-Hsiang Wu, Prem Seetharaman, Kun- dan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP, 2022.","title":"Wav2clip: Learning robust audio representations from clip","authors":"Wu","year":"2022","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2022b] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In CVPR, 2022.","title":"Language as queries for referring video object segmentation","authors":"Wu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang and Yang, 2021] Zongxin Yang and Yi Yang. Associ- ating objects with transformers for video object segmenta- tion. NeurIPS, 2021.","title":"Associating objects with transformers for video object segmentation","authors":"Yang, Yang ;, Zongxin Yang, Yi Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang and Yang, 2022] Zongxin Yang and Yi Yang. Decou- pling features in hierarchical propagation for video object segmentation. NeurIPS, 2022.","title":"Decoupling features in hierarchical propagation for video object segmentation","authors":"Yang, Yang ;, Zongxin Yang, Yi Yang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2020] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground-background integration. In ECCV, 2020.","title":"Collaborative video object segmentation by foreground-background integration","authors":"Yang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yuan et al., 2021] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Ob- ject context for semantic segmentation. IJCV, 2021.","title":"Ocnet: Object context for semantic segmentation","authors":"Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, Jingdong Wang","year":"2021","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhao et al., 2019] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019.","title":"The sound of motions","authors":"Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou et al., 2022] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In ECCV, 2022.","title":"Audio-visual segmentation","authors":"Zhou","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhu et al., 2017] Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei. Deep feature flow for video recognition. In CVPR, 2017.","title":"Deep feature flow for video recognition","authors":"Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, Yichen Wei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu et al., 2021] X Zhu, W Su, LW Lu, B Li, XG Wang, and Deformable DETR Dai J F. Deformable transformers for end-to-end object detection. In ICLR, 2021.","title":"Deformable transformers for end-to-end object detection","authors":"Zhu, Su, B Lu, Li, Deformable Detr Wang, J Dai","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}