{"type":"ScholarlyArticle","title":"Active Contrastive Set Mining for Robust Audio-Visual Instance Discrimination","authors":[{"name":"Hanyu Xuan","slug":"hanyu-xuan","given":"Hanyu","family":"Xuan","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"},{"name":"Yihong Xu","slug":"yihong-xu","given":"Yihong","family":"Xu","affiliation":"Inria , University Grenoble Alpes , CNRS , Grenoble INP , LJK , Grenoble , France"},{"name":"Shuo Chen","slug":"shuo-chen","given":"Shuo","family":"Chen","affiliation":"RIKEN Center for Advanced Intelligence Project , Japan"},{"name":"Zhiliang Wu","slug":"zhiliang-wu","given":"Zhiliang","family":"Wu","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"},{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology , China"},{"name":"Yan Yan","slug":"yan-yan","given":"Yan","family":"Yan","affiliation":"Department of Computer Science , Illinois Institute of Technology , USA"},{"name":"Xavier Alameda-Pineda","slug":"xavier-alameda-pineda","given":"Xavier","family":"Alameda-Pineda","affiliation":"Inria , University Grenoble Alpes , CNRS , Grenoble INP , LJK , Grenoble , France"}],"abstract":"The recent success of audio-visual representation learning can be largely attributed to their pervasive property of audio-visual synchronization, which can be used as self-annotated supervision. As a state-of-the-art solution, Audio-Visual Instance Discrimination (AVID) extends instance discrimination to the audio-visual realm. Existing AVID methods construct the contrastive set by random sampling based on the assumption that the audio and visual clips from all other videos are not semantically related. We argue that this assumption is rough, since the resulting contrastive sets have a large number of faulty negatives. In this paper, we overcome this limitation by proposing a novel Active Contrastive Set Mining (ACSM) that aims to mine the contrastive sets with informative and diverse negatives for robust AVID. Moreover, we also integrate a semantically-aware hard-sample mining strategy into our ACSM. The proposed ACSM is implemented into two most recent state-of-the-art AVID methods and significantly improves their performance. Extensive experiments conducted on both action and  sound recognition on multiple datasets show the remarkably improved performance of our method.","doi":"10.24963/ijcai.2022/506","url":"https://ijcai.org/proceedings/2022/506","pdf":"https://ijcai.org/proceedings/2022/0506.pdf","markdown":"https://ijcai.org/proceedings/2022/506.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/506","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":506,"pages":{"first":3643,"last":3649},"keywords":["Machine Learning: Self-supervised Learning","Machine Learning: Multi-modal learning","Machine Learning: Representation learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986468","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984611","length_sec":244,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alwassel et al., 2018] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. In NeuIPS., 2018.","title":"Selfsupervised learning by cross-modal audio-video clustering","authors":"Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, Du Tran","year":"2018","venue":"NeuIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arandjelovic and Zisserman, 2017] Relja Arandjelovic and An- drew Zisserman. Look, listen and learn. In ICCV., 2017.","title":"Relja Arandjelovic and Andrew Zisserman. Look, listen and learn","authors":"Zisserman Arandjelovic","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arandjelovic and Zisserman, 2018] Relja Arandjelovic and An- drew Zisserman. Objects that sound. In ECCV., 2018.","title":"Relja Arandjelovic and Andrew Zisserman. Objects that sound","authors":"Zisserman Arandjelovic","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aytar et al., 2016] Yusuf Aytar, Carl Vondrick, and Antonio Tor- ralba. Soundnet: Learning sound representations from unlabeled video. In NeurIPS., 2016.","title":"Soundnet: Learning sound representations from unlabeled video","authors":"Yusuf Aytar, Carl Vondrick, Antonio Torralba","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bruno et al., 2018] Korbar Bruno, Tran Du, and Torresani Lorenzo. Cooperative learning of audio and video models from self-supervised synchronization. In NeurIPS., 2018.","title":"Cooperative learning of audio and video models from self-supervised synchronization","authors":"Bruno","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buchler et al., 2018] Uta Buchler, Biagio Brattoli, and Bjorn Om- mer. Improving spatiotemporal self-supervision by deep rein- forcement learning. In ECCV., 2018.","title":"Improving spatiotemporal self-supervision by deep reinforcement learning","authors":"Uta Buchler, Biagio Brattoli, Bjorn Ommer","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Exploring sim- ple siamese representation learning. In CVPR., 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2018] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theoretic perspective on model interpretation. In ICML., 2018.","title":"Learning to explain: An information-theoretic perspective on model interpretation","authors":"Jianbo Chen, Le Song, Martin Wainwright, Michael Jordan","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In ICML., 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. In arXiv., 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Florent Altché, et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS., 2020.","title":"Bootstrap your own latent: A new approach to self-supervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altché","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gutmann and Hyvärinen, 2010] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation : A new estimation principle for unnormalized statistical models. In AISTATS., 2010.","title":"Noise-contrastive estimation : A new estimation principle for unnormalized statistical models","authors":"Hyvärinen Gutmann, Michael Gutmann, Aapo Hyvärinen","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2019] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In CVPR. Workshops, 2019.","title":"Video representation learning by dense predictive coding","authors":"Han","year":"2019","venue":"CVPR. Workshops","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR., 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jaiswal et al., 2021] Ashish Jaiswal, Ashwin Ramesh Babu, Mo- hammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. In Technolo- gies, 2021.","title":"A survey on contrastive self-supervised learning","authors":"Ashish Jaiswal, Ramesh Ashwin, Mohammad Zaki Babu, Debapriya Zadeh, Fillia Banerjee, Makedon","year":"2021","venue":"Technologies","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jing et al., 2018] Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. In arXiv., 2018.","title":"Self-supervised spatiotemporal feature learning via video rotation prediction","authors":"Longlong Jing, Xiaodong Yang, Jingen Liu, Yingli Tian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS., 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Korbar et al., 2018] Bruno Korbar, Du Tran, and Lorenzo Torre- sani. Cooperative learning of audio and video models from self- supervised synchronization. In NeurIPS., 2018.","title":"Cooperative learning of audio and video models from selfsupervised synchronization","authors":"Bruno Korbar, Du Tran, Lorenzo Torresani","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2017] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In ICCV., 2017.","title":"Unsupervised representation learning by sorting sequences","authors":"Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, Ming-Hsuan Yang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ma et al., 2021] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video rep- resentations. In ICLR., 2021.","title":"Active contrastive learning of audio-visual video representations","authors":"Shuang Ma, Zhaoyang Zeng, Daniel Mcduff, Yale Song","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McAllester and Stratos, 2020] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In AISTATS., 2020.","title":"Formal limitations on the measurement of mutual information","authors":"Stratos Mcallester, David Mcallester, Karl Stratos","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Misra et al., 2016] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV., 2016.","title":"Shuffle and learn: unsupervised learning using temporal order verification","authors":"Ishan Misra, Lawrence Zitnick, Martial Hebert","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Morgado et al., 2020] Pedro Morgado, Yi Li, and Nuno Nvascon- celos. Learning representations from audio-visual spatial align- ment. In NeurIPS., 2020.","title":"Learning representations from audio-visual spatial alignment","authors":"Pedro Morgado, Yi Li, Nuno Nvasconcelos","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Morgado et al., 2021a] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In CVPR., 2021.","title":"Robust audio-visual instance discrimination","authors":"Morgado","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Morgado et al., 2021b] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross- modal agreement. In CVPR., 2021.","title":"Audio-visual instance discrimination with crossmodal agreement","authors":"Morgado","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Owens and Efros, 2018] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV., 2018.","title":"Audio-visual scene analysis with self-supervised multisensory features","authors":"Andrew Owens, Alexei A Efros","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Piczak, 2015a] Karol J Piczak. Environmental sound classification with convolutional neural networks. In MLSP. workshops, 2015.","title":"Environmental sound classification with convolutional neural networks","authors":"J Karol, Piczak","year":"2015","venue":"MLSP. workshops","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Piczak, 2015b] Karol J Piczak. Esc: Dataset for environmental sound classification. In ACM Multimedia, 2015.","title":"Esc: Dataset for environmental sound classification","authors":"J Karol, Piczak","year":"2015","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sailor et al., 2017] Hardik B Sailor, Dharmesh M Agrawal, and Hemant A Patil. Unsupervised filterbank learning using convo- lutional restricted boltzmann machine for environmental sound classification. In InterSpeech, 2017.","title":"Unsupervised filterbank learning using convolutional restricted boltzmann machine for environmental sound classification","authors":"Hardik B Sailor, M Dharmesh, Hemant A Agrawal, Patil","year":"2017","venue":"InterSpeech","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Saunshi et al., 2019] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theo- retical analysis of contrastive unsupervised representation learn- ing. In ICML., 2019.","title":"A theoretical analysis of contrastive unsupervised representation learning","authors":"Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, Hrishikesh Khandeparkar","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sheng et al., 2020] Hao Sheng, Yanwei Zheng, Wei Ke, Dongxiao Yu, Xiuzhen Cheng, Weifeng Lyu, and Zhang Xiong. Mining hard samples globally and efficiently for person reidentification. In IoT., 2020.","title":"Mining hard samples globally and efficiently for person reidentification","authors":"Yanwei Hao Sheng, Wei Zheng, Dongxiao Ke, Xiuzhen Yu, Weifeng Cheng, Zhang Lyu, Xiong","year":"2020","venue":"IoT","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Smith and Gasser, 2005] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. In Artificial Life, 2005.","title":"The development of embodied cognition: Six lessons from babies","authors":"Gasser ; Smith, Linda Smith, Michael Gasser","year":"2005","venue":"Artificial Life","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sun et al., 2019] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using con- trastive bidirectional transformer. In arXiv., 2019.","title":"Learning video representations using contrastive bidirectional transformer","authors":"Chen Sun, Fabien Baradel, Kevin Murphy, Cordelia Schmid","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tian et al., 2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV., 2020.","title":"Contrastive multiview coding","authors":"Yonglong Tian, Dilip Krishnan, Phillip Isola","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tran et al., 2018] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spa- tiotemporal convolutions for action recognition. In CVPR., 2018.","title":"A closer look at spatiotemporal convolutions for action recognition","authors":"Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann Lecun, Manohar Paluri","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR., 2018.","title":"Unsupervised feature learning via non-parametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xu et al., 2019] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR., 2019.","title":"Self-supervised spatiotemporal learning via video clip order prediction","authors":"Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, Yueting Zhuang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xuan et al., 2020] Hanyu Xuan, Zhenyu Zhang, Shuo Chen, Jian Yang, and Yan Yan. Cross-modal attention network for temporal inconsistent audio-visual event localization. In AAAI., 2020.","title":"Cross-modal attention network for temporal inconsistent audio-visual event localization","authors":"Zhenyu Hanyu Xuan, Shuo Zhang, Jian Chen, Yan Yang, Yan","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xuan et al., 2022] Hanyu Xuan, Zhiliang Wu, Jian Yang, Yan Yan, and Xavier Alameda-Pineda. A proposal-based paradigm for self-supervised sound source localization in videos. In CVPR., 2022.","title":"A proposal-based paradigm for self-supervised sound source localization in videos","authors":"Zhiliang Hanyu Xuan, Jian Wu, Yan Yang, Xavier Yan, Alameda-Pineda","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhan et al., 2020] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew- Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In CVPR., 2020.","title":"Online deep clustering for unsupervised representation learning","authors":"Zhan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}