{"type":"ScholarlyArticle","title":"Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast","authors":[{"name":"Boqing Zhu","slug":"boqing-zhu","given":"Boqing","family":"Zhu","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Kele Xu","slug":"kele-xu","given":"Kele","family":"Xu","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Changjian Wang","slug":"changjian-wang","given":"Changjian","family":"Wang","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Zheng Qin","slug":"zheng-qin","given":"Zheng","family":"Qin","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Tao Sun","slug":"tao-sun","given":"Tao","family":"Sun","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Huaimin Wang","slug":"huaimin-wang","given":"Huaimin","family":"Wang","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"},{"name":"Yuxing Peng","slug":"yuxing-peng","given":"Yuxing","family":"Peng","affiliation":"College of Computer , National University of Defense Technology , Changsha , China"}],"abstract":"We present an approach to learn voice-face representations from the talking face videos, without any identity labels. Previous works employ cross-modal instance discrimination tasks to establish the correlation of voice and face. These methods neglect the semantic content of different videos, introducing false-negative pairs as training noise. Furthermore, the positive pairs are constructed based on the natural correlation between audio clips and visual frames. However, this correlation might be weak or inaccurate in a large amount of real-world data, which leads to deviating positives into the contrastive paradigm. To address these issues, we propose the cross-modal prototype contrastive learning (CMPC), which takes advantage of contrastive methods and resists adverse effects of false negatives and deviate positives. On one hand, CMPC could learn the intra-class invariance by constructing semantic-wise positives via unsupervised clustering in different modalities. On the other hand, by comparing the similarities of cross-modal instances from that of cross-modal prototypes, we dynamically recalibrate the unlearnable instances' contribution to overall loss. Experiments show that the proposed approach outperforms state-of-the-art unsupervised methods on various voice-face association evaluation protocols. Additionally, in the low-shot supervision setting, our method also has a significant improvement compared to previous instance-wise contrastive learning.","doi":"10.24963/ijcai.2022/526","url":"https://ijcai.org/proceedings/2022/526","pdf":"https://ijcai.org/proceedings/2022/0526.pdf","markdown":"https://ijcai.org/proceedings/2022/526.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/526","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":526,"pages":{"first":3787,"last":3794},"keywords":["Machine Learning: Unsupervised Learning","Machine Learning: Self-supervised Learning","Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Machine Learning: Multi-modal learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986897","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985040","length_sec":304,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alayrac et al., 2020] Jean-Baptiste Alayrac, Adria Re- casens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks. NeurIPS, 2(6):7, 2020.","title":"Self-supervised multimodal versatile networks","authors":"Alayrac","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alwassel et al., 2020] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio- video clustering. In NeruIPS, 2020.","title":"Self-supervised learning by cross-modal audiovideo clustering","authors":"Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, Du Tran","year":"2020","venue":"NeruIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahrick et al., 2005] Lorraine E Bahrick, Maria Hernandez- Reif, and Ross Flom. The development of infant learning about specific face-voice relations. Developmental psy- chology, 41(3):541, 2005.","title":"The development of infant learning about specific face-voice relations","authors":"Lorraine E Bahrick, Maria Hernandez-Reif, Ross Flom","year":"2005","venue":"Developmental psychology","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caron et al., 2018] Mathilde Caron, Piotr Bojanowski, Ar- mand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In ECCV, pages 132-149, 2018.","title":"Deep clustering for unsupervised learning of visual features","authors":"Mathilde Caron, Piotr Bojanowski, Armand Joulin, Matthijs Douze","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caron et al., 2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by con- trasting cluster assignments. In NeruIPS, 2020.","title":"Unsupervised learning of visual features by contrasting cluster assignments","authors":"Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin","year":"2020","venue":"NeruIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Charlier et al., 2021] Benjamin Charlier, Jean Feydy, Joan Alexis Glaunès, Franc ¸ois-David Collin, and Ghislain Durif. Kernel operations on the gpu, with autodiff, without memory overflows. Journal of Machine Learning Research, 22(74):1-6, 2021.","title":"Kernel operations on the gpu, with autodiff, without memory overflows","authors":"Benjamin Charlier, Jean Feydy, Joan Alexis Glaunès, Franc ¸ois-David Collin, Ghislain Durif","year":"2021","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Choi et al., 2020] Hyeong-Seok Choi, Changdae Park, and Kyogu Lee. From inference to generation: End-to- end fully self-supervised generation of human face from speech. In ICLR, 2020.","title":"From inference to generation: End-toend fully self-supervised generation of human face from speech","authors":"Hyeong-Seok Choi, Changdae Park, Kyogu Lee","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chuang et al., 2020] Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NeurIPS, Virtual, On- line, 2020.","title":"Debiased contrastive learning","authors":"Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, Stefanie Jegelka","year":"2020","venue":"NeurIPS, Virtual","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chung et al., 2020] Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong-Goo Kang. Facefilter: Audio-visual speech separation using still images. In INTERSPEECH, volume 2020-October, pages 3481 - 3485, 2020.","title":"Facefilter: Audio-visual speech separation using still images","authors":"Soo-Whan Chung, Soyeon Choe, Joon, Son Chung, Hong-Goo Kang","year":"2020","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2020] Tengda Han, Weidi Xie, and Andrew Zis- serman. Memory-augmented dense predictive coding for video representation learning. In ECCV, pages 312-329. Springer, 2020.","title":"Memory-augmented dense predictive coding for video representation learning","authors":"Han","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Horiguchi et al., 2018] Shota Horiguchi, Naoyuki Kanda, and Kenji Nagamatsu. Face-voice matching using cross- modal embeddings. In ACM MM, pages 1011-1019, 2018.","title":"Face-voice matching using crossmodal embeddings","authors":"Shota Horiguchi, Naoyuki Kanda, Kenji Nagamatsu","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Joassin et al., 2011] Frédéric Joassin, Mauro Pesenti, Pierre Maurage, Emilie Verreckt, Raymond Bruyer, and Salva- tore Campanella. Cross-modal interactions between hu- man faces and voices involved in person recognition. Cor- tex, 47(3):367-376, 2011.","title":"Cross-modal interactions between human faces and voices involved in person recognition","authors":"Frédéric Joassin, Mauro Pesenti, Pierre Maurage, Emilie Verreckt, Raymond Bruyer, Salvatore Campanella","year":"2011","venue":"Cortex","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kalantidis et al., 2020] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NeurIPS, 2020.","title":"Hard negative mixing for contrastive learning","authors":"Yannis Kalantidis, Bulent Mert, Noe Sariyildiz, Philippe Pion, Diane Weinzaepfel, Larlus","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. arXiv preprint arXiv:2004.11362, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kim et al., 2018] Changil Kim, Hijung Valentina Shin, Tae- Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wo- jciech Matusik. On learning associations of faces and voices. In ACCV, pages 276-292. Springer, 2018.","title":"On learning associations of faces and voices","authors":"Kim","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of un- supervised representations. In ICLR, 2021.","title":"Prototypical contrastive learning of unsupervised representations","authors":"Li","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Morgado et al., 2021a] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrim- ination. In CVPR, pages 12934-12945, 2021.","title":"Robust audio-visual instance discrimination","authors":"Morgado","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Morgado et al., 2021b] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In CVPR, pages 12475- 12486, 2021.","title":"Audio-visual instance discrimination with cross-modal agreement","authors":"Morgado","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nagrani et al., 2018a] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable pins: Cross-modal embed- dings for person identity. In ECCV, pages 71-88, 2018.","title":"Learnable pins: Cross-modal embeddings for person identity","authors":"Nagrani","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nagrani et al., 2018b] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Seeing voices and hearing faces: Cross-modal biometric matching. In CVPR, pages 8427- 8436, 2018.","title":"Seeing voices and hearing faces: Cross-modal biometric matching","authors":"Nagrani","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nagraniy et al., 2017] Arsha Nagraniy, Joon Son Chungy, and Andrew Zisserman. Voxceleb: A large-scale speaker identification dataset. In INTERSPEECH, volume 2017- August, pages 2616 -2620, 2017.","title":"Voxceleb: A large-scale speaker identification dataset","authors":"Nagraniy","year":"2017","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nawaz et al., 2021] Shah Nawaz, Muhammad Saad Saeed, Pietro Morerio, Arif Mahmood, Ignazio Gallo, Muham- mad Haroon Yousaf, and Alessio Del Bue. Cross-modal speaker verification and recognition: A multilingual per- spective. In CVPR (Workshop), pages 1682-1691, 2021.","title":"Cross-modal speaker verification and recognition: A multilingual perspective","authors":"Muhammad Shah Nawaz, Pietro Saad Saeed, Arif Morerio, Ignazio Mahmood, Muhammad Gallo, Alessio Haroon Yousaf, Del Bue","year":"2021","venue":"CVPR (Workshop)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pan et al., 2021] Tian Pan, Yibing Song, Tianyu Yang, Wen- hao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial exam- ples. In CVPR, pages 11205-11214, 2021.","title":"Videomoco: Contrastive video representation learning with temporally adversarial examples","authors":"Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, Wei Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Patrick et al., 2021] Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, Joao F Henriques, Ge- offrey Zweig, and Andrea Vedaldi. Multi-modal self- supervision from generalized data transformations. In ICCV, 2021.","title":"Multi-modal selfsupervision from generalized data transformations","authors":"Patrick","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wen et al., 2019a] Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh. Disjoint mapping network for cross-modal matching of voices and faces. In ICLR, 2019.","title":"Disjoint mapping network for cross-modal matching of voices and faces","authors":"Wen","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wen et al., 2019b] Yandong Wen, Bhiksha Raj, and Rita Singh. Face reconstruction from voice using generative adversarial networks. In NeurIPS, volume 32, 2019.","title":"Face reconstruction from voice using generative adversarial networks","authors":"Wen","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wen et al., 2021] Peisong Wen, Qianqian Xu, Yangbangyan Jiang, Zhiyong Yang, Yuan He, and Qingming Huang. Seeking the shape of sound: An adaptive framework for learning voice-face association. In CVPR, pages 16347- 16356, 2021.","title":"Seeking the shape of sound: An adaptive framework for learning voice-face association","authors":"Peisong Wen, Qianqian Xu, Yangbangyan Jiang, Zhiyong Yang, Yuan He, Qingming Huang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2021] Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for contrastive learning of visual representations. In ICLR, 2021.","title":"Conditional negative sampling for contrastive learning of visual representations","authors":"Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, Noah Goodman","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhan et al., 2020] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clus- tering for unsupervised representation learning. In CVPR, pages 6688-6697, 2020.","title":"Online deep clustering for unsupervised representation learning","authors":"Zhan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zolfaghari et al., 2021] Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal contrastive learning for multi-modal video representations. In ICCV, pages 1450-1459, 2021.","title":"Crossclr: Cross-modal contrastive learning for multi-modal video representations","authors":"Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, Thomas Brox","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}