{"type":"ScholarlyArticle","title":"Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data","authors":[{"name":"Haytham M. Fayek","slug":"haytham-m-fayek","given":"Haytham M.","family":"Fayek","affiliation":"Facebook Reality Labs , Redmond , WA , USA"},{"name":"Anurag Kumar","slug":"anurag-kumar","given":"Anurag","family":"Kumar","affiliation":"Facebook Reality Labs , Redmond , WA , USA"}],"abstract":"Recognizing sounds is a key aspect of computational audio scene analysis and machine perception. In this paper, we advocate that sound recognition is inherently a multi-modal audiovisual task in that it is easier to differentiate sounds using both the audio and visual modalities as opposed to one or the other. We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. The proposed fusion model utilizes an attention mechanism to dynamically combine the outputs of the individual audio and visual models. Experiments on the large scale sound events dataset, AudioSet, demonstrate\r\nthe efficacy of the proposed model, which outperforms the single-modal models, and state-of-the-art fusion and multi-modal models. We achieve a mean Average Precision (mAP) of 46.16 on Audioset, outperforming prior state of the art by approximately +4.35 mAP (relative: 10.4%).","doi":"10.24963/ijcai.2020/78","url":"https://ijcai.org/proceedings/2020/78","pdf":"https://ijcai.org/proceedings/2020/0078.pdf","markdown":"https://ijcai.org/proceedings/2020/78.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/78","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":78,"pages":{"first":558,"last":565},"keywords":["Natural Language Processing: Speech","Computer Vision: Video: Events, Activities and Surveillance","Machine Learning: Multi-instance;Multi-label;Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Adavanne et al., 2019] Sharath Adavanne, Haytham M. Fayek, and Vladimir Tourbabin. Sound event classifica- tion and detection with weakly labeled data. In Detection and Classification of Acoustic Scenes and Events Work- shop (DCASE), pages 15-19, New York, USA, October 2019.","title":"Sound event classification and detection with weakly labeled data","authors":"Sharath Adavanne, M Haytham, Vladimir Fayek, Tourbabin","year":"2019","venue":"Detection and Classification of Acoustic Scenes and Events Workshop (DCASE)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alwassel et al., 2019] Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video cluster- ing. arXiv preprint arXiv:1911.12667, 2019.","title":"Selfsupervised learning by cross-modal audio-video clustering","authors":"Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, Du Tran","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andrews et al., 2003] Stuart Andrews, Ioannis Tsochan- taridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. In NIPS, pages 577-584, 2003.","title":"Support vector machines for multiple-instance learning","authors":"Stuart Andrews, Ioannis Tsochantaridis, Thomas Hofmann","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arandjelovic and Zisserman, 2017] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In CVPR, pages 609-617, 2017.","title":"Relja Arandjelovic and Andrew Zisserman. Look, listen and learn","authors":"Zisserman Arandjelovic","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Aytar et al., 2016] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NIPS, pages 892-900, 2016.","title":"Soundnet: Learning sound representations from unlabeled video","authors":"Yusuf Aytar, Carl Vondrick, Antonio Torralba","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Baltrušaitis et al., 2018] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41(2):423-443, 2018.","title":"Multimodal machine learning: A survey and taxonomy","authors":"Tadas Baltrušaitis, Chaitanya Ahuja, Louis-Philippe Morency","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ford et al., 2019] Logan Ford, Hao Tang, Franc ¸ois Grondin, and James Glass. A deep residual network for large-scale acoustic scene analysis. In Interspeech, 2019.","title":"A deep residual network for large-scale acoustic scene analysis","authors":"Logan Ford, Hao Tang, James Franc ¸ois Grondin, Glass","year":"2019","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2019] Ruohan Gao, Tae-Hyun Oh, Kristen Grau- man, and Lorenzo Torresani. Listen to look: Ac- tion recognition by previewing audio. arXiv preprint arXiv:1912.04487, 2019.","title":"Listen to look: Action recognition by previewing audio","authors":"Gao","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gemmeke et al., 2017] J Gemmeke, D Ellis, D Freedman, A Jansen, W Lawrence, C Moore, M Plakal, and M Rit- ter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, pages 776-780. IEEE, 2017.","title":"Audio set: An ontology and human-labeled dataset for audio events","authors":"Gemmeke, D Ellis, Freedman, W Jansen, C Lawrence, M Moore, M Plakal, Ritter","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ghanem et al., 2018] B Ghanem, J Niebles, C Snoek, F Heilbron, H Alwassel, V Escorcia, R Khrisna, S Buch, and C Dao. The activitynet large-scale activ- ity recognition challenge 2018 summary. arXiv preprint arXiv:1808.03766, 2018.","title":"The activitynet large-scale activity recognition challenge 2018 summary","authors":"J Ghanem, C Niebles, F Snoek, H Heilbron, V Alwassel, R Escorcia, S Khrisna, C Buch, Dao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gu et al., 2018] C Gu, C Sun, D Ross, C Vondrick, C Panto- faru, Y Li, S Vijayanarasimhan, G Toderici, S Ricco, R Sukthankar, et al. AVA: A video dataset of spatio- temporally localized atomic visual actions. In CVPR, pages 6047-6056, 2018.","title":"AVA: A video dataset of spatiotemporally localized atomic visual actions","authors":"C Gu, D Sun, C Ross, C Vondrick, Y Pantofaru, S Li, G Vijayanarasimhan, S Toderici, Ricco, Sukthankar","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hori et al., 2018] Chiori Hori, Takaaki Hori, Gordon Wich- ern, Jue Wang, Teng-Yok Lee, Anoop Cherian, and Tim K Marks. Multimodal attention for fusion of audio and spa- tiotemporal features for video description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2528-2531, 2018.","title":"Multimodal attention for fusion of audio and spatiotemporal features for video description","authors":"Chiori Hori, Takaaki Hori, Gordon Wichern, Jue Wang, Teng-Yok Lee, Anoop Cherian, Tim K Marks","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kay et al., 2017] W Kay, J Carreira, K Simonyan, B Zhang, C Hillier, S Vijayanarasimhan, F Viola, T Green, T Back, P Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"J Kay, K Carreira, B Simonyan, C Zhang, S Hillier, F Vijayanarasimhan, T Viola, T Green, Back, Natsev","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kazakos et al., 2019] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV, October 2019.","title":"Epic-fusion: Audio-visual temporal binding for egocentric action recognition","authors":"Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, Dima Damen","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kong et al., 2019] Qiuqiang Kong, Changsong Yu, Yong Xu, Turab Iqbal, Wenwu Wang, and Mark D Plumbley. Weakly labelled audioset tagging with attention neural net- works. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 27(11):1791-1802, 2019.","title":"Weakly labelled audioset tagging with attention neural networks","authors":"Kong","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Korbar et al., 2018] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In NeurIPS, pages 7763-7774, 2018.","title":"Cooperative learning of audio and video models from self-supervised synchronization","authors":"Bruno Korbar, Du Tran, Lorenzo Torresani","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kumar and Ithapu, 2020] Anurag Kumar and Vamsi Kr- ishna Ithapu. Secost: Sequential co-supervision for weakly labeled audio event detection. In ICASSP. IEEE, 2020.","title":"Secost: Sequential co-supervision for weakly labeled audio event detection","authors":"Anurag Kumar, Krishna Vamsi, Ithapu","year":"2020","venue":"ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kumar and Raj, 2016a] Anurag Kumar and Bhiksha Raj. Audio event detection using weakly labeled data. In ACM Multimedia, pages 1038-1047. ACM, 2016.","title":"Audio event detection using weakly labeled data","authors":"Raj ; Kumar, Anurag Kumar, Bhiksha Raj","year":"2016","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kumar and Raj, 2016b] Anurag Kumar and Bhiksha Raj. Weakly supervised scalable audio content analysis. In 2016 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2016.","title":"Weakly supervised scalable audio content analysis","authors":"Raj ; Kumar, Anurag Kumar, Bhiksha Raj","year":"2016","venue":"IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kumar, 2018] Anurag Kumar. Acoustic Intelligence in Ma- chines. PhD thesis, Carnegie Mellon University, 2018.","title":"Acoustic Intelligence in Machines","authors":"Anurag Kumar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lan et al., 2012] Zhen-zhong Lan, Lei Bao, Shoou-I Yu, Wei Liu, and Alexander G Hauptmann. Double fusion for multimedia event detection. In International Conference on Multimedia Modeling, pages 173-185. Springer, 2012.","title":"Double fusion for multimedia event detection","authors":"Zhen-Zhong Lan, Lei Bao, Shoou-I Yu, Wei Liu, Alexander G Hauptmann","year":"2012","venue":"International Conference on Multimedia Modeling","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lin et al., 2019] Yan-Bo Lin, Yu-Jhe Li, and Yu- Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In ICASSP, pages 2002-2006. IEEE, 2019.","title":"Dual-modality seq2seq network for audio-visual event localization","authors":"Yan-Bo Lin, Yu-Jhe Li, Yu-Chiang Frank, Wang","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Long et al., 2018] Xiang Long, Chuang Gan, Gerard De Melo, Jiajun Wu, Xiao Liu, and Shilei Wen. Atten- tion clusters: Purely attention based local feature integra- tion for video classification. In CVPR, pages 7834-7843, 2018.","title":"Attention clusters: Purely attention based local feature integration for video classification","authors":"Xiang Long, Chuang Gan, Gerard De Melo, Jiajun Wu, Xiao Liu, Shilei Wen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[McFee et al., 2018] Brian McFee, Justin Salamon, and Juan Pablo Bello. Adaptive pooling operators for weakly labeled sound event detection. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(11):2180-2193, 2018.","title":"Adaptive pooling operators for weakly labeled sound event detection","authors":"Brian Mcfee, Justin Salamon, Juan Pablo Bello","year":"2018","venue":"IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Owens and Efros, 2018] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, pages 631-648, 2018.","title":"Audio-visual scene analysis with self-supervised multisensory features","authors":"Andrew Owens, Alexei A Efros","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Parekh et al., 2019] S Parekh, S Essid, A Ozerov, N Duong, P Pérez, and G Richard. Weakly supervised representa- tion learning for audio-visual scene analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 28:416-428, 2019.","title":"Weakly supervised representation learning for audio-visual scene analysis","authors":"Parekh, Essid, N Ozerov, P Duong, G Pérez, Richard","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sterpu et al., 2018] George Sterpu, Christian Saam, and Naomi Harte. Attention-based audio-visual fusion for ro- bust automatic speech recognition. In Proceedings of the 20th ACM International Conference on Multimodal Inter- action, pages 111-115, 2018.","title":"Attention-based audio-visual fusion for robust automatic speech recognition","authors":"George Sterpu, Christian Saam, Naomi Harte","year":"2018","venue":"Proceedings of the 20th ACM International Conference on Multimodal Interaction","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Virtanen et al., 2017] Tuomas Virtanen, Mark D Plumbley, and Dan Ellis. Computational analysis of sound scenes and events. Springer, 2017.","title":"Computational analysis of sound scenes and events","authors":"Tuomas Virtanen, Mark D Plumbley, Dan Ellis","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2019a] Weiyao Wang, Du Tran, and Matt Feis- zli. What makes training multi-modal networks hard? arXiv preprint arXiv:1905.12681, 2019.","title":"What makes training multi-modal networks hard","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2019b] Yun Wang, Juncheng Li, and Florian Metze. A comparison of five multiple instance learning pooling functions for sound event detection with weak la- beling. In ICASSP, pages 31-35, 2019.","title":"A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling","authors":"Wang","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wei et al., 2014] Xiu-Shen Wei, Jianxin Wu, and Zhi-Hua Zhou. Scalable multi-instance learning. In IEEE Inter- national Conference on Data Mining, pages 1037-1042. IEEE, 2014.","title":"Scalable multi-instance learning","authors":"Xiu-Shen Wei, Jianxin Wu, Zhi-Hua Zhou","year":"2014","venue":"IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2016] Zuxuan Wu, Yu-Gang Jiang, Xi Wang, Hao Ye, and Xiangyang Xue. Multi-stream multi-class fusion of deep networks for video classification. In ACM Multi- media, pages 791-800. ACM, 2016.","title":"Multi-stream multi-class fusion of deep networks for video classification","authors":"Zuxuan Wu, Yu-Gang Jiang, Xi Wang, Hao Ye, Xiangyang Xue","year":"2016","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2019] Pan Zhou, Wenwen Yang, Wei Chen, Yanfeng Wang, and Jia Jia. Modality attention for end- to-end audio-visual speech recognition. In ICASSP, pages 6565-6569. IEEE, 2019.","title":"Modality attention for endto-end audio-visual speech recognition","authors":"Zhou","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}