{"type":"ScholarlyArticle","title":"Balancing Multimodal Learning via Online Logit Modulation","authors":[{"name":"Daoming Zong","slug":"daoming-zong","given":"Daoming","family":"Zong","affiliation":"SenseTime Research"},{"name":"Chaoyue Ding","slug":"chaoyue-ding","given":"Chaoyue","family":"Ding","affiliation":"SenseTime Research"},{"name":"Baoxiang Li","slug":"baoxiang-li","given":"Baoxiang","family":"Li","affiliation":"SenseTime Research"},{"name":"Jiakui Li","slug":"jiakui-li","given":"Jiakui","family":"Li","affiliation":"SenseTime Research"},{"name":"Ken Zheng","slug":"ken-zheng","given":"Ken","family":"Zheng","affiliation":"SenseTime Research"}],"abstract":"Multimodal learning is provably superior to unimodal learning. However, in practice, the best-performing unimodal networks often outperform jointly trained multimodal networks. This phenomenon can be attributed to the varying convergence and generalization rates across different modalities, leading to the dominance of one modality and causing underfitting of other modalities in simple multimodal joint training. To mitigate this issue, we propose two key ingredients: i) disentangling the learning of unimodal features and multimodal interaction through an intermediate representation fusion block; ii) modulating the logits of different modalities via dynamic coefficients during training to align their magnitudes with the target values, referred to as online logit modulation (OLM). Remarkably, OLM is model-agnostic and can be seamlessly integrated with most existing multimodal training frameworks. Empirical evidence shows that our approach brings significant enhancements over baselines on a wide range of multimodal tasks, covering video, audio, text, image, and depth modalities.","doi":"10.24963/ijcai.2024/636","url":"https://ijcai.org/proceedings/2024/636","pdf":"https://ijcai.org/proceedings/2024/0636.pdf","markdown":"https://ijcai.org/proceedings/2024/636.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/636","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":636,"pages":{"first":5753,"last":5761},"keywords":["Computer Vision: CV: Multimodal learning","Machine Learning: ML: Optimization","Machine Learning: ML: Applications","Machine Learning: ML: Attention models"],"videos":[],"references":[{"position":1,"raw":"[Alvarez-Melis and Fusi, 2020] David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. NeurIPS, 33:21428-21439, 2020.","title":"Geometric dataset distances via optimal transport","authors":"Fusi Alvarez-Melis, David Alvarez-Melis, Nicolo Fusi","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arandjelovic and Zisserman, 2017] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, pages 609-617, 2017.","title":"Relja Arandjelovic and Andrew Zisserman. Look, listen and learn","authors":"Zisserman Arandjelovic","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brousmiche et al., 2021] Mathilde Brousmiche, Jean Rouat, and Stéphane Dupont. Multi-level attention fusion network for audio-visual event recognition. arXiv e-prints, pages arXiv-2106, 2021.","title":"Multi-level attention fusion network for audio-visual event recognition","authors":"Mathilde Brousmiche, Jean Rouat, Stéphane Dupont","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2014] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Comput- ing, 5(4):377-390, 2014.","title":"Crema-d: Crowd-sourced emotional multimodal actors dataset","authors":"Cao","year":"2014","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pages 794-803.","title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks","authors":"Vijay Zhao Chen, Chen-Yu Badrinarayanan, Andrew Lee, Rabinovich","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2020a] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, pages 721-725. IEEE, 2020.","title":"Vggsound: A large-scale audio-visual dataset","authors":"Chen","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020b] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cui et al., 2020] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre- trained models for chinese natural language processing. In Findings of EMNLP, pages 657-668, 2020.","title":"Revisiting pretrained models for chinese natural language processing","authors":"Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, Guoping Hu","year":"2020","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fan et al., 2023] Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In CVPR, pages 20029-20038, 2023.","title":"Pmr: Prototypical modal rebalance for multimodal learning","authors":"Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, Song Guo","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Feichtenhofer et al., 2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6202-6211, 2019.","title":"Slowfast networks for video recognition","authors":"Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gemmeke et al., 2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, pages 776-780. IEEE, 2017.","title":"Audio set: An ontology and human-labeled dataset for audio events","authors":"Jort F Gemmeke, P W Daniel, Dylan Ellis, Aren Freedman, Wade Jansen, R Channing Lawrence, Manoj Moore, Marvin Plakal, Ritter","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gretton et al., 2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.","title":"A kernel two-sample test","authors":"Gretton","year":"2012","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hazarika et al., 2020] Devamanyu Hazarika, Roger Zim- mermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In ACM MM, pages 1122-1131, 2020.","title":"Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis","authors":"Hazarika","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hsu et al., 2021] Wei-Ning Hsu, Benjamin Bolte, Yao- Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.","title":"Hubert: Selfsupervised speech representation learning by masked prediction of hidden units","authors":"Wei-Ning Hsu, Benjamin Bolte, Hubert Yao-Hung, Kushal Tsai, Ruslan Lakhotia, Abdelrahman Salakhutdinov, Mohamed","year":"2021","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Huang et al., 2022] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competi- tion: What makes joint training of multi-modal network fail in deep learning?(provably). In ICML, pages 9226- 9259. PMLR, 2022.","title":"Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably)","authors":"Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, Longbo Huang","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jia et al., 2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text su- pervision. In ICML, pages 4904-4916. PMLR, 2021.","title":"Scaling up visual and vision-language representation learning with noisy text supervision","authors":"Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, Tom Duerig","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kay et al., 2017] Will Kay, Joao Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2021] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 34:9694-9705, 2021.","title":"Align before fuse: Vision and language representation learning with momentum distillation","authors":"Li","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2022] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In CVPR, pages 4953-4963, 2022.","title":"Align and prompt: Video-and-language pre-training with entity prompts","authors":"Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lian et al., 2023] Zheng Lian, Haiyang Sun, Licai Sun, Jin- ming Zhao, Ye Liu, Bin Liu, Jiangyan Yi, Meng Wang, Erik Cambria, Guoying Zhao, et al. Mer 2023: Multi- label learning, modality robustness, and semi-supervised learning. arXiv preprint arXiv:2304.08981, 2023.","title":"Mer 2023: Multilabel learning, modality robustness, and semi-supervised learning","authors":"Zheng Lian, Haiyang Sun, Licai Sun, Jinming Zhao, Ye Liu, Bin Liu, Jiangyan Yi, Meng Wang, Erik Cambria, Guoying Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lu et al., 2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visi- olinguistic representations for vision-and-language tasks. NeurIPS, 32, 2019.","title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","authors":"Lu","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[McFee et al., 2015] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science con- ference, volume 8, pages 18-25, 2015.","title":"librosa: Audio and music signal analysis in python","authors":"Brian Mcfee, Colin Raffel, Dawen Liang, P Daniel, Matt Ellis, Eric Mcvicar, Oriol Battenberg, Nieto","year":"2015","venue":"Proceedings of the 14th python in science conference","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Nagrani et al., 2021] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. NeurIPS, 34:14200-14213, 2021.","title":"Attention bottlenecks for multimodal fusion","authors":"Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Peng et al., 2022] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In CVPR, pages 8238-8247, 2022.","title":"Balanced multimodal learning via on-the-fly gradient modulation","authors":"Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, Di Hu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Perez et al., 2018] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, volume 32, 2018.","title":"Film: Visual reasoning with a general conditioning layer","authors":"Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, Aaron Courville","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, pages 8748-8763.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shen et al., 2023] Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. arXiv preprint arXiv:2302.05738, 2023.","title":"Cross-modal fine-tuning: Align then refine","authors":"Junhong Shen, Liam Li, Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, Ameet Talwalkar","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Song et al., 2015] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, pages 567-576, 2015.","title":"Sun rgb-d: A rgb-d scene understanding benchmark suite","authors":"Song","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sun et al., 2023] Licai Sun, Zheng Lian, Bin Liu, and Jian- hua Tao. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analy- sis. IEEE Transactions on Affective Computing, 2023.","title":"Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis","authors":"Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao","year":"2023","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tian et al., 2018] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, pages 247-263, 2018.","title":"Audio-visual event localization in unconstrained videos","authors":"Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, Chenliang Xu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2020] Weiyao Wang, Du Tran, and Matt Feis- zli. What makes training multi-modal classification net- works hard? In CVPR, pages 12695-12705, 2020.","title":"What makes training multi-modal classification networks hard?","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wu et al., 2022] Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. Character- izing and overcoming the greedy nature of learning in multi-modal deep neural networks. In ICML, pages 24043-24055.","title":"Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks","authors":"Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, Krzysztof J Geras","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"PMLR, 2022.","title":null,"authors":"Pmlr","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Xia and Zhao, 2022] Yan Xia and Zhou Zhao. Cross-modal background suppression for audio-visual event localiza- tion. In CVPR, pages 19989-19998, 2022.","title":"Cross-modal background suppression for audio-visual event localization","authors":"Yan Xia, Zhou Zhao","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Xiao et al., 2010] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pages 3485-3492, 2010.","title":"Sun database: Large-scale scene recognition from abbey to zoo","authors":"Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, Antonio Torralba","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Xiao et al., 2020] Fanyi Xiao, Yong Jae Lee, Kristen Grau- man, Jitendra Malik, and Christoph Feichtenhofer. Au- diovisual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020.","title":"Audiovisual slowfast networks for video recognition","authors":"Xiao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Xu et al., 2023a] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE TPAMI, 2023.","title":"Multimodal learning with transformers: A survey","authors":"Xu","year":"2023","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Xu et al., 2023b] Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning. In ICASSP, pages 1-5. IEEE, 2023.","title":"Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning","authors":"Xu","year":"2023","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Yang et al., 2021] Jianwei Yang, Yonatan Bisk, and Jian- feng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In ICCV, pages 11562-11572, 2021.","title":"Taco: Token-aware cascade contrastive learning for video-text alignment","authors":"Jianwei Yang, Yonatan Bisk, Jianfeng Gao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhao et al., 2018] Hang Zhao, Chuang Gan, Andrew Rou- ditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, pages 570-586, 2018.","title":"The sound of pixels","authors":"Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh Mcdermott, Antonio Torralba","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhao et al., 2021] Zengqun Zhao, Qingshan Liu, and Shan- min Wang. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Transactions on Image Processing, 30:6544- 6556, 2021.","title":"Learning deep global multi-scale and local attention features for facial expression recognition in the wild","authors":"Zhao","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhou et al., 2014] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. NeurIPS, 27, 2014.","title":"Learning deep features for scene recognition using places database","authors":"Zhou","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhou et al., 2021] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample prop- agation along the audio-visual event line. In CVPR, pages 8436-8444, 2021.","title":"Positive sample propagation along the audio-visual event line","authors":"Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, Meng Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}