{"type":"ScholarlyArticle","title":"Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition","authors":[{"name":"Yuchen Hu","slug":"yuchen-hu","given":"Yuchen","family":"Hu","affiliation":"Nanyang Technological University , Singapore"},{"name":"Ruizhe Li","slug":"ruizhe-li","given":"Ruizhe","family":"Li","affiliation":"University of Aberdeen , UK"},{"name":"Chen Chen","slug":"chen-chen","given":"Chen","family":"Chen","affiliation":"Nanyang Technological University , Singapore"},{"name":"Heqing Zou","slug":"heqing-zou","given":"Heqing","family":"Zou","affiliation":"Nanyang Technological University , Singapore"},{"name":"Qiushi Zhu","slug":"qiushi-zhu","given":"Qiushi","family":"Zhu","affiliation":"University of Science and Technology of China , China"},{"name":"Eng Siong Chng","slug":"eng-siong-chng","given":"Eng Siong","family":"Chng","affiliation":"Nanyang Technological University , Singapore"}],"abstract":"Audio-visual speech recognition (AVSR) research has gained a great success recently by improving the noise-robustness of audio-only automatic speech recognition (ASR) with noise-invariant visual information. However, most existing AVSR approaches simply fuse the audio and visual features by concatenation, without explicit interactions to capture the deep correlations between them, which results in sub-optimal multimodal representations for downstream speech recognition task. In this paper, we propose a cross-modal global interaction and local alignment (GILA) approach for AVSR, which captures the deep audio-visual (A-V) correlations from both global and local perspectives. Specifically, we design a global interaction model to capture the A-V complementary relationship on modality level, as well as a local alignment approach to model the A-V temporal consistency on frame level. Such a holistic view of cross-modal correlations enable better multimodal representations for AVSR. Experiments on public benchmarks LRS3 and LRS2 show that our GILA outperforms the supervised learning state-of-the-art. Code is at https://github.com/YUCHEN005/GILA.","doi":"10.24963/ijcai.2023/564","url":"https://ijcai.org/proceedings/2023/564","pdf":"https://ijcai.org/proceedings/2023/0564.pdf","markdown":"https://ijcai.org/proceedings/2023/564.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/564","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":564,"pages":{"first":5076,"last":5084},"keywords":["Natural Language Processing: NLP: Speech"],"videos":[],"references":[{"position":1,"raw":"[Afouras et al., 2018a] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zis- serman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 2018.","title":"Deep audio-visual speech recognition","authors":"Afouras","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Afouras et al., 2018b] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: a large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496, 2018.","title":"Lrs3-ted: a large-scale dataset for visual speech recognition","authors":"Afouras","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baevski et al., 2019] Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Con- ference on Learning Representations, 2019.","title":"vq-wav2vec: Self-supervised learning of discrete speech representations","authors":"Alexei Baevski, Steffen Schneider, Michael Auli","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baevski et al., 2020] Alexei Baevski, Yuhao Zhou, Abdel- rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations. Advances in Neural Information Processing Systems, 33:12449-12460, 2020.","title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","authors":"Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael Auli","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Biswas et al., 2016] Astik Biswas, Prakash Kumar Sahu, and Mahesh Chandra. Multiple cameras audio visual speech recognition using active appearance model visual features in car environment. International Journal of Speech Technology, 19(1):159-171, 2016.","title":"Multiple cameras audio visual speech recognition using active appearance model visual features in car environment","authors":"Biswas","year":"2016","venue":"International Journal of Speech Technology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chopra et al., 2005] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages 539-546. IEEE, 2005.","title":"Learning a similarity metric discriminatively, with application to face verification","authors":"Sumit Chopra, Raia Hadsell, Yann Lecun","year":"2005","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chung et al., 2017] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pages 3444-3453. IEEE, 2017.","title":"Lip reading sentences in the wild","authors":"Son Joon, Andrew Chung, Oriol Senior, Andrew Vinyals, Zisserman","year":"2017","venue":"IEEE conference on computer vision and pattern recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goncalves and Busso, 2022] Lucas Goncalves and Carlos Busso. Auxformer: Robust approach to audiovisual emo- tion recognition. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 7357-7361. IEEE, 2022.","title":"Auxformer: Robust approach to audiovisual emotion recognition","authors":"Busso Goncalves, Lucas Goncalves, Carlos Busso","year":"2022","venue":"ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Graves, 2012] Alex Graves. Sequence transduction with re- current neural networks. arXiv preprint arXiv:1211.3711, 2012.","title":"Sequence transduction with recurrent neural networks","authors":"Alex Graves, Graves","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2021] Chunbin Gu, Jiajun Bu, Zhen Zhang, Zhi Yu, Dongfang Ma, and Wei Wang. Image search with text feedback by deep hierarchical attention mutual informa- tion maximization. In Proceedings of the 29th ACM In- ternational Conference on Multimedia, pages 4600-4609, 2021.","title":"Image search with text feedback by deep hierarchical attention mutual information maximization","authors":"Gu","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gulati et al., 2020] Anmol Gulati, James Qin, Chiu Chung- Cheng, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented transformer for speech recognition. In Interspeech, pages 5036-5040, 2020.","title":"Conformer: Convolution-augmented transformer for speech recognition","authors":"Anmol Gulati, James Qin, Chiu Chung-Cheng, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, Ruoming Pang","year":"2020","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hori et al., 2017] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. Attention-based multimodal fusion for video description. In Proceedings of the IEEE international conference on computer vision, pages 4193- 4202, 2017.","title":"Attention-based multimodal fusion for video description","authors":"Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, Kazuhiko Sumi","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hsu and Shi, 2022] Wei-Ning Hsu and Bowen Shi. u- hubert: Unified mixed-modal speech pretraining and zero- shot transfer to unlabeled modality. In Advances in Neural Information Processing Systems, 2022.","title":"uhubert: Unified mixed-modal speech pretraining and zeroshot transfer to unlabeled modality","authors":"Wei-Ning Hsu, Bowen Shi","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 448-456. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim et al., 2022] Minsu Kim, Jeong Hun Yeo, and Yong Man Ro. Distinguishing homophenes using multi- head visual-audio memory for lip reading. In Proceed- ings of the 36th AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, volume 22, 2022.","title":"Distinguishing homophenes using multihead visual-audio memory for lip reading","authors":"Minsu Kim, Jeong Hun Yeo, Yong Man Ro","year":"2022","venue":"Proceedings of the 36th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Koguchi et al., 2018] Yuto Koguchi, Kazuya Oharada, Yuki Takagi, Yoshiki Sawada, Buntarou Shizuki, and Shin Takahashi. A mobile command input through vowel lip shape recognition. In International Conference on Human- Computer Interaction, pages 297-305. Springer, 2018.","title":"A mobile command input through vowel lip shape recognition","authors":"Yuto Koguchi, Kazuya Oharada, Yuki Takagi, Yoshiki Sawada, Buntarou Shizuki, Shin Takahashi","year":"2018","venue":"International Conference on Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Korbar et al., 2018] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018.","title":"Cooperative learning of audio and video models from self-supervised synchronization","authors":"Bruno Korbar, Du Tran, Lorenzo Torresani","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lee et al., 2020] Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross-attentional audio-visual fusion for weakly-supervised action localization. In Inter- national Conference on Learning Representations, 2020.","title":"Cross-attentional audio-visual fusion for weakly-supervised action localization","authors":"Lee","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2021] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distilla- tion. Advances in neural information processing systems, 34:9694-9705, 2021.","title":"Align before fuse: Vision and language representation learning with momentum distillation","authors":"Li","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lv et al., 2021] Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. Progressive modality re- inforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2554-2562, 2021.","title":"Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences","authors":"Lv","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ma et al., 2021] Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7613-7617. IEEE, 2021.","title":"End-to-end audio-visual speech recognition with conformers","authors":"Ma","year":"2021","venue":"ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Makino et al., 2019] Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan. Recurrent neural network transducer for audio-visual speech recognition. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pages 905-912. IEEE, 2019.","title":"Recurrent neural network transducer for audio-visual speech recognition","authors":"Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, Olivier Siohan","year":"2019","venue":"IEEE automatic speech recognition and understanding workshop (ASRU)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mercea et al., 2022] Otniel-Bogdan Mercea, Lukas Riesch, A Koepke, and Zeynep Akata. Audio-visual generalised zero-shot learning with cross-modal attention and lan- guage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10553- 10563, 2022.","title":"Audio-visual generalised zero-shot learning with cross-modal attention and language","authors":"Otniel-Bogdan, Lukas Mercea, Riesch, Zeynep Koepke, Akata","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Morgado et al., 2021] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12486, 2021.","title":"Audio-visual instance discrimination with cross-modal agreement","authors":"Pedro Morgado, Nuno Vasconcelos, Ishan Misra","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pan et al., 2022] Xichen Pan, Peiyu Chen, Yichen Gong, Helong Zhou, Xinbing Wang, and Zhouhan Lin. Lever- aging unimodal self-supervised learning for multimodal audio-visual speech recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4491-4503, Dublin, Ireland, May 2022. Association for Computational Linguistics.","title":"Leveraging unimodal self-supervised learning for multimodal audio-visual speech recognition","authors":"Pan","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Petridis et al., 2018] Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pan- tic. Audio-visual speech recognition with a hybrid ctc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 513-520. IEEE, 2018.","title":"Audio-visual speech recognition with a hybrid ctc/attention architecture","authors":"Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, Maja Pantic","year":"2018","venue":"IEEE Spoken Language Technology Workshop (SLT)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Raij et al., 2000] Tommi Raij, Kimmo Uutela, and Riitta Hari. Audiovisual integration of letters in the human brain. Neuron, 28(2):617-625, 2000.","title":"Audiovisual integration of letters in the human brain","authors":"Raij","year":"2000","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shi et al., 2022a] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio- visual speech representation by masked multimodal clus- ter prediction. In International Conference on Learning Representations, 2022.","title":"Learning audiovisual speech representation by masked multimodal cluster prediction","authors":"Shi","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Shi et al., 2022b] Bowen Shi, Wei-Ning Hsu, and Abdel- rahman Mohamed. Robust self-supervised audio-visual speech recognition. arXiv preprint arXiv:2201.01763, 2022.","title":"Robust self-supervised audio-visual speech recognition","authors":"Shi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Snyder et al., 2015] David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise cor- pus. arXiv preprint arXiv:1510.08484, 2015.","title":"Musan: A music, speech, and noise corpus","authors":"David Snyder, Guoguo Chen, Daniel Povey","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sumby and Pollack, 1954] William H Sumby and Irwin Pollack. Visual contribution to speech intelligibility in noise. The journal of the acoustical society of america, 26(2):212-215, 1954.","title":"Visual contribution to speech intelligibility in noise","authors":"Pollack ; Sumby, H William, Irwin Sumby, Pollack","year":"1954","venue":"The journal of the acoustical society of america","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tao and Busso, 2018] Fei Tao and Carlos Busso. Gating neural network for large vocabulary audiovisual speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(7):1290-1302, 2018.","title":"Gating neural network for large vocabulary audiovisual speech recognition","authors":"Busso ; ] Fei Tao, Carlos Tao, Busso","year":"2018","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for un- aligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguis- tics. Meeting, volume 2019, page 6558. NIH Public Ac- cess, 2019.","title":"Multimodal transformer for unaligned multimodal language sequences","authors":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2019","venue":"Proceedings of the conference. Association for Computational Linguistics. Meeting","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2020] Zhong-Qiu Wang, Peidong Wang, and DeLiang Wang. Complex spectral mapping for single- and multi-channel speech enhancement and robust asr. IEEE/ACM transactions on audio, speech, and language processing, 28:1778-1787, 2020.","title":"Complex spectral mapping for singleand multi-channel speech enhancement and robust asr","authors":"Zhong-Qiu Wang, Peidong Wang, Deliang Wang","year":"2020","venue":"IEEE/ACM transactions on audio, speech, and language processing","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2022] Daheng Wang, Tong Zhao, Wenhao Yu, Nitesh V Chawla, and Meng Jiang. Deep multimodal com- plementarity learning. IEEE Transactions on Neural Net- works and Learning Systems, 2022.","title":"Deep multimodal complementarity learning","authors":"Daheng Wang, Tong Zhao, Wenhao Yu, Nitesh V Chawla, Meng Jiang","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Watanabe et al., 2017] Shinji Watanabe, Takaaki Hori, Suy- oun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recogni- tion. IEEE Journal of Selected Topics in Signal Process- ing, 11(8):1240-1253, 2017.","title":"Hybrid ctc/attention architecture for end-to-end speech recognition","authors":"Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, Tomoki Hayashi","year":"2017","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xu et al., 2020] Bo Xu, Cheng Lu, Yandong Guo, and Ja- cob Wang. Discriminative multi-modality speech recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14433- 14442, 2020.","title":"Discriminative multi-modality speech recognition","authors":"Bo Xu, Cheng Lu, Yandong Guo, Jacob Wang","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yang et al., 2022] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre- training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15671-15680, 2022.","title":"Vision-language pretraining with triple contrastive learning","authors":"Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, Junzhou Huang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yu et al., 2020] Jianwei Yu, Shi-Xiong Zhang, Jian Wu, Shahram Ghorbani, Bo Wu, Shiyin Kang, Shansong Liu, Xunying Liu, Helen Meng, and Dong Yu. Audio-visual recognition of overlapped speech for the lrs2 dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6984-6988. IEEE, 2020.","title":"Audio-visual recognition of overlapped speech for the lrs2 dataset","authors":"Jianwei Yu, Shi-Xiong Zhang, Jian Wu, Shahram Ghorbani, Bo Wu, Shiyin Kang, Shansong Liu, Xunying Liu, Helen Meng, Dong Yu","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}