{"type":"ScholarlyArticle","title":"Contextualized Speech Recognition: Rethinking Second-Pass Rescoring with Generative Large Language Models","authors":[{"name":"Yixuan Tang","slug":"yixuan-tang","given":"Yixuan","family":"Tang","affiliation":"Department of Computer Science , National University of Singapore"},{"name":"Anthony K. H. Tung","slug":"anthony-k-h-tung","given":"Anthony K.H:","family":"Tung","affiliation":"Department of Computer Science , National University of Singapore"}],"abstract":"Automatic Speech Recognition (ASR) systems have witnessed notable advancements in recent years. Contextualized ASR tasks require recognizing speech not as isolated utterances but within the broader context in which they occur. Conventional approaches often employ a second-pass paradigm to re-rank initial transcriptions, yet they risk propagating errors across candidate hypotheses, thereby compromising recognition precision. In this study, we introduce a novel framework that diverges from typical second-pass rescoring methods. Given n-best hypotheses, we leverage prompting with a large language model for contextualized second-pass generation. Besides pursuing higher accuracy, we aim to explore the performance boundaries without substantially altering the underlying pre-trained speech and language models. We investigate the effectiveness of the proposed paradigm through zero-shot prompting and strategic low-rank adaptation tuning. On the multi-accent spoken reading comprehension benchmark SQuAD-SRC, both prompting and fine-tuned models outperform the 1-best ASR hypothesis, achieving notable relative Word Error Rate (WER) improvements of 13.6% and 45.9%, respectively. The results suggest that the proposed approach enhances transcription accuracy and contextual understanding.","doi":"10.24963/ijcai.2024/716","url":"https://ijcai.org/proceedings/2024/716","pdf":"https://ijcai.org/proceedings/2024/0716.pdf","markdown":"https://ijcai.org/proceedings/2024/716.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/716","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":716,"pages":{"first":6478,"last":6485},"keywords":["Natural Language Processing: NLP: Speech"],"videos":[],"references":[{"position":1,"raw":"[Chang et al., 2021] Feng-Ju Chang, Jing Liu, Martin Rad- far, Athanasios Mouchtaris, Maurizio Omologo, Ariya Rastrow, and Siegfried Kunzmann. Context-aware trans- former transducer for speech recognition. In ASRU, pages 503-510. IEEE, 2021.","title":"Context-aware transformer transducer for speech recognition","authors":"Chang","year":"2021","venue":"ASRU","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2023] Shuo-Yiin Chang, Chao Zhang, Tara N. Sainath, Bo Li, and Trevor Strohman. Context-aware end- to-end ASR using self-attentive embedding and tensor fu- sion. In ICASSP, pages 1-5. IEEE, 2023.","title":"Context-aware endto-end ASR using self-attentive embedding and tensor fusion","authors":"Shuo-Yiin Chang, Chao Zhang, Tara N Sainath, Bo Li, Trevor Strohman","year":"2023","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2023] Zhehuai Chen, He Huang, Andrei An- drusenko, Oleksii Hrinchuk, Krishna C. Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Gins- burg. SALM: speech-augmented language model with in-context learning for speech recognition and translation. CoRR, abs/2310.09424, 2023.","title":"SALM: speech-augmented language model with in-context learning for speech recognition and translation","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2023] Xuandi Fu, Kanthashree Mysore Sathyen- dra, Ankur Gandhe, Jing Liu, Grant P. Strimel, Ross McGowan, and Athanasios Mouchtaris. Robust acoustic and semantic contextual biasing in neural transducers for speech recognition. In ICASSP, pages 1-5. IEEE, 2023.","title":"Robust acoustic and semantic contextual biasing in neural transducers for speech recognition","authors":"Fu","year":"2023","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2022] Minglun Han, Linhao Dong, Zhenlin Liang, Meng Cai, Shiyu Zhou, Zejun Ma, and Bo Xu. Improving end-to-end contextual speech recognition with fine-grained contextual knowledge selection. In ICASSP, pages 8532-8536. IEEE, 2022.","title":"Improving end-to-end contextual speech recognition with fine-grained contextual knowledge selection","authors":"Han","year":"2022","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hono et al., 2023] Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, and Kei Sawada. An integration of pre-trained speech and lan- guage models for end-to-end speech recognition. CoRR, abs/2312.03668, 2023.","title":"An integration of pre-trained speech and language models for end-to-end speech recognition","authors":"Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, Kei Sawada","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022.","title":"Lora: Low-rank adaptation of large language models","authors":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen","year":"2022","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2023] Kaixun Huang, Ao Zhang, Binbin Zhang, Tianyi Xu, Xingchen Song, and Lei Xie. Spike- triggered contextual biasing for end-to-end mandarin speech recognition. CoRR, abs/2310.04657, 2023.","title":"Spiketriggered contextual biasing for end-to-end mandarin speech recognition","authors":"Huang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825, 2023.","title":"Mistral 7b","authors":"Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego De Las, Florian Casas, Gianna Bressand, Guillaume Lengyel, Lucile Lample, Saulnier, Renard Lélio, Marie-Anne Lavaud, Pierre Lachaux, Teven Stock, Thibaut Le Scao, Thomas Lavril, Timothée Wang, William El Lacroix, Sayed","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le et al., 2021] Duc Le, Mahaveer Jain, Gil Keren, Suyoun Kim, Yangyang Shi, Jay Mahadeokar, Julian Chan, Yuan Shangguan, Christian Fuegen, Ozlem Kalinli, Yatharth Saraf, and Michael L. Seltzer. Contextualized streaming end-to-end speech recognition with trie-based deep bias- ing and shallow fusion. In Interspeech, pages 1772-1776. ISCA, 2021.","title":"Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion","authors":"Duc Le, Mahaveer Jain, Gil Keren, Suyoun Kim, Yangyang Shi, Jay Mahadeokar, Julian Chan, Yuan Shangguan, Christian Fuegen, Ozlem Kalinli, Yatharth Saraf, Michael L Seltzer","year":"2021","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2023] Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. Prompting large language models for zero-shot domain adaptation in speech recognition. CoRR, abs/2306.16007, 2023.","title":"Prompting large language models for zero-shot domain adaptation in speech recognition","authors":"Yuang Li, Yu Wu, Jinyu Li, Shujie Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2023] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9):195:1-195:35, 2023.","title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","authors":"Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, Graham Neubig","year":"2023","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ogawa et al., 2023] Atsunori Ogawa, Takafumi Moriya, Naoyuki Kamo, Naohiro Tawara, and Marc Delcroix. Iter- ative shallow fusion of backward language model for end- to-end speech recognition. In ICASSP, pages 1-5. IEEE, 2023.","title":"Iterative shallow fusion of backward language model for endto-end speech recognition","authors":"Ogawa","year":"2023","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Radford et al., 2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 28492-28518. PMLR, 2023.","title":"Robust speech recognition via large-scale weak supervision","authors":"Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, Ilya Sutskever","year":"2023","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Radhakrishnan et al., 2023] Srijith Radhakrishnan, Chao- Han Huck Yang, Sumeer Ahmad Khan, Rohit Kumar, Nar- sis A. Kiani, David Gomez-Cabrero, and Jesper Tegnér. Whispering llama: A cross-modal generative error correc- tion framework for speech recognition. In EMNLP, pages 10007-10016. Association for Computational Linguistics, 2023.","title":"Whispering llama: A cross-modal generative error correction framework for speech recognition","authors":"Srijith Radhakrishnan, Chao-Han Huck, Sumeer Yang, Rohit Ahmad Khan, Narsis A Kumar, David Kiani, Jesper Gomez-Cabrero, Tegnér","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sathyendra et al., 2022] Kanthashree Mysore Sathyendra, Thejaswi Muniyappa, Feng-Ju Chang, Jing Liu, Jinru Su, Grant P. Strimel, Athanasios Mouchtaris, and Siegfried Kunzmann. Contextual adapters for personalized speech recognition in neural transducers. In ICASSP, pages 8537- 8541. IEEE, 2022.","title":"Contextual adapters for personalized speech recognition in neural transducers","authors":"Sathyendra","year":"2022","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shenoy et al., 2021] Ashish Shenoy, Sravan Bodapati, Monica Sunkara, Srikanth Ronanki, and Katrin Kirchhoff. Adapting long context NLM for ASR rescoring in conver- sational agents. In Interspeech, pages 3246-3250. ISCA, 2021.","title":"Adapting long context NLM for ASR rescoring in conversational agents","authors":"Ashish Shenoy, Sravan Bodapati, Monica Sunkara, Srikanth Ronanki, Katrin Kirchhoff","year":"2021","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shivakumar et al., 2023] Prashanth Gurunath Shivakumar, Jari Kolehmainen, Yile Gu, Ankur Gandhe, Ariya Ras- trow, and Ivan Bulyko. Discriminative speech recogni- tion rescoring with pre-trained language models. arXiv preprint arXiv:2310.06248, 2023.","title":"Discriminative speech recognition rescoring with pre-trained language models","authors":"Gurunath Prashanth, Jari Shivakumar, Yile Kolehmainen, Ankur Gu, Ariya Gandhe, Ivan Rastrow, Bulyko","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shu et al., 2023] Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. Llasm: Large language and speech model. CoRR, abs/2308.15930, 2023.","title":"Llasm: Large language and speech model","authors":"Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, Yemin Shi","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shukor et al., 2023] Mustafa Shukor, Corentin Dancette, Alexandre Ramé, and Matthieu Cord. Unified model for image, video, audio and language tasks. CoRR, abs/2307.16184, 2023.","title":"Unified model for image, video, audio and language tasks","authors":"Mustafa Shukor, Corentin Dancette, Alexandre Ramé, Matthieu Cord","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2023] Chuanneng Sun, Zeeshan Ahmed, Yingyi Ma, Zhe Liu, Lucas Kabela, Yutong Pang, and Ozlem Kalinli. Contextual biasing of named-entities with large language models. CoRR, abs/2309.00723, 2023.","title":"Contextual biasing of named-entities with large language models","authors":"Chuanneng Sun, Zeeshan Ahmed, Yingyi Ma, Zhe Liu, Lucas Kabela, Yutong Pang, Ozlem Kalinli","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang and Tung, 2023] Yixuan Tang and Anthony K. H. Tung. Squad-src: A dataset for multi-accent spoken read- ing comprehension. In IJCAI, pages 5206-5214. ijcai.org, 2023.","title":"Squad-src: A dataset for multi-accent spoken reading comprehension","authors":"Yixuan Tang, Anthony K H Tung","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/578"},{"position":23,"raw":"[Wu et al., 2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any mul- timodal LLM. CoRR, abs/2309.05519, 2023.","title":"Next-gpt: Any-to-any multimodal LLM","authors":"Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2023] Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. Generative speech recognition error correction with large language models and task-activating prompting. pages 1- 8, 2023.","title":"Generative speech recognition error correction with large language models and task-activating prompting","authors":"Chao-Han Huck, Yile Yang, Yi-Chieh Gu, Shalini Liu, Ivan Ghosh, Andreas Bulyko, Stolcke","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2023] Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth Gurunath Shivakumar, Yile Gu, Sungho Ryu, Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, Tuan Dinh, Ankur Gandhe, Denis Filimonov, Shalini Ghosh, Andreas Stolcke, Ariya Rastrow, and Ivan Bulyko. Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition. CoRR, abs/2309.15223, 2023.","title":"Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition","authors":"Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Gurunath Prashanth, Yile Shivakumar, Sungho Gu, Roger Ryu, Qi Ren, Aditya Luo, I-Fan Gourav, Yi-Chieh Chen, Tuan Liu, Ankur Dinh, Denis Gandhe, Shalini Filimonov, Andreas Ghosh, Ariya Stolcke, Ivan Rastrow, Bulyko","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}