{"type":"ScholarlyArticle","title":"SQuAD-SRC: A Dataset for Multi-Accent Spoken Reading Comprehension","authors":[{"name":"Yixuan Tang","slug":"yixuan-tang","given":"Yixuan","family":"Tang","affiliation":"Department of Computer Science , National University of Singapore"},{"name":"Anthony K.H: Tung","slug":"anthony-k-h-tung","given":"Anthony K.H:","family":"Tung","affiliation":"Department of Computer Science , National University of Singapore"}],"abstract":"Spoken Reading Comprehension (SRC) is a challenging problem in spoken natural language retrieval, which automatically extracts the answer from the text-form contents according to the audio-form question. However, the existing spoken question answering approaches are mainly based on synthetically generated audio-form data, which may be ineffectively applied for multi-accent spoken question answering directly in many real-world applications. In this paper, we construct a large-scale multi-accent human spoken dataset SQuAD-SRC, in order to study the problem of multi-accent spoken reading comprehension. We choose 24 native English speakers from six different countries with various English accents and construct audio-form questions to the correspondent text-form contents by the chosen speakers. The dataset consists of 98,169 spoken question answering pairs and 20,963 passages from the popular machine reading comprehension dataset SQuAD. We present a statistical analysis of our SQuAD-SRC dataset and conduct extensive experiments on it by comparing cascaded SRC approaches and the enhanced end-to-end ones. Moreover, we explore various adaption strategies to improve the SRC performance, especially for multi-accent spoken questions.","doi":"10.24963/ijcai.2023/578","url":"https://ijcai.org/proceedings/2023/578","pdf":"https://ijcai.org/proceedings/2023/0578.pdf","markdown":"https://ijcai.org/proceedings/2023/578.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/578","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":578,"pages":{"first":5206,"last":5214},"keywords":["Natural Language Processing: NLP: Speech","Natural Language Processing: NLP: Question answering"],"videos":[],"references":[{"position":1,"raw":"[Abdelnour et al., 2018] Jérôme Abdelnour, Giampiero Salvi, and Jean Rouat. CLEAR: A dataset for composi- tional language and elementary acoustic reasoning. CoRR, abs/1811.10561, 2018.","title":"CLEAR: A dataset for compositional language and elementary acoustic reasoning","authors":"Jérôme Abdelnour, Giampiero Salvi, Jean Rouat","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ao et al., 2022] Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. In ACL (1), pages 5723- 5738. Association for Computational Linguistics, 2022.","title":"Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing","authors":"Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei","year":"2022","venue":"ACL (1)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baevski et al., 2020] Alexei Baevski, Yuhao Zhou, Abdel- rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations. In NeurIPS, 2020.","title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","authors":"Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael Auli","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Nuo Chen, Chenyu You, and Yuexian Zou. Self-supervised dialogue learning for spoken conver- sational question answering. In Interspeech, pages 231- 235. ISCA, 2021.","title":"Self-supervised dialogue learning for spoken conversational question answering","authors":"Nuo Chen, Chenyu You, Yuexian Zou","year":"2021","venue":"Interspeech","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chi et al., 2021] Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-Wen Li, and Hung-yi Lee. Audio albert: A lite bert for self- supervised learning of audio representation. In SLT, pages 344-350. IEEE, 2021.","title":"Audio albert: A lite bert for selfsupervised learning of audio representation","authors":"Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-Wen Li, Hung-Yi Lee","year":"2021","venue":"SLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chuang et al., 2020] Yung-Sung Chuang, Chi-Liang Liu, Hung-yi Lee, and Lin-Shan Lee. Speechbert: An audio- and-text jointly learned language model for end-to-end spoken question answering. In INTERSPEECH, pages 4168-4172. ISCA, 2020.","title":"Speechbert: An audioand-text jointly learned language model for end-to-end spoken question answering","authors":"Yung-Sung Chuang, Chi-Liang Liu, Hung-Yi Lee, Lin-Shan Lee","year":"2020","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171-4186. Association for Computational Linguistics, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fayek and Johnson, 2020] Haytham M. Fayek and Justin Johnson. Temporal reasoning via audio question answer- ing. IEEE ACM Trans. Audio Speech Lang. Process., 28:2283-2294, 2020.","title":"Temporal reasoning via audio question answering","authors":"M Haytham, Justin Fayek, Johnson","year":"2020","venue":"IEEE ACM Trans. Audio Speech Lang. Process","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goyal et al., 2019] Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. Int. J. Comput. Vis., 127(4):398-414, 2019.","title":"Making the V in VQA matter: Elevating the role of image understanding in visual question answering","authors":"Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, Devi Parikh","year":"2019","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Graves et al., 2005] Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Bidirectional LSTM networks for improved phoneme classification and recognition. In ICANN (2), volume 3697 of Lecture Notes in Computer Science, pages 799-804. Springer, 2005.","title":"Bidirectional LSTM networks for improved phoneme classification and recognition","authors":"Alex Graves, Santiago Fernández, Jürgen Schmidhuber","year":"2005","venue":"ICANN (2)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hoy, 2018] Matthew B Hoy. Alexa, siri, cortana, and more: an introduction to voice assistants. Medical reference ser- vices quarterly, 37(1):81-88, 2018.","title":"Alexa, siri, cortana, and more: an introduction to voice assistants","authors":"B Matthew, Hoy","year":"2018","venue":"Medical reference services quarterly","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2021] Zhiqi Huang, Fenglin Liu, Xian Wu, Shen Ge, Helin Wang, Wei Fan, and Yuexian Zou. Audio- oriented multimodal machine comprehension via dynamic inter-and intra-modality attention. In AAAI, pages 13098- 13106, 2021.","title":"Audiooriented multimodal machine comprehension via dynamic inter-and intra-modality attention","authors":"Huang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hudson and Manning, 2019] Drew A. Hudson and Christo- pher D. Manning. GQA: A new dataset for real-world vi- sual reasoning and compositional question answering. In CVPR, pages 6700-6709. Computer Vision Foundation / IEEE, 2019.","title":"GQA: A new dataset for real-world visual reasoning and compositional question answering","authors":"Drew A Hudson, Christopher D Manning","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kuo et al., 2020] Chia-Chih Kuo, Shang-Bao Luo, and Kuan-Yu Chen. An audio-enriched bert-based framework for spoken multiple-choice question answering. In INTER- SPEECH, pages 4173-4177. ISCA, 2020.","title":"An audio-enriched bert-based framework for spoken multiple-choice question answering","authors":"Chia-Chih Kuo, Shang-Bao Luo, Kuan-Yu Chen","year":"2020","venue":"INTER-SPEECH","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kuo et al., 2021] Chia-Chih Kuo, Kuan-Yu Chen, and Shang-Bao Luo. Audio-aware spoken multiple-choice question answering with pre-trained language models. IEEE ACM Trans. Audio Speech Lang. Process., 29:3170- 3179, 2021.","title":"Audio-aware spoken multiple-choice question answering with pre-trained language models","authors":"Chia-Chih Kuo, Kuan-Yu Chen, Shang-Bao Luo","year":"2021","venue":"IEEE ACM Trans. Audio Speech Lang. Process","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2018] Chia-Hsuan Lee, Shang-Ming Wang, Huan-Cheng Chang, and Hung-yi Lee. ODSQA: open- domain spoken question answering dataset. In SLT, pages 949-956, 2018.","title":"ODSQA: opendomain spoken question answering dataset","authors":"Chia-Hsuan Lee, Shang-Ming Wang, Huan-Cheng Chang, Hung-Yi Lee","year":"2018","venue":"SLT","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, and Hung-yi Lee. Spoken squad: A study of mitigating the impact of speech recognition errors on listening com- prehension. In INTERSPEECH, pages 3459-3463, 2018.","title":"Spoken squad: A study of mitigating the impact of speech recognition errors on listening comprehension","authors":"Chia-Hsuan Li, Szu-Lin Wu, Chi-Liang Liu, Hung-Yi Lee","year":"2018","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2022] Guan-Ting Lin, Yung-Sung Chuang, Ho- Lam Chung, Shu-Wen Yang, Hsuan-Jui Chen, Shuyan An- nie Dong, Shang-Wen Li, Abdelrahman Mohamed, Hung- yi Lee, and Lin-Shan Lee. DUAL: discrete spoken unit adaptive learning for textless spoken question answering. In INTERSPEECH, pages 5165-5169. ISCA, 2022.","title":"DUAL: discrete spoken unit adaptive learning for textless spoken question answering","authors":"Guan-Ting Lin, Yung-Sung Chuang, Ho-Lam Chung, Shu-Wen Yang, Hsuan-Jui Chen, Shuyan, Annie Dong, Shang-Wen Li, Abdelrahman Mohamed, Hungyi Lee, Lin-Shan Lee","year":"2022","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lipping et al., 2022] Samuel Lipping, Parthasaarathy Su- darsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-aqa: A crowdsourced dataset for audio question answering. In EUSIPCO, pages 1140-1144. IEEE, 2022.","title":"Clotho-aqa: A crowdsourced dataset for audio question answering","authors":"Lipping","year":"2022","venue":"EUSIPCO","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2016] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@NIPS, volume 1773 of CEUR Workshop Proceedings, 2016.","title":"MS MARCO: A human generated machine reading comprehension dataset","authors":"Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng","year":"2016","venue":"CEUR Workshop Proceedings","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. ACL, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. SQuAD: 100, 000+ questions for machine comprehension of text. In EMNLP, pages 2383-2392, 2016.","title":"SQuAD: 100, 000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Raux et al., 2005] Antoine Raux, Brian Langner, Dan Bo- hus, Alan W. Black, and Maxine Eskénazi. Let's go pub- lic! taking a spoken dialog system to the real world. In INTERSPEECH, pages 885-888. ISCA, 2005.","title":"Let's go public! taking a spoken dialog system to the real world","authors":"Antoine Raux, Brian Langner, Dan Bohus, Alan W Black, Maxine Eskénazi","year":"2005","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Seo et al., 2017] Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In ICLR (Poster). Open- Review.net, 2017.","title":"Bidirectional attention flow for machine comprehension","authors":"Seo","year":"2017","venue":"ICLR (Poster). Open-Review.net","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Su and Fung, 2020] Dan Su and Pascale Fung. Improving spoken question answering using contextualized word rep- resentation. In ICASSP, pages 8004-8008. IEEE, 2020.","title":"Improving spoken question answering using contextualized word representation","authors":"Fung ; ] Su, Dan Su, Pascale Fung","year":"2020","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Trischler et al., 2017] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehen- sion dataset. In Rep4NLP@ACL, pages 191-200, 2017.","title":"NewsQA: A machine comprehension dataset","authors":"Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, Kaheer Suleman","year":"2017","venue":"Rep4NLP@ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2017] Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching net- works for reading comprehension and question answering. In ACL, pages 189-198. Association for Computational Linguistics, 2017.","title":"Gated self-matching networks for reading comprehension and question answering","authors":"Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, Ming Zhou","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369-2380, 2018.","title":"HotpotQA: A dataset for diverse, explainable multi-hop question answering","authors":"Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, Christopher D Manning","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[You et al., 2021a] Chenyu You, Nuo Chen, and Yuexian Zou. Mrd-net: Multi-modal residual knowledge distilla- tion for spoken question answering. In IJCAI, pages 3985- 3991. ijcai.org, 2021.","title":"Mrd-net: Multi-modal residual knowledge distillation for spoken question answering","authors":"You","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/549"},{"position":30,"raw":"[You et al., 2021b] Chenyu You, Nuo Chen, and Yuexian Zou. Self-supervised contrastive cross-modality represen- tation learning for spoken question answering. In EMNLP (Findings), pages 28-39. Association for Computational Linguistics, 2021.","title":"Self-supervised contrastive cross-modality representation learning for spoken question answering","authors":"You","year":"2021","venue":"EMNLP (Findings)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[You et al., 2022] Chenyu You, Nuo Chen, Fenglin Liu, Shen Ge, Xian Wu, and Yuexian Zou. End-to-end spo- ken conversational question answering: Task, dataset and model. In NAACL-HLT (Findings), pages 1219-1232, 2022.","title":"End-to-end spoken conversational question answering: Task, dataset and model","authors":"You","year":"2022","venue":"NAACL-HLT (Findings)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}