{"type":"ScholarlyArticle","title":"Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration","authors":[{"name":"Piyush Singh Pasi","slug":"piyush-singh-pasi","given":"Piyush Singh","family":"Pasi","affiliation":"Indian Institute of Technology , Bombay"},{"name":"Karthikeya Battepati","slug":"karthikeya-battepati","given":"Karthikeya","family":"Battepati","affiliation":"Indian Institute of Technology , Bombay"},{"name":"Preethi Jyothi","slug":"preethi-jyothi","given":"Preethi","family":"Jyothi","affiliation":"Indian Institute of Technology , Bombay"},{"name":"Ganesh Ramakrishnan","slug":"ganesh-ramakrishnan","given":"Ganesh","family":"Ramakrishnan","affiliation":"Indian Institute of Technology , Bombay"},{"name":"Tanmay Mahapatra","slug":"tanmay-mahapatra","given":"Tanmay","family":"Mahapatra","affiliation":"CARE India Solutions for Sustainable Development {piyushsinghpasi ,"},{"name":"Manoj Singh","slug":"manoj-singh","given":"Manoj","family":"Singh","affiliation":"CARE India Solutions for Sustainable Development {piyushsinghpasi ,"}],"abstract":"The problem of audio-to-text alignment has seen significant amount of research using complete supervision during training. However, this is typically not in the context of long audio recordings wherein the text being queried does not appear verbatim within the audio file. This work is a collaboration with a non-governmental organization called CARE India that collects long audio health surveys from young mothers residing in rural parts of Bihar, India. Given a question drawn from a questionnaire that is used to guide these surveys, we aim to locate where the question is asked within a long audio recording. This is of great value to African and Asian organizations that would otherwise have to painstakingly go through long and noisy audio recordings to locate questions (and answers) of interest. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on the temporal ordering of sentences to learn speech embeddings that capture the semantics of the underlying spoken text. These learnt embeddings are used to retrieve the corresponding audio segment based on text queries at inference time. We empirically demonstrate the significant effectiveness (improvement in R-avg of about 3%) of our model over those obtained using text-based heuristics.  We also show how noisy ASR, generated using state-of-the-art ASR models for Indian languages, yields better results when used in place of speech. INDENT, trained only on Hindi data is able to cater to all languages supported by the (semantically) shared text space. We illustrate this empirically on 11 Indic languages.","doi":"10.24963/ijcai.2023/683","url":"https://ijcai.org/proceedings/2023/683","pdf":"https://ijcai.org/proceedings/2023/0683.pdf","markdown":"https://ijcai.org/proceedings/2023/683.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/683","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI for Good","track":"AI for Good","number":683,"pages":{"first":6156,"last":6164},"keywords":["AI for Good: Natural Language Processing","AI for Good: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"R@1 R@5 R@10 R-avg R@1 R@5 R@10 R-avg R@1 R@5 R@10 R-avg Hindi (hi) 19.8 38.5 51.6 36.7 21.1 46.5 61.8 43.1 40.9 69.0 79.4 63.1 Assamese (as) 17.7 36.0 50.3 34.7 16.4 39.4 54.3 36.7 33.8 62.8 74.6 57.1 Bengali (bn) 18.3 37.9 51.7 36.0 20.3 45.4 57.9 41.2 37.2 67.7 78.6 61.2 English (en) 21.5 43.2 55.9 40.2 16.9 40.4 53.7 37.0 37.2 64.5 78.0 59.9 Gujarati (gu) 19.2 38.5 50.9 36.2 20.0 45.1 60.3 41.8 38.7 68.6 78.8 62.0 Kannada (kn) 19.1 37.1 52.8 36.3 18.7 45.0 58.6 40.8 39.2 66.6 78.1 61.3 Malayalam (ml) 18.5 35.1 49.1 34.2 18.3 44.7 57.8 40.3 35.0 65.2 77.4 59.2 Marathi (mr) 19.0 36.2 52.3 35.8 18.7 44.8 59.6 41.0 38.4 65.7 77.0 60.4 Oriya (or) 20.3 39.6 52.4 37.4 18.9 44.4 58.4 40.5 39.0 67.0 78.0 61.4 Punjabi (pa) 19.5 37.9 50.5 36.0 20.3 45.7 58.5 41.5 39.2 67.4 78.3 61.6 Tamil (ta) 18.9 38.5 52.0 36.4 17.7 43.8 59.0 40.2 34.9 66.5 77.5 59.6 Telugu (te) 17.8 37.8 50.9 35.5 18.1 43.8 58.3 40.1 37.3 65.5 76.4 59","title":"@5 R@10 R-avg R@1 R@5 R@10","authors":null,"year":null,"venue":"R-avg R@1 R","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"7 Table 2: Test set results using models trained with Hindi questions and evaluated on questions translated in 11 different languages.","title":"7 Table 2: Test set results using models trained with Hindi questions and evaluated on questions translated in 11 different languages","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anguera et al., 2014] Xavier Anguera, Jordi Luque, and Ciro Gracia. Audio-to-text alignment for speech recogni- tion with very limited resources. In Fifteenth Annual Con- ference of the International Speech Communication Asso- ciation, 2014.","title":"Audio-to-text alignment for speech recognition with very limited resources","authors":"Xavier Anguera, Jordi Luque, Ciro Gracia","year":"2014","venue":"Fifteenth Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anne Hendricks et al., 2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natu- ral language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.","title":"Localizing moments in video with natural language","authors":"Anne Lisa, Oliver Hendricks, Eli Wang, Josef Shechtman, Trevor Sivic, Bryan Darrell, Russell","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ao et al., 2021] Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder- decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021.","title":"Speecht5: Unified-modal encoderdecoder pre-training for spoken language processing","authors":"Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baevski et al., 2020] Alexei Baevski, Yuhao Zhou, Abdel- rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations. Advances in neural information processing sys- tems, 33:12449-12460, 2020.","title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","authors":"Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael Auli","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bredin et al., 2020] Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. Pyannote. audio: neural building blocks for speaker diarization. In ICASSP 2020-2020 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pages 7124-7128. IEEE, 2020.","title":"Pyannote. audio: neural building blocks for speaker diarization","authors":"Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, Marie-Philippe Gill","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[CARE, 2023] CARE. Care india website. https://www. careindia.org/, 2023. Accessed: 2023-03-02.","title":"CARE. Care india website","authors":null,"year":"2023","venue":null,"doi":null,"url":"https://www.careindia.org/,2023","internal_url":null},{"position":9,"raw":"[Chadha et al., 2022] Harveen Singh Chadha, Anirudh Gupta, Priyanshi Shah, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, and Vivek Raghavan. Vakyansh: Asr toolkit for low resource indic languages. arXiv preprint arXiv:2203.16512, 2022.","title":"Vakyansh: Asr toolkit for low resource indic languages","authors":"Chadha","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2018] Yi-Chen Chen, Chia-Hao Shen, Sung- Feng Huang, and Hung-yi Lee. Towards unsupervised automatic speech recognition trained by unaligned speech and text only. arXiv preprint arXiv:1803.10952, 2018.","title":"Towards unsupervised automatic speech recognition trained by unaligned speech and text only","authors":"Yi-Chen Chen, Chia-Hao Shen, Sung-Feng Huang, Hung-Yi Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597- 1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2022] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. Maestro: Matched speech text rep- resentations through modality matching. arXiv preprint arXiv:2204.03409, 2022.","title":"Maestro: Matched speech text representations through modality matching","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chung et al., 2018] Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass. Unsupervised cross- modal alignment of speech and text embedding spaces. Advances in neural information processing systems, 31, 2018.","title":"Unsupervised crossmodal alignment of speech and text embedding spaces","authors":"Yu-An Chung, Wei-Hung Weng, Schrasing Tong, James Glass","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Conneau et al., 2020] Alexis Conneau, Alexei Baevski, Ro- nan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979, 2020.","title":"Unsupervised cross-lingual representation learning for speech recognition","authors":"Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael Auli","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Feng et al., 2022] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 878- 891, Dublin, Ireland, May 2022. Association for Compu- tational Linguistics.","title":"Languageagnostic BERT sentence embedding","authors":"Feng","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gao et al., 2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267-5275, 2017.","title":"Tall: Temporal activity localization via language query","authors":"Jiyang Gao, Chen Sun, Zhenheng Yang, Ram Nevatia","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gupta et al., 2021] Anirudh Gupta, Harveen Singh Chadha, Priyanshi Shah, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, and Vivek Raghavan. Clsril-23: cross lin- gual speech representations for indic languages. arXiv preprint arXiv:2107.07402, 2021.","title":"Clsril-23: cross lingual speech representations for indic languages","authors":"Anirudh Gupta, Harveen Singh Chadha, Priyanshi Shah, Neeraj Chhimwal, Ankur Dhuriya, Rishabh Gaur, Vivek Raghavan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Heinritz et al., 2022] Florian Heinritz, Gisela Will, and Raf- faela Gentile. Surveying illiterate individuals: Are audio files in computer-assisted self-interviews a useful support- ive tool? Migration Research in a Digitized World, page 101, 2022.","title":"Surveying illiterate individuals: Are audio files in computer-assisted self-interviews a useful supportive tool","authors":"Heinritz","year":"2022","venue":"Migration Research in a Digitized World","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.","title":"Gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Javed et al., 2022] Tahir Javed, Sumanth Doddapaneni, Ab- higyan Raman, Kaushal Santosh Bhogale, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra. Towards building asr systems for the next billion users. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10813-10821, 2022.","title":"Towards building asr systems for the next billion users","authors":"Tahir Javed, Sumanth Doddapaneni, Abhigyan Raman, Kaushal Santosh Bhogale, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M Khapra","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jones, 2003] Rachel Jones. Survey data collection using au- dio computer assisted self-interview. Western journal of nursing research, 25:349-58, 05 2003.","title":"Survey data collection using audio computer assisted self-interview","authors":"Rachel Jones","year":"2003","venue":"Western journal of nursing research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2018] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua. Cross-modal moment localization in videos. In Proceedings of the 26th ACM international conference on Multimedia, pages 843- 851, 2018.","title":"Cross-modal moment localization in videos","authors":"Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, Tat-Seng Chua","year":"2018","venue":"Proceedings of the 26th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[López and Luque, 2022] Fernando López and Jordi Luque. Iterative pseudo-forced alignment by acoustic ctc loss for self-supervised asr domain adaptation. arXiv preprint arXiv:2210.15226, 2022.","title":"Iterative pseudo-forced alignment by acoustic ctc loss for self-supervised asr domain adaptation","authors":"Fernando López, Jordi Luque","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mithun et al., 2019] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury. Weakly supervised video moment retrieval from text queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11592-11601, 2019.","title":"Weakly supervised video moment retrieval from text queries","authors":"Mithun","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ramesh et al., 2022] Gowtham Ramesh, Sumanth Dodda- paneni, Aravinth Bheemaraj, Mayank Jobanputra, Ragha- van AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. Samanantar: The Largest Pub- licly Available Parallel Corpora Collection for 11 Indic Languages. Transactions of the Association for Compu- tational Linguistics, 10:145-162, 02 2022.","title":"Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages","authors":"Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, A K Raghavan, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, J Mahalakshmi, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Anoop Vivek Raghavan, Pratyush Kunchukuttan, Mitesh Kumar, Khapra Shantadevi","year":"2022","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ravanelli et al., 2021] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lu- gosch, Cem Subakan, Nauman Dawalatabad, Abdelwa- hab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, Franc ¸ois Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: A general-purpose speech toolkit, 2021. arXiv:2106.04624.","title":"SpeechBrain: A general-purpose speech toolkit","authors":"Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, William Franc ¸ois Grondin, Hwidong Aris, Yan Na, Renato De Gao, Yoshua Mori, Bengio","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Reed et al., 2014] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy la- bels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.","title":"Training deep neural networks on noisy labels with bootstrapping","authors":"Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, Andrew Rabinovich","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Reichmann et al., 2010] William M Reichmann, Elena Losina, George R Seage III, Christian Arbelaez, Steven A Safren, Jeffrey N Katz, Adam Hetland, and Rochelle P Walensky. Does modality of survey administration im- pact data quality: audio computer assisted self interview (acasi) versus self-administered pen and paper? PloS one, 5(1):e8728, 2010.","title":"Does modality of survey administration impact data quality: audio computer assisted self interview (acasi) versus self-administered pen and paper?","authors":"Elena William M Reichmann, George R Losina, Iii Seage, Christian Arbelaez, Steven A Safren, Jeffrey N Katz, Adam Hetland, Rochelle P Walensky","year":"2010","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Synnaeve et al., 2019] Gabriel Synnaeve, Qiantong Xu, Ja- cob Kahn, Tatiana Likhomanenko, Edouard Grave, Vi- neel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end asr: from supervised to semi-supervised learning with modern architectures. arXiv preprint arXiv:1911.08460, 2019.","title":"End-to-end asr: from supervised to semi-supervised learning with modern architectures","authors":"Qiantong Gabriel Synnaeve, Jacob Xu, Tatiana Kahn, Edouard Likhomanenko, Grave, Anuroop Vineel Pratap, Vitaliy Sriram, Ronan Liptchinsky, Collobert","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wiseman, 2017] John Wiseman. Python WebRTC-VAD website. https://pypi.org/project/webrtcvad/, 2017. Ac- cessed: 2023-06-13.","title":"Python WebRTC-VAD website","authors":"John Wiseman, Wiseman","year":"2017","venue":null,"doi":null,"url":"https://pypi.org/project/webrtcvad/,2017.Ac-cessed:2023-06-13","internal_url":null},{"position":31,"raw":"[Xu et al., 2018] Huijuan Xu, Kun He, Leonid Sigal, Stan Sclaroff, and Kate Saenko. Text-to-clip video retrieval with early fusion and re-captioning. arXiv preprint arXiv:1804.05113, 2(6):7, 2018.","title":"Text-to-clip video retrieval with early fusion and re-captioning","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2022] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. The elements of temporal sentence grounding in videos: A survey and future directions. arXiv preprint arXiv:2201.08071, 2022.","title":"The elements of temporal sentence grounding in videos: A survey and future directions","authors":"Hao Zhang, Aixin Sun, Wei Jing, Joey Tianyi Zhou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}