{"type":"ScholarlyArticle","title":"EAT: Self-Supervised Pre-Training with Efficient Audio Transformer","authors":[{"name":"Wenxi Chen","slug":"wenxi-chen","given":"Wenxi","family":"Chen","affiliation":"X-LANCE Lab , Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Yuzhe Liang","slug":"yuzhe-liang","given":"Yuzhe","family":"Liang","affiliation":"X-LANCE Lab , Department of Computer Science and Engineering , Shanghai Jiao Tong University , Shanghai , China"},{"name":"Ziyang Ma","slug":"ziyang-ma","given":"Ziyang","family":"Ma","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute ,"},{"name":"Zhisheng Zheng","slug":"zhisheng-zheng","given":"Zhisheng","family":"Zheng","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute ,"},{"name":"Xie Chen","slug":"xie-chen","given":"Xie","family":"Chen","affiliation":"MoE Key Lab of Artificial Intelligence , AI Institute ,"}],"abstract":"Audio self-supervised learning (SSL) pre-training, which aims to learn good representations from unlabeled audio, has made remarkable progress. However, the extensive computational demands during pre-training pose a significant barrier to the potential application and optimization of audio SSL models. In this paper, inspired by the success of data2vec 2.0 in image modality and Audio-MAE in audio modality, we introduce Efficient Audio Transformer (EAT) to further improve the effectiveness and efficiency in audio SSL. The proposed EAT adopts the bootstrap self-supervised training paradigm to the audio domain. A novel Utterance-Frame Objective (UFO) is designed to enhance the modeling capability of acoustic events. Furthermore, we reveal that the masking strategy is critical in audio SSL pre-training, and superior audio representations can be obtained with large inverse block masks. Experiment results demonstrate that EAT achieves state-of-the-art (SOTA) performance on a range of audio-related tasks, including AudioSet (AS-2M, AS-20K), ESC-50, and SPC-2, along with a significant pre-training speedup up to ~15x compared to existing audio SSL models.","doi":"10.24963/ijcai.2024/421","url":"https://ijcai.org/proceedings/2024/421","pdf":"https://ijcai.org/proceedings/2024/0421.pdf","markdown":"https://ijcai.org/proceedings/2024/421.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/421","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":421,"pages":{"first":3807,"last":3815},"keywords":["Machine Learning: ML: Self-supervised Learning","Natural Language Processing: NLP: Speech","Machine Learning: ML: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baade et al., 2022] Alan Baade, Puyuan Peng, and David Harwath. MAE-AST: Masked autoencoding audio spec- trogram Transformer. arXiv preprint arXiv:2203.16691, 2022.","title":"MAE-AST: Masked autoencoding audio spectrogram Transformer","authors":"Alan Baade, Puyuan Peng, David Harwath","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baevski et al., 2020] Alexei Baevski, Yuhao Zhou, Abdel- rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations. Proc. NeurIPS, 2020.","title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","authors":"Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael Auli","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baevski et al., 2022] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In Proc. ICML, 2022.","title":"Data2vec: A general framework for self-supervised learning in speech, vision and language","authors":"Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli","year":"2022","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baevski et al., 2023] Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learn- ing with contextualized target representations for vision, speech and language. In Proc. ICML, 2023.","title":"Efficient self-supervised learning with contextualized target representations for vision, speech and language","authors":"Alexei Baevski, Arun Babu, Wei-Ning Hsu, Michael Auli","year":"2023","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, et al. Emerging properties in self- supervised vision Transformers. In Proc. ICCV, 2021.","title":"Emerging properties in selfsupervised vision Transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou","year":"2021","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Ex- ploring simple siamese representation learning. In Proc. CVPR, 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597- 1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2021] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision Transformers. In Proc. ICCV, 2021.","title":"An empirical study of training self-supervised vision Transformers","authors":"Chen","year":"2021","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2022a] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. HTS- AT: A hierarchical token-semantic audio Transformer for sound classification and detection. In Proc. ICASSP. IEEE, 2022.","title":"HTS-AT: A hierarchical token-semantic audio Transformer for sound classification and detection","authors":"Chen","year":"2022","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2022b] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing. In Proc. JSTSP, 2022.","title":"WavLM: Large-scale self-supervised pre-training for full stack speech processing","authors":"Chen","year":"2022","venue":"Proc. JSTSP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2022c] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. Proc. ICML, 2022.","title":"BEATs: Audio pre-training with acoustic tokenizers","authors":"Chen","year":"2022","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chong et al., 2023] Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng. Masked spectrogram pre- diction for self-supervised audio pre-training. In Proc. ICASSP. IEEE, 2023.","title":"Masked spectrogram prediction for self-supervised audio pre-training","authors":"Chong","year":"2023","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"BERT: Pre-training of deep bidirectional Transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Dinkel et al., 2024] Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Junbo Zhang, and Yujun Wang. CED: Con- sistent ensemble distillation for audio tagging. In Proc. ICASSP, 2024.","title":"CED: Consistent ensemble distillation for audio tagging","authors":"Heinrich Dinkel, Yongqing Wang, Zhiyong Yan, Junbo Zhang, Yujun Wang","year":"2024","venue":"Proc. ICASSP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gemmeke et al., 2017] Jort Gemmeke, Daniel Ellis, Dylan Freedman, Aren Jansen, et al. Audio set: An ontology and human-labeled dataset for audio events. In Proc. ICASSP. IEEE, 2017.","title":"Audio set: An ontology and human-labeled dataset for audio events","authors":"Jort Gemmeke, Daniel Ellis, Dylan Freedman, Aren Jansen","year":"2017","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gong et al., 2021a] Yuan Gong, Yu-An Chung, and James Glass. AST: Audio spectrogram Transformer. arXiv preprint arXiv:2104.01778, 2021.","title":"AST: Audio spectrogram Transformer","authors":"Gong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gong et al., 2021b] Yuan Gong, Yu-An Chung, and James Glass. PSLA: Improving audio tagging with pretrain- ing, sampling, labeling, and aggregation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3292-3306, 2021.","title":"PSLA: Improving audio tagging with pretraining, sampling, labeling, and aggregation","authors":"Gong","year":"2021","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gong et al., 2022] Yuan Gong, Cheng Lai, Yu-An Chung, and James Glass. SSAST: Self-supervised audio spectro- gram Transformer. In Proc. AAAI, 2022.","title":"SSAST: Self-supervised audio spectrogram Transformer","authors":"Yuan Gong, Cheng Lai, Yu-An Chung, James Glass","year":"2022","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, et al. Bootstrap your own latent-a new approach to self-supervised learning. Proc. NeurIPS, 2020.","title":"Bootstrap your own latent-a new approach to self-supervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Guzhov et al., 2022] Andrey Guzhov, Federico Raue, Jorn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In Proc. ICASSP. IEEE, 2022.","title":"Audioclip: Extending clip to image, text and audio","authors":"Andrey Guzhov, Federico Raue, Jorn Hees, Andreas Dengel","year":"2022","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proc. CVPR, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proc. CVPR, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, Ross Girshick","year":"2022","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.","title":"Gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Hsu et al., 2021] Wei-Ning Hsu, Benjamin Bolte, Yao- Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. HuBERT: Self- supervised speech representation learning by masked pre- diction of hidden units. In Proc. TASLP, 2021.","title":"HuBERT: Selfsupervised speech representation learning by masked prediction of hidden units","authors":"Wei-Ning Hsu, Benjamin Bolte, Hubert Yao-Hung, Kushal Tsai, Ruslan Lakhotia, Abdelrahman Salakhutdinov, Mohamed","year":"2021","venue":"Proc. TASLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Proc. ECCV. Springer, 2016.","title":"Deep networks with stochastic depth","authors":"Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q Weinberger","year":"2016","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Huang et al., 2022] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, et al. Masked autoencoders that listen. In Proc. NeurIPS, 2022.","title":"Masked autoencoders that listen","authors":"Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski","year":"2022","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Kong et al., 2020] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark Plumbley. PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.","title":"PANNs: Large-scale pretrained audio neural networks for audio pattern recognition","authors":"Kong","year":"2020","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Koutini et al., 2021] Khaled Koutini, Jan Schlüter, Hamid Eghbal-Zadeh, and Gerhard Widmer. Efficient training of audio Transformers with patchout. arXiv preprint arXiv:2110.05069, 2021.","title":"Efficient training of audio Transformers with patchout","authors":"Koutini","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Li and Li, 2022] Xian Li and Xiaofei Li. ATST: Audio representation learning with teacher-student Transformer. arXiv preprint arXiv:2204.12076, 2022.","title":"ATST: Audio representation learning with teacher-student Transformer","authors":"Xian Li, Xiaofei Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Li et al., 2023] Xian Li, Nian Shao, and Xiaofei Li. Self-supervised audio teacher-student Transformer for both clip-level and frame-level tasks. arXiv preprint arXiv:2306.04186, 2023.","title":"Self-supervised audio teacher-student Transformer for both clip-level and frame-level tasks","authors":"Xian Li, Nian Shao, Xiaofei Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Lillicrap et al., 2015] Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, et al. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.","title":"SGDR: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov, Ilya Loshchilov, Frank Hutter","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ma et al., 2023] Ziyang Ma, Zhisheng Zheng, Changli Tang, Yujin Wang, and Xie Chen. MT4SSL: Boosting self-supervised speech representation learning by integrat- ing multiple targets. In Proc. Interspeech, 2023.","title":"MT4SSL: Boosting self-supervised speech representation learning by integrating multiple targets","authors":"Ma","year":"2023","venue":"Proc. Interspeech","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Nagrani et al., 2021] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. At- tention bottlenecks for multimodal fusion. Proc. NeurIPS, 2021.","title":"Attention bottlenecks for multimodal fusion","authors":"Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun","year":"2021","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Niizumi et al., 2021] Daisuke Niizumi, Daiki Takeuchi, Ya- sunori Ohishi, Noboru Harada, and Kunio Kashino. BYOL for audio: Self-supervised learning for general-purpose audio representation. In Proc. IJCNN. IEEE, 2021.","title":"BYOL for audio: Self-supervised learning for general-purpose audio representation","authors":"Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino","year":"2021","venue":"Proc. IJCNN. IEEE","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Niizumi et al., 2022] Daisuke Niizumi, Daiki Takeuchi, Ya- sunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoen- coders for learning general-purpose audio representation. In HEAR: Holistic Evaluation of Audio Representations, pages 1-24. PMLR, 2022.","title":"Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation","authors":"Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino","year":"2022","venue":"HEAR: Holistic Evaluation of Audio Representations","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Niizumi et al., 2023] Daisuke Niizumi, Daiki Takeuchi, Ya- sunori Ohishi, Noboru Harada, and Kunio Kashino. Masked modeling duo: Learning representations by en- couraging both networks to model the input. In Proc. ICASSP. IEEE, 2023.","title":"Masked modeling duo: Learning representations by encouraging both networks to model the input","authors":"Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, Kunio Kashino","year":"2023","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Ott et al., 2019] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. Fairseq: A fast, extensible toolkit for se- quence modeling. arXiv preprint arXiv:1904.01038, 2019.","title":"Fairseq: A fast, extensible toolkit for sequence modeling","authors":"Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Park et al., 2019] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779, 2019.","title":"Specaugment: A simple data augmentation method for automatic speech recognition","authors":"William Daniel S Park, Yu Chan, Chung-Cheng Zhang, Barret Chiu, Ekin D Zoph, Quoc V Cubuk, Le","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Piczak, 2015] Karol J Piczak. ESC: Dataset for environ- mental sound classification. In Proc. ACM MM, 2015.","title":"ESC: Dataset for environmental sound classification","authors":"J Karol, Piczak","year":"2015","venue":"Proc. ACM MM","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Srivastava et al., 2022] Sangeeta Srivastava, Yun Wang, Andros Tjandra, Anurag Kumar, Chunxi Liu, Kritika Singh, and Yatharth Saraf. Conformer-based self- supervised learning for non-speech audio tasks. In Proc. ICASSP. IEEE, 2022.","title":"Conformer-based selfsupervised learning for non-speech audio tasks","authors":"Sangeeta Srivastava, Yun Wang, Andros Tjandra, Anurag Kumar, Chunxi Liu, Kritika Singh, Yatharth Saraf","year":"2022","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Warden, 2018] Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.","title":"Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition","authors":"Warden","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Wu et al., 2022] Ho-Hsiang Wu, Prem Seetharaman, Kun- dan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In Proc. ICASSP. IEEE, 2022.","title":"Wav2clip: Learning robust audio representations from clip","authors":"Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, Juan Pablo Bello","year":"2022","venue":"Proc. ICASSP. IEEE","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yang et al., 2021] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng Jeff Lai, et al. Superb: Speech pro- cessing universal performance benchmark. arXiv preprint arXiv:2105.01051, 2021.","title":"Superb: Speech processing universal performance benchmark","authors":"Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Jeff Cheng, Lai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 2017] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Be- yond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, David Yann N Dauphin, Lopez-Paz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}