{"type":"ScholarlyArticle","title":"End-to-End Signal Factorization for Speech: Identity, Content, and Style","authors":[{"name":"Jennifer Williams","slug":"jennifer-williams","given":"Jennifer","family":"Williams","affiliation":"Centre for Speech Technology Research , The University of Edinburgh , United Kingdom"}],"abstract":"Preliminary experiments in this dissertation show that it is possible to factorize specific types of information from the speech signal in an abstract embedding space using machine learning. This information includes characteristics of the recording environment, speaking style, and speech quality. Based on these findings, a new technique is proposed to factorize multiple types of information from the speech signal simultaneously using a combination of state-of-the-art machine learning methods for speech processing. Successful speech signal factorization will lead to advances across many speech technologies, including improved speaker identification, detection of speech audio deep fakes, and controllable expression in speech synthesis.","doi":"10.24963/ijcai.2020/746","url":"https://ijcai.org/proceedings/2020/746","pdf":"https://ijcai.org/proceedings/2020/0746.pdf","markdown":"https://ijcai.org/proceedings/2020/746.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/746","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":746,"pages":{"first":5212,"last":5213},"keywords":["Natural Language Processing: Speech","Machine Learning: Deep Learning","Machine Learning: Multi-instance;Multi-label;Multi-view learning","Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24878/file_24878.m3u8","length_sec":285,"provider":"hls"}],"references":[{"position":1,"raw":"[Ding and Gutierrez-Osuna, 2019] Shaojin Ding and Ri- cardo Gutierrez-Osuna. Group Latent Embedding for Vector Quantized Variational Autoencoder in Non-Parallel Voice Conversion. In Proc. Interspeech 2019, pages 724- 728, 2019.","title":"Shaojin Ding and Ricardo Gutierrez-Osuna. Group Latent Embedding for Vector Quantized Variational Autoencoder in Non-Parallel Voice Conversion","authors":"Gutierrez-Osuna Ding","year":"2019","venue":"Proc. Interspeech 2019","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning Deep Represen- tations by Mutual Information Estimation and Maximiza- tion. arXiv preprint arXiv:1808.06670, 2018.","title":"Learning Deep Representations by Mutual Information Estimation and Maximization","authors":"Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Kalchbrenner et al., 2018] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Ed- ward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient Neural Au- dio Synthesis. In International Conference on Machine Learning, pages 2410-2419, 2018.","title":"Efficient Neural Audio Synthesis","authors":"Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, Koray Kavukcuoglu","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Raj et al., 2019] Desh Raj, David Snyder, Daniel Povey, and Sanjeev Khudanpur. Probing the information encoded in x-vectors. In IEEE Automatic Speech Recognition and Un- derstanding Workshop, ASRU 2019, Singapore, December 14-18, 2019, pages 726-733. IEEE, 2019.","title":"Probing the information encoded in x-vectors","authors":"Desh Raj, David Snyder, Daniel Povey, Sanjeev Khudanpur","year":"2019","venue":"IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[van den Oord et al., 2017] Aäaron van den Oord, Oriol Vinyals, et al. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, pages 6306-6315, 2017.","title":"Neural Discrete Representation Learning","authors":"Oord Van Den","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Wang et al., 2019] Xin Wang, Shinji Takaki, Junichi Yam- agishi, Simon King, and Keiichi Tokuda. A Vector Quan- tized Variational Autoencoder (VQ-VAE) Autoregressive Neural F0 Model for Statistical Parametric Speech Syn- thesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:157-170, 2019.","title":"A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural F0 Model for Statistical Parametric Speech Synthesis","authors":"Xin Wang, Shinji Takaki, Junichi Yamagishi, Simon King, Keiichi Tokuda","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Williams and King, 2019] Jennifer Williams and Simon King. Disentangling style factors from speaker represen- tations. Proc. Interspeech 2019, pages 3945-3949, 2019.","title":"Disentangling style factors from speaker representations","authors":"Jennifer Williams, Simon King","year":"2019","venue":"Proc. Interspeech 2019","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Williams and Rownicka, 2019] Jennifer Williams and Joanna Rownicka. Speech replay detection with x- vector attack embeddings and spectral features. Proc. Interspeech 2019, pages 1053-1057, 2019.","title":"Speech replay detection with xvector attack embeddings and spectral features","authors":"Jennifer Williams, Joanna Rownicka","year":"2019","venue":"Proc. Interspeech 2019","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Williams et al., 2020] Jennifer Williams, Joanna Rown- icka, Pilar Oplustil, and Simon King. Comparison of Speech Representations for Automatic Quality Estima- tion in Multi-Speaker Text-to-Speech Synthesis. Speaker Odyssey, 2020.","title":"Comparison of Speech Representations for Automatic Quality Estimation in Multi-Speaker Text-to-Speech Synthesis","authors":"Jennifer Williams, Joanna Rownicka, Pilar Oplustil, Simon King","year":"2020","venue":"Speaker Odyssey","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}