{"type":"ScholarlyArticle","title":"Towards Robust Unsupervised Disentanglement of Sequential Data — A Case Study Using Music Audio","authors":[{"name":"Yin-Jyun Luo","slug":"yin-jyun-luo","given":"Yin-Jyun","family":"Luo","affiliation":"Centre for Digital Music , Queen Mary University of London"},{"name":"Sebastian Ewert","slug":"sebastian-ewert","given":"Sebastian","family":"Ewert","affiliation":"Spotify"},{"name":"Simon Dixon","slug":"simon-dixon","given":"Simon","family":"Dixon","affiliation":"Centre for Digital Music , Queen Mary University of London"}],"abstract":"Disentangled sequential autoencoders (DSAEs) represent a class of probabilistic graphical models that describes an observed sequence with dynamic latent variables and a static latent variable. The former encode information at a frame rate identical to the observation, while the latter globally governs the entire sequence. This introduces an inductive bias and facilitates unsupervised disentanglement of the underlying local and global factors. In this paper, we show that the vanilla DSAE suffers from being sensitive to the choice of model architecture and capacity of the dynamic latent variables, and is prone to collapse the static latent variable. As a countermeasure, we propose TS-DSAE, a two-stage training framework that first learns sequence-level prior distributions, which are subsequently employed to regularise the model and facilitate auxiliary objectives to promote disentanglement. The proposed framework is fully unsupervised and robust against the global factor collapse problem across a wide range of model configurations. It also avoids typical solutions such as adversarial training which usually involves laborious parameter tuning, and domain-specific data augmentation. We conduct quantitative and qualitative evaluations to demonstrate its robustness in terms of disentanglement on both artificial and real-world music audio datasets.","doi":"10.24963/ijcai.2022/458","url":"https://ijcai.org/proceedings/2022/458","pdf":"https://ijcai.org/proceedings/2022/0458.pdf","markdown":"https://ijcai.org/proceedings/2022/458.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/458","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":458,"pages":{"first":3299,"last":3305},"keywords":["Machine Learning: Explainable/Interpretable Machine Learning","Machine Learning: Unsupervised Learning","Machine Learning: Autoencoders","Machine Learning: Time-series; Data Streams","Machine Learning: Representation learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986948","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985091","length_sec":318,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bai et al., 2021] Junwen Bai, Weiran Wang, and Carla Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 2021.","title":"Contrastively disentangled sequential variational autoencoder","authors":"Bai","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio, 2013] Yoshua Bengio. Deep learning of representa- tions: Looking forward. In Proceedings of the International Conference on Statistical Language and Speech Processing, 2013.","title":"Deep learning of representations: Looking forward","authors":"Yoshua Bengio","year":"2013","venue":"Proceedings of the International Conference on Statistical Language and Speech Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cífka et al., 2021] Ondřej Cífka, Alexey Ozerov, Umut S ¸ims ¸ekli, and Gaël Richard. Self-supervised VQ-VAE for one-shot music style transfer. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2021.","title":"Self-supervised VQ-VAE for one-shot music style transfer","authors":"Ondřej Cífka, Alexey Ozerov, S Umut, Gaël Richard","year":"2021","venue":"Proceedings of the International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han et al., 2021] Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li, and Xuan Zhang. Disentangled recurrent Wasserstein autoencoder. In Proceedings of the Interna- tional Conference on Learning Representations, 2021.","title":"Disentangled recurrent Wasserstein autoencoder","authors":"Han","year":"2021","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hayes et al., 2021] Ben Hayes, Charalampos Saitis, and György Fazekas. Neural waveshaping synthesis. In Pro- ceedings of the International Society for Music Information Retrieval Conference, 2021.","title":"Neural waveshaping synthesis","authors":"Ben Hayes, Charalampos Saitis, György Fazekas","year":"2021","venue":"Proceedings of the International Society for Music Information Retrieval Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hsu et al., 2017] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in Neural Information Processing Systems, 2017.","title":"Unsupervised learning of disentangled and interpretable representations from sequential data","authors":"Wei-Ning Hsu, Yu Zhang, James Glass","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Khurana et al., 2019] Sameer Khurana, Shafiq Rayhan Joty, Ahmed Ali, and James Glass. A factorial deep Markov model for unsupervised disentangled representation learn- ing from speech. In Proceedings of the International Con- ference on Acoustics, Speech and Signal Processing, 2019.","title":"A factorial deep Markov model for unsupervised disentangled representation learning from speech","authors":"Sameer Khurana, Rayhan Shafiq, Ahmed Joty, James Ali, Glass","year":"2019","venue":"Proceedings of the International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kilgour et al., 2019] Kevin Kilgour, Mauricio Zuluaga, Do- minik Roblek, and Matthew Sharifi. Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. In Proceedings of INTERSPEECH, 2019.","title":"Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms","authors":"Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, Matthew Sharifi","year":"2019","venue":"Proceedings of INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2018] Jong Wook Kim, Justin Salamon, Peter Qi Li, and Juan Pablo Bello. CREPE: A convolutional rep- resentation for pitch estimation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2018.","title":"CREPE: A convolutional representation for pitch estimation","authors":"Jong Wook, Kim, Justin Salamon, Peter Qi Li, Juan Pablo Bello","year":"2018","venue":"Proceedings of the International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representa- tions, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lezama, 2019] José Lezama. Overcoming the disentangle- ment vs reconstruction trade-off via Jacobian supervision. In Proceedings of the International Conference on Learning Representations, 2019.","title":"Overcoming the disentanglement vs reconstruction trade-off via Jacobian supervision","authors":"José Lezama, Lezama","year":"2019","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Mandt, 2018] Yingzhen Li and Stephan Mandt. Dis- entangled sequential autoencoder. In Proceedings of the International Conference on Machine Learning, 2018.","title":"Disentangled sequential autoencoder","authors":"Mandt ; Li, Yingzhen Li, Stephan Mandt","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, and Gaurav Sharma. Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 2019.","title":"Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications","authors":"Li","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progressive learning and disentanglement of hierarchical representa- tions. In Proceedings of the International Conference on Learning Representations, 2020.","title":"Progressive learning and disentanglement of hierarchical representations","authors":"Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, Linwei Wang","year":"2020","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Locatello et al., 2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the International Con- ference on Machine Learning, 2019.","title":"Challenging common assumptions in the unsupervised learning of disentangled representations","authors":"Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem","year":"2019","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2020] Yin-Jyun Luo, Kin Wai Cheuk, Tomoyasu Nakano, Masataka Goto, and Dorien Herremans. Unsu- pervised disentanglement of pitch and timbre for isolated musical instrument sounds. In Proceedings of the Interna- tional Society for Music Information Retrieval Conference, 2020.","title":"Unsupervised disentanglement of pitch and timbre for isolated musical instrument sounds","authors":"Yin-Jyun Luo, Kin Wai Cheuk, Tomoyasu Nakano, Masataka Goto, Dorien Herremans","year":"2020","venue":"Proceedings of the International Society for Music Information Retrieval Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pati et al., 2020] Ashis Pati, Siddharth Gururani, and Alexander Lerch. dMelodies: A music dataset for dis- entanglement learning. In Proceedings of the International Society for Music Information Retrieval Conference, 2020.","title":"dMelodies: A music dataset for disentanglement learning","authors":"Ashis Pati, Siddharth Gururani, Alexander Lerch","year":"2020","venue":"Proceedings of the International Society for Music Information Retrieval Conference","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Salamon et al., 2014] Justin Salamon, Emilia Gómez, Daniel P. W. Ellis, and Gaël Richard. Melody extraction from polyphonic music signals: Approaches, applications, and challenges. IEEE Signal Processing Magazine, 2014.","title":"Melody extraction from polyphonic music signals: Approaches, applications, and challenges","authors":"Justin Salamon, Emilia Gómez, P W Daniel, Gaël Ellis, Richard","year":"2014","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saxena et al., 2021] Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 2021.","title":"Clockwork variational autoencoders","authors":"Vaibhav Saxena, Jimmy Ba, Danijar Hafner","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vowels et al., 2020] Matthew James Vowels, Necati Cihan Camgöz, and Richard Bowden. NestedVAE: Isolating com- mon factors via weak supervision. In Proceedings of the International Conference on Computer Vision and Pattern Recognition, 2020.","title":"NestedVAE: Isolating common factors via weak supervision","authors":"Matthew James Vowels, Necati Cihan Camgöz, Richard Bowden","year":"2020","venue":"Proceedings of the International Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vowels et al., 2021] Matthew James Vowels, Necati Cihan Camgoz, and Richard Bowden. VDSM: Unsupervised video disentanglement with state-space modeling and deep mixtures of experts. In Proceedings of the International Conference on Computer Vision and Pattern Recognition, 2021.","title":"VDSM: Unsupervised video disentanglement with state-space modeling and deep mixtures of experts","authors":"Matthew James Vowels, Necati Cihan Camgoz, Richard Bowden","year":"2021","venue":"Proceedings of the International Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2016] Weiran Wang, Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint arxiv:1610.03454, 2016.","title":"Deep variational canonical correlation analysis","authors":"Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhu et al., 2020] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3VAE: Self-supervised se- quential VAE for representation disentanglement and data generation. In Proceedings of the International Conference on Computer Vision and Pattern Recognition, 2020.","title":"S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation","authors":"Zhu","year":"2020","venue":"Proceedings of the International Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}