{"type":"ScholarlyArticle","title":"Overlooked Implications of the Reconstruction Loss for VAE Disentanglement","authors":[{"name":"Nathan Michlo","slug":"nathan-michlo","given":"Nathan","family":"Michlo","affiliation":"University of the Witwatersrand , Johannesburg , South Africa"},{"name":"Richard Klein","slug":"richard-klein","given":"Richard","family":"Klein","affiliation":"University of the Witwatersrand , Johannesburg , South Africa"},{"name":"Steven James","slug":"steven-james","given":"Steven","family":"James","affiliation":"University of the Witwatersrand , Johannesburg , South Africa"}],"abstract":"Learning disentangled representations with variational autoencoders (VAEs) is often attributed to the regularisation component of the loss. In this work, we highlight the interaction between data and the reconstruction term of the loss as the main contributor to disentanglement in VAEs. We show that standard benchmark datasets have unintended correlations between their subjective ground-truth factors and perceived axes in the data according to typical VAE reconstruction losses. Our work exploits this relationship to provide a theory for what constitutes an adversarial dataset under a given reconstruction loss. We verify this by constructing an example dataset that prevents disentanglement in state-of-the-art frameworks while maintaining human-intuitive ground-truth factors. Finally, we re-enable disentanglement by designing an example reconstruction loss that is once again able to perceive the ground-truth factors. Our findings demonstrate the subjective nature of disentanglement and the importance of considering the interaction between the ground-truth factors, data and notably, the reconstruction loss, which is under-recognised in the literature.","doi":"10.24963/ijcai.2023/453","url":"https://ijcai.org/proceedings/2023/453","pdf":"https://ijcai.org/proceedings/2023/0453.pdf","markdown":"https://ijcai.org/proceedings/2023/453.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/453","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":453,"pages":{"first":4073,"last":4081},"keywords":["Machine Learning: ML: Representation learning","Machine Learning: ML: Explainable/Interpretable machine learning","Machine Learning: ML: Unsupervised learning","Machine Learning: ML: Autoencoders"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Burgess and Kim, 2018] Christopher Burgess and Hyunjik Kim. 3D shapes dataset. https://github.com/deepmind/ 3dshapes-dataset, 2018. Accessed: 2023-01-01.","title":"3D shapes dataset","authors":"Christopher Burgess, Hyunjik Kim","year":"2023","venue":null,"doi":null,"url":"https://github.com/deepmind/3dshapes-dataset,2018","internal_url":null},{"position":3,"raw":"[Burgess et al., 2017] Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentan- gling in β-VAE. In Workshop on Learning Disentangled Representations at the 31st Conference on Neural Informa- tion Processing Systems,, 2017.","title":"Understanding disentangling in β-VAE","authors":"Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, Alexander Lerchner","year":"2017","venue":"Workshop on Learning Disentangled Representations at the 31st Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Ricky Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2615-2625, 2018.","title":"Isolating sources of disentanglement in variational autoencoders","authors":"Ricky Chen, Xuechen Li, Roger Grosse, David Duvenaud","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eastwood and Williams, 2018] Cian Eastwood and Christo- pher Williams. A framework for the quantitative evaluation of disentangled representations. In International Confer- ence on Learning Representations, 2018.","title":"A framework for the quantitative evaluation of disentangled representations","authors":"Cian Eastwood, Christopher Williams","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gondal et al., 2019] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Mar- tin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. Advances in Neural Information Processing Systems, 32:15740-15751, 2019.","title":"On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset","authors":"Waleed Muhammad, Manuel Gondal, Djordje Wuthrich, Francesco Miladinovic, Martin Locatello, Valentin Breidt, Joel Volchkov, Olivier Akpo, Bernhard Bachem, Stefan Schölkopf, Bauer","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Higgins et al., 2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained varia- tional framework. 2016.","title":"beta-vae: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hou et al., 2017] Xianxu Hou, Linlin Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoen- coder. In 2017 IEEE Winter Conference on Applications of Computer Vision, pages 1133-1141. IEEE, 2017.","title":"Deep feature consistent variational autoencoder","authors":"Hou","year":"2017","venue":"IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim and Mnih, 2018] Hyunjik Kim and Andriy Mnih. Dis- entangling by factorising. In International Conference on Machine Learning, pages 2649-2658. PMLR, 2018.","title":"Hyunjik Kim and Andriy Mnih. Disentangling by factorising","authors":"Mnih Kim","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Inter- national Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2014] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.","title":"Auto-encoding variational bayes","authors":"Diederik Kingma, Max Welling","year":"2014","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Infor- mation Processing Systems, 25:1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2018] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In In- ternational Conference on Learning Representations, 2018.","title":"Variational inference of disentangled latent concepts from unlabeled observations","authors":"Abhishek Kumar, Prasanna Sattigeri, Avinash Balakrishnan","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 2004] Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition, 2004.","title":"Learning methods for generic object recognition with invariance to pose and lighting","authors":"Lecun","year":"2004","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Locatello et al., 2019a] Francesco Locatello, Gabriele Ab- bati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled repre- sentations. In Advances in Neural Information Processing Systems, pages 14584-14597, 2019.","title":"On the fairness of disentangled representations","authors":"Locatello","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Locatello et al., 2019b] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114-4124. PMLR, 2019.","title":"Challenging common assumptions in the unsupervised learning of disentangled representations","authors":"Locatello","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Locatello et al., 2020] Francesco Locatello, Ben Poole, Gun- nar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Ma- chine Learning, pages 6348-6359. PMLR, 2020.","title":"Weakly-supervised disentanglement without compromises","authors":"Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, Michael Tschannen","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mathieu et al., 2019] Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentangle- ment in variational autoencoders. In International Con- ference on Machine Learning, pages 4402-4412. PMLR, 2019.","title":"Disentangling disentanglement in variational autoencoders","authors":"Mathieu","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Matthey et al., 2017] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentangle- ment testing sprites dataset. https://github.com/deepmind/ dsprites-dataset, 2017. Accessed: 2023-01-01.","title":"dSprites: Disentanglement testing sprites dataset","authors":"Loic Matthey, Irina Higgins, Demis Hassabis, Alexander Lerchner","year":"2023","venue":null,"doi":null,"url":"https://github.com/deepmind/dsprites-dataset,2017","internal_url":null},{"position":20,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017.","title":"Automatic differentiation in PyTorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reed et al., 2015] Scott Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. Advances in Neural Information Processing Systems, 28:1252-1260, 2015.","title":"Deep visual analogy-making","authors":"Scott Reed, Yi Zhang, Yuting Zhang, Honglak Lee","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rolinek et al., 2019] Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue PCA di- rections (by accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406-12415, 2019.","title":"Variational autoencoders pursue PCA directions (by accident)","authors":"Michal Rolinek, Dominik Zietlow, Georg Martius","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shao et al., 2020] Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable varia- tional autoencoder. In International Conference on Ma- chine Learning, pages 8655-8664. PMLR, 2020.","title":"Controlvae: Controllable variational autoencoder","authors":"Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, Tarek Abdelzaher","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zaidi et al., 2020] Julian Zaidi, Jonathan Boilard, Ghys- lain Gagnon, and Marc-André Carbonneau. Measuring disentanglement: A review of metrics. arXiv preprint arXiv:2012.09276, 2020.","title":"Measuring disentanglement: A review of metrics","authors":"Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, Marc-André Carbonneau","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2017] Shengjia Zhao, Jiaming Song, and Ste- fano Ermon. InfoVAE: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262, 2017.","title":"Shengjia Zhao, Jiaming Song, and Stefano Ermon","authors":"Zhao","year":"2017","venue":"InfoVAE: Information maximizing variational autoencoders","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zietlow et al., 2021] Dominik Zietlow, Michal Rolinek, and Georg Martius. Demystifying inductive biases for β-VAE based architectures. arXiv preprint arXiv:2102.06822, 2021.","title":"Demystifying inductive biases for β-VAE based architectures","authors":"Dominik Zietlow, Michal Rolinek, Georg Martius","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}