{"type":"ScholarlyArticle","title":"Neural PCA for Flow-Based Representation Learning","authors":[{"name":"Shen Li","slug":"shen-li","given":"Shen","family":"Li","affiliation":"Institute of Data Science , National University of Singapore"},{"name":"Bryan Hooi","slug":"bryan-hooi","given":"Bryan","family":"Hooi","affiliation":"Institute of Data Science , National University of Singapore"}],"abstract":"Of particular interest is to discover useful representations solely from observations in an unsupervised generative manner. However, the question of whether existing normalizing flows provide effective representations for downstream tasks remains mostly unanswered despite their strong ability for sample generation and density estimation. This paper investigates this problem for such a family of generative models that admits exact invertibility. We propose Neural Principal Component Analysis (Neural-PCA) that operates in full dimensionality while capturing principal components in descending order. Without exploiting any label information, the principal components recovered store the most informative elements in their leading dimensions and leave the negligible in the trailing ones, allowing for clear performance improvements of 5%-10% in downstream tasks. Such improvements are empirically found consistent irrespective of the number of latent trailing dimensions dropped. Our work suggests that necessary inductive bias be introduced into generative modeling when representation quality is of interest.","doi":"10.24963/ijcai.2022/448","url":"https://ijcai.org/proceedings/2022/448","pdf":"https://ijcai.org/proceedings/2022/0448.pdf","markdown":"https://ijcai.org/proceedings/2022/448.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/448","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":448,"pages":{"first":3229,"last":3235},"keywords":["Machine Learning: Representation learning","Computer Vision: Neural generative models, auto encoders, GANs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986501","length_sec":78,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984644","length_sec":356,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aris- tide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Ben- gio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062, 2018.","title":"Mohamed Ishmael Belghazi, Aristide Baratin","authors":"Belghazi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chapelle et al., 2009] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.","title":"Semi-supervised learning","authors":"Olivier Chapelle, Bernhard Scholkopf, Alexander Zien","year":"2006","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in neural information pro- cessing systems, pages 6571-6583, 2018.","title":"Neural ordinary differential equations","authors":"Yulia Ricky Tq Chen, Jesse Rubanova, David K Bettencourt, Duvenaud","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Ricky T. Q. Chen, Jens Behrmann, David K Duvenaud, and Joern-Henrik Jacobsen. Residual flows for invertible generative modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 9916-9926.","title":"Residual flows for invertible generative modeling","authors":"T Q Ricky, Jens Chen, David K Behrmann, Joern-Henrik Duvenaud, Jacobsen","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dempster et al., 1977] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incom- plete data via the em algorithm. Journal of the Royal Sta- tistical Society: Series B (Methodological), 39(1):1-22, 1977.","title":"Maximum likelihood from incomplete data via the em algorithm","authors":"Nan M Arthur P Dempster, Donald B Laird, Rubin","year":"1977","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dinh et al., 2014] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent compo- nents estimation. arXiv preprint arXiv:1410.8516, 2014.","title":"Nice: Non-linear independent components estimation","authors":"Laurent Dinh, David Krueger, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dinh et al., 2016] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. Interna- tional Conference on Learning Representations, 2016.","title":"Density estimation using real nvp","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Durkan et al., 2019] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pages 7509-7520, 2019.","title":"Neural spline flows","authors":"Conor Durkan, Artur Bekasov, Iain Murray, George Papamakarios","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fefferman et al., 2016] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypoth- esis. Journal of the American Mathematical Society, 29(4):983-1049, 2016.","title":"Testing the manifold hypothesis","authors":"Charles Fefferman, Sanjoy Mitter, Hariharan Narayanan","year":"2016","venue":"Journal of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grathwohl et al., 2018] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable re- versible generative models. In International Conference on Learning Representations, 2018.","title":"Ffjord: Free-form continuous dynamics for scalable reversible generative models","authors":"Grathwohl","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Dhariwal, 2018] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.","title":"Glow: Generative flow with invertible 1x1 convolutions","authors":"Dhariwal ; Kingma, P Durk, Prafulla Kingma, Dhariwal","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kobyzev et al., 2020] Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, page 1-1, 2020.","title":"Normalizing flows: An introduction and review of current methods","authors":"Ivan Kobyzev, Simon Prince, Marcus Brubaker","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Meng et al., 2020] Chenlin Meng, Yang Song, Jiaming Song, and Stefano Ermon. Gaussianization flows. In Inter- national Conference on Artificial Intelligence and Statis- tics, 2020.","title":"Gaussianization flows","authors":"Chenlin Meng, Yang Song, Jiaming Song, Stefano Ermon","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Narayanan and Mitter, 2010] Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In Proceedings of the 23rd International Conference on Neural Information Processing Systems- Volume 2, pages 1786-1794, 2010.","title":"Sample complexity of testing the manifold hypothesis","authors":"Hariharan Narayanan, Sanjoy Mitter","year":"2010","venue":"Proceedings of the 23rd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papamakarios et al., 2019] George Papamakarios, Eric Nal- isnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019.","title":"Normalizing flows for probabilistic modeling and inference","authors":"George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, Balaji Lakshminarayanan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Poole et al., 2019] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Confer- ence on Machine Learning, pages 5171-5180. PMLR, 2019.","title":"On variational bounds of mutual information","authors":"Ben Poole, Sherjil Ozair, Aaron Van Den, Alex Oord, George Alemi, Tucker","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rezende and Mohamed, 2015] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with nor- malizing flows. International Conference on Machine Learning, 2015.","title":"Variational inference with normalizing flows","authors":"Mohamed ; Rezende, Danilo Jimenez, Rezende, Shakir Mohamed","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Theis et al., 2015] Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844, 2015.","title":"A note on the evaluation of generative models","authors":"Lucas Theis, Aäron Van Den Oord, Matthias Bethge","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tsai et al., 2020] Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Neural methods for point-wise depen- dency estimation. arXiv preprint arXiv:2006.05553, 2020.","title":"Neural methods for point-wise dependency estimation","authors":"Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, Ruslan Salakhutdinov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Uria et al., 2013] Benigno Uria, Iain Murray, and Hugo Larochelle. Rnade: the real-valued neural autoregressive density-estimator. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume 2, pages 2175-2183, 2013.","title":"Rnade: the real-valued neural autoregressive density-estimator","authors":"Uria","year":"2013","venue":"Proceedings of the 26th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}