{"type":"ScholarlyArticle","title":"Unsupervised Learning via Total Correlation Explanation","authors":[{"name":"Greg Ver Steeg","slug":"greg-ver-steeg","given":"Greg Ver","family":"Steeg","affiliation":"University of Southern California Information Sciences Institute"}],"abstract":"Learning by children and animals occurs effortlessly and largely without obvious supervision. Successes in automating supervised learning have not translated to the more ambiguous realm of unsupervised learning where goals and labels are not provided. Barlow (1961) suggested that the signal that brains leverage for unsupervised learning is dependence, or redundancy, in the sensory environment. Dependence can be characterized using the information-theoretic multivariate mutual information measure called total correlation. The principle of Total Cor-relation Ex-planation (CorEx) is to learn representations of data that \"explain\" as much dependence in the data as possible. We review some manifestations of this principle along with successes in unsupervised learning problems across diverse domains including human behavior, biology, and language.","doi":"10.24963/ijcai.2017/740","url":"https://ijcai.org/proceedings/2017/740","pdf":"https://ijcai.org/proceedings/2017/0740.pdf","markdown":"https://ijcai.org/proceedings/2017/740.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/740","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Early Career","track":"Early Career","number":740,"pages":{"first":5151,"last":5155},"keywords":["Machine Learning: Machine Learning","Machine Learning: Deep Learning","Machine Learning: Unsupervised Learning","Multidisciplinary Topics and Applications: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Achille and Soatto, 2016] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. ArXiv e-prints, 2016.","title":"Information dropout: Learning optimal representations through noisy computation","authors":"Alessandro Achille, Stefano Soatto","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barlow et al., 1989] Horace B Barlow, Tej P Kaushal, and Graeme J Mitchison. Finding minimum entropy codes. Neural Computation, 1(3):412-423, 1989.","title":"Finding minimum entropy codes","authors":"Horace B Barlow, Tej P Kaushal, Graeme J Mitchison","year":"1989","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bell and Sejnowski, 1995] Anthony J Bell and Terrence J Se- jnowski. An information-maximization approach to blind sep- aration and blind deconvolution. Neural computation, 7(6):1129- 1159, 1995.","title":"An information-maximization approach to blind separation and blind deconvolution","authors":"J Anthony, Terrence J Bell, Sejnowski","year":"1995","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2012] Yoshua Bengio, Aaron C Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538, 1, 2012.","title":"Unsupervised feature learning and deep learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron C Courville, Pascal Vincent","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. Pattern Analysis and Machine Intelligence, 35(8):1798- 1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Button et al., 2013] Katherine S Button, John PA Ioannidis, Claire Mokrysz, Brian A Nosek, Jonathan Flint, Emma SJ Robinson, and Marcus R Munafò. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neu- roscience, 14(5):365-376, 2013.","title":"Power failure: why small sample size undermines the reliability of neuroscience","authors":"Katherine S Button, P A John, Claire Ioannidis, Brian A Mokrysz, Jonathan Nosek, Emma Sj Flint, Marcus R Robinson, Munafò","year":"2013","venue":"Nature Reviews Neuroscience","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2016a] Peixian Chen, Nevin L Zhang, Leonard KM Poon, and Zhourong Chen. Progressive em for latent tree models and hierarchical topic detection. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.","title":"Progressive em for latent tree models and hierarchical topic detection","authors":"Chen","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2016b] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Inter- pretable representation learning by information maximizing gen- erative adversarial nets. In Advances In Neural Information Pro- cessing Systems 29 (NIPS), pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Chen","year":"2016","venue":"Advances In Neural Information Processing Systems 29 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[DeDeo, 2015] Simon DeDeo. Information theory for intelligent people, 2015.","title":"Information theory for intelligent people","authors":"Simon Dedeo","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dinh et al., 2014] Laurent Dinh, David Krueger, and Yoshua Ben- gio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.","title":"Nice: Non-linear independent components estimation","authors":"Laurent Dinh, David Krueger, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gallagher et al., 2016] R.J. Gallagher, K. Reing, D. Kale, and G. Ver Steeg. Anchored correlation explanation: Topic modeling with minimal domain knowledge. arXiv preprint arXiv:1611.10277, 2016.","title":"Anchored correlation explanation: Topic modeling with minimal domain knowledge","authors":"R J Gallagher, K Reing, D Kale, G Ver, Steeg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goerg and Shalizi, 2012] G.M. Goerg and C.R. Shalizi. Licors: Light cone reconstruction of states for non-parametric forecasting of spatio-temporal systems. arXiv:1206.2398, 2012.","title":"Licors: Light cone reconstruction of states for non-parametric forecasting of spatio-temporal systems","authors":"G M Goerg, C R Shalizi","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hodas et al., 2015] N. Hodas, G. Ver Steeg, J. Harrison, S. Chikk- agoudar, E. Bell, and C. Corley. Disentangling the lexicons of disaster response in twitter. In The 3rd International Workshop on Social Web for Disaster Management (SWDM'15), 2015.","title":"Disentangling the lexicons of disaster response in twitter","authors":"N Hodas, G Ver, J Steeg, S Harrison, E Chikkagoudar, C Bell, Corley","year":"2015","venue":"The 3rd International Workshop on Social Web for Disaster Management (SWDM'15)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hyvärinen and Oja, 2000] A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural net- works, 13(4):411-430, 2000.","title":"Independent component analysis: algorithms and applications","authors":"Oja Hyvärinen, A Hyvärinen, E Oja","year":"2000","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jaynes, 2003] Edwin T Jaynes. Probability theory: the logic of science. Cambridge university press, 2003.","title":"Probability theory: the logic of science","authors":"T Edwin, Jaynes","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kolchinsky et al., 2017] Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. arXiv preprint arXiv:1705.02436, 2017.","title":"Nonlinear information bottleneck","authors":"Artemy Kolchinsky, Brendan D Tracey, David H Wolpert","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Linsker, 1988] Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):105-117, 1988.","title":"Ralph Linsker. Self-organization in a perceptual network","authors":"Linsker","year":"1988","venue":"Computer","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Madsen et al., 2015] Sarah K. Madsen, Greg Ver Steeg, Adam Mezher, Neda Jahanshad, Talia M. Nir, Xue Hua, Boris A. Gut- man, Aram Galstyan, and Paul M. Thompson. Information- theoretic characterization of blood panel predictors for brain at- rophy and cognitive decline in the elderly. IEEE International Symposium on Biomedical Imaging, 2015.","title":"Informationtheoretic characterization of blood panel predictors for brain atrophy and cognitive decline in the elderly","authors":"Sarah K Madsen, Greg Ver Steeg, Adam Mezher, Neda Jahanshad, Talia M Nir, Xue Hua, Boris A Gutman, Aram Galstyan, Paul M Thompson","year":"2015","venue":"IEEE International Symposium on Biomedical Imaging","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Madsen et al., 2016] Sarah K. Madsen, Greg Ver Steeg, Madelaine Daianu, Adam Mezher, Neda Jahanshad, Talia M. Nir, Xue Hua, Boris A. Gutman, Aram Galstyan, and Paul M. Thompson. Rel- ative value of diverse brain mri and blood-based measures for predicting cognitive decline in the elderly. SPIE Medical Imaging, 2016.","title":"Relative value of diverse brain mri and blood-based measures for predicting cognitive decline in the elderly","authors":"Sarah K Madsen, Greg Ver Steeg, Madelaine Daianu, Adam Mezher, Neda Jahanshad, Talia M Nir, Xue Hua, Boris A Gutman, Aram Galstyan, Paul M Thompson","year":"2016","venue":"SPIE Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Minsky, 1961] Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-30, 1961.","title":"Steps toward artificial intelligence","authors":"Marvin Minsky, Minsky","year":"1961","venue":"Proceedings of the IRE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oizumi et al., 2014] Masafumi Oizumi, Larissa Albantakis, and Giulio Tononi. From the phenomenology to the mechanisms of consciousness: integrated information theory 3.0. 2014.","title":"From the phenomenology to the mechanisms of consciousness: integrated information theory","authors":"Oizumi","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Op't Veld and Gastpar, 2016] Giel Op't Veld and Michael C Gast- par. Caching gaussians: Minimizing total correlation on the gray- wyner network. In 50th Annual Conference on Information Sys- tems and Sciences (CISS), 2016.","title":"Giel Op't Veld and Michael C Gastpar. Caching gaussians: Minimizing total correlation on the graywyner network","authors":"' Op, Gastpar Veld","year":"2016","venue":"50th Annual Conference on Information Systems and Sciences (CISS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pepke and Ver Steeg, 2017] Shirley Pepke and Greg Ver Steeg. Comprehensive discovery of subsample gene expression com- ponents by information explanation: therapeutic implications in cancer. BMC medical genomics, 10(1):12, 2017.","title":"Shirley Pepke and Greg Ver Steeg. Comprehensive discovery of subsample gene expression components by information explanation: therapeutic implications in cancer","authors":"Ver Pepke, Steeg","year":"2017","venue":"BMC medical genomics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reing et al., 2016] Kyle Reing, David C Kale, Greg Ver Steeg, and Aram Galstyan. Toward interpretable topic discovery via anchored correlation explanation. In Proceedings of the 2016 ICML Work- shop on Human Interpretability in Machine Learning (WHI 2016), number arXiv:1606.07043, 2016.","title":"Toward interpretable topic discovery via anchored correlation explanation","authors":"Kyle Reing, Greg David C Kale, Aram Ver Steeg, Galstyan","year":"2016","venue":"Proceedings of the 2016 ICML Work-shop on Human Interpretability in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shannon, 1948] C.E. Shannon. A mathematical theory of commu- nication. The Bell System Technical Journal, 27:379-423, 1948.","title":"A mathematical theory of communication","authors":"C E Shannon","year":"1948","venue":"The Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shannon, 1956] Claude Shannon. The bandwagon (edtl.). IRE Transactions on Information Theory, 1(2):3, 1956.","title":"The bandwagon (edtl.)","authors":"Claude Shannon","year":"1956","venue":"IRE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Simoncelli and Olshausen, 2001] Eero Simoncelli and Bruno Ol- shausen. Natural image statistics and neural representation. Annu. Rev. Neurosci., 24, 2001.","title":"Eero Simoncelli and Bruno Olshausen. Natural image statistics and neural representation","authors":"Olshausen Simoncelli","year":"2001","venue":"Annu. Rev. Neurosci","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy et al., 2014] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR, 2014.","title":"Intriguing properties of neural networks","authors":"C Szegedy, W Zaremba, I Sutskever, J Bruna, D Erhan, I Goodfellow, R Fergus","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[TensorFlow, 2015] TensorFlow. Large-scale machine learning on heterogeneous systems, 2015. Software available from tensor- flow.org.","title":"TensorFlow. Large-scale machine learning on heterogeneous systems","authors":"Tensorflow","year":"2015","venue":"Software available from tensorflow","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tishby and Zaslavsky, 2015] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In Infor- mation Theory Workshop (ITW), 2015 IEEE. IEEE, 2015.","title":"Deep learning and the information bottleneck principle","authors":"Naftali Tishby, Noga Zaslavsky","year":"2015","venue":"Information Theory Workshop (ITW)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tishby et al., 2000] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv:physics/0004057, 2000.","title":"The information bottleneck method","authors":"Naftali Tishby, Fernando C Pereira, William Bialek","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ver Steeg and Galstyan, 2014] G. Ver Steeg and A. Galstyan. Dis- covering structure in high-dimensional data through correlation explanation. In Advances in Neural Information Processing Sys- tems (NIPS), 2014.","title":"Discovering structure in high-dimensional data through correlation explanation","authors":"Ver Steeg, ; G Galstyan, A Ver Steeg, Galstyan","year":"2014","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ver Steeg and Galstyan, 2015] Greg Ver Steeg and Aram Galstyan. Maximally informative hierarchical representations of high- dimensional data. In Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 2015. http://arxiv.org/abs/1410.7404.","title":"Maximally informative hierarchical representations of highdimensional data","authors":"Ver Steeg, Greg Galstyan, Steeg Ver, Aram Galstyan","year":"2015","venue":"Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":"http://arxiv.org/abs/1410.7404","internal_url":null},{"position":34,"raw":"[Ver Steeg and Galstyan, 2016] Greg Ver Steeg and Aram Galstyan. The information sieve. In International Conference on Machine Learning (ICML), 2016.","title":"Greg Ver Steeg and Aram Galstyan. The information sieve","authors":"Ver Steeg, Galstyan","year":"2016","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ver Steeg and Galstyan, 2017] Greg Ver Steeg and Aram Galstyan. Low complexity gaussian latent factor models and a blessing of dimensionality. arXiv:1706.03353 [stat.ML], 2017.","title":"Low complexity gaussian latent factor models and a blessing of dimensionality","authors":"Ver Steeg, ; Galstyan, Aram Steeg, Galstyan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ver Steeg et al., 2014] Greg Ver Steeg, Aram Galstyan, Fei Sha, and Simon DeDeo. Demystifying information-theoretic clustering. In International Conference on Machine Learning, 2014.","title":"Demystifying information-theoretic clustering","authors":"Ver Steeg","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ver Steeg et al., 2017] Greg Ver Steeg, Shuyang Gao, Kyle Reing, and Aram Galstyan. Sifting common information from many variables. IJCAI, 2017.","title":"Sifting common information from many variables","authors":"Ver Steeg","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/402"},{"position":38,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Ben- gio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103. ACM, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Watanabe, 1960] Satosi Watanabe. Information theoretical anal- ysis of multivariate correlation. IBM Journal of research and development, 4(1):66-82, 1960.","title":"Information theoretical analysis of multivariate correlation","authors":"Satosi Watanabe, Watanabe","year":"1960","venue":"IBM Journal of research and development","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wolpert et al., 2014] David H Wolpert, Joshua A Grochow, Eric Libby, and Simon DeDeo. A framework for optimal high-level de- scriptions in science and engineering-preliminary report. arXiv preprint arXiv:1409.7403, 2014.","title":"A framework for optimal high-level descriptions in science and engineering-preliminary report","authors":"Joshua A David H Wolpert, Eric Grochow, Simon Libby, Dedeo","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wyner, 1975] Aaron D Wyner. The common information of two dependent random variables. Information Theory, IEEE Transac- tions on, 21(2):163-179, 1975.","title":"The common information of two dependent random variables","authors":"D Aaron, Wyner","year":"1975","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zavaliangos-Petropulu et al., 2017] Artemis Zavaliangos- Petropulu, Emily L Dennis, Greg Ver Steeg, Talin Babikian, Richard Mink, Christopher Babbitt, Jeffrey Johnson, Christo- pher C Giza, Robert F Asarnow, and Paul M Thompson. Variable clustering reveals associations between subcortical brain volume and cognitive changes in pediatric traumatic brain injury. In 12th International Symposium on Medical Information Processing and Analysis. International Society for Optics and Photonics, 2017.","title":"Variable clustering reveals associations between subcortical brain volume and cognitive changes in pediatric traumatic brain injury","authors":"Zavaliangos-Petropulu","year":"2017","venue":"12th International Symposium on Medical Information Processing and Analysis","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}