{"type":"ScholarlyArticle","title":"Sifting Common Information from Many Variables","authors":[{"name":"Greg Ver Steeg","slug":"greg-ver-steeg","given":"Greg Ver","family":"Steeg","affiliation":"University of Southern California Information Sciences Institute"},{"name":"Shuyang Gao","slug":"shuyang-gao","given":"Shuyang","family":"Gao","affiliation":"University of Southern California Information Sciences Institute"},{"name":"Kyle Reing","slug":"kyle-reing","given":"Kyle","family":"Reing","affiliation":"University of Southern California Information Sciences Institute"},{"name":"Aram Galstyan","slug":"aram-galstyan","given":"Aram","family":"Galstyan","affiliation":"University of Southern California Information Sciences Institute"}],"abstract":"Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice. In contrast, characterizing the common information among any group of variables is typically a theoretical exercise with few practical methods for high-dimensional data. A promising solution would be a multivariate generalization of the famous Wyner common information, but this approach relies on solving an apparently intractable optimization problem. We leverage the recently introduced information sieve decomposition to formulate an incremental version of the common information problem that admits a simple fixed point solution, fast convergence, and complexity that is linear in the number of variables. This scalable approach allows us to demonstrate the usefulness of common information in high-dimensional learning problems.The sieve outperforms standard methods on dimensionality reduction tasks, solves \r\na blind source separation problem that cannot be solved with ICA, and accurately recovers structure in brain imaging data.","doi":"10.24963/ijcai.2017/402","url":"https://ijcai.org/proceedings/2017/402","pdf":"https://ijcai.org/proceedings/2017/0402.pdf","markdown":"https://ijcai.org/proceedings/2017/402.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/402","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":402,"pages":{"first":2885,"last":2892},"keywords":["Machine Learning: Unsupervised Learning","Uncertainty in AI: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Allen et al., 2012] Elena A Allen, Erik B Erhardt, Yonghua Wei, Tom Eichele, and Vince D Calhoun. Capturing inter-subject vari- ability with group independent component analysis of fmri data: a simulation study. Neuroimage, 59(4), 2012.","title":"Capturing inter-subject variability with group independent component analysis of fmri data: a simulation study","authors":"Erik B Elena A Allen, Yonghua Erhardt, Tom Wei, Vince D Eichele, Calhoun","year":"2012","venue":"Neuroimage","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baldi and Sadowski, 2015] Pierre Baldi and Peter Sadowski. The ebb and flow of deep learning: a theory of local learning. arXiv preprint arXiv:1506.06472, 2015.","title":"The ebb and flow of deep learning: a theory of local learning","authors":"Sadowski Baldi, Pierre Baldi, Peter Sadowski","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2007] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep net- works. Advances in neural information processing systems, 19:153, 2007.","title":"Greedy layer-wise training of deep networks","authors":"Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. Pattern Analysis and Machine Intelligence, 35(8):1798- 1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Bengio","year":"2013","venue":"Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cattell, 1952] Raymond B Cattell. Factor analysis: an introduction and manual for the psychologist and social scientist. 1952. [","title":"Factor analysis: an introduction and manual for the psychologist and social scientist","authors":"Raymond B Cattell","year":"1952","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chechik et al., 2005] Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. In Journal of Machine Learning Research, pages 165-188, 2005.","title":"Information bottleneck for gaussian variables","authors":"Chechik","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Comon, 1994] Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.","title":"Pierre Comon. Independent component analysis, a new concept?","authors":"Comon","year":"1994","venue":"Signal processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cover and Thomas, 2006] Thomas M Cover and Joy A Thomas. Elements of information theory. Wiley-Interscience, 2006.","title":"Elements of information theory","authors":"M Thomas, Joy A Cover, Thomas","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dinh et al., 2014] Laurent Dinh, David Krueger, and Yoshua Ben- gio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.","title":"Nice: Non-linear independent components estimation","authors":"Laurent Dinh, David Krueger, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Erhardt et al., 2012] Erik B Erhardt, Elena A Allen, Yonghua Wei, Tom Eichele, and Vince D Calhoun. Simtb, a simulation tool- box for fmri data under a model of spatiotemporal separability. Neuroimage, 59(4):4160-4167, 2012.","title":"Simtb, a simulation toolbox for fmri data under a model of spatiotemporal separability","authors":"Erhardt","year":"2012","venue":"Neuroimage","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Foster and Grassberger, 2011] David V Foster and Peter Grass- berger. Lower bounds on mutual information. Phys. Rev. E, 83(1):010101, 2011.","title":"Lower bounds on mutual information","authors":"V David, Peter Foster, Grassberger","year":"2011","venue":"Phys. Rev. E","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Friedman, 1987] Jerome H Friedman. Exploratory projection pur- suit. Journal of the American statistical association, 82(397), 1987.","title":"Exploratory projection pursuit","authors":"Jerome H Friedman","year":"1987","venue":"Journal of the American statistical association","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goerg, 2014] Georg M. Goerg. The lambert way to gaussianize heavy-tailed data with the inverse of tukey's h transformation as a special case. The Scientific World Journal: Special Issue on Probability and Statistics with Applications in Finance and Economics, 2014.","title":"The lambert way to gaussianize heavy-tailed data with the inverse of tukey's h transformation as a special case","authors":"Georg M Goerg","year":"2014","venue":"The Scientific World Journal: Special Issue on Probability and Statistics with Applications in Finance and Economics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Griffith et al., 2014] Virgil Griffith, Edwin KP Chong, Ryan G James, Christopher J Ellison, and James P Crutchfield. Inter- section information based on common randomness. Entropy, 16(4):1985-2000, 2014.","title":"Intersection information based on common randomness","authors":"Griffith","year":"2014","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Guyon et al., 2004] Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the nips 2003 feature selection challenge. In Advances in neural information processing systems, pages 545-552, 2004.","title":"Result analysis of the nips 2003 feature selection challenge","authors":"Guyon","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Halko et al., 2011] Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217-288, 2011.","title":"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions","authors":"Halko","year":"2011","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hinton and Salakhutdinov, 2006] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.","title":"Reducing the dimensionality of data with neural networks","authors":"Geoffrey E Hinton, Ruslan R Salakhutdinov","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hyvärinen and Oja, 2000] Aapo Hyvärinen and Erkki Oja. Inde- pendent component analysis: algorithms and applications. Neural networks, 13(4):411-430, 2000.","title":"Independent component analysis: algorithms and applications","authors":"Oja Hyvärinen, Aapo Hyvärinen, Erkki Oja","year":"2000","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kumar et al., 2014] G Ramesh Kumar, Cheuk Ting Li, and Abbas El Gamal. Exact common information. In Information Theory (ISIT), 2014 IEEE International Symposium on. IEEE, 2014.","title":"Exact common information","authors":"Ramesh Kumar, Cheuk Ting Li, Abbas El Gamal","year":"2014","venue":"IEEE International Symposium on","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin, 2007] Chih-Jen Lin. Projected gradient methods for nonnega- tive matrix factorization. Neural computation, 19(10):2756-2779, 2007.","title":"Projected gradient methods for nonnegative matrix factorization","authors":"Chih-Jen Lin","year":"2007","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Op't Veld and Gastpar, 2016a] Giel Op't Veld and Michael C Gast- par. Caching gaussians: Minimizing total correlation on the gray- wyner network. In 50th Annual Conference on Information Sys- tems and Sciences (CISS), 2016.","title":"Giel Op't Veld and Michael C Gastpar. Caching gaussians: Minimizing total correlation on the graywyner network","authors":"' Op, Gastpar Veld","year":"2016","venue":"50th Annual Conference on Information Systems and Sciences (CISS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Op't Veld and Gastpar, 2016b] Giel J Op't Veld and Michael C Gastpar. Total correlation of gaussian vector sources on the gray- wyner network. In Communication, Control, and Computing (Allerton), 2016 54th Annual Allerton Conference on, pages 385- 392. IEEE, 2016.","title":"Total correlation of gaussian vector sources on the graywyner network","authors":"' Op, Veld, ; Gastpar, J Giel, ' Op, Michael C Veld, Gastpar","year":"2016","venue":"Communication, Control, and Computing (Allerton)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pedregosa et al., 2011] Fabian Pedregosa, Gaël Varoquaux, Alexan- dre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research, 12:2825-2830, 2011.","title":"Scikit-learn: Machine learning in python","authors":"Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg","year":"2011","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Roweis and Saul, 2000] Sam T Roweis and Lawrence K Saul. Non- linear dimensionality reduction by locally linear embedding. Sci- ence, 290(5500):2323-2326, 2000.","title":"Nonlinear dimensionality reduction by locally linear embedding","authors":"T Sam, Lawrence K Roweis, Saul","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sculley, 2010] David Sculley. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, pages 1177-1178. ACM, 2010.","title":"Web-scale k-means clustering","authors":"David Sculley","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shalizi, 2013] Cosma Shalizi. Advanced data analysis from an elementary point of view, 2013.","title":"Advanced data analysis from an elementary point of view","authors":"Cosma Shalizi","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Singh and Pøczos, 2017] Shashank Singh and Barnabás Pøczos. Nonparanormal information estimation. arXiv preprint arXiv:1702.07803, 2017.","title":"Nonparanormal information estimation","authors":"Shashank Singh, Barnabás Pøczos","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Stögbauer et al., 2004] Harald Stögbauer, Alexander Kraskov, Sergey A Astakhov, and Peter Grassberger. Least-dependent- component analysis based on mutual information. Physical Re- view E, 70(6):066123, 2004.","title":"Least-dependentcomponent analysis based on mutual information","authors":"Stögbauer","year":"2004","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tenenbaum et al., 2000] Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319-2323, 2000.","title":"A global geometric framework for nonlinear dimensionality reduction","authors":"Tenenbaum","year":"2000","venue":"science","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tishby et al., 2000] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv:physics/0004057, 2000.","title":"The information bottleneck method","authors":"Naftali Tishby, Fernando C Pereira, William Bialek","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Van der Waerden, 1952] BL Van der Waerden. Order tests for the two-sample problem and their power. In Indagationes Mathemati- cae (Proceedings), volume 55, pages 453-458. Elsevier, 1952.","title":"Order tests for the two-sample problem and their power","authors":"; Bl Van Der Waerden, Van Der Waerden","year":"1952","venue":"Indagationes Mathematicae (Proceedings)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ver Steeg and Galstyan, 2014] Greg Ver Steeg and Aram Galstyan. Discovering structure in high-dimensional data through correlation explanation. Advances in Neural Information Processing Systems (NIPS), 2014.","title":"Discovering structure in high-dimensional data through correlation explanation","authors":"Ver Steeg, Greg Galstyan, Steeg Ver, Aram Galstyan","year":"2014","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ver Steeg and Galstyan, 2015] Greg Ver Steeg and Aram Galstyan. Maximally informative hierarchical representations of high- dimensional data. In Artificial Intelligence and Statistics (AIS- TATS), 2015.","title":"Maximally informative hierarchical representations of highdimensional data","authors":"Ver Steeg, Greg Galstyan, Steeg Ver, Aram Galstyan","year":"2015","venue":"Artificial Intelligence and Statistics (AIS-TATS)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ver Steeg and Galstyan, 2016] Greg Ver Steeg and Aram Galstyan. The information sieve. In International Conference on Machine Learning (ICML), 2016.","title":"Greg Ver Steeg and Aram Galstyan. The information sieve","authors":"Ver Steeg, Galstyan","year":"2016","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ver Steeg, 2016] Greg Ver Steeg. Linear information sieve code. http://github.com/gregversteeg/LinearSieve, 2016.","title":"Greg Ver Steeg. Linear information sieve code","authors":"Ver Steeg","year":"2016","venue":null,"doi":null,"url":"http://github.com/gregversteeg/LinearSieve","internal_url":null},{"position":36,"raw":"[Wang et al., 2010] Meihong Wang, Fei Sha, and Michael I Jordan. Unsupervised kernel dimension reduction. In Advances in Neural Information Processing Systems, pages 2379-2387, 2010.","title":"Unsupervised kernel dimension reduction","authors":"Wang","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Watanabe, 1960] Satosi Watanabe. Information theoretical anal- ysis of multivariate correlation. IBM Journal of research and development, 4(1):66-82, 1960.","title":"Information theoretical analysis of multivariate correlation","authors":"Satosi Watanabe, Watanabe","year":"1960","venue":"IBM Journal of research and development","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wyner, 1975] Aaron D Wyner. The common information of two dependent random variables. Information Theory, IEEE Transac- tions on, 21(2):163-179, 1975.","title":"The common information of two dependent random variables","authors":"D Aaron, Wyner","year":"1975","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xu et al., 2013] Ge Xu, Wei Liu, and Biao Chen. Wyner's com- mon information: Generalizations and a new lossy source coding interpretation. arXiv preprint arXiv:1301.2237, 2013.","title":"Wyner's common information: Generalizations and a new lossy source coding interpretation","authors":"Ge Xu, Wei Liu, Biao Chen","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}