{"type":"ScholarlyArticle","title":"Parametric Manifold Learning of Gaussian Mixture Models","authors":[{"name":"Ziquan Liu","slug":"ziquan-liu","given":"Ziquan","family":"Liu","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Lei Yu","slug":"lei-yu","given":"Lei","family":"Yu","affiliation":"Department of Computer Science , City University of Hong Kong"},{"name":"Janet H. Hsiao","slug":"janet-h-hsiao","given":"Janet H.","family":"Hsiao","affiliation":"Department of Psychology , The University of Hong Kong"},{"name":"Antoni B. Chan","slug":"antoni-b-chan","given":"Antoni B.","family":"Chan","affiliation":"Department of Computer Science , City University of Hong Kong"}],"abstract":"The Gaussian Mixture Model (GMM) is among the most widely used parametric probability distributions for representing data. However, it is complicated to analyze the relationship among GMMs since they lie on a high-dimensional manifold. Previous works either perform clustering of GMMs, which learns a limited discrete latent representation, or kernel-based embedding of GMMs, which is not interpretable due to difficulty in computing the inverse mapping. In this paper, we propose Parametric Manifold Learning of GMMs (PML-GMM), which learns a parametric mapping from a low-dimensional latent space to a high-dimensional GMM manifold. Similar to PCA, the proposed mapping is parameterized by the principal axes for the component weights, means, and covariances, which are optimized to minimize the reconstruction loss measured using Kullback-Leibler divergence (KLD). As the KLD between two GMMs is intractable, we approximate the objective function by a variational upper bound, which is optimized by an EM-style algorithm. Moreover, We derive an efficient solver by alternating optimization of subproblems and exploit Monte Carlo sampling to escape from local minima. We demonstrate the effectiveness of PML-GMM through experiments on synthetic, eye-fixation, flow cytometry, and social check-in data.","doi":"10.24963/ijcai.2019/426","url":"https://ijcai.org/proceedings/2019/426","pdf":"https://ijcai.org/proceedings/2019/0426.pdf","markdown":"https://ijcai.org/proceedings/2019/426.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/426","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":426,"pages":{"first":3073,"last":3079},"keywords":["Machine Learning: Probabilistic Machine Learning","Machine Learning: Unsupervised Learning","Machine Learning: Interpretability","Machine Learning: Dimensionality Reduction and Manifold Learning"],"videos":[],"references":[{"position":1,"raw":"[Aghaeepour et al., 2013] Nima Aghaeepour, Greg Finak, Holger Hoos, Tim R Mosmann, Ryan Brinkman, Raphael Gottardo, Richard H Scheuermann, FlowCAP Consortium, DREAM Consortium, et al. Critical assessment of auto- mated flow cytometry data analysis techniques. Nature methods, 10(3):228, 2013.","title":"Critical assessment of automated flow cytometry data analysis techniques","authors":"Nima Aghaeepour, Greg Finak, Holger Hoos, Ryan Tim R Mosmann, Raphael Brinkman, Richard H Gottardo, ; Scheuermann","year":"2013","venue":"Nature methods","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amari and Nagaoka, 2007] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American Mathematical Soc., 2007.","title":"Shun-ichi Amari and Hiroshi Nagaoka","authors":"Nagaoka Amari","year":"2007","venue":"Methods of information geometry","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carter et al., 2009] Kevin M Carter, Raviv Raich, William G Finn, and Alfred O Hero III. Fine: Fisher information non- parametric embedding. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 31(11):2093-2098, 2009.","title":"Fine: Fisher information nonparametric embedding","authors":"Kevin M Carter, Raviv Raich, William G Finn, Alfred O Hero, Iii","year":"2009","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2010] Antoni B Chan, Emanuele Coviello, and Gert RG Lanckriet. Clustering dynamic textures with the hierarchical EM algorithm. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2022-2029. IEEE, 2010.","title":"Clustering dynamic textures with the hierarchical EM algorithm","authors":"Emanuele Antoni B Chan, Gert Rg Coviello, Lanckriet","year":"2010","venue":"Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chan et al., 2018] Cynthia YH Chan, Antoni B Chan, Ta- tia MC Lee, and Janet H Hsiao. Eye-movement patterns in face recognition are associated with cognitive decline in older adults. Psychonomic bulletin and review, pages 1-8, 2018.","title":"Eye-movement patterns in face recognition are associated with cognitive decline in older adults","authors":"Antoni B Cynthia Yh Chan, Tatia Mc Chan, Janet H Lee, Hsiao","year":"2018","venue":"Psychonomic bulletin and review","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2011] Eunjoon Cho, Seth A. Myers, and Jure Leskovec. Friendship and mobility:user movement in location-based social networks. In ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, San Diego, Ca, Usa, August, pages 1082-1090, 2011.","title":"Friendship and mobility:user movement in location-based social networks","authors":"Cho","year":"2011","venue":"ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Coviello et al., 2012] Emanuele Coviello, Gert R Lanckriet, and Antoni B Chan. The variational hierarchical EM algo- rithm for clustering hidden Markov models. In Advances in neural information processing systems, pages 404-412, 2012.","title":"The variational hierarchical EM algorithm for clustering hidden Markov models","authors":"Emanuele Coviello, Gert R Lanckriet, Antoni B Chan","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Doretto et al., 2003] Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. International Journal of Computer Vision, 51(2):91-109, 2003.","title":"Dynamic textures","authors":"Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, Stefano Soatto","year":"2003","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Hastings, 1970","title":null,"authors":"Hastings","year":"1970","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.","title":"Monte carlo sampling methods using markov chains and their applications","authors":"Hastings Keith","year":"1970","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hershey and Olsen, 2007] John R Hershey and Peder A Olsen. Approximating the Kullback Leibler divergence between Gaussian mixture models. In ICASSP, 2007.","title":"Approximating the Kullback Leibler divergence between Gaussian mixture models","authors":"R John, Peder A Hershey, Olsen","year":"2007","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jebara et al., 2004] Tony Jebara, Risi Kondor, and Andrew Howard. Probability product kernels. Journal of Machine Learning Research, 5(Jul):819-844, 2004.","title":"Probability product kernels","authors":"Tony Jebara, Risi Kondor, Andrew Howard","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kullback, 1997] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.","title":"Information theory and statistics","authors":"Kullback","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lawrence, 2004] Neil D Lawrence. Gaussian process la- tent variable models for visualisation of high dimensional data. In Advances in neural information processing systems, pages 329-336, 2004.","title":"Gaussian process latent variable models for visualisation of high dimensional data","authors":"Lawrence Neil","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Ziquan Liu, Lei Yu, Janet H. Hsiao, and Antoni B. Chan. Parametric manifold learning of Gaussian mixture models-Supplementary. http://visal.cs.cityu.edu. hk/static/pubs/conf/ijcai19-pml-gmm-supp.pdf, 2019.","title":"Parametric manifold learning of Gaussian mixture models-Supplementary","authors":"Liu","year":"2019","venue":null,"doi":null,"url":"http://visal.cs.cityu.edu.hk/static/pubs/conf/ijcai19-pml-gmm-supp.pdf","internal_url":null},{"position":16,"raw":"[Metropolis et al., 1953] Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1092, 1953.","title":"Equation of state calculations by fast computing machines","authors":"Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, Edward Teller","year":"1953","venue":"The journal of chemical physics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moreno et al., 2004] Pedro J Moreno, Purdy P Ho, and Nuno Vasconcelos. A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications. In Ad- vances in neural information processing systems, pages 1385-1392, 2004.","title":"A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications","authors":"Pedro J Moreno, P Purdy, Nuno Ho, Vasconcelos","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Perronnin et al., 2006] Florent Perronnin, Christopher Dance, Gabriela Csurka, and Marco Bressan. Adapted vocabularies for generic visual categorization. In European Conference on Computer Vision, pages 464-475, 2006.","title":"Adapted vocabularies for generic visual categorization","authors":"Florent Perronnin, Christopher Dance, Gabriela Csurka, Marco Bressan","year":"2006","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rabiner, 1989] Lawrence R Rabiner. A tutorial on hidden Markov models and selected applications in speech recog- nition. Proceedings of the IEEE, 77(2):257-286, 1989.","title":"A tutorial on hidden Markov models and selected applications in speech recognition","authors":"Lawrence R Rabiner","year":"1989","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rose, 1998] Kenneth Rose. Deterministic annealing for clustering, compression, classification, regression, and re- lated optimization problems. Proceedings of the IEEE, 86(11):2210-2239, 1998.","title":"Deterministic annealing for clustering, compression, classification, regression, and related optimization problems","authors":"Kenneth Rose","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sánchez et al., 2013] Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. Image classification with the Fisher vector: theory and practice. International Journal of Computer Vision, 105(3):222-245, Dec 2013.","title":"Image classification with the Fisher vector: theory and practice","authors":"Jorge Sánchez, Florent Perronnin, Thomas Mensink, Jakob Verbeek","year":"2013","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schölkopf et al., 1998] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural computa- tion, 10(5):1299-1319, 1998.","title":"Nonlinear component analysis as a kernel eigenvalue problem","authors":"Bernhard Schölkopf, Alexander Smola, Klaus-Robert Müller","year":"1998","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vasconcelos and Lippman, 1999] Nuno Vasconcelos and Andrew Lippman. Learning mixture hierarchies. In Ad- vances in Neural Information Processing Systems, pages 606-612, 1999.","title":"Learning mixture hierarchies","authors":"Nuno Vasconcelos, Andrew Lippman","year":"1999","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Winn et al., 2005] J. Winn, A. Criminisi, and T. Minka. Ob- ject categorization by learned universal visual dictionary. In Tenth IEEE International Conference on Computer Vision, pages 1800-1807 Vol. 2, 2005.","title":"Object categorization by learned universal visual dictionary","authors":"J Winn, A Criminisi, T Minka","year":"2005","venue":"Tenth IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2016] Lei Yu, Tianyu Yang, and Antoni B Chan. Approximate inference for generic likelihoods via density- preserving GMM simplification. In Workshop on Advances in Neural Information Processing Systems, 2016.","title":"Approximate inference for generic likelihoods via densitypreserving GMM simplification","authors":"Lei Yu, Tianyu Yang, Antoni B Chan","year":"2016","venue":"Workshop on Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2018] Lei Yu, Tianyu Yang, and Antoni B Chan. Density-preserving hierarchical EM algorithm: Simplifying Gaussian mixture models for approximate inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1):1-1, 2018.","title":"Density-preserving hierarchical EM algorithm: Simplifying Gaussian mixture models for approximate inference","authors":"Lei Yu, Tianyu Yang, Antoni B Chan","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}