{"type":"ScholarlyArticle","title":"Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent (Extended Abstract)","authors":[{"name":"Frederik Kunstner","slug":"frederik-kunstner","given":"Frederik","family":"Kunstner","affiliation":"University of British Columbia"},{"name":"Raunak Kumar","slug":"raunak-kumar","given":"Raunak","family":"Kumar","affiliation":"Cornell University"},{"name":"Mark Schmidt","slug":"mark-schmidt","given":"Mark","family":"Schmidt","affiliation":"University of British Columbia; Canada CIFAR AI Chair (Amii)"}],"abstract":"Expectation maximization (EM) is the default algorithm for fitting probabilistic models with missing or latent variables, yet we lack a full understanding of its non-asymptotic convergence properties. Previous works show results along the lines of “EM converges at least as fast as gradient descent” by assuming the conditions for the convergence of gradient descent apply. This approach is not only loose, in that it does not capture that EM can make more progress than a gradient step, but the assumptions fail to hold for textbook examples of EM like Gaussian mixtures. In this work, we show that for the common setting of exponential family distributions, viewing EM as a mirror descent algorithm leads to convergence rates in Kullback-Leibler (KL) divergence and how the KL divergence is related to first-order stationarity via Bregman divergences. In contrast to previous works, the analysis is invariant to the choice of parametrization and holds with minimal assumptions. We also show applications of these ideas to local linear (and superlinear) convergence rates, generalized EM, and non-exponential family distributions.","doi":"10.24963/ijcai.2022/738","url":"https://ijcai.org/proceedings/2022/738","pdf":"https://ijcai.org/proceedings/2022/0738.pdf","markdown":"https://ijcai.org/proceedings/2022/738.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/738","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":738,"pages":{"first":5294,"last":5298},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987254","length_sec":93,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985397","length_sec":450,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amari and Nagaoka, 2000] Shun-ichi Amari and Hiroshi Nagaoka. Methods of Information Geometry. Oxford University Press, 2000.","title":"Shun-ichi Amari and Hiroshi Nagaoka","authors":"Nagaoka Amari","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amid and Warmuth, 2020] Ehsan Amid and Manfred K. Warmuth. Divergence-based motivation for online EM and combining hid- den variable models. In UAI, volume 124, pages 81-90, 2020.","title":"Divergence-based motivation for online EM and combining hidden variable models","authors":"Ehsan Amid, Manfred K Warmuth","year":"2020","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balakrishnan et al., 2017] Sivaraman Balakrishnan, Martin J. Wain- wright, and Bin Yu. Statistical guarantees for the EM algorithm: From population to sample-based analysis. Ann. Stat., 45(1):77- 120, 2017.","title":"Statistical guarantees for the EM algorithm: From population to sample-based analysis","authors":"Sivaraman Balakrishnan, Martin J Wainwright, Bin Yu","year":"2017","venue":"Ann. Stat","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banerjee et al., 2005] Arindam Banerjee, Srujana Merugu, Inder- jit S. Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. JMLR, 6:1705-1749, 2005.","title":"Clustering with bregman divergences","authors":"Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, Joydeep Ghosh","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bauschke et al., 2017] Heinz H. Bauschke, Jérôme Bolte, and Marc Teboulle. A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications. Math. Oper. Res., 42(2):330-348, 2017.","title":"A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications","authors":"H Heinz, Jérôme Bauschke, Marc Bolte, Teboulle","year":"2017","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bishop, 2007] Christopher M. Bishop. Pattern recognition and machine learning, 5th Edition. Springer, 2007.","title":"Pattern recognition and machine learning","authors":"Christopher M Bishop","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Boyles, 1983] Russell A. Boyles. On the convergence of the EM algorithm. J. R. Stat. Soc. Ser. B Methodol., 45(1):47-50, 1983.","title":"On the convergence of the EM algorithm","authors":"Russell A Boyles","year":"1983","venue":"J. R. Stat. Soc. Ser. B Methodol","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cappé and Moulines, 2009] Olivier Cappé and Eric Moulines. On- line expectation-maximization algorithm for latent data models. J. R. Stat. Soc. Ser. B Methodol., 71(3):593-613, 2009.","title":"Online expectation-maximization algorithm for latent data models","authors":"Moulines Cappé, Olivier Cappé, Eric Moulines","year":"2009","venue":"J. R. Stat. Soc. Ser. B Methodol","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Caron and Doucet, 2008] Franc ¸ois Caron and Arnaud Doucet. Sparse Bayesian nonparametric regression. In ICML, pages 88-95, 2008.","title":"Franc ¸ois Caron and Arnaud Doucet. Sparse Bayesian nonparametric regression","authors":"Doucet Caron","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2018] Jianfei Chen, Jun Zhu, Yee Whye Teh, and Tong Zhang. Stochastic expectation maximization with variance reduc- tion. In NeurIPS, pages 7978-7988, 2018.","title":"Stochastic expectation maximization with variance reduction","authors":"Jianfei Chen, Jun Zhu, Yee Whye Teh, Tong Zhang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chrétien and Hero, 2000] Stéphane Chrétien and Alfred O. Hero. Kullback proximal algorithms for maximum likelihood estimation. IEEE Trans. Inf. Theory, 46(5):1800-1810, 2000.","title":"Kullback proximal algorithms for maximum likelihood estimation","authors":"Stéphane Chrétien, Alfred O Hero","year":"2000","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Csiszár and Tusnády, 1984] Imre Csiszár and Gábor Tusnády. In- formation geometry and alternating minimization procedures. Stat. decis., 1:205-237, 1984.","title":"Imre Csiszár and Gábor Tusnády. Information geometry and alternating minimization procedures","authors":"Tusnády Csiszár","year":"1984","venue":"Stat. decis","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Daskalakis et al., 2017] Constantinos Daskalakis, Christos Tzamos, and Manolis Zampetakis. Ten steps of EM suffice for mixtures of two gaussians. In PMLR, volume 65, pages 704-710, 2017.","title":"Ten steps of EM suffice for mixtures of two gaussians","authors":"Constantinos Daskalakis, Christos Tzamos, Manolis Zampetakis","year":"2017","venue":"PMLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dempster et al., 1977] Arthur P. Dempster, Nan M. Laird, and Don- ald B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B Methodol., 39(1):1-38, 1977.","title":"Maximum likelihood from incomplete data via the EM algorithm","authors":"Arthur P Dempster, Nan M Laird, Donald B Rubin","year":"1977","venue":"J. R. Stat. Soc. Ser. B Methodol","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Figueiredo and Nowak, 2003] Mário A. T. Figueiredo and Robert D. Nowak. An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Process., 12(8):906-916, 2003.","title":"An EM algorithm for wavelet-based image restoration","authors":"A T Mário, Robert D Figueiredo, Nowak","year":"2003","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ghahramani and Jordan, 1994] Zoubin Ghahramani and Michael I. Jordan. Supervised learning from incomplete data via an EM approach. In NeurIPS, pages 120-127, 1994.","title":"Supervised learning from incomplete data via an EM approach","authors":"Zoubin Ghahramani, Michael I Jordan","year":"1994","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jordan and Xu, 1995] Michael I. Jordan and Lei Xu. Convergence results for the EM approach to mixtures of experts architectures. Neural Networks, 8(9):1409-1431, 1995.","title":"Convergence results for the EM approach to mixtures of experts architectures","authors":"Michael I Jordan, Lei Xu","year":"1995","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Karimi et al., 2019] Belhal Karimi, Hoi-To Wai, Eric Moulines, and Marc Lavielle. On the global convergence of (fast) incre- mental expectation maximization methods. In NeurIPS, pages 2837-2847, 2019.","title":"On the global convergence of (fast) incremental expectation maximization methods","authors":"Karimi","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lange et al., 2000] Kenneth Lange, David R. Hunter, and Ilsoon Yang. Optimization transfer using surrogate objective functions. J. Comput. Graph. Stat., 9(1):1-20, 2000.","title":"Optimization transfer using surrogate objective functions","authors":"Kenneth Lange, David R Hunter, Ilsoon Yang","year":"2000","venue":"J. Comput. Graph. Stat","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Le Roux et al., 2012] Nicolas Le Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponential conver- gence rate for finite training sets. In NeurIPS, pages 2672-2680, 2012.","title":"A stochastic gradient method with an exponential convergence rate for finite training sets","authors":"Le Roux","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2018] Haihao Lu, Robert M. Freund, and Yurii Nesterov. Relatively smooth convex optimization by first-order methods, and applications. SIAM J. Optim, 28(1):333-354, 2018.","title":"Relatively smooth convex optimization by first-order methods, and applications","authors":"Lu","year":"2018","venue":"SIAM J. Optim","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mairal, 2015] Julien Mairal. Incremental majorization- minimization optimization with application to large-scale machine learning. SIAM J. Optim, 25(2):829-855, 2015.","title":"Incremental majorizationminimization optimization with application to large-scale machine learning","authors":"Julien Mairal","year":"2015","venue":"SIAM J. Optim","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Meng and Rubin, 1993] Xiao-Li Meng and Donald B. Rubin. Max- imum likelihood estimation via the ECM algorithm: A general framework. Biometrika, 80(2):267-278, 1993.","title":"Maximum likelihood estimation via the ECM algorithm: A general framework","authors":"Xiao-Li Meng, Donald B Rubin","year":"1993","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Murphy, 2012] Kevin P. Murphy. Machine learning: A probabilistic perspective. MIT Press, 2012.","title":"Machine learning: A probabilistic perspective","authors":"Kevin P Murphy","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Neal and Hinton, 1998] Radford M. Neal and Geoffrey E. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pages 355-368. Springer, 1998.","title":"A view of the EM algorithm that justifies incremental, sparse, and other variants","authors":"M Radford, Geoffrey E Neal, Hinton","year":"1998","venue":"Learning in graphical models","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nemirovski and Yudin, 1983] Arkadi Nemirovski and David B. Yudin. Problem complexity and method efficiency in optimization. Wiley, NY, 1983. translated by E.R. Dawson. [Nemirovski et al., 2009]","title":"Problem complexity and method efficiency in optimization","authors":"Arkadi Nemirovski, David B Yudin","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approx- imation approach to stochastic programming. SIAM J. Optim., 19(4):1574-1609, 2009.","title":"Robust stochastic approximation approach to stochastic programming","authors":"Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, Alexander Shapiro","year":"2009","venue":"SIAM J. Optim","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Nesterov and Nemirovski, 1994] Yurii Nesterov and Arkadi Ne- mirovski. Interior-Point Polynomial Algorithms in Convex Pro- gramming. SIAM, 1994.","title":"Interior-Point Polynomial Algorithms in Convex Programming","authors":"Yurii Nesterov, Arkadi Nemirovski","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Orchard and Woodbury, 1972] Terence Orchard and Max A. Wood- bury. A missing information principle: theory and applications. In Berkeley Symp. on Math. Stat. and Prob., pages 697-715, 1972.","title":"A missing information principle: theory and applications","authors":"Terence Orchard, Max A Woodbury","year":"1972","venue":"Berkeley Symp. on Math. Stat. and Prob","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rabiner, 1989] Lawrence R. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE, 77(2):257-286, 1989.","title":"A tutorial on hidden Markov models and selected applications in speech recognition","authors":"Lawrence R Rabiner","year":"1989","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Razaviyayn, 2014] Meisam Razaviyayn. Successive convex ap- proximation: Analysis and applications. PhD thesis, University of Minnesota, 2014.","title":"Meisam Razaviyayn. Successive convex approximation: Analysis and applications","authors":"Razaviyayn","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Robbins and Monro, 1951] Herbert Robbins and Sutton Monro. A stochastic approximation method. Ann. Math. Stat., 22(3):400- 407, 1951.","title":"Herbert Robbins and Sutton Monro. A stochastic approximation method","authors":"Monro Robbins","year":"1951","venue":"Ann. Math. Stat","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Salakhutdinov et al., 2003] Ruslan Salakhutdinov, Sam T. Roweis, and Zoubin Ghahramani. Optimization with EM and expectation- conjugate-gradient. In ICML, pages 672-679, 2003.","title":"Optimization with EM and expectationconjugate-gradient","authors":"Ruslan Salakhutdinov, Sam T Roweis, Zoubin Ghahramani","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sato, 1999] Masa-aki Sato. Fast learning of on-line EM algorithm. Technical report, ATR Human Inf. Proc. Res. Lab., 1999.","title":"Fast learning of on-line EM algorithm","authors":"Masa-Aki Sato","year":"1999","venue":"ATR Human Inf. Proc. Res. Lab","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tseng, 2004] Paul Tseng. An analysis of the EM algorithm and entropy-like proximal point methods. Math. Oper. Res., 29(1):27- 44, 2004.","title":"An analysis of the EM algorithm and entropy-like proximal point methods","authors":"Paul Tseng","year":"2004","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Varadhan and Roland, 2004] Ravi Varadhan and Christophe Roland. Squared extrapolation methods (SQUAREM): A new class of simple and efficient numerical schemes for accelerating the convergence of the EM algorithm. Technical Report 63, Johns Hopkins, 2004.","title":"Squared extrapolation methods (SQUAREM): A new class of simple and efficient numerical schemes for accelerating the convergence of the EM algorithm","authors":"Ravi Varadhan, Christophe Roland","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2015] Zhaoran Wang, Quanquan Gu, Yang Ning, and Han Liu. High dimensional EM algorithm: Statistical optimization and asymptotic normality. In NeurIPS, pages 2521-2529, 2015.","title":"High dimensional EM algorithm: Statistical optimization and asymptotic normality","authors":"Wang","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wu, 1983] C. F. Jeff Wu. On the convergence properties of the EM algorithm. Ann. Stat., 11(1):95-103, 1983.","title":"On the convergence properties of the EM algorithm","authors":"C F, Jeff Wu","year":"1983","venue":"Ann. Stat","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xu and Jordan, 1996] Lei Xu and Michael I. Jordan. On conver- gence properties of the EM algorithm for gaussian mixtures. Neu- ral Comp., 8(1):129-151, 1996.","title":"On convergence properties of the EM algorithm for gaussian mixtures","authors":"Lei Xu, Michael I Jordan","year":"1996","venue":"Neural Comp","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}