{"type":"ScholarlyArticle","title":"Scalable Estimation of Dirichlet Process Mixture Models on Distributed Data","authors":[{"name":"Ruohui Wang","slug":"ruohui-wang","given":"Ruohui","family":"Wang","affiliation":"Department of Information Engineering , The Chinese University of Hong Kong"},{"name":"Dahua Lin","slug":"dahua-lin","given":"Dahua","family":"Lin","affiliation":"Department of Information Engineering , The Chinese University of Hong Kong"}],"abstract":"We consider the estimation of Dirichlet Process Mixture Models (DPMMs) in distributed environments, where data are distributed across multiple computing nodes. A key advantage of Bayesian nonparametric models such as DPMMs is that they allow new components to be introduced on the fly as needed. This, however, posts an important challenge to distributed estimation -- how to handle new components efficiently and consistently. To tackle this problem, we propose a new estimation method, which allows new components to be created locally in individual computing nodes. Components corresponding to the same cluster will be identified and merged via a probabilistic consolidation scheme. In this way, we can maintain the consistency of estimation with very low communication cost. Experiments on large real-world data sets show that the proposed method can achieve high scalability in distributed and asynchronous environments without compromising the mixing performance.","doi":"10.24963/ijcai.2017/646","url":"https://ijcai.org/proceedings/2017/646","pdf":"https://ijcai.org/proceedings/2017/0646.pdf","markdown":"https://ijcai.org/proceedings/2017/646.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/646","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Uncertainty in AI","track":"Main track","number":646,"pages":{"first":4632,"last":4639},"keywords":["Uncertainty in AI: Bayesian Networks","Uncertainty in AI: Approximate Probabilistic Inference"],"videos":[],"references":[{"position":1,"raw":"[Antoniak, 1974] Charles E Antoniak. Mixtures of dirich- let processes with applications to bayesian nonparamet- ric problems. The annals of statistics, pages 1152-1174, 1974.","title":"Mixtures of dirichlet processes with applications to bayesian nonparametric problems","authors":"Charles E Antoniak","year":"1974","venue":"The annals of statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blei and Jordan, 2005] David M. Blei and Michael I. Jor- dan. Variational inference for dirichlet process mixtures. Bayesian Analysis, 1:121-144, 2005.","title":"Variational inference for dirichlet process mixtures","authors":"David M Blei, Michael I Jordan","year":"2005","venue":"Bayesian Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022, 2003.","title":"Latent dirichlet allocation","authors":"Andrew Y David M Blei, Michael I Ng, Jordan","year":"2003","venue":"Journal of machine Learning research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blei, 2016] David M Blei. The exponential family. 2016.","title":"The exponential family","authors":"M David, Blei","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Campbell et al., 2015] Trevor Campbell, Julian Straub, John W Fisher III, and Jonathan P How. Streaming, dis- tributed variational inference for bayesian nonparametrics. In Advances in Neural Information Processing Systems, pages 280-288, 2015.","title":"Streaming, distributed variational inference for bayesian nonparametrics","authors":"Trevor Campbell, Julian Straub, John W Fisher, Iii, Jonathan P How","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang and Fisher III, 2013] Jason Chang and John W Fisher III. Parallel sampling of dp mixture models using sub-cluster splits. In Advances in Neural Information Pro- cessing Systems, pages 620-628, 2013.","title":"Parallel sampling of dp mixture models using sub-cluster splits","authors":"Chang, Fisher Iii ; ], Jason Chang, John W Fisher, Iii","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gal and Ghahramani, 2014] Yarin Gal and Zoubin Ghahra- mani. Pitfalls in the use of parallel inference for the dirich- let process. In Proceedings of the 31st International Con- ference on Machine Learning (ICML-14), pages 208-216, 2014.","title":"Yarin Gal and Zoubin Ghahramani. Pitfalls in the use of parallel inference for the dirichlet process","authors":"Ghahramani Gal","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning (ICML-14)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ge et al., 2015] Hong Ge, Yutian Chen, Moquan Wan, and Zoubin Ghahramani. Distributed inference for dirichlet process mixture models. In Proceedings of the 32nd In- ternational Conference on Machine Learning (ICML-15), pages 2276-2284, 2015.","title":"Distributed inference for dirichlet process mixture models","authors":"Ge","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jain and Neal, 2004] Sonia Jain and Radford M Neal. A split-merge markov chain monte carlo procedure for the dirichlet process mixture model. Journal of Computa- tional and Graphical Statistics, 13(1):158-182, 2004.","title":"A split-merge markov chain monte carlo procedure for the dirichlet process mixture model","authors":"Sonia Jain, M Radford, Neal","year":"2004","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2010] Dahua Lin, Eric Grimson, and John Fisher. Construction of dependent dirichlet processes based on poisson processes. In Proceedings of the 23rd Interna- tional Conference on Neural Information Processing Sys- tems, pages 1396-1404.","title":"Construction of dependent dirichlet processes based on poisson processes","authors":"Lin","year":"2010","venue":"Proceedings of the 23rd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[MacEachern and Müller, 1998] Steven N MacEachern and Peter Müller. Estimating mixture of dirichlet process mod- els. Journal of Computational and Graphical Statistics, 7(2):223-238, 1998.","title":"Estimating mixture of dirichlet process models","authors":"N Steven, Peter Maceachern, Müller","year":"1998","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Meilȃ, 2003] Marina Meilȃ. Comparing clusterings by the variation of information. In Learning theory and kernel machines, pages 173-187. Springer, 2003.","title":"Marina Meilȃ. Comparing clusterings by the variation of information","authors":"Meilȃ","year":"2003","venue":"Learning theory and kernel machines","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Neal, 2000] Radford M Neal. Markov chain sampling meth- ods for dirichlet process mixture models. Journal of com- putational and graphical statistics, 9(2):249-265, 2000.","title":"Markov chain sampling methods for dirichlet process mixture models","authors":"M Radford, Neal","year":"2000","venue":"Journal of computational and graphical statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Newman et al., 2007] David Newman, Padhraic Smyth, Max Welling, and Arthur U Asuncion. Distributed infer- ence for latent dirichlet allocation. In Advances in neural information processing systems, pages 1081-1088, 2007.","title":"Distributed inference for latent dirichlet allocation","authors":"David Newman, Padhraic Smyth, Max Welling, Arthur U Asuncion","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Newman et al., 2009] David Newman, Arthur Asuncion, Padhraic Smyth, and Max Welling. Distributed algorithms for topic models. The Journal of Machine Learning Re- search, 10:1801-1828, 2009.","title":"Distributed algorithms for topic models","authors":"David Newman, Arthur Asuncion, Padhraic Smyth, Max Welling","year":"2009","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Ima- geNet Large Scale Visual Recognition Challenge. Inter- national Journal of Computer Vision (IJCV), 115(3):211- 252, 2015.","title":"Ima-geNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sandhaus, 2008] Evan Sandhaus. The new york times anno- tated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752, 2008.","title":"The new york times annotated corpus","authors":"Evan Sandhaus","year":"2008","venue":"Linguistic Data Consortium","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sethuraman, 1994] Jayaram Sethuraman. A constructive definition of dirichlet priors. Statistica sinica, pages 639- 650, 1994.","title":"A constructive definition of dirichlet priors","authors":"Jayaram Sethuraman","year":"1994","venue":"Statistica sinica","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Smyth et al., 2009] Padhraic Smyth, Max Welling, and Arthur U Asuncion. Asynchronous distributed learning of topic models. In Advances in Neural Information Process- ing Systems, pages 81-88, 2009.","title":"Asynchronous distributed learning of topic models","authors":"Padhraic Smyth, Max Welling, Arthur U Asuncion","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Szegedy et al., 2016] Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR, abs/1602.07261, 2016.","title":"Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Walker, 2007] Stephen G Walker. Sampling the dirichlet mixture model with slices. Communications in Statistic- sSimulation and Computation R , 36(1):45-54, 2007.","title":"Sampling the dirichlet mixture model with slices","authors":"G Stephen, Walker","year":"2007","venue":"Communications in Statistic-sSimulation and Computation R","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williamson et al., 2013] Sinead Williamson, Avinava Dubey, and Eric Xing. Parallel markov chain monte carlo for nonparametric mixture models. In Proceedings of the 30th International Conference on Machine Learning, pages 98-106, 2013.","title":"Parallel markov chain monte carlo for nonparametric mixture models","authors":"Sinead Williamson, Avinava Dubey, Eric Xing","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}