{"type":"ScholarlyArticle","title":"Ensuring Rapid Mixing and Low Bias for Asynchronous Gibbs Sampling","authors":[{"name":"Christopher De Sa","slug":"christopher-de-sa","given":"Christopher","family":"De Sa","affiliation":"Stanford University , Stanford , California"},{"name":"Kunle Olukotun","slug":"kunle-olukotun","given":"Kunle","family":"Olukotun","affiliation":"Stanford University , Stanford , California"},{"name":"Christopher Ré","slug":"christopher-re","given":"Christopher","family":"Ré","affiliation":"Stanford University , Stanford , California"}],"abstract":"Gibbs sampling is a Markov chain Monte Carlo technique commonly used for estimating marginal distributions. To speed up Gibbs sampling, there has recently been interest in parallelizing it by executing asynchronously.  While empirical results suggest that many models can be efficiently sampled asynchronously, traditional Markov chain analysis does not apply to the asynchronous case, and thus asynchronous Gibbs sampling is poorly understood. In this paper, we derive a better understanding of the two main challenges of asynchronous Gibbs: bias and mixing time.  We show experimentally that our theoretical results match practical outcomes.","doi":"10.24963/ijcai.2017/672","url":"https://ijcai.org/proceedings/2017/672","pdf":"https://ijcai.org/proceedings/2017/0672.pdf","markdown":"https://ijcai.org/proceedings/2017/672.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/672","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":672,"pages":{"first":4811,"last":4815},"keywords":["Artificial Intelligence: machine learning"],"videos":[],"references":[{"position":1,"raw":"[De Sa et al., 2015] Christopher De Sa, Ce Zhang, Kunle Olukotun, and Christopher Ré. Taming the wild: A unified analysis of HOGWILD!-style algorithms. In NIPS, 2015.","title":"Taming the wild: A unified analysis of HOGWILD!-style algorithms","authors":"Christopher De Sa, Ce Zhang, Kunle Olukotun, Christopher Ré","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dobrushin, 1956] RL Dobrushin. Central limit theorem for nonstationary markov chains. i. Theory of Probability & Its Applications, 1(4):329-383, 1956.","title":"Central limit theorem for nonstationary markov chains","authors":"Rl Dobrushin","year":"1956","venue":"i. Theory of Probability & Its Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dyer et al., 2006] Martin Dyer, Leslie Ann Goldberg, and Mark Jerrum. Dobrushin conditions and systematic scan. In in Proc. 10th International Workshop on Randomiza- tion and Computation, Lecture Notes in Computer Science 4110, pages 327-338. Springer, 2006.","title":"Dobrushin conditions and systematic scan","authors":"Martin Dyer, Leslie, Ann Goldberg, Mark Jerrum","year":"2006","venue":"Proc. 10th International Workshop on Randomization and Computation","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hayes, 2006] Thomas P Hayes. A simple condition imply- ing rapid mixing of single-site dynamics on spin systems. In Foundations of Computer Science, 2006. FOCS'06. 47th Annual IEEE Symposium on, pages 39-46. IEEE, 2006.","title":"A simple condition implying rapid mixing of single-site dynamics on spin systems","authors":"P Thomas, Hayes","year":"2006","venue":"FOCS'06. 47th Annual IEEE Symposium on","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Johnson et al., 2013] Matthew Johnson, James Saunderson, and Alan Willsky. Analyzing hogwild parallel gaussian Gibbs sampling. In NIPS, pages 2715-2723, 2013.","title":"Analyzing hogwild parallel gaussian Gibbs sampling","authors":"Matthew Johnson, James Saunderson, Alan Willsky","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Koller and Friedman, 2009] Daphne Koller and Nir Fried- man. Probabilistic graphical models: principles and tech- niques. MIT press, 2009.","title":"Probabilistic graphical models: principles and techniques","authors":"Friedman Koller","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Levin et al., 2009] David Asher Levin, Yuval Peres, and Elizabeth Lee Wilmer. Markov chains and mixing times. American Mathematical Soc., 2009.","title":"Markov chains and mixing times","authors":"David Asher, Levin, Yuval Peres, Elizabeth Lee, Wilmer","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu and Wright, 2015] Ji Liu and Stephen J. Wright. Asyn- chronous stochastic coordinate descent: Parallelism and convergence properties. SIOPT, 25(1):351-376, 2015.","title":"Asynchronous stochastic coordinate descent: Parallelism and convergence properties","authors":"Ji Liu, Stephen J Wright","year":"2015","venue":"SIOPT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2015] Ji Liu, Stephen J Wright, Christopher Ré, Victor Bittorf, and Srikrishna Sridhar. An asynchronous parallel stochastic coordinate descent algorithm. JMLR, 16:285-322, 2015.","title":"An asynchronous parallel stochastic coordinate descent algorithm","authors":"Ji Liu, Stephen J Wright, Christopher Ré, Victor Bittorf, Srikrishna Sridhar","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lunn et al., 2009] David Lunn, David Spiegelhalter, An- drew Thomas, and Nicky Best. The BUGS project: evolu- tion, critique and future directions. Statistics in medicine, (25):3049-3067, 2009.","title":"The BUGS project: evolution, critique and future directions","authors":"David Lunn, David Spiegelhalter, Andrew Thomas, Nicky Best","year":"2009","venue":"Statistics in medicine","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mania et al., 2015] Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, and Michael I Jordan. Perturbed iterate analysis for asynchronous stochastic optimization. arXiv preprint arXiv:1507.06970, 2015.","title":"Perturbed iterate analysis for asynchronous stochastic optimization","authors":"Horia Mania, Xinghao Pan, Dimitris Papailiopoulos, Benjamin Recht, Kannan Ramchandran, Michael I Jordan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[McCallum et al., 2009] Andrew McCallum, Karl Schultz, and Sameer Singh. Factorie: Probabilistic programming via imperatively defined factor graphs. In NIPS, pages 1249-1257, 2009.","title":"Factorie: Probabilistic programming via imperatively defined factor graphs","authors":"Andrew Mccallum, Karl Schultz, Sameer Singh","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Neubig, 2014] Graham Neubig. Simple, correct paralleliza- tion for blocked Gibbs sampling. Technical report, Nara Institute of Science and Technology, 2014.","title":"Simple, correct parallelization for blocked Gibbs sampling","authors":"Graham Neubig","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Newman et al., 2007] David Newman, Padhraic Smyth, Max Welling, and Arthur U Asuncion. Distributed infer- ence for latent dirichlet allocation. In NIPS, pages 1081- 1088, 2007.","title":"Distributed inference for latent dirichlet allocation","authors":"David Newman, Padhraic Smyth, Max Welling, Arthur U Asuncion","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nguyen et al., 2013] Duc Thien Nguyen, William Yeoh, and Hoong Chuin Lau. Distributed Gibbs: A memory- bounded sampling-based DCOP algorithm. In Proceed- ings of the 2013 international conference on Autonomous agents and multi-agent systems, pages 167-174. Interna- tional Foundation for Autonomous Agents and Multiagent Systems, 2013.","title":"Distributed Gibbs: A memorybounded sampling-based DCOP algorithm","authors":"William Duc Thien Nguyen, Hoong Chuin Yeoh, Lau","year":"2013","venue":"Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Niu et al., 2011] Feng Niu, Benjamin Recht, Christopher Re, and Stephen Wright. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, pages 693-701, 2011.","title":"Hogwild: A lock-free approach to parallelizing stochastic gradient descent","authors":"Feng Niu, Benjamin Recht, Christopher Re, Stephen Wright","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shin et al., 2015] Jaeho Shin, Sen Wu, Feiran Wang, Christopher De Sa, Ce Zhang, Feiran Wang, and Christo- pher Ré. Incremental knowledge base construction using DeepDive. PVLDB, 2015.","title":"Incremental knowledge base construction using DeepDive","authors":"Jaeho Shin, Sen Wu, Feiran Wang, Christopher De Sa, Ce Zhang, Feiran Wang, Christopher Ré","year":"2015","venue":"PVLDB","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Smola and Narayanamurthy, 2010] Alexander Smola and Shravan Narayanamurthy. An architecture for parallel topic models. PVLDB, 2010.","title":"An architecture for parallel topic models","authors":"Alexander Smola, Shravan Narayanamurthy","year":"2010","venue":"PVLDB","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutter, 2005] Herb Sutter. The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software. Dr. Dobb's Journal, 30(3), 2005.","title":"The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software","authors":"Herb Sutter, Sutter","year":"2005","venue":"Dr. Dobb's Journal","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Theis et al., 2012] Lucas Theis, Jascha Sohl-dickstein, and Matthias Bethge. Training sparse natural image models with a fast Gibbs sampler of an extended state space. In NIPS, pages 1124-1132. 2012.","title":"Training sparse natural image models with a fast Gibbs sampler of an extended state space","authors":"Lucas Theis, Jascha Sohl-Dickstein, Matthias Bethge","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Ré, 2014] Ce Zhang and Christopher Ré. DimmWitted: A study of main-memory statistical analytics. PVLDB, 2014.","title":"DimmWitted: A study of main-memory statistical analytics","authors":"Ce Zhang, Christopher Ré","year":"2014","venue":"PVLDB","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}