{"type":"ScholarlyArticle","title":"General Purpose MRF Learning with Neural Network Potentials","authors":[{"name":"Hao Xiong","slug":"hao-xiong","given":"Hao","family":"Xiong","affiliation":"The University of Texas at Dallas"},{"name":"Nicholas Ruozzi","slug":"nicholas-ruozzi","given":"Nicholas","family":"Ruozzi","affiliation":"The University of Texas at Dallas"}],"abstract":"Maximum likelihood learning is a well-studied approach for fitting discrete Markov random fields (MRFs) to data.  However, general purpose maximum likelihood estimation for fitting MRFs with continuous variables have only been studied in much more limited settings.  In this work, we propose a generic MLE estimation procedure for MRFs whose potential functions are modeled by neural networks.  To make learning effective in practice, we show how to leverage a highly parallelizable variational inference method that can easily fit into popular machining learning frameworks like TensorFlow. We demonstrate experimentally that our approach is capable of effectively modeling the data distributions of a variety of real data sets and that it can compete effectively with other common methods on multilabel classification and generative modeling tasks.","doi":"10.24963/ijcai.2020/384","url":"https://ijcai.org/proceedings/2020/384","pdf":"https://ijcai.org/proceedings/2020/0384.pdf","markdown":"https://ijcai.org/proceedings/2020/384.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/384","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":384,"pages":{"first":2769,"last":2776},"keywords":["Uncertainty in AI: Approximate Probabilistic Inference","Machine Learning: Learning Graphical Models","Uncertainty in AI: Graphical Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25057/file_25057.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2015] L.-C. Chen, A. Schwing, A. Yuille, and R. Urtasun. Learning deep structured models. In Interna- tional Conference on Machine Learning (ICML), 2015.","title":"Learning deep structured models","authors":"L.-C Chen, A Schwing, A Yuille, R Urtasun","year":"2015","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Do and Artieres, 2010] T. Do and T. Artieres. Neural con- ditional random fields. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.","title":"Neural conditional random fields","authors":"T Do, T Artieres","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Domke, 2013] J. Domke. Learning graphical model param- eters with approximate marginal inference. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 35(10):2454-2467, 2013.","title":"Learning graphical model parameters with approximate marginal inference","authors":"J Domke","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ganapathi et al., 2008] V. Ganapathi, D. Vickrey, J. Duchi, and D. Koller. Constrained approximate maximum en- tropy learning of Markov random fields. In Proceedings of the Twenty-Fourth Conference on Uncertainty in Artifi- cial Intelligence (UAI), 2008.","title":"Constrained approximate maximum entropy learning of Markov random fields","authors":"V Ganapathi, D Vickrey, J Duchi, D Koller","year":"2008","venue":"Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gershman et al., 2012] S. J. Gershman, M. D. Hoffman, and D. M. Blei. Nonparametric variational inference. In Pro- ceedings of the 29th International Conference on Machine Learning (ICML), pages 235-242, 2012.","title":"Nonparametric variational inference","authors":"S J Gershman, M D Hoffman, D M Blei","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Golub and Welsch, 1969] G. H. Golub and J. H. Welsch. Calculation of Gauss quadrature rules. Mathematics of computation, 23(106):221-230, 1969.","title":"Calculation of Gauss quadrature rules","authors":"G H Golub, J H Welsch","year":"1969","venue":"Mathematics of computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2019] Y. Guo, H. Xiong, and N. Ruozzi. Marginal inference in continuous Markov random fields using mixtures. In 33rd Conference on Artificial Intelli- gence (AAAI), Jan. 2019.","title":"Marginal inference in continuous Markov random fields using mixtures","authors":"Y Guo, H Xiong, N Ruozzi","year":"2019","venue":"33rd Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hazan and Urtasun, 2010] T. Hazan and R. Urtasun. A primal-dual message-passing algorithm for approximated large scale structured prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2010.","title":"A primal-dual message-passing algorithm for approximated large scale structured prediction","authors":"T Hazan, R Urtasun","year":"2010","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heskes and Zoeter, 2002] T. Heskes and O. Zoeter. Expec- tation propagation for approximate inference in dynamic bayesian networks. In Proceedings of the Eighteenth Con- ference on Uncertainty in Artificial Intelligence (UAI), pages 216-223, 2002.","title":"Expectation propagation for approximate inference in dynamic bayesian networks","authors":"T Heskes, O Zoeter","year":"2002","venue":"Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heskes and Zoeter, 2005] T. Heskes and O. Zoeter. Gaus- sian quadrature based expectation propagation. In Pro- ceedings of the Eighth International Conference on Artifi- cial Intelligence and Statistics (AISTATS), 2005.","title":"Gaussian quadrature based expectation propagation","authors":"T Heskes, O Zoeter","year":"2005","venue":"Proceedings of the Eighth International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ihler and McAllester, 2009] A. T. Ihler and D. A. McAllester. Particle belief propagation. In Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS), pages 256-263, 2009.","title":"Particle belief propagation","authors":"A T Ihler, D A Mcallester","year":"2009","venue":"Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Klambauer et al., 2017] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Self-normalizing neural net- works, 2017.","title":"Self-normalizing neural networks","authors":"G Klambauer, T Unterthiner, A Mayr, S Hochreiter","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lacoste-Julien et al., 2013] S. Lacoste-Julien, M. Jaggi, M. Schmidt, and P. Pletscher. Block-coordinate Frank- Wolfe optimization for structural svms. In International Conference on Machine Learning (ICML), 2013.","title":"Block-coordinate Frank-Wolfe optimization for structural svms","authors":"S Lacoste-Julien, M Jaggi, M Schmidt, P Pletscher","year":"2013","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lienart et al., 2015] T. Lienart, Y. W. Teh, and A. Doucet. Expectation particle belief propagation. In Advances in Neural Information Processing Systems (NIPS), pages 3609-3617, 2015.","title":"Expectation particle belief propagation","authors":"T Lienart, Y W Teh, A Doucet","year":"2015","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Malioutov et al., 2006] D. M. Malioutov, J. K. Johnson, and A. S. Willsky. Walk-sums and belief propagation in Gaus- sian graphical models. Journal of Machine Learning Re- search, 7:2031-2064, 2006.","title":"Walk-sums and belief propagation in Gaussian graphical models","authors":"D M Malioutov, J K Johnson, A S Willsky","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Minka, 2001] T. P. Minka. Expectation propagation for ap- proximate Bayesian inference. In Proceedings of the Sev- enteenth conference on Uncertainty in Artificial Intelli- gence (UAI), pages 362-369, 2001.","title":"Expectation propagation for approximate Bayesian inference","authors":"T P Minka","year":"2001","venue":"Proceedings of the Seventeenth conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Moallemi and Van Roy, 2009] C. C. Moallemi and B. Van Roy. Convergence of min-sum message passing for quadratic optimization. Information Theory, IEEE Transactions on, 55(5):2413 -2423, May 2009.","title":"Convergence of min-sum message passing for quadratic optimization","authors":"C C Moallemi, B Van Roy","year":"2009","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Noorshams and Wainwright, 2013] N. Noorshams and M. J. Wainwright. Belief propagation for continuous state spaces: stochastic message-passing with quantitative guar- antees. Journal of Machine Learning Research (JMLR), 14(1):2799-2835, 2013.","title":"Belief propagation for continuous state spaces: stochastic message-passing with quantitative guarantees","authors":"N Noorshams, M J Wainwright","year":"2013","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2009] J Peng, L Bo, and J Xu. Conditional neu- ral fields. In Advances in Neural Information Processing Systems (NeurIPS), 2009.","title":"Conditional neural fields","authors":"L Peng, Bo, Xu","year":"2009","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ruozzi and Tatikonda, 2013] N. Ruozzi and S. Tatikonda. Message-passing algorithms for quadratic minimization. The Journal of Machine Learning Research, 14(1):2287- 2314, 2013.","title":"Message-passing algorithms for quadratic minimization","authors":"N Ruozzi, S Tatikonda","year":"2013","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2011] L. Song, A. Gretton, D. Bickson, Y. Low, and C. Guestrin. Kernel belief propagation. In Proceed- ings of the Fourteenth International Conference on Artifi- cial Intelligence and Statistics (AISTATS), pages 707-715, 2011.","title":"Kernel belief propagation","authors":"L Song, A Gretton, D Bickson, Y Low, C Guestrin","year":"2011","venue":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sudderth et al., 2003] E. B. Sudderth, A. T. Ihler, M. Isard, W. T. Freeman, and A. S. Willsky. Nonparametric belief propagation. In Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society Conference on, 2003.","title":"Nonparametric belief propagation","authors":"E B Sudderth, A T Ihler, M Isard, W T Freeman, A S Willsky","year":"2003","venue":"Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2016] K. Tang, N. Ruozzi, D. Belanger, and T. Jebara. Bethe learning of graphical models via MAP de- coding. In Artificial Intelligence and Statistics (AISTATS), 2016.","title":"Bethe learning of graphical models via MAP decoding","authors":"K Tang, N Ruozzi, D Belanger, T Jebara","year":"2016","venue":"Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Weiss and Freeman, 2001] Y. Weiss and W. T. Freeman. Correctness of belief propagation in Gaussian graph- ical models of arbitrary topology. Neural Comput., 13(10):2173-2200, Oct. 2001.","title":"Correctness of belief propagation in Gaussian graphical models of arbitrary topology","authors":"Y Weiss, W T Freeman","year":"2001","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiong et al., 2019] H. Xiong, Y. Guo, Y. Yang, and N. Ruozzi. One-shot marginal map inference in Markov random fields. In Proc. 35th Uncertainty in Artifical Intel- ligence (UAI), Tel Aviv, Israel, July 2019.","title":"One-shot marginal map inference in Markov random fields","authors":"H Xiong, Y Guo, Y Yang, N Ruozzi","year":"2019","venue":"Proc. 35th Uncertainty in Artifical Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yedidia et al., 2005] J. S. Yedidia, W. T. Freeman, and Y. Weiss. Constructing free-energy approximations and generalized belief propagation algorithms. Information Theory, IEEE Transactions on, 51(7):2282 -2312, July 2005.","title":"Constructing free-energy approximations and generalized belief propagation algorithms","authors":"J S Yedidia, W T Freeman, Y Weiss","year":"2005","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2015] S. Zheng, S. Jayasumana, B. Romera- Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.","title":"Conditional random fields as recurrent neural networks","authors":"S Zheng, S Jayasumana, B Romera-Paredes, V Vineet, Z Su, D Du, C Huang, P Torr","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}