{"type":"ScholarlyArticle","title":"Knowledge-Based Regularization in Generative Modeling","authors":[{"name":"Naoya Takeishi","slug":"naoya-takeishi","given":"Naoya","family":"Takeishi","affiliation":"RIKEN Center for Advanced Intelligence Project"},{"name":"Yoshinobu Kawahara","slug":"yoshinobu-kawahara","given":"Yoshinobu","family":"Kawahara","affiliation":"Institute of Mathematics for Industry , Kyushu University; RIKEN Center for Advanced Intelligence Project"}],"abstract":"Prior domain knowledge can greatly help to learn generative models. However, it is often too costly to hard-code prior knowledge as a specific model architecture, so we often have to use general-purpose models. In this paper, we propose a method to incorporate prior knowledge of feature relations into the learning of general-purpose generative models. To this end, we formulate a regularizer that makes the marginals of a generative model to follow prescribed relative dependence of features. It can be incorporated into off-the-shelf learning methods of many generative models, including variational autoencoders and generative adversarial networks, as its gradients can be computed using standard backpropagation techniques. We show the effectiveness of the proposed method with experiments on multiple types of datasets and generative models.","doi":"10.24963/ijcai.2020/331","url":"https://ijcai.org/proceedings/2020/331","pdf":"https://ijcai.org/proceedings/2020/0331.pdf","markdown":"https://ijcai.org/proceedings/2020/331.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/331","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":331,"pages":{"first":2390,"last":2396},"keywords":["Machine Learning: Knowledge-based Learning","Machine Learning: Deep Generative Models","Machine Learning: Learning Generative Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24017/file_24017.m3u8","length_sec":253,"provider":"hls"}],"references":[{"position":1,"raw":"[Bounliphone et al., 2015] Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus, and Matthew B. Blaschko. A low variance consistent test of relative dependency. In Proc. of the 32nd Int. Conf. on Machine Learning, pages 20-29, 2015.","title":"A low variance consistent test of relative dependency","authors":"Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus, Matthew B Blaschko","year":"2015","venue":"Proc. of the 32nd Int. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cussens et al., 2017] James Cussens, Matti Järvisalo, Janne H. Korhonen, and Mark Bartlett. Bayesian network structure learning with integer programming: Polytopes, facets and complexity. Journal of Artificial Intelligence Research, 58:185-229, 2017.","title":"Bayesian network structure learning with integer programming: Polytopes, facets and complexity","authors":"James Cussens, Matti Järvisalo, Janne H Korhonen, Mark Bartlett","year":"2017","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dinh et al., 2018] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proc. of the 6th Int. Conf. on Learning Representations, 2018.","title":"Density estimation using real NVP","authors":"Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio","year":"2018","venue":"Proc. of the 6th Int. Conf. on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Downs and Vogel, 1993] James J. Downs and Ernest F. Vogel. A plant-wide industrial process control prob- lem. Computers & Chemical Engineering, 17(3):245-255, 1993.","title":"A plant-wide industrial process control problem","authors":"James J Downs, Ernest F Vogel","year":"1993","venue":"Computers & Chemical Engineering","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ganchev et al., 2010] Kuzman Ganchev, João Grac ¸a, Jen- nifer Gillenwater, and Ben Taskar. Posterior regulariza- tion for structured latent variable models. J. of Machine Learning Research, 11:2001-2049, 2010.","title":"Posterior regularization for structured latent variable models","authors":"Kuzman Ganchev, João Grac ¸a, Jennifer Gillenwater, Ben Taskar","year":"2010","venue":"J. of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in Neural Informa- tion Processing Systems 27, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gretton et al., 2005] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statis- tical dependence with Hilbert-Schmidt norms. In Algo- rithmic Learning Theory, pages 63-77, 2005.","title":"Measuring statistical dependence with Hilbert-Schmidt norms","authors":"Arthur Gretton, Olivier Bousquet, Alex Smola, Bernhard Schölkopf","year":"2005","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018] Zhiting Hu, Zichao Yang, Ruslan Salakhut- dinov, Xiaodan Liang, Lianhui Qin, Haoye Dong, and Eric P. Xing. Deep generative models with learnable knowledge constraints. In Advances in Neural Informa- tion Processing Systems 31, 2018.","title":"Deep generative models with learnable knowledge constraints","authors":"Hu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2011] Junzhou Huang, Tong Zhang, and Dim- itris Mataxas. Learning with structured sparsity. J. of Ma- chine Learning Research, 12:3371-3412, 2011.","title":"Learning with structured sparsity","authors":"Huang","year":"2011","venue":"J. of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proc. of the 2nd Int. Conf. on Learning Representations, 2014.","title":"Auto-encoding variational Bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"Proc. of the 2nd Int. Conf. on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koller and Friedman, 2009] Daphne Koller and Nir Fried- man. Probabilistic Graphical Models: Principles and Techniques. The MIT Press, 2009.","title":"Probabilistic Graphical Models: Principles and Techniques","authors":"Friedman Koller","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koller and Pfeffer, 1997] Daphne Koller and Avi Pfeffer. Object-oriented Bayesian networks. In Proc. of the 13th Conf. on Uncertainty in Artificial Intelligence, pages 302- 313, 1997.","title":"Object-oriented Bayesian networks","authors":"Pfeffer Koller, Daphne Koller, Avi Pfeffer","year":"1997","venue":"Proc. of the 13th Conf. on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krupka and Tishby, 2007] Eyal Krupka and Naftali Tishby. Incorporating prior knowledge on features into learning. In Proc. of the 11th Int. Conf. on Artificial Intelligence and Statistics, pages 227-234, 2007.","title":"Incorporating prior knowledge on features into learning","authors":"Tishby Krupka, Eyal Krupka, Naftali Tishby","year":"2007","venue":"Proc. of the 11th Int. Conf. on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Li, 2008] Caiyan Li and Hongzhe Li. Network- constrained regularization and variable selection for anal- ysis of genomic data. Bioinformatics, 24(9):1175-1182, 2008.","title":"Networkconstrained regularization and variable selection for analysis of genomic data","authors":"Li ; Li, Caiyan Li, Hongzhe Li","year":"2008","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and van Beek, 2018] Andrew C Li and Peter van Beek. Bayesian network structure learning with side constraints. In Proc. of the 9th Int. Conf. on Probabilistic Graphical Models, pages 225-236, 2018.","title":"Bayesian network structure learning with side constraints","authors":"C Andrew, Peter Li, Van Beek","year":"2018","venue":"Proc. of the 9th Int. Conf. on Probabilistic Graphical Models","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Yingming Li, Ming Yang, Zenglin Xu, and Zhongfei (Mark) Zhang. Learning with feature network and label network simultaneously. In Proc. of the 31st AAAI Conf. on Artificial Intelligence, pages 1410-1416, 2017.","title":"Learning with feature network and label network simultaneously","authors":"Li","year":"2017","venue":"Proc. of the 31st AAAI Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2015] Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. Learning semantic word embeddings based on ordinal knowledge constraints. In Proc. of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Int. Joint Conf. on Natural Lan- guage Processing, pages 1501-1511, 2015.","title":"Learning semantic word embeddings based on ordinal knowledge constraints","authors":"Quan Liu, Hui Jiang, Si Wei, Zhen-Hua Ling, Yu Hu","year":"2015","venue":"Proc. of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Int. Joint Conf. on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lopez et al., 2018] Romain Lopez, Jeffrey Regier, Michael I Jordan, and Nir Yosef. Information con- straints on auto-encoding variational Bayes. In Advances in Neural Information Processing Systems 31, 2018.","title":"Information constraints on auto-encoding variational Bayes","authors":"Romain Lopez, Jeffrey Regier, Michael I Jordan, Nir Yosef","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mei et al., 2014] Shike Mei, Jun Zhu, and Xiaojin Zhu. Ro- bust RegBayes: Selectively incorporating first-order logic domain knowledge into Bayesian models. In Proc. of the 31st Int. Conf. on Machine Learning, pages 253-261, 2014.","title":"Robust RegBayes: Selectively incorporating first-order logic domain knowledge into Bayesian models","authors":"Shike Mei, Jun Zhu, Xiaojin Zhu","year":"2014","venue":"Proc. of the 31st Int. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Metz et al., 2017] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In Proc. of the 5th Int. Conf. on Learning Rep- resentations, 2017.","title":"Unrolled generative adversarial networks","authors":"Metz","year":"2017","venue":"Proc. of the 5th Int. Conf. on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mollaysa et al., 2017] Amina Mollaysa, Pablo Strasser, and Alexandros Kalousis. Regularising non-linear models us- ing feature side-information. In Proc. of the 34th Int. Conf. on Machine Learning, pages 2508-2517, 2017.","title":"Regularising non-linear models using feature side-information","authors":"Amina Mollaysa, Pablo Strasser, Alexandros Kalousis","year":"2017","venue":"Proc. of the 34th Int. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pfister et al., 2017] Niklas Pfister, Peter Bühlmann, Bern- hard Schölkopf, and Jonas Peters. Kernel-based tests for joint independence. J. of the Royal Statistical Society: Se- ries B, 80(1):5-31, 2017.","title":"Kernel-based tests for joint independence","authors":"Pfister","year":"2017","venue":"J. of the Royal Statistical Society: Series B","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sandler et al., 2009] Ted Sandler, John Blitzer, Partha P. Talukdar, and Lyle H. Ungar. Regularized learning with networks of features. In Advances in Neural Information Processing Systems 21, pages 1401-1408, 2009.","title":"Regularized learning with networks of features","authors":"Ted Sandler, John Blitzer, Partha P Talukdar, Lyle H Ungar","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2012] Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. J. of Machine Learning Re- search, 13:1393-1434, 2012.","title":"Feature selection via dependence maximization","authors":"Le Song, Alex Smola, Arthur Gretton, Justin Bedo, Karsten Borgwardt","year":"2012","venue":"J. of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[van den Oord et al., 2016] Aäron van den Oord, Nal Kalch- brenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proc. of the 33rd Int. Conf. on Machine Learning, pages 1747-1756, 2016.","title":"Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks","authors":"Oord Van Den","year":"2016","venue":"Proc. of the 33rd Int. Conf. on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2015] Pengtao Xie, Diyi Yang, and Eric Xing. In- corporating word correlation knowledge into topic model- ing. In Proc. of the 2015 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 725-734, 2015.","title":"Incorporating word correlation knowledge into topic modeling","authors":"Xie","year":"2015","venue":"Proc. of the 2015 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Qinyi Zhang, Sarah Filippi, Arthur Gretton, and Dino Sejdinovic. Large-scale kernel meth- ods for independence testing. Statistics and Computing, 28(1):113-130, 2018.","title":"Large-scale kernel methods for independence testing","authors":"Zhang","year":"2018","venue":"Statistics and Computing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang, 2009] Yi Zhang. Smart PCA. In Proc. of the 21st Int. Joint Conf. on Artificial Intelligence, pages 1351-1356, 2009.","title":"Smart PCA","authors":"Yi Zhang","year":"2009","venue":"Proc. of the 21st Int. Joint Conf. on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2014] Jun Zhu, Ning Chen, and Eric P. Xing. Bayesian inference with posterior regularization and ap- plications to infinite latent SVMs. J. of Machine Learning Research, 15:1799-1847, 2014.","title":"Bayesian inference with posterior regularization and applications to infinite latent SVMs","authors":"Jun Zhu, Ning Chen, Eric P Xing","year":"2014","venue":"J. of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}