{"type":"ScholarlyArticle","title":"Learning Interpretable Representations with Informative Entanglements","authors":[{"name":"Ege Beyazıt","slug":"ege-beyaz-t","given":"Ege","family":"Beyazıt","affiliation":"University of Louisiana at Lafayette , Lafayette , LA , USA"},{"name":"Doruk Tuncel","slug":"doruk-tuncel","given":"Doruk","family":"Tuncel","affiliation":"Johannes Kepler University Linz , Linz , Austria"},{"name":"Xu Yuan","slug":"xu-yuan","given":"Xu","family":"Yuan","affiliation":"University of Louisiana at Lafayette , Lafayette , LA , USA"},{"name":"Nian-Feng Tzeng","slug":"nian-feng-tzeng","given":"Nian-Feng","family":"Tzeng","affiliation":"University of Louisiana at Lafayette , Lafayette , LA , USA"},{"name":"Xindong Wu","slug":"xindong-wu","given":"Xindong","family":"Wu","affiliation":"Mininglamp Academy of Sciences , Beijing , China"}],"abstract":"Learning interpretable representations in an unsupervised setting is an important yet a challenging task. Existing unsupervised interpretable methods focus on extracting independent salient features from data. However they miss out the fact that the entanglement of salient features may also be informative. Acknowledging these entanglements can improve the interpretability, resulting in extraction of higher quality and a wider variety of salient features. In this paper, we propose a new method to enable Generative Adversarial Networks (GANs) to discover salient features that may be entangled in an informative manner, instead of extracting only disentangled features. Specifically, we propose a regularizer to punish the disagreement between the extracted feature interactions and a given dependency structure while training. We model these interactions using a Bayesian network, estimate the maximum likelihood parameters and calculate a negative likelihood score to measure the disagreement. Upon qualitatively and quantitatively evaluating the proposed method using both synthetic and real-world datasets, we show that our proposed regularizer guides GANs to learn representations with disentanglement scores competing with the state-of-the-art, while extracting a wider variety of salient features.","doi":"10.24963/ijcai.2020/273","url":"https://ijcai.org/proceedings/2020/273","pdf":"https://ijcai.org/proceedings/2020/0273.pdf","markdown":"https://ijcai.org/proceedings/2020/273.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/273","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":273,"pages":{"first":1970,"last":1976},"keywords":["Machine Learning: Interpretability","Machine Learning: Deep Learning","Machine Learning: Deep Generative Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24869/file_24869.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Adel et al., 2018] Tameem Adel, Zoubin Ghahramani, and Adrian Weller. Discovering interpretable representations for both deep generative and discriminative models. In International Conference on Machine Learning, pages 50- 59, 2018.","title":"Discovering interpretable representations for both deep generative and discriminative models","authors":"Tameem Adel, Zoubin Ghahramani, Adrian Weller","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.","title":"Pattern recognition and machine learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dupont, 2018] Emilien Dupont. Learning disentangled joint continuous and discrete representations. In Advances in Neural Information Processing Systems, pages 710-720, 2018.","title":"Learning disentangled joint continuous and discrete representations","authors":"Emilien Dupont","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Esmaeili et al., 2018] Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan-Willem van de Meent. Structured disentangled representations. arXiv preprint arXiv:1804.02086, 2018.","title":"Structured disentangled representations","authors":"Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, Jan-Willem Van De Meent","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gilpin et al., 2018] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Con- ference on data science and advanced analytics (DSAA), pages 80-89. IEEE, 2018.","title":"Explaining explanations: An overview of interpretability of machine learning","authors":"David Leilani H Gilpin, Bau, Ayesha Ben Z Yuan, Michael Bajwa, Lalana Specter, Kagal","year":"2018","venue":"IEEE 5th International Conference on data science and advanced analytics (DSAA)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow, 2016] Ian Goodfellow. Nips 2016 tuto- rial: Generative adversarial networks. arXiv preprint arXiv:1701.00160, 2016.","title":"Nips 2016 tutorial: Generative adversarial networks","authors":"Ian Goodfellow","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Higgins et al., 2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained varia- tional framework. ICLR, 2(5):6, 2017.","title":"beta-vae: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim and Mnih, 2018] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv:1802.05983, 2018.","title":"Disentangling by factorising","authors":"Mnih Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kurutach et al., 2018] Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learn- ing plannable representations with causal infogan. In Ad- vances in Neural Information Processing Systems, pages 8733-8744, 2018.","title":"Learning plannable representations with causal infogan","authors":"Kurutach","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 2010] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2:18, 2010.","title":"Mnist handwritten digit database","authors":"Lecun","year":"2010","venue":null,"doi":null,"url":"http://yann.lecun.com/exdb/mnist,2:18","internal_url":null},{"position":14,"raw":"[Lipton, 2018] Zachary C Lipton. The mythos of model in- terpretability. Queue, 16(3):31-57, 2018.","title":"The mythos of model interpretability","authors":"Zachary C Lipton","year":"2018","venue":"Queue","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Martin and Lon, 2017] Arjovsky Martin and B Lon. To- wards principled methods for training generative adver- sarial networks. In NIPS 2016 Workshop on Adversarial Training. In review for ICLR, volume 2016, 2017.","title":"Towards principled methods for training generative adversarial networks","authors":"Martin, Lon ;, Arjovsky Martin, B Lon","year":"2016","venue":"NIPS 2016 Workshop on Adversarial Training. In review for ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Matthey et al., 2017] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disen- tanglement testing sprites dataset. URL https://github. com/deepmind/dsprites-dataset/.[Accessed on: 2018-05- 08], 2017.","title":"dsprites: Disentanglement testing sprites dataset","authors":"Loic Matthey, Irina Higgins, Demis Hassabis, Alexander Lerchner","year":"2017","venue":null,"doi":null,"url":"https://github.com/deepmind/dsprites-dataset/","internal_url":null},{"position":17,"raw":"[Nielsen and Jensen, 2009] Thomas Dyhre Nielsen and Finn Verner Jensen. Bayesian networks and decision graphs. Springer Science & Business Media, 2009.","title":"Bayesian networks and decision graphs","authors":"Jensen ; Nielsen, Dyhre Thomas, Finn Nielsen, Jensen Verner","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paysan et al., 2009] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recogni- tion. In 2009 Sixth IEEE International Conference on Ad- vanced Video and Signal Based Surveillance, pages 296- 301. Ieee, 2009.","title":"A 3d face model for pose and illumination invariant face recognition","authors":"Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, Thomas Vetter","year":"2009","venue":"Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Perry et al., 2010] Gavin Perry, ET Rolls, and SM Stringer. Continuous transformation learning of translation in- variant representations. Experimental brain research, 204(2):255-270, 2010.","title":"Continuous transformation learning of translation invariant representations","authors":"Gavin Perry, Rolls, Stringer","year":"2010","venue":"Experimental brain research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ranzato et al., 2007] Marc'Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In 2007 IEEE conference on computer vision and pattern recognition, pages 1-8. IEEE, 2007.","title":"Unsupervised learning of invariant feature hierarchies with applications to object recognition","authors":"Ranzato","year":"2007","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ross and Doshi-Velez, 2018] Andrew Slavin Ross and Fi- nale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regulariz- ing their input gradients. In Thirty-second AAAI confer- ence on artificial intelligence, 2018.","title":"Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients","authors":"Ross, Doshi-Velez ;, Andrew Slavin, Ross, Finale Doshi-Velez","year":"2018","venue":"Thirty-second AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schmidhuber, 1992] Jürgen Schmidhuber. Learning facto- rial codes by predictability minimization. Neural Compu- tation, 4(6):863-879, 1992.","title":"Learning factorial codes by predictability minimization","authors":"Jürgen Schmidhuber","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shen et al., 2019] Yujia Shen, Anchal Goyanka, Adnan Darwiche, and Arthur Choi. Structured bayesian net- works: From inference to learning with routes. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7957-7965, 2019.","title":"Structured bayesian networks: From inference to learning with routes","authors":"Yujia Shen, Anchal Goyanka, Adnan Darwiche, Arthur Choi","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2013] Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. Tensor analyzers. In International conference on machine learning, pages 163-171, 2013.","title":"Tensor analyzers","authors":"Yichuan Tang, Ruslan Salakhutdinov, Geoffrey Hinton","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}