{"type":"ScholarlyArticle","title":"IID-GAN: an IID Sampling Perspective for Regularizing Mode Collapse","authors":[{"name":"Yang Li","slug":"yang-li","given":"Yang","family":"Li","affiliation":"Department of Computer Science and Engineering , MoE Key Lab of Artificial Intelligence , Shanghai Jiao Tong University"},{"name":"Liangliang Shi","slug":"liangliang-shi","given":"Liangliang","family":"Shi","affiliation":"Department of Computer Science and Engineering , MoE Key Lab of Artificial Intelligence , Shanghai Jiao Tong University"},{"name":"Junchi Yan","slug":"junchi-yan","given":"Junchi","family":"Yan","affiliation":"Department of Computer Science and Engineering , MoE Key Lab of Artificial Intelligence , Shanghai Jiao Tong University"}],"abstract":"Despite its success, generative adversarial networks (GANs) still suffer from mode collapse, i.e., the generator can only map latent variables to a partial set of modes in the target distribution. In this paper, we analyze and seek to regularize this issue with an independent and identically distributed (IID) sampling perspective and emphasize that holding the IID property referring to the target distribution for generation can naturally avoid mode collapse. This is based on the basic IID assumption for real data in machine learning. However, though the source samples {z} obey IID, the generations {G(z)} may not necessarily be IID sampling from the target distribution. Based on this observation, considering a necessary condition of IID generation, we propose a new loss to encourage the closeness between the inverse samples of real data and the Gaussian source in the latent space to regularize the generation to be IID from the target distribution. The logic is that the inverse samples from target data should also be IID in the source distribution. Experiments on both synthetic and real-world data show the effectiveness of our model.","doi":"10.24963/ijcai.2023/437","url":"https://ijcai.org/proceedings/2023/437","pdf":"https://ijcai.org/proceedings/2023/0437.pdf","markdown":"https://ijcai.org/proceedings/2023/437.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/437","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":437,"pages":{"first":3929,"last":3938},"keywords":["Computer Vision: CV: Neural generative models, auto encoders, GANs","Machine Learning: ML: Generative adverserial networks"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In ICML, pages 214-223, 2017.","title":"Wasserstein generative adversarial networks","authors":"Arjovsky","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bang and Shim, 2021] Duhyeon Bang and Hyunjung Shim. Mggan: Solving mode collapse using manifold-guided training. In ICCV, 2021.","title":"Duhyeon Bang and Hyunjung Shim. Mggan: Solving mode collapse using manifold-guided training","authors":"Shim Bang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Coates et al., 2011] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceed- ings, 2011.","title":"An analysis of single-layer networks in unsupervised feature learning","authors":"Adam Coates, Andrew Ng, Honglak Lee","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Meulemeester et al., 2021] Hannes De Meulemeester, Joachim Schreurs, Michaël Fanuel, Bart De Moor, and Johan AK Suykens. The bures metric for generative adver- sarial networks. In ECML PKDD, 2021.","title":"The bures metric for generative adversarial networks","authors":"Hannes De Meulemeester, Joachim Schreurs, Michaël Fanuel, Bart De Moor, Johan Ak Suykens","year":"2021","venue":"ECML PKDD","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dieng et al., 2019] Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias. Prescribed genera- tive adversarial networks. arXiv preprint arXiv:1910.04302, 2019.","title":"Prescribed generative adversarial networks","authors":"Francisco Jr Adji B Dieng, David M Ruiz, Michalis K Blei, Titsias","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Donahue et al., 2017] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Durugkar et al., 2017] I. Durugkar, I. Gemp, and S. Mahade- van. Generative multi-adversarial networks. In ICLR, 2017.","title":"Generative multi-adversarial networks","authors":"Durugkar","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Elfeki et al., 2019] Mohamed Elfeki, Camille Couprie, Mor- gane Riviére, and Mohamed Elhoseiny. Gdpp: Learning diverse generations using determinantal point processes. In ICML, 2019.","title":"Gdpp: Learning diverse generations using determinantal point processes","authors":"Mohamed Elfeki, Camille Couprie, Morgane Riviére, Mohamed Elhoseiny","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghosh et al., 2018] A. Ghosh, V. Kulharia, V. Namboodiri, P. H. Torr, and P. K. Dokania. Multi-agent diverse genera- tive adversarial networks. In CVPR, 2018.","title":"Multi-agent diverse generative adversarial networks","authors":"A Ghosh, V Kulharia, V Namboodiri, P H Torr, P K Dokania","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] I.","title":null,"authors":"Goodfellow","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"J Goodfellow, M Pouget-Abadie, B Mirza, D Xu, S Warde-Farley, A Ozair, Y Courville, Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gosset, 1987] Eric Gosset. A three-dimensional extended kolmogorov-smirnov test as a useful tool in astronomy. Astronomy and Astrophysics, 188:258-264, 1987.","title":"A three-dimensional extended kolmogorov-smirnov test as a useful tool in astronomy","authors":"Eric Gosset","year":"1987","venue":"Astronomy and Astrophysics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Grover et al., 2018] Aditya Grover, Manik Dhar, and Ste- fano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Proceed- ings of the AAAI conference on artificial intelligence, 2018.","title":"Flow-gan: Combining maximum likelihood and adversarial learning in generative models","authors":"Aditya Grover, Manik Dhar, Stefano Ermon","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gulrajani et al., 2017] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In NIPS, 2017.","title":"Improved training of wasserstein gans","authors":"Gulrajani","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Higgins et al., 2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Mohamed Shakir, and Alexander Lerchner. Learning basic visual concepts with a constrained variational framework. ICLR, 2017.","title":"Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Mohamed Shakir, Alexander Lerchner","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Karras et al., 2020] Tero Karras, Miika Aittala, Janne Hell- sten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-12114, 2020.","title":"Training generative adversarial networks with limited data","authors":"Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, Timo Aila","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kim and Mnih, 2018] Hyunjik Kim and Andriy Mnih. Dis- entangling by factorising. ICML, 2018.","title":"Disentangling by factorising","authors":"Mnih Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Tech Report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Ben- gio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2022] Yang Li, Yichuan Mo, Liangliang Shi, and Junchi Yan. Improving generative adversarial networks via adversarial learning in latent space. In Advances in Neural Information Processing Systems, 2022.","title":"Improving generative adversarial networks via adversarial learning in latent space","authors":"Yang Li, Yichuan Mo, Liangliang Shi, Junchi Yan","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu and Tuzel, 2016] M.-Y. Liu and O. Tuzel. Coupled gen- erative adversarial networks. In NIPS, 2016.","title":"Coupled generative adversarial networks","authors":"M.-Y Liu, O Tuzel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2020] Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba. Diverse image genera- tion via self-conditioned gans. In CVPR, 2020.","title":"Diverse image generation via self-conditioned gans","authors":"Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mangalam and Garg, 2021] Karttikeya Mangalam and Ro- hin Garg. Overcoming mode collapse with adaptive multi adversarial training. arXiv preprint arXiv:2112.14406, 2021.","title":"Karttikeya Mangalam and Rohin Garg. Overcoming mode collapse with adaptive multi adversarial training","authors":"Garg Mangalam","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mao et al., 2019] Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.","title":"Mode seeking generative adversarial networks for diverse image synthesis","authors":"Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, Ming-Hsuan Yang","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[McCloskey and Cohen, 1989] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.","title":"Catastrophic interference in connectionist networks: The sequential learning problem","authors":"Cohen Mccloskey, Michael Mccloskey, Neal J Cohen","year":"1989","venue":"Psychology of learning and motivation","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Metz et al., 2017] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. In ICLR, 2017.","title":"Unrolled generative adversarial networks","authors":"Metz","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Miyato et al., 2018] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normal- ization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.","title":"Spectral normalization for generative adversarial networks","authors":"Miyato","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Nguyen et al., 2017] T. Nguyen, T. Le, H. Vu, and D. Phung. Dual discriminator generative adversarial nets. In NIPS, 2017.","title":"Dual discriminator generative adversarial nets","authors":"T Nguyen, T Le, H Vu, D Phung","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Peyré et al., 2019] Gabriel Peyré, Marco Cuturi, et al. Com- putational optimal transport: With applications to data sci- ence. Foundations and Trends R in Machine Learning, 11(5-6):355-607, 2019.","title":"Computational optimal transport: With applications to data science","authors":"Gabriel Peyré, Marco Cuturi","year":"2019","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Radford et al., 2016] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Rezende and Mohamed, 2015] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.","title":"Variational inference with normalizing flows","authors":"Danilo Jimenez, Rezende, Shakir Mohamed","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Richardson and Weiss, 2018] Eitan Richardson and Yair Weiss. On gans and gmms. arXiv preprint arXiv:1805.12462, 2018.","title":"On gans and gmms","authors":"Weiss Richardson, Eitan Richardson, Yair Weiss","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Salimans et al., 2017] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In NIPS, 2017.","title":"Improved techniques for training gans","authors":"T Salimans, I Goodfellow, W Zaremba, V Cheung, A Radford, X Chen","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Shapiro and Wilk, 1965] Samuel Sanford Shapiro and Mar- tin B Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591-611, 1965.","title":"Samuel Sanford Shapiro and Martin B Wilk. An analysis of variance test for normality (complete samples)","authors":"Wilk Shapiro","year":"1965","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Srivastava et al., 2017] Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton. Vee- gan: Reducing mode collapse in gans using implicit varia- tional learning. In NIPS, 2017.","title":"Veegan: Reducing mode collapse in gans using implicit variational learning","authors":"Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, Charles Sutton","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tran et al., 2018] Ngoc-Trung Tran, Tuan-Anh Bui, and Ngai-Man Cheung. Dist-gan: An improved gan using distance constraints. In ECCV, 2018.","title":"Dist-gan: An improved gan using distance constraints","authors":"Ngoc-Trung Tran, Tuan-Anh Bui, Ngai-Man Cheung","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yang et al., 2019] Dingdong Yang, Seunghoon Hong, Yun- seok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In ICLR, 2019.","title":"Diversitysensitive conditional generative adversarial networks","authors":"Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, Honglak Lee","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yu et al., 2015] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.","title":"Lsun: Construction of a largescale image dataset using deep learning with humans in the loop","authors":"Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, Jianxiong Xiao","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yu et al., 2019] Xianwen Yu, Xiaoning Zhang, Yang Cao, and Min Xia. Vaegan: A collaborative filtering framework based on adversarial variational autoencoders. In IJCAI, pages 4206-4212, 2019.","title":"Vaegan: A collaborative filtering framework based on adversarial variational autoencoders","authors":"Yu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/584"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}