{"type":"ScholarlyArticle","title":"Can We Find Neurons that Cause Unrealistic Images in Deep Generative Networks?","authors":[{"name":"Hwanil Choi","slug":"hwanil-choi","given":"Hwanil","family":"Choi","affiliation":"Graduate School of AI , KAIST , Seongnam , Republic of Korea"},{"name":"Wonjoon Chang","slug":"wonjoon-chang","given":"Wonjoon","family":"Chang","affiliation":"Graduate School of AI , KAIST , Seongnam , Republic of Korea"},{"name":"Jaesik Choi","slug":"jaesik-choi","given":"Jaesik","family":"Choi","affiliation":"Graduate School of AI , KAIST , Seongnam , Republic of Korea; INEEJI , Republic of Korea"}],"abstract":"Even though Generative Adversarial Networks (GANs) have shown a remarkable ability to generate high-quality images, GANs do not always guarantee the generation of photorealistic images. Occasionally, they generate images that have defective or unnatural objects, which are referred to as `artifacts'. Research to investigate why these artifacts emerge and how they can be detected and removed has yet to be sufficiently carried out. To analyze this, we first hypothesize that rarely activated neurons and frequently activated neurons have different purposes and responsibilities for the progress of generating images. In this study, by analyzing the statistics and the roles for those neurons, we empirically show that rarely activated neurons are related to the failure results of making diverse objects and inducing artifacts. In addition, we suggest a correction method, called `Sequential Ablation’, to repair the defective part of the generated images without high computational cost and manual efforts.","doi":"10.24963/ijcai.2022/400","url":"https://ijcai.org/proceedings/2022/400","pdf":"https://ijcai.org/proceedings/2022/0400.pdf","markdown":"https://ijcai.org/proceedings/2022/400.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/400","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":400,"pages":{"first":2888,"last":2894},"keywords":["Machine Learning: Generative Adverserial Networks","Machine Learning: Explainable/Interpretable Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986831","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984974","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bau et al., 2019] David Bau, Jun-Yan Zhu, Hendrik Stro- belt, Bolei Zhou, Joshua B. Tenenbaum, William T. Free- man, and Antonio Torralba. Visualizing and understanding generative adversarial networks. In Proceedings of ICLR, 2019.","title":"Visualizing and understanding generative adversarial networks","authors":"David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, Antonio Torralba","year":"2019","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brock et al., 2019] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fi- delity natural image synthesis. In Proceedings of ICLR, 2019.","title":"Large scale GAN training for high fidelity natural image synthesis","authors":"Andrew Brock, Jeff Donahue, Karen Simonyan","year":"2019","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Yang, 2021] Zehao Chen and Hua Yang. Atten- tive semantic exploring for manipulated face detection. Proceedings of ICASSP, Jun 2021.","title":"Attentive semantic exploring for manipulated face detection","authors":"Chen, Yang ; ] Zehao Chen, Hua Yang","year":"2021","venue":"Proceedings of ICASSP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022] Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu. Sofgan: A por- trait image generator with dynamic styling. ACM Trans. Graph., 41(1), feb 2022.","title":"Sofgan: A portrait image generator with dynamic styling","authors":"Chen","year":"2022","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Esser et al., 2021] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of CVPR, pages 12873- 12883, June 2021.","title":"Taming transformers for high-resolution image synthesis","authors":"Patrick Esser, Robin Rombach, Bjorn Ommer","year":"2021","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in NeurIPS, 27, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in NeurIPS, 30, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hung et al., 2021] Tun-Min Hung, Bo-Yu Chen, Yen-Tung Yeh, and Yi-Hsuan Yang. A benchmarking initiative for audio-domain music generation using the freesound loop dataset. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy, editors, Proceedings of ISMIR, 2021, Online, November 7-12, 2021, pages 310-317, 2021.","title":"A benchmarking initiative for audio-domain music generation using the freesound loop dataset","authors":"Tun-Min Hung, Bo-Yu Chen, Yen-Tung Yeh, Yi-Hsuan Yang","year":"2021","venue":"Proceedings of ISMIR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karras et al., 2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In Proceedings of ICLR, 2018.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2018","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karras et al., 2019] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of CVPR, pages 4401-4410, 2019.","title":"A style-based generator architecture for generative adversarial networks","authors":"Karras","year":"2019","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karras et al., 2020] Tero Karras, Samuli Laine, Miika Ait- tala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. An- alyzing and improving the image quality of stylegan. In Proceedings of CVPR, pages 8110-8119, 2020.","title":"Analyzing and improving the image quality of stylegan","authors":"Karras","year":"2020","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karras et al., 2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in NeurIPS, 34, 2021.","title":"Alias-free generative adversarial networks","authors":"Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, Timo Aila","year":"2021","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2021] Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploiting spatial di- mensions of latent in gan for real-time image editing. In Proceedings of CVPR, pages 852-861, June 2021.","title":"Exploiting spatial dimensions of latent in gan for real-time image editing","authors":"Kim","year":"2021","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kynkäänniemi et al., 2019] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing gener- ative models. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Ad- vances in NeurIPS, volume 32.","title":"Improved precision and recall metric for assessing generative models","authors":"Kynkäänniemi","year":"2019","venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. Blendgan: Implicitly gan blending for arbitrary stylized face gener- ation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in NeurIPS, volume 34, pages 29710-29722.","title":"Blendgan: Implicitly gan blending for arbitrary stylized face generation","authors":"Mingcong Liu, Qiang Li, Zekui Qin, Guoxin Zhang, Pengfei Wan, Wen Zheng","year":null,"venue":"Advances in NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Odena et al., 2016] Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 2016.","title":"Deconvolution and checkerboard artifacts","authors":"Odena","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Selvaraju et al., 2019] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Ad- vances in IJCV, 128(2):336-359, Oct 2019.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"R Ramprasaath, Michael Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2019","venue":"Advances in IJCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shen et al., 2020] Yujun Shen, Yinghao Xu, Ceyuan Yang, Jiapeng Zhu, and Bolei Zhou. Genforce. https://github. com/genforce/genforce, 2020.","title":"Genforce","authors":"Yujun Shen, Yinghao Xu, Ceyuan Yang, Jiapeng Zhu, Bolei Zhou","year":"2020","venue":null,"doi":null,"url":"https://github.com/genforce/genforce","internal_url":null},{"position":20,"raw":"[Tousi et al., 2021] Ali Tousi, Haedong Jeong, Jiyeon Han, Hwanil Choi, and Jaesik Choi. Automatic correction of internal units in generative neural networks. In Proceed- ings of CVPR, pages 7932-7940, 2021.","title":"Automatic correction of internal units in generative neural networks","authors":"Ali Tousi, Haedong Jeong, Jiyeon Han, Hwanil Choi, Jaesik Choi","year":"2021","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] T. Y. Wang, D. Ceylan, K. Singh, and N. J. Mitra. Dance in the wild: Monocular human ani- mation with neural dynamic appearance synthesis. In Pro- ceedings of 3DV, pages 268-277. IEEE Computer Society, dec 2021.","title":"Dance in the wild: Monocular human animation with neural dynamic appearance synthesis","authors":"T Y Wang, D Ceylan, K Singh, N J Mitra","year":"2021","venue":"Proceedings of 3DV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2021] Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. Advances in IJCV, 129(5):1451-1466, 2021.","title":"Semantic hierarchy emerges in deep generative representations for scene synthesis","authors":"Yang","year":"2021","venue":"Advances in IJCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2021] Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier. A latent transformer for disentangled face editing in images and videos. In Proceedings of ICCV, pages 13789-13798, October 2021.","title":"A latent transformer for disentangled face editing in images and videos","authors":"Xu Yao, Alasdair Newson, Yann Gousseau, Pierre Hellier","year":"2021","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}