{"type":"ScholarlyArticle","title":"CoCoG: Controllable Visual Stimuli Generation Based on Human Concept Representations","authors":[{"name":"Chen Wei","slug":"chen-wei","given":"Chen","family":"Wei","affiliation":"Southern University of Science and Technology , Shenzhen , China; University of Birmingham , Birmingham , United Kingdom"},{"name":"Jiachen Zou","slug":"jiachen-zou","given":"Jiachen","family":"Zou","affiliation":"Southern University of Science and Technology , Shenzhen , China"},{"name":"Dietmar Heinke","slug":"dietmar-heinke","given":"Dietmar","family":"Heinke","affiliation":"University of Birmingham , Birmingham , United Kingdom"},{"name":"Quanying Liu","slug":"quanying-liu","given":"Quanying","family":"Liu","affiliation":"Southern University of Science and Technology , Shenzhen , China"}],"abstract":"A central question for cognitive science is to understand how humans process visual scenes, i.e, to uncover human low-dimensional concept representation space from high-dimensional visual stimuli. Generating visual stimuli with controlling concepts is the key. However, there are currently no generative models in AI to solve this problem.  Here, we present the Concept based Controllable Generation (CoCoG) framework. CoCoG consists of two components, a simple yet efficient AI agent for extracting interpretable concept and predicting human decision-making in visual similarity judgment tasks, and a conditional generation model for generating visual stimuli given the concepts.  We quantify the performance of CoCoG from two aspects, the human behavior prediction accuracy and the controllable generation ability. The experiments with CoCoG indicate that 1) the reliable concept embeddings in CoCoG allows to predict human behavior with 64.07% accuracy in the THINGS-similarity dataset; 2) CoCoG can generate diverse stimuli through the control of concepts; 3) CoCoG can manipulate human similarity judgment behavior by intervening key concepts.\r\nCoCoG offers visual objects with controlling concepts to advance our understanding of causality in human cognition. The code of CoCoG framework is available at https://github.com/ncclab-sustech/CoCoG.","doi":"10.24963/ijcai.2024/352","url":"https://ijcai.org/proceedings/2024/352","pdf":"https://ijcai.org/proceedings/2024/0352.pdf","markdown":"https://ijcai.org/proceedings/2024/352.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/352","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Humans and AI","track":"Main Track","number":352,"pages":{"first":3178,"last":3186},"keywords":["Humans and AI: HAI: Cognitive modeling","Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Cognitive systems"],"videos":[],"references":[{"position":1,"raw":"[Bansal et al., 2023] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.","title":"Universal guidance for diffusion models","authors":"Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chung et al., 2022] Hyungjin Chung, Byeongsu Sim, Do- hoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683- 25696, 2022.","title":"Improving diffusion models for inverse problems using manifold constraints","authors":"Chung","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan et al., 2023] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models. arXiv preprint arXiv:2305.16381, 2023.","title":"Dpok: Reinforcement learning for fine-tuning text-toimage diffusion models","authors":"Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, Kimin Lee","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2023] Stephanie Fu, Netanel Tamir, Shobhita Sun- daram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306.09344, 2023.","title":"Dreamsim: Learning new dimensions of human visual similarity using synthetic data","authors":"Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, Phillip Isola","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2020] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.","title":"Generative adversarial networks","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2020","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hebart et al., 2020] Martin N Hebart, Charles Y Zheng, Francisco Pereira, and Chris I Baker. Revealing the multi- dimensional mental representations of natural objects un- derlying human similarity judgements. Nature human be- haviour, 4(11):1173-1185, 2020.","title":"Revealing the multidimensional mental representations of natural objects underlying human similarity judgements","authors":"Charles Y Martin N Hebart, Francisco Zheng, Chris I Pereira, Baker","year":"2020","venue":"Nature human behaviour","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hebart et al., 2023] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris I Baker. Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12:e82580, 2023.","title":"Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior","authors":"Oliver Martin N Hebart, Lina Contier, Adam H Teichmann, Charles Y Rockter, Alexis Zheng, Anna Kidder, Maryam Corriveau, Chris I Vaziri-Pashkam, Baker","year":"2023","venue":"Elife","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ho and Salimans, 2022] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.","title":"Classifier-free diffusion guidance","authors":"Jonathan Ho, Tim Salimans","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jha et al., 2023] Aditi Jha, Joshua C Peterson, and Thomas L Griffiths. Extracting low-dimensional psy- chological representations from convolutional neural networks. Cognitive science, 47(1):e13226, 2023.","title":"Extracting low-dimensional psychological representations from convolutional neural networks","authors":"Aditi Jha, Joshua C Peterson, Thomas L Griffiths","year":"2023","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kawar et al., 2022] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restora- tion models. Advances in Neural Information Processing Systems, 35:23593-23606, 2022.","title":"Denoising diffusion restoration models","authors":"Bahjat Kawar, Michael Elad, Stefano Ermon, Jiaming Song","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirstain et al., 2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to- image generation. arXiv preprint arXiv:2305.01569, 2023.","title":"Pick-a-pic: An open dataset of user preferences for text-toimage generation","authors":"Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Marjieh et al., 2022a] Raja Marjieh, Ilia Sucholutsky, Theodore R Sumers, Nori Jacoby, and Thomas L Grif- fiths. Predicting human similarity judgments using large language models. arXiv preprint arXiv:2202.04728, 2022.","title":"Predicting human similarity judgments using large language models","authors":"Marjieh","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Marjieh et al., 2022b] Raja Marjieh, Pol Van Rijn, Ilia Su- cholutsky, Theodore Sumers, Harin Lee, Thomas L Grif- fiths, and Nori Jacoby. Words are all you need? language as an approximation for human similarity judgments. In The Eleventh International Conference on Learning Rep- resentations, 2022.","title":"Words are all you need? language as an approximation for human similarity judgments","authors":"Marjieh","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Medin et al., 1993] Douglas L Medin, Robert L Goldstone, and Dedre Gentner. Respects for similarity. Psychological review, 100(2):254, 1993.","title":"Respects for similarity","authors":"Robert L Douglas L Medin, Dedre Goldstone, Gentner","year":"1993","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Meng et al., 2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Er- mon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Confer- ence on Learning Representations, 2021.","title":"Sdedit: Guided image synthesis and editing with stochastic differential equations","authors":"Meng","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Murphy and Medin, 1985] Gregory L Murphy and Dou- glas L Medin. The role of theories in conceptual coher- ence. Psychological review, 92(3):289, 1985.","title":"The role of theories in conceptual coherence","authors":"L Gregory, Douglas L Murphy, Medin","year":"1985","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Muttenthaler et al., 2022a] Lukas Muttenthaler, Jonas Dip- pel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Kornblith. Human alignment of neural network repre- sentations. In The Eleventh International Conference on Learning Representations, 2022.","title":"Human alignment of neural network representations","authors":"Muttenthaler","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Muttenthaler et al., 2022b] Lukas Muttenthaler, Charles Y Zheng, Patrick McClure, Robert A Vandermeulen, Mar- tin N Hebart, and Francisco Pereira. Vice: Variational in- terpretable concept embeddings. Advances in Neural In- formation Processing Systems, 35:33661-33675, 2022.","title":"Vice: Variational interpretable concept embeddings","authors":"Muttenthaler","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Muttenthaler et al., 2023] Lukas Muttenthaler, Lorenz Lin- hardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, Andrew K Lampinen, and Simon Kornblith. Im- proving neural network representations using human simi- larity judgments. arXiv preprint arXiv:2306.04507, 2023.","title":"Improving neural network representations using human similarity judgments","authors":"Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, Andrew K Lampinen, Simon Kornblith","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Özbey et al., 2023] Muzaffer Özbey, Onat Dalmaz, Salman UH Dar, Hasan A Bedel, S ¸aban Özturk, Alper Güngör, and Tolga C ¸ukur. Unsupervised medical image translation with adversarial diffusion models. IEEE Transactions on Medical Imaging, 2023.","title":"Unsupervised medical image translation with adversarial diffusion models","authors":"Muzaffer Özbey, Onat Dalmaz, Salman Uh Dar, A Hasan, S Bedel, Alper ¸aban Özturk, Tolga C Güngör, ¸ukur","year":"2023","venue":"IEEE Transactions on Medical Imaging","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Peterson et al., 2018] Joshua C Peterson, Joshua T Abbott, and Thomas L Griffiths. Evaluating (and improving) the correspondence between deep neural networks and hu- man representations. Cognitive science, 42(8):2648-2669, 2018.","title":"Evaluating (and improving) the correspondence between deep neural networks and human representations","authors":"Joshua T Joshua C Peterson, Thomas L Abbott, Griffiths","year":"2018","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Podell et al., 2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving la- tent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.","title":"Sdxl: Improving latent diffusion models for high-resolution image synthesis","authors":"Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, Robin Rombach","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rainforth et al., 2023] Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith. Modern bayesian experimental design. arXiv preprint arXiv:2302.14545, 2023.","title":"Modern bayesian experimental design","authors":"Rainforth","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Ramesh et al., ] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. [Roads and Love, 2021]","title":"Hierarchical textconditional image generation with clip latents","authors":"Ramesh","year":"2021","venue":"Roads and Love","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Brett D Roads and Bradley C Love. Enriching imagenet with human similarity judgments and psychological embeddings. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 3547-3557, 2021.","title":"Enriching imagenet with human similarity judgments and psychological embeddings","authors":"D Brett, Roads, C Bradley, Love","year":"2021","venue":"Proceedings of the ieee/cvf conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Roads and Love, 2023] Brett D Roads and Bradley C Love. Modeling similarity and psychological space. Annual Re- view of Psychology, 75, 2023.","title":"Modeling similarity and psychological space","authors":"D Brett, Roads, C Bradley, Love","year":"2023","venue":"Annual Review of Psychology","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sauer et al., 2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023.","title":"Adversarial diffusion distillation","authors":"Axel Sauer, Dominik Lorenz, Andreas Blattmann, Robin Rombach","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Song et al., 2020] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.","title":"Score-based generative modeling through stochastic differential equations","authors":"Yang Song, Jascha Sohl-Dickstein, P Diederik, Abhishek Kingma, Stefano Kumar, Ben Ermon, Poole","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Song et al., 2021] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In Interna- tional Conference on Learning Representations, 2021.","title":"Solving inverse problems in medical imaging with score-based generative models","authors":"Yang Song, Liyue Shen, Lei Xing, Stefano Ermon","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tang et al., 2023] Zhiwei Tang, Dmitry Rybin, and Tsung- Hui Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303.03751, 2023.","title":"Zeroth-order optimization meets human feedback: Provable learning via ranking oracles","authors":"Zhiwei Tang, Dmitry Rybin, Tsung-Hui Chang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tao et al., 2022] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthe- sis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16515- 16525, 2022.","title":"Df-gan: A simple and effective baseline for text-to-image synthesis","authors":"Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, Changsheng Xu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[von Rütte et al., 2023] Dimitri von Rütte, Elisabetta Fedele, Jonathan Thomm, and Lukas Wolf. Fabric: Personalizing diffusion models with iterative feedback. arXiv preprint arXiv:2307.10159, 2023.","title":"Fabric: Personalizing diffusion models with iterative feedback","authors":"Elisabetta Dimitri Von Rütte, Jonathan Fedele, Lukas Thomm, Wolf","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu et al., 2023] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Bet- ter aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096-2105, 2023.","title":"Human preference score: Better aligning text-to-image models with human preference","authors":"Wu","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ye et al., 2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023.","title":"Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models","authors":"Jun Hu Ye, Sibo Zhang, Xiao Liu, Wei Han, Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yu et al., 2023] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided conditional diffusion model. arXiv preprint arXiv:2303.09833, 2023.","title":"Freedom: Training-free energy-guided conditional diffusion model","authors":"Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, Jian Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 3836-3847, 2023.","title":"Adding conditional control to text-to-image diffusion models","authors":"Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zheng et al., 2018] Charles Y Zheng, Francisco Pereira, Chris I Baker, and Martin N Hebart. Revealing inter- pretable object representations from human behavior. In International Conference on Learning Representations, 2018.","title":"Revealing interpretable object representations from human behavior","authors":"Francisco Charles Y Zheng, Chris I Pereira, Martin N Baker, Hebart","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}