{"type":"ScholarlyArticle","title":"GEM: Generating Engaging Multimodal Content","authors":[{"name":"Chongyang Gao","slug":"chongyang-gao","given":"Chongyang","family":"Gao","affiliation":"Northwestern University"},{"name":"Yiren Jian","slug":"yiren-jian","given":"Yiren","family":"Jian","affiliation":"Dartmouth College"},{"name":"Natalia Denisenko","slug":"natalia-denisenko","given":"Natalia","family":"Denisenko","affiliation":"Northwestern University"},{"name":"Soroush Vosoughi","slug":"soroush-vosoughi","given":"Soroush","family":"Vosoughi","affiliation":"Dartmouth College"},{"name":"V. S. Subrahmanian","slug":"v-s-subrahmanian","given":"V. S.","family":"Subrahmanian","affiliation":"Northwestern University"}],"abstract":"Generating engaging multimodal content is a key objective in numerous applications, such as the creation of online advertisements that captivate user attention through a synergy of images and text. In this paper, we introduce GEM, a novel framework engineered for the generation of engaging multimodal image-text posts. The GEM framework operates in two primary phases. Initially, GEM integrates a pre-trained engagement discriminator with a technique for deriving an effective continuous prompt tailored for the stable diffusion model. Subsequently, GEM unveils an iterative algorithm dedicated to producing coherent and compelling image-sentence pairs centered around a specified topic of interest. Through a combination of experimental analysis and human evaluations, we establish that the image-sentence pairs generated by GEM not only surpass several established baselines in terms of engagement but also in achieving superior alignment.","doi":"10.24963/ijcai.2024/847","url":"https://ijcai.org/proceedings/2024/847","pdf":"https://ijcai.org/proceedings/2024/0847.pdf","markdown":"https://ijcai.org/proceedings/2024/847.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/847","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"AI, Arts & Creativity","track":"AI, Arts & Creativity","number":847,"pages":{"first":7654,"last":7662},"keywords":["Application domains: Images, movies and visual arts","Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI","Application domains: Text, literature and creative language","Methods and resources: Machine learning, deep learning, neural models, reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"2: Iterative Generation Process 1 T P : text paraphraser 2 T xt: Input text 3 Sim: the similarity score provided by CLIP 4 S: threshold of the termination 5 I e = f d (Concate(cp, Emb(T xt))) 6 Sim O = Sim(T xt, I e ) 7 T xt ′ = T P (T xt) 8 for Number of iteration do References [Agarap, 2018] Abien Fred Agarap. Deep learning using rec- tified linear units (relu). arXiv preprint arXiv:1803.08375, 2018.","title":"Iterative Generation Process 1 T P : text paraphraser 2 T xt: Input text 3 Sim: the similarity score provided by CLIP 4 S: threshold of the termination 5 I e = f d (Concate(cp, Emb(T xt))) 6 Sim O = Sim(T xt, I e ) 7 T xt ′ = T P (T xt) 8 for Number of iteration do References","authors":null,"year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal and Kann, 2020] Rajat Agarwal and Katharina Kann. Acrostic poem generation. arXiv preprint arXiv:2010.02239, 2020.","title":"Acrostic poem generation","authors":"Rajat Agarwal, Katharina Kann","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chandrasekaran et al., 2017] Arjun Chandrasekaran, Devi Parikh, and Mohit Bansal. Punny captions: Witty wordplay in image descriptions. arXiv preprint arXiv:1704.08224, 2017.","title":"Punny captions: Witty wordplay in image descriptions","authors":"Arjun Chandrasekaran, Devi Parikh, Mohit Bansal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2024] Chongyang Gao, Sushil Jajodia, Andrea Pugliese, and VS Subrahmanian. Fakedb: Generating fake synthetic databases. IEEE Transactions on Dependable and Secure Computing, 2024.","title":"Fakedb: Generating fake synthetic databases","authors":"Gao","year":"2024","venue":"IEEE Transactions on Dependable and Secure Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2020] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.","title":"Generative adversarial networks","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2020","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2022] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified dis- crete diffusion for simultaneous vision-language genera- tion. arXiv preprint arXiv:2211.14842, 2022.","title":"Unified discrete diffusion for simultaneous vision-language generation","authors":"Hu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 709-727. Springer, 2022.","title":"Visual prompt tuning","authors":"Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jian et al., 2023] Yiren Jian, Tingkai Liu, Yunzhe Tao, Soroush Vosoughi, and Hongxia Yang. Simvlg: Sim- ple and efficient pretraining of visual language generative models. arXiv preprint arXiv:2310.03291, 2023.","title":"Simvlg: Simple and efficient pretraining of visual language generative models","authors":"Yiren Jian, Tingkai Liu, Yunzhe Tao, Soroush Vosoughi, Hongxia Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jian et al., 2024] Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Bootstrapping vision-language learning with decoupled language pre-training. Advances in Neural In- formation Processing Systems, 36, 2024.","title":"Bootstrapping vision-language learning with decoupled language pre-training","authors":"Yiren Jian, Chongyang Gao, Soroush Vosoughi","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2020] Seungbae Kim, Jyun-Yu Jiang, Masaki Nakada, Jinyoung Han, and Wei Wang. Multimodal post attentive profiling for influencer marketing. In Proceed- ings of The Web Conference 2020, pages 2878-2884, 2020.","title":"Multimodal post attentive profiling for influencer marketing","authors":"Kim","year":"2020","venue":"Proceedings of The Web Conference 2020","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2021] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without con- volution or region supervision. In International Confer- ence on Machine Learning, pages 5583-5594. PMLR, 2021.","title":"Vilt: Vision-and-language transformer without convolution or region supervision","authors":"Kim","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2021] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Train- ing gans with vision transformers. arXiv preprint arXiv:2107.04589, 2021.","title":"Vitgan: Training gans with vision transformers","authors":"Lee","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2022] Hyungyung Lee, Sungjin Park, Joonseok Lee, and Edward Choi. Unconditional image-text pair gen- eration with multimodal cross quantizer. arXiv preprint arXiv:2204.07537, 2022.","title":"Unconditional image-text pair generation with multimodal cross quantizer","authors":"Lee","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguistics.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 4582-4597, 2021.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.","title":"Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2021] Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Miti- gating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Ar- tificial Intelligence, volume 35, pages 14857-14866, 2021.","title":"Mitigating political bias in language models through reinforced calibration","authors":"Liu","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2022] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 61-68, Dublin, Ireland, May 2022. Association for Computational Linguistics.","title":"P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks","authors":"Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, Jie Tang","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Luccioni et al., 2023] Alexandra Sasha Luccioni, Christo- pher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion mod- els. arXiv preprint arXiv:2303.11408, 2023.","title":"Stable bias: Analyzing societal representations in diffusion models","authors":"Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, Yacine Jernite","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mathews et al., 2016] Alexander Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descriptions with sentiments. In Proceedings of the AAAI conference on ar- tificial intelligence, volume 30, 2016.","title":"Senticap: Generating image descriptions with sentiments","authors":"Alexander Mathews, Lexing Xie, Xuming He","year":"2016","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Perera and Patel, 2023] Malsha V Perera and Vishal M Pa- tel. Analyzing bias in diffusion-based face generation models. arXiv preprint arXiv:2305.06402, 2023.","title":"Analyzing bias in diffusion-based face generation models","authors":"Patel Perera","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qiao et al., 2019] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image gen- eration by redescription. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1505-1514, 2019.","title":"Mirrorgan: Learning text-to-image generation by redescription","authors":"Tingting Qiao, Jing Zhang, Duanqing Xu, Dacheng Tao","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ramesh et al., 2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.","title":"Hierarchical text-conditional image generation with clip latents","authors":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rombach et al., 2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. [Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234- 241. Springer, 2015.","title":"U-net: Convolutional networks for biomedical image segmentation","authors":"Rombach","year":"2015","venue":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rozière et al., 2023] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.","title":"Code llama: Open foundation models for code","authors":"Jonas Baptiste Rozière, Fabian Gehring, Sten Gloeckle, Itai Sootla, Gat, Ellen Xiaoqing, Yossi Tan, Jingyu Adi, Tal Liu, Jérémy Remez, Rapin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shin et al., 2020] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automati- cally generated prompts. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 4222-4235, 2020.","title":"Autoprompt: Eliciting knowledge from language models with automatically generated prompts","authors":"Taylor Shin, Yasaman Razeghi, Robert L Logan, I V, Eric Wallace, Sameer Singh","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. 2011.","title":"The caltechucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2021] Xu Yang, Chongyang Gao, Hanwang Zhang, and Jianfei Cai. Auto-parsing network for image captioning and visual question answering. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 2197-2207, 2021.","title":"Auto-parsing network for image captioning and visual question answering","authors":"Xu Yang, Chongyang Gao, Hanwang Zhang, Jianfei Cai","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yoshida et al., 2018] Kota Yoshida, Munetaka Minoguchi, Kenichiro Wani, Akio Nakamura, and Hirokatsu Kataoka. Neural joking machine: Humorous image captioning. arXiv preprint arXiv:1805.11850, 2018.","title":"Neural joking machine: Humorous image captioning","authors":"Yoshida","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2022] Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-based gan for high-resolution image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11304-11314, 2022.","title":"Styleswin: Transformer-based gan for high-resolution image generation","authors":"Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, Baining Guo","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.","title":"Conditional prompt learning for vision-language models","authors":"Zhou","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}