{"type":"ScholarlyArticle","title":"Integrating LLM, VLM, and Text-to-Image Models for Enhanced Information Graphics: A Methodology for Accurate and Visually Engaging Visualizations","authors":[{"name":"Chao-Ting Chen","slug":"chao-ting-chen","given":"Chao-Ting","family":"Chen","affiliation":"Department of Computer Science , National Chengchi University , Taipei , Taiwan"},{"name":"Hen-Hsen Huang","slug":"hen-hsen-huang","given":"Hen-Hsen","family":"Huang","affiliation":"Institute of Information Science , Academia Sinica , Taipei , Taiwan"}],"abstract":"This study presents an innovative approach to the creation of information graphics, where the accuracy of content and aesthetic appeal are of paramount importance. Traditional methods often struggle to balance these two aspects, particularly in complex visualizations like phylogenetic trees. Our methodology integrates the strengths of Large Language Models (LLMs), Vision Language Models (VLMs), and advanced text-to-image models to address this challenge. Initially, an LLM plans the layout and structure, employing Mermaid—a JavaScript-based tool that uses Markdown-like scripts for diagramming—to establish a precise and structured foundation. This structured script is crucial for ensuring data accuracy in the graphical representation. Following this, text-to-image models are employed to enhance the vector graphic generated by Mermaid, adding rich visual elements and enhancing overall aesthetic appeal. The integration of text-to-image models is a key innovation, enabling the creation of graphics that are not only informative but also visually captivating. Finally, a VLM performs quality control, ensuring that the visual enhancements align with the informational accuracy. This comprehensive approach effectively combines the accuracy of structured data representation, the creative potential of text-to-image models, and the validation capabilities of VLMs. The result is a new standard in information graphic creation, suitable for diverse applications ranging from education to scientific communication, where both information integrity and visual engagement are essential.","doi":"10.24963/ijcai.2024/995","url":"https://ijcai.org/proceedings/2024/995","pdf":"https://ijcai.org/proceedings/2024/0995.pdf","markdown":"https://ijcai.org/proceedings/2024/995.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/995","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":995,"pages":{"first":8627,"last":8630},"keywords":["Humans and AI: HAI: Intelligent user interfaces","Computer Vision: CV: Vision and language","Multidisciplinary Topics and Applications: MDA: Arts and creativity"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.","title":"Gpt-4 technical report","authors":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dibia, 2023] Victor Dibia. Lida: A tool for automatic generation of grammar-agnostic visualizations and info- graphics using large language models. arXiv preprint arXiv:2303.02927, 2023.","title":"Lida: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models","authors":"Victor Dibia","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Huang et al., 2024] Qirui Huang, Min Lu, Joel Lanir, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Graphi- mind: Llm-centric interface for information graphics de- sign, 2024.","title":"Graphimind: Llm-centric interface for information graphics design","authors":"Qirui Huang, Min Lu, Joel Lanir, Dani Lischinski, Daniel Cohen-Or, Hui Huang","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Schulman et al., 2022] John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2022.","title":"Chatgpt: Optimizing language models for dialogue","authors":"John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron, Liam Uribe, Luke Fedus, Michael Metz, Pokorny","year":"2022","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Shi et al., 2020] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xi- aodan Zhu. Improving image captioning with better use of captions, 2020.","title":"Improving image captioning with better use of captions","authors":"Zhan Shi, Xu Zhou, Xipeng Qiu, Xiaodan Zhu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Zhang et al., 2023] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image dif- fusion models in generative ai: A survey, 2023.","title":"Text-to-image diffusion models in generative ai: A survey","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}