--- title: "AK4Prompts: Aesthetics-driven Automatically Keywords-Ranking for Prompts in Text-To-Image Models" authors: - "Haiyang Zhang" - "Mengchao Wang" - "Shuai He" - "Anlong Ming" doi: "10.24963/ijcai.2024/184" url: "https://ijcai.org/proceedings/2024/184" pdf: "https://ijcai.org/proceedings/2024/0184.pdf" venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence" year: 2024 pages: "1661-1669" track: "Main Track" keywords: - "Computer Vision: CV: Machine learning for vision" - "Computer Vision: CV: Computational photography" - "Computer Vision: CV: Image and video synthesis and generation" --- # AK4Prompts: Aesthetics-driven Automatically Keywords-Ranking for Prompts in Text-To-Image Models Haiyang Zhang, Mengchao Wang, Shuai He, Anlong Ming *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 1661-1669. DOI: [10.24963/ijcai.2024/184](https://doi.org/10.24963/ijcai.2024/184) ## Abstract Current text-to-image synthesis (TIS) models have demonstrated the ability to generate high-fidelity images based on textual prompts. However, the efficacy of these models heavily relies on the keywords present in the prompts, and there is a dearth of objective analysis regarding how different keywords impact the ultimate quality of generated results. Therefore, manual evaluation becomes necessary but limited and inefficient to ascertain the role played by keywords. In this paper, we propose automated keywords-ranking for prompts (AK4Prompts), a keyword evaluation model based on mainstream TIS models that explicitly quantifies the multidimensional impact of various keywords on image generation based on prompts. To enable personalized keyword evaluation based on prompt content, we propose decoupling the latent representations of keywords and prompts in TIS models, followed by integrating the semantic features of prompts into keywords. For quantitative and multidimensional evaluation, we align the fused features of keywords using HPSv2, aesthetic score, and CLIP score, each representing distinct factors contributing to keyword impact. Our AK4Prompts can flexibly and automatically select the keywords that best match the original prompt based on individual user preferences. Extensive experimental results show the superiority of AK4Prompts to improve the quality of generated images significantly over strong baselines. Our approach not only enhances usability and user experience but also addresses the current gap in automated analysis and evaluation of keyword effects. Our code is availableat https://github.com/mRobotit/AK4Prompts. ## Full text ## 1 Introduction Text-to-Image Synthesis (TIS) is a highly advanced and extensively utilized technique in generative Artificial Intelligence, aimed at generating realistic images based on textual input [Zhang et al., 2023]. Recently, with the advancement in the modeling capabilities of large models, TIS is undergoing a revolution. Cutting-edge text-to-image diffusion models like DALLE [Ramesh et al., 2021] and Latent Diffusion Models (LDMs), such as Stable Diffusion [Rombach et al., 2022], have emerged as pioneers in image generation, leveraging training data at a web scale. Additionally, Consistency models [Song et al., 2023], a new class of generative models capable of producing high-quality samples through a single network evaluation, have inspired further advancements. Building upon this concept, Luo et al. [Luo et al., 2023a] propose Latent Consistency Models (LCMs), which enable rapid inference with minimal steps on any pre-trained LDM and significantly enhance the speed of TIS. However, the image generation quality in these existing models relies heavily on the sophisticated design of keyword-based text prompts [Zhong et al., 2023]. This reliance stems from the training data quality, necessitating detailed prompts to produce high-quality images [Betker et al., 2023]. Determining the most effective keywords for a given prompt content often requires iterative experimentation and research within online communities [Oppenlaender, 2022]. In real-world scenarios, individuals lacking expertise often encounter challenges in selecting appropriate keywords for composing detailed prompts. This necessity leads to subjective assessments of the impact and quality of chosen keywords through repeated generation attempts, resulting in significant time and resource losses. ∗Corresponding author. Prompt engineering is an emerging research field focused on developing more effective prompts for deep generative models. In the TIS community, a common practice involves investigating the effects of various keywords. BestPrompt [Pavlichenko and Ustalov, 2023] initially explores these effects by employing human evaluation to assess a limited number of text prompts and keywords, ultimately selecting a set of exceptional keywords. However, this approach has inherent limitations and inefficiencies, posing challenges in comprehensively and objectively exploring the impact of all keywords on generated results. Furthermore, image generation is jointly controlled by the interaction between keywords and basic prompt content. However, this study only considers the individual influence of keywords, neglecting potential variations in their effects across different prompt categories. Another prevalent approach involves fine-tuning a language model for TIS prompt generation using com- ## Original Prompt: "vase of mixed flowers." Figure 1: The evolution of images is facilitated by customizing keyword preferences with AK4Prompts, leveraging the SDXL-Turbo model. ![Image](https://ijcai.org/proceedings/2024/fulltext/184/image_000000_cbe7623b6cc9b492bed8681ae145b3bdffb1938f9dadf03f4c82364424660746.png) plex prompts datasets, such as MagicPrompt 1 and BeautifulPrompt [Cao et al., 2023]. However, these methods primarily focus on completing or rewriting prompts, potentially altering the content entities. Moreover, the generated prompts often lack flexibility and exhibit excessive rigidity. In this paper, we propose a keyword evaluation model named AK4Prompts that can explicitly quantify the multidimensional impact of different keywords on the quality of generated images in response to user prompts. Our AK4Prompts automatically selects the keywords that best match the original prompt based on users' preferences, effectively enhancing the quality of the generated images. Our method preserves the content of the original prompt and allows users to customize their final outputs, thus overcoming the limitations of previous approaches. Meanwhile, it significantly alleviates the complexity of prompt design and the difficulty of selecting keywords, thereby saving users considerable trial and error time. Figure 1 illustrates the evolution of images generated by setting different degrees of specific preferences. In summary, the main contributions of this study are as follows: - We curate a high-quality set of keywords and develop an evaluation framework based on established TIS models. This framework enables automated and efficient assessment of these keywords, eliminating the need for manual intervention. - We propose AK4Prompts, a keyword evaluation model that can explicitly quantify the multidimensional impact of different keywords on the quality of images generated by TIS models based on user prompts. 1 https://huggingface.co/Gustavosta/MagicPrompt-StableDiffusion - Our approach demonstrates the judicious selection of the most appropriate keywords based on user prompts and preferences. Extensive experimental results showcase its superiority over strong baselines. ## 2 RelatedWork ## 2.1 Text-to-Image Synthesis (TIS) TIS is a multi-modal task involving the generation of images based on textual inputs. In the early years, popular image generation tasks were mainly based on Generative Adversarial Network (GAN) [Goodfellow et al., 2014; Reed et al., 2016]. More recently, diffusion models [Ho et al., 2020; Song et al., 2020; Sohl-Dickstein et al., 2015; Dhariwal and Nichol, 2021] have achieved remarkable results. Subsequently, LDMs (e.g., Stable Diffusion [Rombach et al., 2022]) apply a diffusion model in the latent space using a pre-trained autoencoder for efficient training and inference. This development has resulted in the emergence of high-performance generative models trained on billions of data points, including DALLE-2 [Ramesh et al., 2022], Imagen [Saharia et al., 2022], Imagic [Kawar et al., 2023], and SDXL [Podell et al., 2023]. Among these models, the Stable Diffusion series stands out as open-source models with an active user community. Recently, LCMs [Luo et al., 2023a], LCM-LoRA [Luo et al., 2023b], and SDXL-Turbo [Sauer et al., 2023] have emerged as solutions to the slow sampling issue in image generation, inspired by Consistency Models [Song et al., 2023]. LCM-LoRA serves as a universal, training-free acceleration module. It can be directly integrated into various Stable Diffusion fine-tuned models or LoRAs [Hu et al., 2021], facilitating fast inference with minimal steps. This allows us to efficiently infer and explore a large number of keyword and prompt category combinations. However, the qualities of generated images are dependent on the keywords provided in the prompts. TIS models usually utilize a text encoder, which employs pre-trained language models like CLIP [Radford et al., 2021] to convert textual inputs into latent vectors. These latent vectors associated with individual keywords represent specific control influences on images, which can be utilized for our quantitative evaluation task. ## 2.2 TIS Evaluation The aesthetic score [Schuhmann et al., 2022] assesses the aesthetic quality of individual images, while the CLIP score [Radford et al., 2021] quantifies the similarity between generated images and prompts. Additionally, there exist metrics specifically trained to align with human preferences, such as [Wu et al., 2023b; He et al., 2022; Wu et al., 2023a; Xu et al., 2023; Kirstain et al., 2023; He et al., 2023a; He et al., 2023b]. Human preferences are intricate and encompass various dimensions, such as text-image similarity, image fidelity, aesthetics, and other factors. Among these metrics, HPSv2 stands out for its consistent scoring and use of larger, more diverse training datasets. Significantly, the CLIP model has been fine-tuned on an extensive dataset, featuring 798,090 human ranking choices across 433,760 image pairs [Wu et al., 2023a]. These evaluation metrics provide valuable visual feedback and enable a quantitative assessment of the multidimensional impact of keywords on TIS. In this paper, we adopt a quantitative and multidimensional approach to evaluate the fused features of keywords. We use multiple TIS evaluation metrics, including HPSv2, the aesthetic score, and the CLIP score, with each metric representing distinct aspects of keyword effects. ## 2.3 Prompt Engineering for TIS Due to the remarkable potential of TIS, there has been a surge of interest in prompt engineering. MagicPrompt, a widely adopted model for automatic prompt completion, is trained on high-quality prompts sourced from the Internet. BeautifulPrompt [Cao et al., 2023] utilizes a reinforcement learning technique, complemented by visual AI feedback, to finetune language models and maximize the rewards of generated prompts. However, these methods primarily focus on completing or rewriting prompts, which may inadvertently alter content entities, resulting in rigid and inflexible generated prompts. Liu et al. [Liu and Chilton, 2022] conducted a series of experiments, proposing several design guidelines for text-to-image prompt engineering that confirm the beneficial impact of incorporating keywords. Additionally, [Oppenlaender, 2023] identified six distinct types of keywords through an ethnographic study spanning three months within the online generative art community. However, previous studies have been constrained by the laborious and time-consuming manual engineering of prompts. In contrast, BestPrompt [Pavlichenko and Ustalov, 2023] employs a genetic algorithm to identify keywords that compose prompts, aiming to achieve optimal aesthetic quality in images. Nevertheless, this study solely relies on selecting a fixed set of keywords based on average performance determined through human evaluation. Conversely, AK4Prompts can assess the varying impact of keywords and select the most suitable ones for specific prompts, thereby enhancing the visual appeal of generated images according to user preferences. To address the potential variations in the effects of keywords across different prompt categories and provide personalized keyword evaluation based on prompt content, we decouple the latent representations of keywords and prompts in TIS models, which have distinct impacts on the generated images. Subsequently, to further enhance this integration, we employ multiple crossattention layers to merge the semantic features of prompts with those of keywords. ## 3 Data Collection for Training Simple Prompt datasets. The image-caption datasets, such as COCO Captions [Chen et al., 2015] and LAION [Schuhmann et al., 2022] primarily describing real images, may not align with the interests of generative model users. DiffusionDB [Wang et al., 2022] comprises a large-scale dataset filled with a wide range of user-written prompts and generated images. However, it features numerous complex keyword-based text prompts. Regarding our prompt dataset, we require simple text prompts to maximize the impact of different keywords. SUR-adapter [Zhong et al., 2023] employs a pre-trained BLIP model to generate captions for images associated with high-quality prompts, treating the brief and less detailed resulting captions as low-quality prompts. BeautifulPrompt [Cao et al., 2023] enhances this approach by using ChatGPT [OpenAI, 2023] to summarize highquality prompts, treating these summaries as low-quality prompts. The dataset includes 143k simple prompts and 2k test prompts. We train our model using BeautifulPrompt's training set and evaluate its performance on the test set. Keywords Collection. We extract a series of keywords from complex prompts in DiffusionDB, separating them by commas and sorting them based on frequency of occurrence. After manually removing non-keywords and those with lower frequencies, we obtain a list of 51,291 keywords. However, the keyword set contains numerous semantically consistent and repetitive terms. Since the SD utilizes CLIP to encode text inputs into latent vectors for conditional control, keywords with similar CLIP encodings have comparable effects on the generation outcome. To reduce keyword redundancy and prevent the model from developing a specific bias, we employ CLIP to uniformly encode the keywords, compute their similarity matrix, and apply the k-means clustering algorithm for grouping them into clusters. We retain the most frequently occurring keywords in each cluster while removing redundant ones to reduce the overall number of keywords further. Through extensive experimentation, we ultimately select 954 distinct semantic representations as our final keyword set. Our AK4Prompts can generalize to all 51,291 keywords via CLIP encoding. Keywords with similar semantics receive similar scores, even rare ones, matched based on learned semantic similarities. Figure 2: Illustration of AK4Prompts. (Top) The overall architecture of our training pipeline, where only the newly added module (highlighted in red) is trained while the pretrained text-to-image model remains frozen. (Bottom) The network structure of our module, which utilizes the AK4Prompts module to assess the impact of keywords on image quality generated in response to user prompts. ![Image](https://ijcai.org/proceedings/2024/fulltext/184/image_000001_b28334655ba3b4fe8247dfd1b56d8901d8ead9f1bec66861de398b295a1d298a.png) ## 4 AK4Prompts As shown in Figure 2 (Top), let us consider the keywordprompt pairs (pi, ki)Ni=1 , where pi represents the original prompt and ki denotes a randomly selected keyword from the collected set of keywords. Firstly, we freeze all learnable parameters of the text encoder fEn and the predictor fpre in the pre-trained TIS model, as well as all the evaluation models used. Then, we utilize the TIS model to generate images Ii corresponding to keyword-based prompt ˆ pi and calculate average scores for both images Ii and the original prompt pi using various evaluation models. Finally, we propose our trainable AK4Prompts model to predict these scores based on the features of pi and ki from the text encoder. Our model, as illustrated in Figure 2 (Bottom), comprises two components: the Semantic Fusion Module (Section 4.1) and the Multidimensional Score Prediction Module (Section 4.2). Section 4.3 demonstrates how to utilize AK4Prompts for inference by identifying suitable keywords according to user prompts and customized preferences. ## 4.1 Semantic Fusion of Keywords and Prompts The structure of the Semantic Fusion Module is depicted in Figure 2 (Bottom), which comprises L cross-attention layers. These layers are responsible for fusing the semantic features of keywords and prompts, generating attention maps for each textual token of prompts. Each layer consists of four learnable transformations, denoted as hj(·) for j = 1, 2, 3, 4, implemented using a multi-layer perceptron (MLP). We utilize the outputs fEn(ki) and fEn(pi) from the text encoder as initial embeddings for keyword features embki 0 and prompt features embpi , respectively. Then, we construct Qi = h3[embki ℓ ] and Ki = h2[embpi ], and calculate attention values as described in [Devlin et al., 2018; Vaswani et al., 2017] where d represents the feature dimension of Qi and Ki, and the cell atti,j ℓ defines the weight of the value of the j-th token in pi on ki. Additionally, we construct Vi = h1[embpi ], and obtain the calibrated semantic information as ˆ Vi = Vi ⊗ atti, which is then used to update the keyword features embki . Afterwards, the output of the first cross-attention sublayer is transformed by a learnable transformation function denoted as h4(·) to obtain the fused semantic features. Here, We employ a residual connection [He et al., 2016] around each of the two sub-layers, followed by layer normalization [Ba et al., 2016]. Finally, the output of the Semantic Fusion Module is obtained as embki L through the crossattention layers. Table 1: Results on the testing set. "Original" refers to the method that directly sends the original prompts to TIS models without modification. "Top-n" refers to using the top-n keywords from different methods. | TISModel | Method | Aesthetic Score - Top-1 | Aesthetic Score - Top-5 | Aesthetic Score - Top-10 | HPSv2 - Top-1 | HPSv2 - Top-5 | HPSv2 - Top-10 | CLIP Score - Top-1 | CLIP Score - Top-5 | CLIP Score - Top-10 | |------------|-------------------|---------------------------|---------------------------|----------------------------|-----------------|-----------------|------------------|----------------------|----------------------|-----------------------| | | Original | 5.70 | 5.70 | 5.70 | 0.256 | 0.256 | 0.256 | 0.258 | 0.258 | 0.258 | | SD (1.5) + | BestPrompt | 5.80 | 5.97 | 6.12 | 0.257 | 0.256 | 0.260 | 0.255 | 0.245 | 0.243 | | LCM-LoRA | MostPopular | 5.76 | 6.04 | 6.19 | 0.256 | 0.257 | 0.257 | 0.259 | 0.251 | 0.246 | | LCM-LoRA | AK4Prompts (ours) | 5.82 | 6.14 | 6.23 | 0.258 | 0.261 | 0.261 | 0.259 | 0.252 | 0.247 | | SD (1.5) | Original | 5.70 | 5.70 | 5.70 | 0.261 | 0.261 | 0.261 | 0.287 | 0.287 | 0.287 | | SD (1.5) | BestPrompt | 5.80 | 6.14 | 6.26 | 0.261 | 0.263 | 0.265 | 0.283 | 0.276 | 0.270 | | SD (1.5) | MostPopular | 5.80 | 6.19 | 6.37 | 0.261 | 0.261 | 0.260 | 0.286 | 0.277 | 0.272 | | SD (1.5) | AK4Prompts (ours) | 6.00 | 6.29 | 6.43 | 0.262 | 0.263 | 0.266 | 0.286 | 0.278 | 0.270 | | SDXL-Turbo | Original | 6.21 | 6.21 | 6.21 | 0.275 | 0.275 | 0.275 | 0.286 | 0.286 | 0.286 | | SDXL-Turbo | BestPrompt | 6.27 | 6.46 | 6.42 | 0.275 | 0.276 | 0.277 | 0.278 | 0.273 | 0.268 | | SDXL-Turbo | MostPopular | 6.31 | 6.53 | 6.62 | 0.275 | 0.275 | 0.276 | 0.282 | 0.272 | 0.269 | | SDXL-Turbo | AK4Prompts (ours) | 6.45 | 6.57 | 6.71 | 0.276 | 0.276 | 0.278 | 0.282 | 0.275 | 0.276 | ## 4.2 Multidimensional Score PredictionModule The Score Prediction Module consists of three MLP score prediction heads, ˆ Shps(·), ˆ Sclip(·) and ˆ Saes(·), as shown in Figure 2 (Bottom). Each prediction head utilizes the output keyword embedding from the Semantic Fusion Module to predict the average HPSv2 score [Wu et al., 2023a], CLIP similarity score [Radford et al., 2021], and aesthetic score [Schuhmann et al., 2022] of the generated image based on the corresponding keyword prompt. Briefly, to enhance the prompt pi with keyword ki, append ki to the end of pi and form ˆ pi. Then input ˆ pi into the downstream TIS model to generate image Ii: HPSv2 is a human preference model trained on a large dataset of text-to-image prompts and real user preferences. To mitigate the impact of random seeds on the quality of images generated by the TIS model, we employ 8 different random seeds to generate images and average the results. The calculated averaged HPSv2 score HPS is used as the ground truth for training our model. The loss function is as follows: where ˆ Shps(embki L ) represents the scalar output of the HPSv2 prediction head for the keyword embedding embki L obtained from the Semantic Fusion Module. MSE refers to Mean Squared Error, and N denotes the total number of samples. CLIP score is a metric used to evaluate the correlation between the image generated by the model and the original text. In our work, we use a MLP prediction head to assess the consistency between the original prompt pi and generated image Ii under the influence of keyword ki. The calculated average CLIP score CLIP is used as the ground truth to train our model. The loss function is: The aesthetic score model is trained to predict the rating that people give when asked "how much do you like this image on a scale from 1 to 10". To maintain consistency with HPS and CLIP, we scale the aesthetic scores to a range of 0 to 1. Similarly, a MLP prediction head is trained to fit the corresponding keywords to the aesthetic scores AES of the generated images: Finally, we combine the losses of the three MLP prediction heads as the final Ltotal using loss coefficients h, a, and c. ## 4.3 Customized Keywords-Ranking After training, our AK4Prompts gak(·) can predict scores for each keyword based on simple prompts, allowing us to determine the optimal set of keywords that align with these prompts. For a given simple prompt p, we input it into our model to simultaneously predict HPSv2 scores, CLIP scores, and aesthetic scores for all keywords in a parallel inference where ki represents a keyword and K denotes the total number of keywords. Then, we can calculate a comprehensive score for each keyword ki as follows: Figure 3: The qualities of images generated from prompts with different keyword selection methods using the SDXL-Turbo model. ![Image](https://ijcai.org/proceedings/2024/fulltext/184/image_000002_150b34cde52b6ff1842e249af54b48046e964310069e0fbf7474d8f81e5f8ba6.png) where (ˆh, ˆ c, ˆa) represent the weights for the HPS score, CLIP score, and aesthetic score, respectively. Finally, a comprehensive ranking of all keywords can be obtained, where higher-ranked keywords exhibit a more pronounced enhancing effect on the prompt p. Moreover, by customizing the values of (ˆh, ˆ c, ˆa) based on user preferences, keywords with different emphases can be sorted and selected. For instance, assigning a larger value to ˆa enables the selection of keywords that exert a greater impact on aesthetics. Based on this keyword ranking, we can choose the topperforming keywords and append them to simple prompts in order to enhance overall image generation performance. Our method achieves higher computational efficiency compared to prompt generation models due to its use of precomputed semantic embeddings, avoidance of self-attention computation during keyword processing, and smaller model size. ## 5 Experiments Training Settings. Our model was trained on 143KBeautifulPrompt training samples. We utilized the pre-trained stable diffusion 1.5 as our TIS model and employed LCM-LoRA for accelerated inference. We generated 512x512 resolution images through a four-step inference with a CFG scale ω set to 1.0, leveraging FLOAT16 formats to save GPU memory and speed up training. For the semantic fusion module, we set L = 3. Regarding Ltotal, we set h = 2.25, c = 2.25, and a = 1. Due to different ranges in the distributions of these scores, we assigned a larger weight to the one with smaller ranges to balance gradients and enhance prediction accuracy. The learning rate was set at 1e-4, weight decay at 1e-2, batch size at 32, and training step at 88,000 steps. All experiments were implemented in PyTorch and run on a single server with NVIDIA RTX3090TI GPUs. Baselines. We consider two strong baselines: BestPrompt [Pavlichenko and Ustalov, 2023] and MostPopular. BestPrompt is the first work to identify the best keywords for TIS, using a genetic algorithm that identifies a set of high-quality keywords through extensive human evaluations. MostPopular represents the most frequently used keywords in TIS, which have gained widespread recognition within the user community for their proven effectiveness in enhancing the quality of generated images. Our method requires setting the number of selected keywords and specific preferences. Whereas the generation methods may produce new prompts with different keyword lengths, making comparison challenging. Evaluation Protocols. We assess the images generated by the TIS model using 2K BeautifulPrompt testing prompts employing various keyword selection methods. Multiple TIS models are employed to generate images, and we compute the aesthetic score [Schuhmann et al., 2022], HPSv2 [Wu et al., 2023a], and CLIP score [Radford et al., 2021] for both the images and original prompts. For the baselines, we evaluate using different numbers of keywords, specifically top-1, top5, and top-10. Regarding our AK4Prompts approach, we set (ˆh=3, ˆ c=5, ˆa=1) to determine a comprehensive ranking for all keywords and select an equivalent number of top keywords for evaluation and comparison purposes. As a comparative measure due to baselines' emphasis on clip scores, we also assign a higher weight to ˆ c in order to enhance the clip score. Table 2: Performance of selecting the most impactful keywords on the aesthetic score, CLIP score, and HPSv2 score, respectively. Each score is calculated independently based on the top-5 highest-scoring keywords. | TISModel | Method | Aesthetic Score | HPSv2 | CLIP Score | |------------|------------|-------------------|---------|--------------| | SD (1.5)+ | Original | 5.70 | 0.256 | 0.258 | | | BestPrompt | 5.97 | 0.256 | 0.245 | | LCM-LoRA | MostCommon | 6.04 | 0.257 | 0.251 | | | AK4Prompts | 6.49 | 0.261 | 0.254 | | SD (1.5) | Original | 5.70 | 0.261 | 0.287 | | SD (1.5) | BestPrompt | 6.14 | 0.263 | 0.276 | | SD (1.5) | MostCommon | 6.19 | 0.261 | 0.277 | | SD (1.5) | AK4Prompts | 6.66 | 0.265 | 0.283 | | SDXL-Turbo | Original | 6.21 | 0.275 | 0.286 | | | BestPrompt | 6.46 | 0.277 | 0.273 | | | MostCommon | 6.53 | 0.275 | 0.272 | | | AK4Prompts | 7.03 | 0.279 | 0.283 | Additionally, we set (ˆh, ˆ c, ˆa) as (1, 0, 0), (0, 1, 0), and (0, 0, 1) respectively to select the top-5 keywords that have a significant impact on the aesthetic score, CLIP similarity score, and human preference score for evaluation. ## 5.1 Overall Results Our method consistently outperforms the other baselines in most scores, as demonstrated by Table 1. Given that the CLIP score reflects semantic consistency between text and image, it is unsurprising that sending the original prompts without any additional keywords to Stable Diffusion unchanged yields the highest score. Our approach achieves superior aesthetic scores, HPSv2, and CLIP scores compared to the baselines while maintaining a comparable CLIP score to the Original prompt. This highlights the effectiveness of our proposed method, AK4Prompts, in selecting optimal keywords for input prompts. Our AK4Prompts is applicable to all SD-style models. The experimental results of SD1.5 and SDXL-Turbo demonstrate the transferability of AK4Prompts to other diffusion-based TIS models. Taking into consideration the widely adopted model SDXL-Turbo 2 , Figure 3 illustrates that, despite already performing well with most vanilla prompts, SDXLTurbo can further enhance its image generation capabilities when utilizing AK4Prompts. As depicted in Table 2, the selection of only the top 5 keywords, predicted by the model with the highest aesthetic scores for image generation, leads to a significant enhancement in aesthetic scores, as well as CLIP and HPS scores. This observation signifies that our approach proficiently captures diverse keyword effects, offering guidance to users and enabling customization of final output preferences. Additional examples are provided in Appendix A and B in our code repository, including qualitative analyses of different keywords and the specific effects of different categories of keywords on image generation. We also feature a qualitative comparison of images generated using keywords selected with different preferences. 2 https://huggingface.co/stabilityai/sdxl-turbo ## 5.2 Ablation Study In Table 3, we analyze the various design choices implemented throughout our experiments. As discussed in Section 4.1, the Semantic Fusion Module integrates the semantics of simple prompts with keywords. This integration enables a more effective combination of different prompt semantics, thereby providing personalized keywords. To validate the module's effectiveness, we initially compare it with a simpler method that omits the Semantic Fusion Module. This method involves directly using the text encoder to obtain semantic encodings of user prompts and additional keywords, followed by score prediction using MLP prediction heads. Subsequently, we employ self-attention instead of cross-attention for integrating semantic features from user prompts and additional keywords. Finally, we evaluate different scheduling steps using LCM-LoRA. For a fair comparison, both models were trained for 88,000 steps under identical configurations. Table 3: The proposed method consistently outperforms all other baselines in terms of achieved rewards, as evidenced by the table featuring the top-5 keywords. | | Aesthetic Score | |---------------------|-------------------| | AK4Prompts (ours) | 6.49 | | w/o SF | 6.34 | | with self-attention | 6.43 | | with step=1 | 6.28 | ## 6 Conclusion In this paper, we propose a universal algorithmic framework named AK4Prompts that quantitatively assesses the different effects of various keywords in TIS models based on simple prompt. Our approach significantly reduces the complexity of prompt design and the difficulty of keyword selection, thereby saving users considerable trial-and-error time. While AK4Prompts can automatically select keywords for user prompts and generate more aesthetically pleasing images, we find it challenging to strike a balance among factors such as image aesthetics, human preferences, and text-image consistency in order to choose the best keywords. At times, in pursuit of better text-image consistency, we have to compromise on certain aspects of image aesthetics. Additionally, since we use SD1.5 and employ LCM-LoRA with CFG scale ω set to 1.0, only positive keywords are used to control generation while negative keywords are not considered. Similar to positive prompts, negative prompts are also crucial inputs for many TIS models. The final image generation results depend on both positive and negative prompts. Therefore, discovering the best negative keywords can effectively improve the quality of generated images as well. The incorporation of negative keywords into our framework is achievable, but addressing this aspect will be a key focus in the development of AK4Prompts 2.0. ## Acknowledgements This work was supported by the Innovation Research Group Project of NSFC (61921003). ## Contributions - † on Page 1 indicates that Haiyang Zhang, Mengchao Wang and Shuai He, three authors of this paper, have made equal contributions. ## References 1. [Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 2. [Betker et al., 2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Jun- tang Zhuang, Joyce Lee, Yufei Guo, et al. Improving im- age generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2023. 3. [Cao et al., 2023] Tingfeng Cao, Chengyu Wang, Bingyan Liu, Ziheng Wu, Jinhui Zhu, and Jun Huang. Beau- tifulprompt: Towards automatic prompt engineering for text-to-image synthesis. arXiv preprint arXiv:2311.06752, 2023. 4. [Chen et al., 2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 5. [Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018. 6. [Dhariwal and Nichol, 2021] Prafulla Dhariwal and Alexan- der Nichol. Diffusion models beat gans on image synthe- sis. Advances in neural information processing systems, 34:8780-8794, 2021. 7. [Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in neural information processing systems, 27, 2014. 8. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 9. [He et al., 2022] Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming. Rethinking image aesthetics assessment: Models, datasets and benchmarks. In IJCAI, pages 942-948, 2022. 10. [He et al., 2023a] Shuai He, Anlong Ming, Yaqi Li, Jinyuan Sun, ShunTian Zheng, and Huadong Ma. Thinking im- age color aesthetics assessment: Models, datasets and benchmarks. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 21838- 21847, 2023. 11. [He et al., 2023b] Shuai He, Anlong Ming, Shuntian Zheng, Haobin Zhong, and Huadong Ma. Eat: An enhancer for aesthetics-oriented transformers. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1023-1032, 2023. 12. [Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 13. [Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 14. [Kawar et al., 2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023. 15. [Kirstain et al., 2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to- image generation. arXiv preprint arXiv:2305.01569, 2023. 16. [Liu and Chilton, 2022] Vivian Liu and Lydia B Chilton. De- sign guidelines for prompt engineering text-to-image gen- erative models. In Proceedings of the 2022 CHI Confer- ence on Human Factors in Computing Systems, pages 1- 23, 2022. 17. [Luo et al., 2023a] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 18. [Luo et al., 2023b] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lcm-lora: A univer- sal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05556, 2023. 19. [OpenAI, 2023] OpenAI. Chatgpt. https:// openai.com/ chatgpt, 2023. 20. [Oppenlaender, 2022] Jonas Oppenlaender. The creativity of text-to-image generation. In Proceedings of the 25th International Academic Mindtrek Conference, Academic Mindtrek '22, page 192-202, New York, NY, USA, 2022. Association for Computing Machinery. 21. [Oppenlaender, 2023] Jonas Oppenlaender. A taxonomy of prompt modifiers for text-to-image generation. Behaviour & Information Technology, pages 1-14, 2023. 22. [Pavlichenko and Ustalov, 2023] Nikita Pavlichenko and Dmitry Ustalov. Best prompts for text-to-image models and how to find them. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2067-2071, 2023. 23. [Podell et al., 2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving la- tent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 24. [Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 25. [Ramesh et al., 2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021. 26. [Ramesh et al., 2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022. 27. [Reed et al., 2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Interna- tional conference on machine learning, pages 1060-1069. 28. PMLR, 2016. 29. [Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 30. [Saharia et al., 2022] Chitwan 31. Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language un- derstanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022. 32. [Sauer et al., 2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023. 33. [Schuhmann et al., 2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wight- man, Mehdi Cherti, Theo Coombes, Aarush Katta, Clay- ton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Sys- tems, 35:25278-25294, 2022. 34. [Sohl-Dickstein et al., 2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermody- namics. In International conference on machine learning, pages 2256-2265. PMLR, 2015. 35. [Song et al., 2020] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 36. [Song et al., 2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023. 37. [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 38. [Wang et al., 2022] Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.14896, 2022. 39. [Wu et al., 2023a] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evalu- ating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341, 2023. 40. [Wu et al., 2023b] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference. arXiv preprint arXiv:2303.14420, 2023. 41. [Xu et al., 2023] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yux- uan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yux- iao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977, 2023. 42. [Zhang et al., 2023] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image dif- fusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909, 2023. 43. [Zhong et al., 2023] Shanshan Zhong, Zhongzhan Huang, Weushao Wen, Jinghui Qin, and Liang Lin. Sur- adapter: Enhancing text-to-image pre-trained diffusion models with large language models. In Proceedings of the 31st ACM International Conference on Multimedia, pages 567-578, 2023.