Music Atelier: Exploring the Knowing–Doing Gap in LLM Creativity via Symbolic Music Composition

1 Introduction

The maxim "knowing is easy, doing is hard" has long been recognized in philosophical traditions as capturing the enduring gap between understanding and action [Crisp, 2014]. In creative practice, theoretical understanding or technical mastery is often only a starting point, rather than a guarantee of successful artistic realization. Artistic creation therefore centers on this process of transformation, through which understanding is carried into practice and ideas become concrete works.

As large language models (LLMs) become deeply embedded in creative tasks [Peng, 2022; Huang et al., 2025; Wan et al., 2024], the knowing-doing gap takes on a new, model-specific form. In this context, LLMs often make user intent and intent-relevant knowledge explicit through reasoning traces that outline how a creative task could be carried out ("knowing") [Wei et al., 2022], and subsequently produce a final generated artifact as the realization of this reasoning ("doing") [Zhou et al., 2024; OpenAI, 2025b]. However, it remains unclear how user intent, as expressed through intermediate creative ideas during model reasoning, is translated, or fails to be translated, into concrete elements of the generated artifact in a systematic and analyzable way. This leaves a gap in creative research, reflected in the lack of interpretable and tractable representations that allow humans to observe, understand, and meaningfully interact with how a model applies its articulated knowledge during the creative process to shape identifiable elements of an artistic artifact.

∗Corresponding author.

When applied to symbolic music composition, the knowing-doing gap becomes particularly salient. LLMs can competently articulate music-theoretic knowledge, such as harmony, tonality, and rhythm, and reason about how these concepts should be applied to construct musical structure [Wang et al., 2025a; OpenAI, 2025a]. However, the explicit articulation of such knowledge does not ensure its consistent realization in the generated music. In practice, reasoning traces that describe coherent musical plans often fail to align with the concrete musical elements instantiated in the final artifact, revealing a structural mismatch between articulated theory and realized composition. Because music composition relies on rule-guided rather than rule-determined creation, symbolic music provides a natural testbed for examining how knowing-doing gaps emerge between planning and realization.

Prior analysis paradigms for music composition primarily focus on evaluating final artifacts, emphasizing output quality or preference alignment [Bian et al., 2023; Hu et al., 2023; Cideron et al., 2024; Hu et al., 2025b]. Attribute-level statistics and LLM-based judging pipelines further provide useful descriptive and post-hoc perspectives on generated music [Wang et al., 2021; Yuan et al., 2023; Zheng et al., 2023]. While effective for outcome-oriented evaluation, these artifact-centered approaches are not explicitly designed to trace how a model's articulated reasoning or knowledge is realized during the creative process, leaving open questions about the correspondence between reasoning and realized decisions. This motivates the need for a complementary process-level analytical framework.

We introduce a process-level perspective on creative generation that treats artistic creation as a realizational process linking user intent, articulated reasoning, and the final artifact. Building on this perspective, we propose Music Atelier (Mutelier), a systematic analytical framework that makes the knowing-doing gap in creative LLMs explicit by tracing how intent-aligned ideas are articulated during reasoning and selectively realized, or lost, in generated artworks. At a technical level, Mutelier instantiates a novel LLM-as-a-Judge paradigm that treats creative realization itself as the object of analysis rather than merely evaluating final outcomes. We validate Mutelier through a series of controlled studies across three creative tasks and fifteen LLMs, through which Mutelier reveals consistent patterns in how creative intentions fail to materialize in practice-patterns that remain largely invisible to artifact-centered analyses. By treating the knowing-doing gap as a realizational phenomenon, this work provides the field with a principled process-level lens for studying machine creativity beyond artifact-centered evaluation.

2 RelatedWork

2.1 LLM-as-a-Judge

Recent advances in LLMs have given rise to the LLM-asa-Judge paradigm, where LLMs are employed as generalpurpose evaluators or decision-makers in generative systems [Wang et al., 2023; Panickssery et al., 2024; Chen et al., 2024; Desmond et al., 2025; Zhang et al., 2025; Hu et al., 2025a]. In this line of work, LLMs are primarily used to approximate human judgments over generated outputs, such as preference alignment, factual correctness, or task compliance, enabling scalable evaluation without extensive human annotation [D'Souza et al., 2025; Ye et al., 2025]. Empirical studies show that models such as GPT-4 can achieve strong agreement with human judges in benchmarks like MT-Bench and Chatbot Arena [Zheng et al., 2023], while subsequent frameworks further improve robustness and consistency in judgment output quality [Yu et al., 2025]. Despite their effectiveness, prior LLM-as-a-Judge approaches implicitly assume a closed artifact as the object of judgment, where an output is generated first and then evaluated against a criterion. In contrast, the problem we study requires judging the correspondence between articulated reasoning and its realization in the generated output, making the judgment object a cross-stage mapping rather than a single artifact. Unlike prior LLM-as-a-Judge work that treats the model mainly as a scoring or evaluation tool for final outputs, we extend this paradigm by using LLM judgment as an analytical operator that explicitly analyzes cross-stage mappings, enabling principled study of how the knowing-doing gap manifests in creative AI.

2.2 Creativity in Generative Models

Research on creativity in generative models has predominantly focused on properties of generated outputs, such as novelty, diversity, surprise, or stylistic deviation [Friedman and Dieng, 2023; Zhang et al., 2024; Zhong et al., 2024; Franceschelli and Musolesi, 2024]. Some studies draw on psychological theories of creativity and apply multidimensional annotations or ratings to artifacts [Chakrabarty et al., 2024], while others design structured tasks to indirectly probe associative reasoning or conceptual recombination abilities [Huang et al., 2025]. Together, these approaches provide valuable perspectives for characterizing creative performance across domains [Meng et al., 2025; Wang et al., 2025b]. However, most existing work treats creativity as an observable attribute of final outputs, rather than as a dynamic process unfolding during generation. Such outcome-centered perspectives make it difficult to examine how creative ideas emerge, how knowledge is invoked, or how abstract intentions are translated into concrete generative realizations. In contrast, this paper adopts a process-oriented view of creativity, framing creative generation as a realizational phenomenon that links reasoning-stage ideas to their manifestations in artifacts.

3 Mutelier

We develop a process-level realizational theory of creative generation and introduce Mutelier.

3.1 ARealizational Process Theory

We conceptualize creative generation in LLMs as a realizational process across three stages (Figure 1): (i) user intent specification, (ii) idea articulation during reasoning, and (iii) artifact realization. Formally, a creative generation instance is represented as a triplet (x, r, y), where x denotes the user input encoding creative intent, r denotes the unstructured reasoning trace produced by the model, and y denotes the final generated artifact. This formulation explicitly separates internal idea articulation from external realization.

Within this framework, we define the knowing-doing gap as a realization failure at the reasoning-to-artifact transition. Let I(r) = {ik} denote the set of creative ideas articulated in a reasoning trace r, and let ρ(ik, y) ∈ {0, 1} indicate whether an articulated idea ik is realized in the artifact y. The knowing-doing gap occurs if there exists a non-empty subset I + ⊆ I(r) such that each ik ∈ I + is aligned with the user intent x but ρ(ik, y) = 0. This localizes the gap to the r → y transition and necessitates idea-level analysis.

3.2 The Thought Graph as an Intermediate Representation

To operationalize the realizational process theory defined above, we introduce the thought graph, an explicit intermediate representation that instantiates articulated creative ideas as analyzable structural units.

A thought graph is defined as a directed graph

where each node ni ∈ N corresponds to a distinct creative idea articulated during reasoning. Formally, the node set N provides a structured representation of the idea set I(r), with each node ni corresponding to an element ik. Each directed edge (u, v) ∈ L captures a structural relation between ideas (e.g., refinement, dependency, or progression).

Each idea node ni is annotated with two complementary attributes:

• an idea-artifact realization label ℓRi ∈ {0, 1}, which instantiates the realization indicator ρ(ik, y) for the corresponding idea, specifying whether it is realized in the final artifact y;
• an intent-idea relation label ℓUi ∈ K, which maps each idea node to a relation category in K with respect to the user intent x (e.g., preserve, elaborate, shift, or diverge).

We next describe how this abstract representation is instantiated from an unstructured reasoning trace r. Given an unstructured reasoning trace r, the goal of graph construction is to extract a structured representation that captures the model's articulated ideas and their relations. We first generate an initial graph

where the prompting constraint Pinit induces the LLM to output idea nodes and directed relations. Each node in N (0) is a short textual description of a creative idea, and each edge in L(0) denotes a structural relation between ideas.

Raw reasoning traces often contain redundant or overlapping idea articulations. To obtain a semantically minimal yet expressive representation, we apply an adaptive pruning procedure. For all node pairs (ni, nj) ∈ N (0) with i < j, we compute

where Sim(·, ·) denotes cosine similarity between sentence embeddings. An adaptive threshold τP = µ + σ is computed from the distribution of {sPij }. For node pairs satisfying sPij > τP , we query an LLM with prompt Pprune to determine whether the nodes express the same underlying idea. Confirmed duplicates are removed, yielding a pruned graph G = (N,L) that serves as the basis for all subsequent analysis. Table 2 reports the effect of adaptive pruning on redundancy reduction in the thought graph.

3.3 Realization-Centered Dual-Dimension Analysis

With the annotated thought graph, we analyze the realizational process along two complementary dimensions: ideaartifact realization and intent-idea relation. Together, these dimensions operationalize the abstract mappings defined in the realizational process theory.

Idea-Artifact Realization

For each idea node ni corresponding to an articulated idea ik ∈ I(r), we determine whether it is realized in the artifact y, thereby instantiating the realization indicator ρ(ik, y). To ensure rigorous and conservative judgment, we adopt a strategy that decouples evidence extraction from realization verification.

Specifically, each idea is first converted into a verifiable, artifact-grounded question

which is answered using the artifact as contextual evidence:

We then compute a semantic consistency score

which serves as a continuous measure of evidential support for realization rather than a direct realization decision. Based on the distribution of {sRi }, we adopt a two-threshold evidence triage strategy with τhi = µ+σ and τlo = µ-σ. Only ambiguous cases are passed to explicit verification:

The final realization label is defined as

This procedure yields a conservative instantiation of the realization function ρ at the level of individual ideas. Table 3 empirically validates its effectiveness in reducing false positives.

Intent-Idea Relation

Each idea node ni ∈ N is independently classified according to its semantic relation to the original user intent x:

thereby instantiating the intent-idea alignment relation required for conditioning realization analysis.

Finally, given intent and realization labels on the thought graph, we summarize realization outcomes conditioned on intent-idea relations. For each relation category c ∈ K, we define

where Φ(x, y;G) constitutes a structural characterization of the knowing-doing gap, revealing how different classes of intent-aligned ideas are selectively realized, weakened, or lost during execution, rather than a single scalar performance score.

4 Experiment

Our experiments consist of three parts: (1) validating Mutelier by comparison with human expert annotations; (2) analyzing the knowing-doing gap across current state-of-theart LLMs; and (3) demonstrating the practical usefulness of Mutelier for human users.

4.1 Experimental Setups

Tasks and Data. We study three representative symbolic composition settings: melody-to-accompaniment generation, melody continuation, and text-to-music composition. For first two tasks, we use phrase-level data from Pop909 [Wang et al., 2020b; Hu et al., 2024]. For the third task, we sample prompts from MidiCaps [Melechovsk´y et al., 2024], each containing at least two controllable attributes (e.g., mood, tempo). Each task includes 150 test samples, balancing diversity with feasibility for expert annotation [Chakrabarty et al., 2024].

Compared Analysis Methods. We compare three analysis configurations under the same backbone (GPT-4o). Outputoriented analysis evaluates the (x, y, r) pair without eliciting reasoning. Process-oriented analysis elicits reasoning but does not construct a thought graph for analysis. Mutelier conducts analysis over a structured knowing-doing graph.

Evaluation Protocol. Both human experts and models assess intent-idea and idea-artifact alignment using shared metrics. The Intent-Idea alignment rate (I-I) measures the percentage of sub-ideas in a reasoning trace that are consistent with the input intent, where ideas labeled as preserve or elaborate are treated as intent-aligned. The Idea-Artifact alignment rate (I-A) measures the percentage of articulated ideas that are actually reflected in the generated artifact. In Experiment 1, agreement between model predictions and human annotations is evaluated using Pearson correlation (PC) and mean absolute error (MAE) on continuous rates, together with non-redundancy and false positive rate (FPR), where the most-agreed human annotations are treated as ground truth. In Experiment 2, we directly analyze model behavior by reporting I-I, I-A, and the Intent-Artifact alignment rate (II-A), enabling structural assessment based on the induced thought graphs.

Human Experts and EvaluationModels. For Experiment (1), we recruited 15 music-domain experts (9 male, 6 female; aged 28-47, M=35.4) with backgrounds in composition, music theory, and music education. Experts received rubric training and evaluated fixed prompts and model outputs to avoid confounds. Each sample was independently rated by 10 experts, yielding 4,500 total ratings, with good to excellent inter-rater reliability, as measured by the intraclass correlation coefficient (ICC > 0.8). For Experiment (2), we evaluate 15 LLMs spanning open-source and proprietary families, including Olmo2-7B, 13B [OLMO Team, 2024], Qwen2.5-7B, 32B, 72B [Yang et al., 2024], LLaMA3-8B, 72B [Grattafiori et al., 2024], Mistral-7B [Mistral, 2025], Phi-4 14B [Abdin et al., 2024], DeepSeek-R1-70B, V3-671B [Liu et al., 2024], Claude-4 [Claude, 2025], Gemini 1.5-Pro [Gemini Team, 2024], and GPT-4o, GPT-o3 [OpenAI, 2025a]. All models use standardized prompting templates to ensure comparability.

User Study. For Experiment (3), we conducted a withinsubject, counterbalanced user study with 180 composition enthusiasts familiar with LLM-based tools. Participants sequentially used three AI music creation systems over six weeks (two weeks per system): an output-oriented baseline producing only final outputs, a process-oriented system, and Mutelier, which allows inspection and editing of individual ideas. Across conditions, participants performed identical music creation tasks. After each period, they rated system support for the knowing-doing cycle using 7-point Likert scales measuring transparency (visibility and traceability of reasoning), controllability (ability to intervene in and adjust generation), and collaboration (perceived sense of human-AI co-creation).

Implementation Details. Across all experiments, we use all-MiniLM-L6-v2 as the sentence embedding model [Reimers and Gurevych, 2019; Wang et al., 2020a]. Symbolic music is represented in ABC notation [Yuan et al., 2024]. Intent-idea relations are categorized as preserve (ideas that directly follow the intent), elaborate (ideas that extend the intent), shift (ideas that partially deviate from the intent), or diverge (ideas that depart from the intent). All LLM prompts use fixed templates across experiments. Opensource models (Olmo2, Qwen2.5, LLaMA3, Mistral, Phi, and DeepSeek-R1) are deployed locally using Ollama, while API-based models, including GPT-4o, GPT-o3, DeepSeekV3-671B, Claude-4, and Gemini-1.5-Pro, are accessed via standard APIs and usage quotas.

4.2 Experiment 1: ValidatingMutelier Against Human Experts Main Results

Figure 2 reports agreement between different analysis methods and human experts on the same LLM-generated reasoning traces and musical outputs. Across all three tasks, Mutelier consistently achieves stronger alignment with expert judgments than both baselines on Intent-Idea (I-I) and Idea-Artifact (I-A) dimensions, yielding stable Pearson correlations together with low MAE. In contrast, the Outputoriented baseline ignores how ideas are formed, while the Process-oriented baseline relies on unstructured reasoning traces that remain noisy without grounding in the output. This result validates Mutelier as a reliable process-level analysis framework for artistic creation.

Ablation Analysis

Table 1 analyzes the contribution of individual components in Mutelier by progressively enabling structured representation, adaptive pruning, and realization verification. Introducing structured representation substantially improves both II and I-A alignment by decomposing raw reasoning traces into explicit ideas, yielding more stable correspondence with human annotations. Adaptive pruning further reduces noise by removing redundant or semantically overlapping ideas, leading to consistent reductions in MAE for both alignment stages. The largest performance gain arises from realization verification, which explicitly checks whether articulated ideas are reflected in the generated artifact. Together, these components enable Mutelier to achieve accurate and robust process-level alignment assessment, closely matching expert judgments across tasks.

Efficiency-Accuracy Tradeoff. This experiment is conducted on a subset of samples with human annotations for idea deduplication, where human judgments are treated as ground truth for whether multiple sub-ideas express the same underlying creative idea. Table 2 shows that similarity-based filtering alone is computationally efficient but limited in accuracy, as it relies on heuristic cosine similarity without explicit semantic validation. LLM-based verification substantially improves deduplication accuracy; however, exhaustive pair-wise verification is prohibitively expensive, while allat-once verification suffers from reduced reliability due to contextual interference. By combining lightweight similarity screening with selective LLM-based verification, adaptive pruning achieves the same accuracy as pair-wise verification with an order-of-magnitude reduction in runtime.

Verification Strategy Ablation. This experiment is conducted on a subset of samples with human annotations indicating whether each articulated idea is realized in the final artifact. Table 3 shows that direct LLM judgment without QA exhibits a severe confirmation bias, achieving moderate accuracy but an extremely high false positive rate, as the model tends to affirm realization even in the absence of supporting evidence. Introducing QA-based judgment substantially reduces false positives by decoupling idea interpretation from artifact inspection. Further adding evidence triage improves overall accuracy while keeping the false positive rate low, demonstrating that verification is essential for reliably distinguishing articulated knowledge from realized outcomes.

4.3 Experiment 2: Diagnosing Knowing-Doing Gaps Across 15 LLMs

Model-wise Performance

Table 4 reveals a consistent knowing-doing gap across models. While I-I scores are uniformly high, indicating that current LLMs can generally understand and articulate intentaligned ideas, execution-related metrics remain substantially lower. In particular, the I-I-Ametric shows that many intentaligned ideas fail to be realized in the final artifact, directly reflecting a knowing-doing gap with respect to user intent. At the same time, the lower I-A scores indicate that even ideas articulated by the model itself are frequently not realized, revealing a broader self-execution gap in structured music composition. Model scale consistently improves both I-A and I-I-A, but even the strongest models fall short of full alignment, suggesting that increased scale mitigates but does not eliminate execution failures.

Idea-to-Artifact Realization Diagnostics

Figures 3 and 4 further examine how ideas expressed in reasoning traces of current LLMs are translated into realized musical structure. These figures reveal the following patterns:

Insight 1: Ideation is concentrated in a narrow conceptual space. As shown in Figure 3(a), models repeatedly propose ideas from a small set of categories, while others, such as genre and tempo, are rarely explored. This clustering suggests limited ideational breadth, even before considering realization.

Insight 2: Realization, not ideation, is the primary bottleneck. Figure 3(b) shows that only a minority of proposed ideas are ultimately realized, with substantial variability in realization outcomes. This gap indicates that many ideas remain confined to the reasoning stage and fail to materialize into symbolic music composition.

Insight 3: Realized ideas exhibit shallow elaboration. As illustrated in Figure 4, models generate ideas across multiple depth levels, but execution is strongly biased toward shallow

(a) Histogram of total ideas in the reasoning trace and

those realized in the final music across three tasks.

(b) I-A realization rates by conceptual category.

Figure 5: Example of shallow idea realization. Green indicates realized ideas and red indicates unrealized ones; numbers mark intended correspondences between the thought graph and the artifact.

ideas, with realization probability dropping sharply as depth increases. In Figure 5, although the thought graph spans three hierarchical layers, only the top two are reflected in the realized music. While concept-level ideas such as G# inflection, three-voice texture, and long-sustained tones are realized, descriptive ideas are weakly concretized and inter-idea coordination remains limited. In particular, imagery such as church-bell resonance is not translated into explicit structural actions, nor differentiated across voices, and all parts are rendered with a default piano timbre. Overall, realized ideas exhibit limited multi-step refinement, constraining coherent hierarchical musical planning.

Intent Drift Analysis

Insight 4: Intent drift is introduced during realization through selective idea instantiation. We further analyze how intent-idea relations are transformed from the reasoning trace to the final artifact by categorizing ideas, for each model, into four relation types: preserve, elaborate, shift, and diverge. Figure 6 compares the distribution of intent-idea relations at the thought level with that observed among ideas ultimately realized in the artifact. At the reasoning stage, the majority of ideas align with the user intent through elaboration, indicating that models primarily reason by expanding upon the given intent. However, among realized ideas, the proportion of elaborate decreases substantially, while shift and diverge increase markedly. This redistribution suggests that intent deviation is introduced during the execution process, rather than originating from the reasoning stage itself. A plausible explanation is that reasoning favors intent expansion, while realization is constrained by coherence, leading to selective instantiation that favors shifted or simplified ideas.

Stability of Realization under Fixed Reasoning

Insight 5: Realization remains unstable even under fixed reasoning. Based on the same example in Figure 5, we generate artifacts using an identical intent and reasoning trace, and observe substantial variation in idea realization across 11 repeated runs. Structurally grounded ideas (e.g., threevoice texture) are consistently realized, whereas expressive or stylistic ideas are highly unstable: church-bell imagery and natural minor with G# inflection appear in only 9% and 18% of samples, respectively. This indicates a lack of enforceable mechanisms that bind high-level intentions to stable execution, resulting in selective and inconsistent realization.

4.4 Experiment 3: Exposing the Knowing-Doing Process via a Large-Scale User Study

Table 5 shows Mutelier significantly outperforms outputoriented and process-oriented baselines in transparency, controllability, and collaboration (p < 0.01). While transparency alone improves understanding, it does not restore user agency. In contrast, Mutelier provides an explicit, editable representation of model reasoning as a thought graph, through which users can adjust the interpretation of articulated ideas without altering internal model reasoning, influencing how intent and ideas are analyzed and reflected in generated artifacts. These results show that effective human-AI co-creation requires not just visibility into reasoning, but shared control over the knowing-doing cycle.

4.5 Limitations

While Mutelier shows strong alignment with human evaluations, its performance depends on the symbolic music understanding of the underlying LLM used for analysis. Stronger models consistently yield higher correlation and lower error. For example, GPT-4o achieves an average PC of 0.67 with an MAE of 0.12, compared to 0.50 / 0.21 for LLaMA3-72B and 0.42 / 0.28 for LLaMA3-8B, indicating that weaker models can limit the performance. As more capable models become accessible, this limitation is expected to diminish. In addition, Mutelier analyzes reasoning traces that are explicitly produced by models; internal latent processes that are not externalized into text remain inaccessible, and thus certain internal aspects of the knowing-doing gap may go unobserved.

5 Conclusion

This paper investigates the knowing-doing gap in LLMbased artistic creation, where models articulate coherent understanding during reasoning but fail to consistently realize it in generated artifacts. We characterize this gap as a realizational failure localized at the reasoning-to-artifact transition and introduce Mutelier, a process-level analytical framework that traces how reasoning-stage ideas are selectively realized, weakened, or lost in the final artifact. Across multiple experiments, we show that Mutelier reveals systematic realization failures, distinguishes knowing-doing behaviors across models, and improves human understanding when the generation process is exposed. Overall, this work shifts the analysis of machine creativity from outcome-level evaluation to process-level realization dynamics, providing a foundation for studying and addressing knowing-doing gaps in creative AI systems. Future work will explore integrating processlevel analysis into training and interactive generation loops, and extending the framework to other creative domains and modalities.

Acknowledgments

This work is supported by the project Towards Nextgeneration Artificial Auditory System with Brain-inspired Technologies (C5052-23G).