Tree-of-AdEditor: Heuristic Tree Reasoning for Automated Video Advertisement Editing with Large Language Model

1 Introduction

Video advertising has become an increasingly popular marketing strategy to capture consumers' attention and drive purchase behavior on e-commerce platforms like Taobao and Amazon [Liu and Yu, 2023]. Crafting effective video advertisements requires adherence to several fundamental principles, such as highlighting abundant product selling points, maintaining a coherent narrative, and aligning with the preferred editing styles of the target consumers [Armstrong, 2010]. Creating such video advertisements necessitates both editing expertise and marketing acumen, and it is timeconsuming to meticulously select and sequence shots from large footage repositories. All these make large-scale, effective product promotion on e-commerce platforms unfeasible.

∗Corresponding authors

Supplementary: github.com/GeniusEditor/Tree-of-AdEditor

Previous works on video advertisement editing mostly rely on predefined rules to constrain the visual quality and coherence [Lin et al., 2021; Liu et al., 2021; Liu et al., 2019]. While these fixed patterns enhance the overall visual attractiveness, this closed-world setting struggles with narrative organization and selling points discovery. Other learning-based methods try to mine professional knowledge from well-edited videos, like cut-trigger patterns [Pardo et al., 2021] and longrange associations among shots [Argaw et al., 2023]. However, this branch of studies lacks an explicit explanation and requires massive datasets and training costs.

Recently, Large Language Models (LLMs) have exhibited impressive zero/few-shot reasoning capabilities across various planning tasks [Huang et al., 2024]. The high-level semantic understanding and abundant domain knowledge of LLMs provide a golden opportunity to advance intelligent video advertisement editing. Typically, video advertisement editing involves selling point discovery, narrative organization, and style understanding. It is challenging for vanilla input-output (IO) prompting to consider all components in only one step. Chain-of-Thought (CoT) [Wei et al., 2022;

Kojima et al., 2022; Wang et al., 2022] can decompose hard problems into several intermediate steps and infer the final answer step by step along this linear thinking chain. Although the CoT achieves significant improvement, it hardly adapts to non-linear and iterative thinking hierarchy of the video editing process, where editors iteratively try different shots or put them back to explore multiple potential editing paths, ultimately fulfilling the creation purpose. For the improved structure based on CoT, Tree-of-Thoughts (ToT) [Yao et al., 2024] builds a reasoning tree to globally explore and selfevaluate each branch of thought, which closely mirrors the manual editing process illustrated above. However, ToT still faces two significant challenges in video advertisement editing: 1) Alignment with Effective Video Advertising Principles: To align the LLMs with the goal of effective video advertisement (e.g., informative, coherent, and stylistic) [Armstrong, 2010; Scott Armstrong, 2011; Valentini et al., 2018; Liu and Yu, 2023; Bowen, 2017], how to design domainspecific prompts to guide the node generation and optimal path selection in the tree inference structure is a challenging issue. 2) External Fact-Checking for Sub-Step: ToT simply conducts self-evaluation without any external knowledge, resulting in unfeasible and cost-ineffective searches.

To address the above challenges, we propose a novel framework, Tree-of-AdEditor (ToAE), which constructs a reasoning tree that can iteratively look ahead ("select") and backtrack ("put back") to precisely imitate human editors. Given the video footage of the product and the brief description of the editing goal, ToAE achieves systematic planning and identifies the optimal editing solution regarding informativeness, coherence, and stylistic preference alignment with target consumers (also easily adaptable to other criteria if needed). Specifically, to extract the essential information for the editing task and address the modality misalignment between multi-shot data and LLM-understandable format, we first leverage an explicit shot semantic representation module. This module encodes each shot, including content details and cinematographic elements like shot scale. Then, we develop a 'local-global' mechanism that precisely guides the LLMs through both shot-level decisions and overall sequence evaluation. The Local Next Shot Selector recommends a good next shot as a tree node, while the Global Sequence Evaluator holistically assesses each potential editing plans derived from the reasoning tree. Finally, to ensure visual attractiveness, we leverage the heuristics of visual coherence errors to provide external fact-checking, which effectively cuts down the the computation cost on the visually abrupt nodes. To conclude, our contributions are summarized below:

• We propose Tree-of-AdEditor (ToAE), a novel framework that uses a reasoning tree to mimic manual video editing with LLM knowledge and visual heuristics, to our knowledge, the first of its kind.
• Motivated by Effective Video Advertising Principles, we design a "local-global" mechanism that guides the LLM at both the shot level and sequence level decisionmaking.
• Quantitative experiments on MovingFashion and our novel dataset, ProductAVE, demonstrate the effective-

ness and efficiency of ToAE compared to advanced baselines. Human evaluations further validate ToAE's superiority, with 43% of professionals favoring it over 32% for ToT.

2 RelatedWork

2.1 Video Advertisement Editing

The video advertisement editing process consists of shot selection and sequencing, both essential for creating an informative and engaging visual narrative. Many methods rely on manually annotated datasets to train classifiers for shot selection, such as narrative importance [Liu et al., 2019]. However, these approaches incur high costs and rely on subjective and hard-to-standardize annotations. For shot sequencing, existing methods often relies on predefined rules [Lin et al., 2021; Liu et al., 2021; Tang et al., 2022; Liu et al., 2019; Galvane et al., 2015; Leake et al., 2017; Arev et al., 2014; Wang et al., 2019], and they are hardly imitate complicated open-domain editing techniques and struggle with highlevel tasks like narrative organization, selling point discovery, and style understanding. Some learning-based methods in the movie domain [Argaw et al., 2022; Pardo et al., 2021; Argaw et al., 2023] learn professional editing knowledge like cut-trigger patterns [Pardo et al., 2021]. However, they lack interpretability and require large datasets and significant training costs.

2.2 Planning with LLMs

Recently, Large Language Models (LLMs) have demonstrated impressive reasoning abilities in tasks like mathematics, creative writing, and multi-hop question answering [Huang et al., 2024; Hazra et al., 2024; Zhang et al., 2024; Huang et al., 2022]. Various methods, such as few-shot demonstrations and refinement mechanisms [Madaan et al., 2024; Gou et al., 2023; Shinn et al., 2024], and accessing external knowledge [Guan et al., 2023; Press et al., 2022], have been proposed to enhance their capabilities. One distinct branch constructs the thought structure to solve the task little by little. The Chain-of-Thoughts (CoT) method [Wei et al., 2022; Kojima et al., 2022] reasons along a linear thought chain, and CoT-SC [Wang et al., 2022] was introduced to enhance robustness. While CoT-like methods struggle with the non-linear nature of video editing and lack a fact-checking mechanism to prevent error propagation. Tree-of-Thoughts (ToT) [Yao et al., 2024] constructs a reasoning tree that iteratively looks ahead and backtracks, mimicking human editors' cognitive process. However, general ToT lacks reliable factchecking and effective guidance to induce shot-level decision and overall sequence-level planning for unique video advertisement editing tasks.

3 Preliminary

3.1 Characteristics of Video Advertisement Editing

Video advertisement editing involves selecting and arranging footage to highlight product features, engage the audience, and drive purchases. Editors typically select shots iteratively, organize them into a coherent narrative, and add captions, music, or special effects [Bowen, 2017]. Among these steps, shot selection and assembly are crucial, as they shape the narrative and determine the ad's resonance with the target audience. This paper focuses on optimizing the iterative process of shot selection and assembly, with advanced effects and post-production to be explored in future work.

Motivated by the theories in advertisement marketing and video editing [Armstrong, 2010; Scott Armstrong, 2011; Valentini et al., 2018; Liu and Yu, 2023; Bowen, 2017], we summarize three pivotal characteristics of effective ad video advertisement editing:

• Informativeness. The advertisement should showcase key product selling points, functional features, and usage scenarios to provide customers with the necessary information to make informed decisions. Irrelevant or non-essential content should be avoided to keep the message concise and easily understood.
• Coherence. A coherent narrative ensures a smooth storyline, reducing the cognitive load of viewers and preventing confusion. Smooth shot transitions, such as scale changes and angle shifts, allow each shot to flow naturally into the next, while also enhancing visual variety to sustain viewer interest and anticipation.
• Attractiveness. Visual quality, narrative structure, cutting rhythm significantly impact attractiveness. Aligning these elements with target audience preferences further boosts appeal and encourages purchase behaviors.

While our framework emphasizes these aspects, it is highly adaptable to incorporate other relevant factors if needed.

3.2 ProblemFormulation

Motivated by these fundamental principles, we aim to generate informative, coherent and stylistic video advertisement A, given the shot repository S = {s1, .., sn} and editing goal description (e.g., selling points) D provided by product business. Each shot si is cut from the original recordings, representing a unit that captures an uninterrupted action, event, or scene. We formulate this task as an optimal path-finding problem in the tree search. The "search" and "backtrack" operations in the tree construction can precisely mimic the "select" and "put back" actions of human editors. Specifically, our Tree-of-Editor (ToAE) framework takes the LLM pθ, shot repository S, editing goal description D, and instructions P as input to construct a reasoning tree T and to find the optimal editing solution A∗ . Here we denote T = (N, E) as the reasoning tree. Each tree node n ∈ N represents a selected shot from the shot repository, with edges E ⊆ N ×N from parent nodes to child nodes indicating shot transitions. The reasoning path Ai within the tree is represented as a series of connected nodes, beginning at the root and extending to a leaf node. The optimal editing solution A∗ is derived from the set of all possible editing plans {Ai} in the searched tree T (S,D, P, pθ), aligning best with the editing goal D. The ultimate goal is formulated as:

where function U(Ai;D) evaluates video advertisements based on informativeness, coherence, and attractiveness.

4 Methodology

4.1 Overview

As Figure 2 shows, our ToAE framework consists of: 1) Explicit Shot Semantic Representation encodes each video clip as textual descriptions of its content details and cinematographic metrics. This serves as the node representation in the tree reasoning, bridging video-text modality gaps between the multi-clips data and LLM-understandable formats. 2) Local-Global Guidance navigates the LLM in shot-level and sequence-level decision-makings. Local Next Shot Selector integrates selling points and narrative structure to determine next shots as child nodes. And Global Sequence Evaluator assesses all potential plans derived from the reasoning tree, focusing primarily on stylistic alignment with the target audience and other content-related attributes. 3) To enhance visual attractiveness, Visual Incoherence Pruning provides external fact-checking to identify visual coherence errors between newly selected shots and the historical sequence. If the transition is visually abrupt or incoherent, the new branch is pruned. Finally, the plan with the highest score from the reasoning tree is selected as the optimal solution. The last subsection details the DFS search process and inference cost.

4.2 Explicit Shot Semantic Representation

LLMs demonstrate superior performance across various NLP tasks. However, emerging multi-modal LLMs like DALL-E [Ramesh et al., 2021] and Videollama [Zhang et al., 2023], struggle with reasoning in the video domain, particularly when handling multiple clips as input. Video ad editing also requires the understanding of content details and cinematographic elements, making the above methods barely applicable. To enable effective LLM inference in multi-clip editing, we employ a video-text model to encode the content and cinematographic elements of each clip. They serve as node representations in the reasoning tree, effectively bridging the gap between video inputs and LLM-compatible formats.

Specifically, editors usually focus on the content of each shot, including the main object, the action, and the background, and the cinematographic elements like shot scale and camera angle [Bowen, 2017; Smith, 2006]. Hence, the textual description of each shot si is represented as sdi = R(si) =<main object detail, action, background, shot scale, camera angle>, and R(·) is the text-video model. We design explicit prompt to encode the representation as follows: This is a shot from a product advertisement video of [product name]. What does this shot describe (include main object detail, action, background, shot scale, camera angle)?

4.3 Local-Global Guidance of Effective Advertising

To align with effective video advertising principles, we design the "local-global" guide to navigate the LLM for node generation and path evaluation, namely, Local Next Shot Selector and Global Sequence Evaluator.

defined as the proportion of the main object in the whole screen, e.g., the close-up, the full shot.

Local Next Shot Selector

In the shot selection and assembly process, editors usually consider two primary aspects: the shot itself and its transition from the historically edited shot sequence. Motivated by the principles of effective advertising, a good video should showcase product features, avoid irrelevant content, attract viewers, and maintain narrative coherence. Hence, we propose Conveyance, Diversity, Attractiveness for the LLM to select shots that effectively convey the selling points and present certain differences from other shots in the ad. And Coherence evaluates the narrative coherency between the candidate shot and historical sequence. According to the above criteria, the next shot selector N(·) recommends feasible subsequent nodes n(1...k) t+1 based on the historical path (nroot 1 , ni2 2 , ..., nit t ) from the root node to the current node, which is calculated as:

Where nit t denotes the it-th node at the t level, k represents the number of recommended next nodes, and P select is the prompt for the next shot selector.

Global Sequence Evaluator

The Next Shot Selector guides shot selection, but controlling style such as cutting rhythm and narrative structure, remains challenging at the shot level. Therefore, we propose the Global Sequence Evaluator to holistically assess each potential editing plan from the reasoning tree, focusing on stylistic alignment and content-related aspects at the sequence level.

According to video editing theories [Bowen, 2017; Choudhary et al., 2019], style can be formulated as the distribution of shot attributes and their evolving patterns over time. For instance, a transition from a full shot to a close-up can intensify emotions. Hence the style of an editing plan Ai is computed as: L(Ai) = pθ({nroot, . . . , nleaf}), nj =<main object detail, action, background, shot scale, camera angle>, Ai ∈ T . To evaluate Stylistic Alignment, we first induce LLM to infer the target audience's preferred style ˆL = pθ(Sd ,D) and then let LLM evaluate the style of the given editing plan L(Ai) , finally infer its match degree with target users' preferred style pθ(ˆL,L(Ai)). In addition, inspired by the principles of effective advertisement practice, we also summarize Conveyance, Coherence, Attractiveness altogether. The process of the global sequence evaluator is computed as follows:

where z ∈ {Conveyance, Coherence, Attractiveness, Stylistic Alignment}.

4.4 Visual Incoherence Pruning

The Local-Global Guidance relies solely on textual descriptions, often causing visually abrupt transitions that impair narrative clarity and aesthetic appeal. To address this, we introduce the Visual Incoherence Pruning module, which factchecks visual coherence at each sub-step in the reasoning tree, focusing on transitions between newly selected shots and the historical sequence. This module ensures visual cohesion, reduces unnecessary exploration, and lowers inference steps. Here we focus on four common types of errors for video advertisement scenarios, as shown in Figure 3:

• jump cut: appears when two subsequent shots present very similar content from slightly different angles or positions, leading to a noticeable gap. Cjc(·) is calculated by the similarity of SURF keypoints [Bay et al., 2008].
• opposite motion: Copp(·) is calculated by the angle difference of these motion vectors. The motion feature is derived from the optical flow tracking [Sun et al., 2010].
• intensity difference: Cint(·) computes as the speed difference of motion features.
• tonal discontinuity: Ctonal(·) compares the similarity of the HSV color histograms.

We use these heuristics to fact-check the visual transition from the last shot of the historical sequence and newly selected shots (from Next Shot Selector). If the visual incoherence error exceeds the threshold (Cprune(n(q) t+1 ) >ε), this branch will be pruned.

where fl(·) judges whether the corresponding errors exists. 0 for "not exist", 1 for "exist", and function I(·) maps a shot's textual description to the corresponding video shot.

4.5 Cost-Efficient Optimal Path Search Algorithm

Algorithm 1 outlines the depth-first search (DFS) process for constructing the ToAE reasoning tree, mimicking manual video editing by deeply exploring potential shots and backtracking. Starting with an empty sequence seq0, next shot selector N(·) generates candidate nodes and explore them from high to low confidence until a stop condition is met (either the tree size exceeds num > M or no candidates remain). For each newly selected node n′ , visual incoherence with the historical sequence is evaluated. If the coherence error exceeds threshold ε, the subtree is pruned and backtracks. Otherwise, this new node n′ will be added to the current sequence seq and exploration continues. Finally, all potential plans {Ai} from the reasoned tree are evaluated by global sequence evaluator in terms of the above four criteria, with the highestscored plan chosen as the optimal solution A∗ .

where P = (P select , P eval , Cprune).

Inference Cost. The computation complexity of ToAE is O((p ∗ n)d ), where n is the number of shot candidates and d denotes the depth of the reasoning tree.

Proof: Each node is selected without replacement (i.e., once a shot is selected, it is no longer available unless backtracked). Initially, there are n possible choices at the first

Algorithm 1 Tree of Editor-DFS

1: procedure TOAE(pθ, S, D, P, M, ε) 2: Sd ←R(S, P encode ) 3: seq0 ← empty 4: {Ai} ←DFS(seq0, Sd ,D, 0) 5: A∗ ← argmaxi ∑z G(pθ,Ai, Sd ,D, P eval ) 6: return A 7: end procedure 8: procedure DFS(seq, Sd , num) 9: nexts ←N(pθ, seq, S d ,D, P select ) 10: if num > M or nexts is empty then 11: A ← record(seq) 12: return A 13: end if 14: for n′ ∈ nexts do 15: sorted candidates 16: q1 ← I(seq[last]), q2 ← I(n′ ) 17: if Cprune(q2) > ε then 18: prune 19: else 20: update seq′ ← assembly(seq, n′ ) 21: num ← num + 1 22: DFS(seq′ , Sd , num) 23: end if 24: end for 25: end procedure

level, n - 1 at the second level, and so on, resulting in a tree size of O(n(n - 1)(n - 2)(n - d)). When visual pruning is applied and only a fraction pd of the branches is retained at each level d, thereby the total nodes of reasoning tree can be expressed as ∏Dd=0 pd (n - d). Thus, the total complexity of ToAE is O((p ∗ n)d ).

Compared to the complexity for na¨ıve permutation O(n!), ToAE cuts down much computation costs and becomes computationally feasible for larger values.

5 Experiment

5.1 Dataset and Preprocessing

MovingFashion-AVE: MovingFashion [Godi et al., 2022] includes fashion video advertisements from the e-commerce platform Net-A-Porter . We create 458 editing tasks ranging from 3 to 6 shots in the fashion advertisement scenario. ProductAVE (Ad Video Editing): comprises 152 Taobao video advertisements across various categories, including furniture, kitchenware, home decor, toys, etc. These videos are typically longer and more challenging than fashion ads, with lengths ranging from 3 to 10 shots after preprocessing. More details are explained in Supplementary.

5.2 Metrics

We employ the LLMs scorings (Conveyance, Coherence, Attractiveness, Stylistic Alignment), and Visual heuristics (Jump cut, Opposite Motion, Intensity Difference, Tonal Mismatch)

https://www.net-a-porter.com

https://www.taobao.com/

for evaluation. Higher scores in the first four dimensions and lower visual incoherence errors indicate better outcomes.

5.3 Implementation Details

We harness the llama3.1:70b [Touvron et al., 2023] deployed in the local device for our experiment (to verify the applicability of ToAE in different LLM foundation, we also compare the performance in Mistral and GPT-4o-mini in the Supplementary). We employ a multi-modal large language model, VideoLLama [Zhang et al., 2023], to encode the raw video footage into the textual description. Our initial observations suggest that VideoLLama extracts more detailed and nuanced information, making it better suited to our application scenario compared to VideoChat [Li et al., 2023] and VideoChatGPT [Maaz et al., 2023]. For the next shot selector, the number of generated answers is determined by LLMs. We limit the DFS process to a maximum of 100 inference steps to balance the performance and efficiency.

5.4 Baselines

Inter-Intra Contrastive (IIC) [Tao et al., 2020] and Longrange Multimodal Pretraining (LMP) [Argaw et al., 2023] both utilize their learned representation for shot sequencing (retrained on both two datasets using 3-fold cross-validation). IO generates directly with LLM. CoT (Chain-of-Thought) [Kojima et al., 2022] generates in a linear chain based on solely textual information. ToT (Tree-of-Thought) [Yao et al., 2024] infers in a reasoning tree based on only textual information. ToAE w/o Visual generates without visual checking.

5.5 Comparison with Baselines

Global Evaluator Scoring. Table 1 shows that ToAE achieves the best performance across both datasets, showcasing its effectiveness and adaptability in diverse scenarios. Compared to learning-based methods like IIC and LMP, this highlights the limitations of small models and the advantage of using domain knowledge from LLMs for automatic video editing. ToT outperforms IO and CoT, demonstrating the effectiveness of tree-structured reasoning. Comparing ToAE w/o Visual with ToT underscores the importance of "localglobal" guidance in tree construction. Table 3 deomstrates its consistent performance on different LLM foundations, highlighting its robustness with varying model capabilities. Additional comparisons can be found in the Supplementary.

Visual Coherence Error. Given that error values vary with different footage, we focus on the relative reduction. Table 1 demonstrates that ToAE achieves significant improvements, reducing Jump Cut error, Tonal Mismatch, and Intensity Difference by at least 10 % and minimizing Opposite Camera Motion by at least 15 % compared to IO, CoT, and ToT.

Effects of the Length of Candidate Shots. As the shot repository grows, the algorithm needs to consider more information and combinations, making inference challenging. Figure 4 (a)(b) shows that ToAE achieves higher performance even under more challenging settings when n = 5, 6. ToAE achieves at least 10.2%, 6.4%, and 5.2% improvement regarding Coherence, Attractiveness, and Stylistic compared to IO and CoT. The improvement may stem from a richer pool, enabling more diverse shot assembly. This demonstrates the effectiveness and robustness of ToAE in typical-length ecommerce ads, highlighting its real-world applicability.

Efficiency of ToAE. Table 4 shows the exponential growth in computational cost for naive enumeration, with inference steps reaching 64 for an input size of only 4. ToAE w/o Visual demonstrates the effectiveness of LLM domain knowledge and 'local-global' guidance in avoiding unlikely exploration. Additionally, ToAE outperforms ToAE w/o Visual by reducing approximately 10 steps when n = 6. It validates visual fact-checking's effectiveness in reducing abrupt branches and lowering computation costs. Despite the effectiveness of ToAE in short video ad production, it still faces challenges with longer ads, which will be addressed in the future work.

5.6 Ablation Study

As Table 2 shows, ToAE w/o encoding relies on general video-text LLM descriptions instead of leveraging explicit cinematographic details, which are critical for ensuring coherence, attractiveness, and stylistic alignment. ToAE w/o GSE (Global Sequence Evaluator) generates less coherent and stylistically aligned video advertisements. The comparison of ToAE w/o Visual with ToAE highlights the importance of visual heuristic fact-checking in avoiding coherence errors.

5.7 Human Evaluation

To validate our method in real-world scenarios, we conducted a human evaluation with six highly qualified professionals, each with extensive experience in video editing and advertising, including some with over 15 years of expertise. Specifically, participants are asked to evaluate generated video advertisements for ten products, each with three generated ads by CoT, ToT, and ToAE. The evaluation scale ranges from 1 to 10, with 1 indicating poor and 10 indicating excellent. Additionally, participants are required to select their preferred advertisement for each product. The results of human evaluation are shown in Figure 4 (c). ToAE outperforms other methods, achieving the highest scores across all dimensions and being favored by 43% of professionals, compared to 25% for CoT and 32% for ToT. One example is shown in Figure 4 (c), and more examples could be found in Supplementary.

6 Conclusion

This paper presents the first exploration of leveraging LLMs for intelligent video editing, aiming to generate engaging short ecommerce advertisements based on user-provided video footage and editing goals. We propose a novel Treeof-AdEditor (ToAE) framework that constructs a reasoning tree to mimic the cognitive process of human editors. This frame work integrates fundamental principles in video editing and advertising marketing and external visual heuristics to address the significant challenges of adapting ToT in the video advertisement editing. Future research could explore integrating music and special effects or addressing the challenges of producing long advertisements.

Acknowledgments

This work was supported by the National Science Fund for Distinguished Young Scholars (No. 62025205), the Hong Kong Research Grants Council under General Research Fund (No. 15200023), Research Impact Fund (No. R1015-23), and National Natural Science Foundation of China (No. 62432007, No. 62272390).