CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG

1 Introduction

Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for enhancing Large Language Models (LLMs) based on external knowledge [Qiao et al., 2025; Guo et al., 2025], serving as a foundation in many AI applications [Mridul et al., 2025; Niu et al., 2024]. As from Figure 1, typically the RAG contains a retriever (i.e., the retrieval stage linked with a vector/graph database) and a generator (i.e., the generation stage by a LLM): for a given batch of queries, the retriever retrieves relevant contexts that consolidate the query prompts for the generator to produce better answers.

Like many other systems [Chen et al., 2018; Chen et al., 2025; Ye et al., 2026], the effectiveness of RAG in handling queries highly depends on its hyperparameter values [Fu et al., 2024], i.e., those for the retrieval (e.g., chunk size) and the generation stage (e.g., generation temperature), thus the their optimization is crucial. However, optimizing RAG hyperparameters using given queries is challenging, because:

∗ Pengzhou Chen is also supervised in the IDEAS Lab.

†Tao Chen is the corresponding author.

• With the combined retriever and generator in RAG, the dimension of tunable hyperparameters increases dramatically with more complex interactions, making optimization difficult [Barker et al., 2025; Stefano et al., 2024].
• Evaluating the quality of a hyperparameter configurations as the feedback under full retrieval-generation round can be expensive, e.g., configuration evaluation on a RAG with Llama-3.1-8B over the Agriculture benchmark can take up to 30 minutes [Guo et al., 2025].

To that end, many algorithms have been proposed. For example, there exist general algorithms for tuning hyperparameters [Bergstra et al., 2011; Cowen-Rivers et al., 2022]; specific ones such as AutoRAG-HP [Fu et al., 2024] and METIS [Ray et al., 2025] for RAG that focus on the entire retrieval-generation process as a whole; as well as others that solely perform hyperparameter optimization/HPO related to the prompt or the retriever of RAG [Agrawal et al., 2025; Jeong et al., 2024]. Yet, those algorithms are limited in the sense that they merely treat RAG as a monolithic black box or restrict themselves only to the hyperparameters at a certain stage. This ignores the highly structural internal information of RAG, causing slow convergence and devastating results.

In this paper, we take a different perspective to bridge these gaps: drawing on our understandings and empirical experiments, we observe that the retriever and generator in RAG are interrelated but also partially separable, each plays different roles that determine the final generation outcomes of RAG. As such, we hypothesize that "if the hyperparameters of retriever and generator can be optimized loosely, in a way that they maintain merely the basic interactions, then it should yield better results using less budget." Thus, we propose CDS4RAG, a cyclic dual-sequential framework that heuristically optimizes RAG's hyperparameters using given queries. What make CDS4RAG special is that it distinguishes the retriever and generator in RAG, optimizing their corresponding hyperparameters sequentially with different objectives, but doing so in a cyclic manner 1 , permitting a fine-grained budget provision and seeding 2 therein. This allows CDS4RAG to focus on the sliced hyperparameter space, hence mitigating the issue of complex hyperparameter interactions, while relieving the optimization from evaluating the costly full retrievalgeneration via a good proportion of retrieval-only evaluations.

Since the cyclic dual-sequential hyperparameter optimization is compatible with any general algorithms, CDS4RAG is also algorithm-agnostic. Specifically, our contributions are:

• We newly reformulate the RAG hyperparameter optimization as a cyclic dual-sequential problem.
• We present a way to sequentially optimize the hyperparameters of the retriever and generator for retrieval and generation quality, respectively, in a cyclic manner.
• To achieve fine-grained control over the budget provisioning of hyperparameter optimization between retrieval and generation within a cycle, we propose an adaptive mechanism that stops optimizing the retriever when it detects a retrieval plateau, while capping that for the generator with a fixed amount of evaluations. This reduces the costly full retrieval-generation evaluations needed.
• To expedite the optimization with a warm-start, we design a cross-cycle seeding, such that promising generation hyperparameter configurations in previous cycles, based on knowledge retrieved under different retrieval configurations, are reused in the current cycle for the generator.

Experimental results on four common benchmarks and two backbone LLMs reveal that CDS4RAG can considerably boost general algorithms in 21/24 cases while significantly outperforms other state-of-the-art hyperparameter optimization algorithms for RAG in all cases with quality improvement of up to 1.54× and better speedup. All data, code, and appendices can be accessed at: https://github.com/ideas-labo/cds4rag.

2 RelatedWork

2.1 General Hyperparameter Optimization

Over the decades, general algorithms like Bayesian Optimization (BO) [Snoek et al., 2012] and its variants [Snoek et al., 2012; Bergstra et al., 2011; Jiang et al., 2024; Chen and Chen, 2026; Li et al., 2020] have been proposed for hyperparameter optimization. Those methods rely on a surrogate model [Gong et al., 2025; Xiang et al., 2026; Gong and Chen, 2023; Gong and Chen, 2024] to predict the next evaluated configuration, balancing the exploration and exploitation with reduced evaluation cost. State-of-the-art algorithms like HEBO [CowenRivers et al., 2022] have also been developed to handle more complex and noisy search spaces robustly. Other metaheuristics without a model also exist [Xiong and Chen, 2025; Chen et al., 2024; Chen and Li, 2021; Chen and Li, 2024; Ye et al., 2025; Chen, 2022].

1 A cycle means the completion of one sequence of optimizing retriever and then generator.

2 Seeding means initialize an optimization with pre-defined hyperparameter configurations to start working with, enabling a warm-start.

However, the above often do not use any specific knowledge/information of the algorithm/system to tune, which can waste the valuable information of the known structure in RAG.

2.2 Full Hyperparameter Optimization for RAG

There are algorithms designed/tailored for optimizing RAG hyperparameters using given queries. Among others, RayTune [Liaw et al., 2018] tailors the general black-box optimization algorithms TPE to optimize the RAG hyperparameters. AutoRAG-HP [Fu et al., 2024] utilizes Hierarchical Multi-Armed Bandits to optimize the RAG hyperparameters. While effective for discrete/combinatorial hyperparameters, it struggles with continuous ones. METIS [Ray et al., 2025] is a multi-objective optimization approach to balance quality and latency in RAG, but it can struggle under high-dimensionality.

Yet, again those works often treat the RAG pipeline as a monolithic black box, relying on the evaluation of the full retrieval-generation rounds throughout the optimization.

2.3 Partial Hyperparameter Optimization for RAG

A parallel thread of research focuses on optimizing the hyperparameters of a specific stage for RAG, e.g., GEPA [Agrawal et al., 2025] and MIPROv2 [Opsahl-Ong et al., 2024] use LLMs to iteratively refine instructions for the prompt at the generator, but overlook the hyperparameters of the retriever (e.g., chunk size) that dictate the underlying context. In contrast, methods like AdaptiveRAG [Jeong et al., 2024] dynamically adjust retrieval intensity only (e.g., top-k) based on query complexity. Other studies justify the importance of hyperparameters at diverse stages in RAG [Orbach et al., 2025].

However, focusing on a single stage in RAG hyperparameter optimization can miss the valuable configurations that can only be found when different stages are considered together. Yet, the complex interaction and expensive evaluation make a direct joint optimization complex. This work tackles exactly such via cyclic dual-sequential hyperparameter optimization.

3 ProblemFormulation andMotivation

3.1 Classic RAGHyperparameter Optimization

A hyperparameter in RAG can be a continuous, enumerated, or categorical variable. Traditionally, hyperparameter optimization for RAG works in a combined hyperparameter space of configurations Ω = Φ × Θ from that of the retriever Φ and generator Θ. Formally, given a query batch q ∈ Q and a knowledge corpus D, this can be expressed as:

whereby f is the RAG system; M is the end-to-end performance metric over Q; G ∗ is the ground truth for answering all the queries. The goal is to jointly find the best hyperparameter configuration for both retriever and generator using Q, hence M, e.g., the F1-score of the generated answers, is maximized.

Albeit straightforward and widely adopted in existing work, this formulation forces one to work on the full combined space of retriever and generator under the costly full retrievalgeneration evaluations, which can be slow and ineffective [Orbach et al., 2025; Fu et al., 2024; Barker et al., 2025].

3.2 Cyclic Dual-Sequential Hyperparameter Optimization for RAG

While we can intuitively understand that the retriever and generator are two independent components in RAG, there is also stronger empirical evidence to motivate this work. From Figure 2, we see that the quality of contexts retrieved and answers generated under some queries, i.e., lexical precision and F1-score, commonly exhibit good positively monotonic correlation, which is also evidenced by prior studies [Salemi and Zamani, 2024; Orbach et al., 2025]. Further, we found that the evaluation cost of retrievers is often ≈ 50% cheaper than that of the generator. This motivates us to reformulate Equation 1 as:

where ω = (ϕ, θ); fg and fr are the generator and retriever, respectively; ϕ∗t and θ∗t are their hyperparameter configurations optimized at cycle t out of T cycles. P is the quality metric for the retriever, e.g., the lexical precision; R∗ is the ground truth of the contexts for all queries. Using Q, we seek to tackle Equation 2a by exploring the best hyperparameter configuration ϕ∗t for retriever under P first, and then use the retrieved contexts C ∗ t under ϕ∗t to jointly optimize and find the optimal generator configuration, leading to the best θ∗t on M thereunder at cycle t. This process is then repeated in a cyclic manner, eventually resulting in the best global (ϕ∗ , θ∗ ) for Q. Note that the above is not a classic bi-level/block-coordinate optimization, but rather a newly emerged, cyclic dual-sequential optimization at the same level for RAG. This is because in those classic paradigms, each step often optimizes one block/level of variables with respect to the same global objective [Sinha et al., 2018]. In our case, the cyclic dual-sequential formula uses distinct stagespecific objectives (Equations 2b and 2c), which are implicitly related to each other in the ultimate goal (Equation 2a).

4 CDS4RAG Designs

CDS4RAG is precisely designed to address the reformulated problem of cyclic dual-sequential hyperparameter optimization using some queries to RAG, as shown in Figure 3 and Algorithm 1. For that, it has several main mechanisms:

• Cyclic Dual-Sequential Optimization: Given a batch of queries, CDS4RAG cyclically optimizes the hyperparameters of the retriever (by Sobol sampling [Sobol', 1967]) and generator (by any general algorithm, e.g., HEBO or TPE) in turn (lines 7-11 and 20-23).
• Within-Cycle Budget Provisioning: To determine how the optimization of retriever and generator share the budget, CDS4RAG uses an adaptive mechanism to decide when to stop optimizing the retriever (lines 6 and 13), while for the generator, we set a cap of evaluations (line 20), reducing the costly full retrieval-generation evaluations.
• Cross-Cycle Generator Seeding To expedite the expensive optimization for the generator with a warm-start, CDS4RAG selects high-quality "seeds" of the hyperparameter configurations for generators archived across the cycles to be reused under the current cycle (lines 15-19).

The above naturally enable CDS4RAG to be a general framework, such that the underlying algorithm for optimizing generator can be replaced in a plug-and-play manner.

4.1 Dual-Sequential Optimization in Cycles

Optimization in CDS4RAG occurs in two sequential stages with multiple cycles, for which we provide the details below.

Algorithm 1 Pseudo code of CDS4RAG

• 1: Input: Budget B; generator cap N; archive of seeds A; retriever quality P; generator quality M; corpus D; given query batch Q;

retrieval ground truth R∗ ; generation ground truth G∗ 2: Output: Best hyperparameter configuration (ϕ∗ , θ∗ ) 3: while B has not been exhausted do 4: t = 1 5: ϕ∗t = 0; C ∗ t = ∅ ▷ Optimizing retriever 6: while Pi of ϕi < Pideal do 7: ϕi ← find the next retriever configuration using Sobol sampling 8: Evaluate the Pi of ϕi via D, Q, R∗ , Equations 3 9: Ci ← retrieve contexts with ϕi . 10: if Pi of ϕi > the P of ϕ∗t then 11: ϕ∗t = ϕi ; C ∗ t = Ci 12: end if ▷ Provisioning budget for retriever 13: Update Pideal using Equations 4 and 5 14: end while ▷ Seeding the optimization of generator 15: A′ ←De-duplicating A 16: H= ∅ 17: H← N4 generator configurations with top 50%MfromA′ 18: H=H∪ randomly generate N4 generator configurations 19: Re-evaluate theMof generator configurations in Hwith C ∗ i ▷ Optimizing generator with provisioned budget 20: H=H∪← run, e.g., HEBO, with C ∗ i , Q, G∗ , and N2 21: ifMi of ∀θi ∈ H> theMof θ∗ then 22: θ∗ = θi; ϕ∗ = ϕ∗t 23: end if 24: A = A∪H; t = t +1 25: end while 26: Return (ϕ∗ , θ∗ )

Retriever Optimization. Recall from Equation 2c, the goal of optimizing retriever is to find its hyperparameter configuration with the best P value, for which we define as the average similarity to the ground truth across the top K documents retrieved from the corpus D for the given query batch Q:

Here, we use lexical precision to measure the similarity between a retrieved document dq,k and the ground truth of context r∗ q for a query q, where W denotes the unique tokens in the corresponding text. The lexical precision is chosen because it is a more efficient signal for finding useful configurations for the generator later on. Importantly, we prefer such over other semantic metrics as it is more strict, biased to prevent false positives, which is more critical for optimizing the generator.

To solve Equation 2c, we use Sobol sampling [Sobol', 1967] in CDS4RAG. This is because, since optimizing retriever only implicitly influences the final generation outcome, we seek to find its diverse yet promising hyperparameter configurations, hence increasing the likelihood to jump out of local optima when optimizing the generator later on. To that end, the Sobol sampling works as a quasi-random low-discrepancy sequence to outperform uniform sampling in high-dimensional cases. Importantly, it provides a distribution-aware exploration and stronger convergence/exploitation than random search. This is crucial for providing the subsequent generation stage with a robust variety of high-quality retrieval foundations. To adopt Sobol sampling to cope with the mixed hyperparameter for retriever, we apply randomized digital scrambling and densitypreserving quantization to the Sobol sequences [Cowen-Rivers et al., 2022], ensuring that the low-discrepancy benefits are preserved across the mixed-integer boundary.

The ϕ∗t , which is the best hyperparameter configuration of the retriever found at cycle t, evaluated under the retrieval quality P, would be fixed, and its retrieved contexts C ∗ t would be applied in the subsequent generator optimization.

Generator Optimization with Fixed Retrieved Contexts.

At cycle t, unlike optimizing the retriever hyperparameters, which focuses on exploration, here we need better exploitation for Equation 2b as the evaluation of the generator is much more expensive. CDS4RAG can be seamlessly paired with any general algorithm, e.g., Bayesian optimization, to optimize the generation quality of RAG using the given queries, e.g., the F1-score. Yet, the retriever hyperparameters are frozen to be ϕ∗t and so do the contexts C ∗ t retrieved thereunder for query batch Q; only the generator's hyperparameters θi are perturbed. The best configuration at cycle t (ϕ∗t , θ∗t ) is stored.

When all budgets are exhausted, the best generator configuration found and its corresponding retrieval configuration across all cycles, (ϕ∗ , θ∗ ), are returned. Otherwise, we repeat from the retriever optimization in the next cycle t + 1. All explored hyperparameter configurations of the generators and their generation quality (θi,Mi) are archived for the possible cross-cycle seeding next.

4.2 Within-Cycle Budget Provisioning

Since the nature of dual-sequential hyperparameter optimization offers fine-grained control for budget provision, we set different mechanisms for the retriever and generator.

Adaptive Termination for Retriever. For the retriever, we design an adaptive termination mechanism based on the expected progress of retrieval quality, considering all cycles so far. This makes sense since optimizing the retriever is cheap, and it often reaches an ideal retrieval plateau early, after which any more budgets spent on it would not be much beneficial.

At each cycle, we compute an ideal threshold Pideal of the obtained retrieval quality over query batch Q:

where (P1,P2, · · · ,Pn) denote the locally achieved values of retrieval quality for all n explored hyperparameter configurations of the retriever at the current cycle. ∆ denotes the globally expected quality considering the current and all previous cycles. α controls the relative contribution, and we set equal weights α = 0.5 by default. ∆ is formulated as:

whereby P95th and Pbest denote the 95th percentile and the best value of all retrieval quality found so far, respectively. t denotes the tth cycle; t t+β is the function that produces a cycleprogress aware weight; we found that β = 2 is appropriate.

As from Figure 4, essentially, P95th is a more relaxed value compared with Pbest, serving as the lower bound of ∆. As the cycle increases, t t+β would become closer to 1, hence ∆ would be closer to P95th. In this way, we can reduce the contribution of ∆ for Pideal at the latter stage of the optimization, since the results of the current cycle would gradually become more important as the optimization converges.

Finally, if a newly explored configuration of the retriever has a retrieval quality greater than or equal to Pideal, then we terminate the retriever optimization at the current cycle.

Fixed Evaluation Count for Generator. Since optimizing the generator aligned with our main objective, we cap its budget via a fixed number of evaluations N, for which we set N = 10 in this work. This is a more intuitive way for one to directly control the trade-off between budget and RAG quality. Clearly, an N that is too large would reduce the contribution from optimizing/exploring the retriever, while a too small N can restrict the exploitation in optimizing the generator. The sensitivity of CDS4RAG to N can be found in Appendix C.3.

4.3 Cross-Cycle Generator Seeding

A naive method would be to treat the dual-sequential optimization between cycles as independent runs, but this ignores the optimization knowledge that has been accumulated so far. Further, we observe that the hyperparameter performance of the generator exhibits significant cross-context transferability even when the frozen hyperparameters of the retriever differ.

To capitalize on that, we reuse/seed certain hyperparameter configurations, found in previous cycles, to the generator optimization at the current cycle for a warm start. The seeds can also initialize the surrogate model in, e.g., BO. The steps are:

Step 1-Archive de-duplication: In previous cycles, it is possible to have multiple identical hyperparameter configurations of the generator with different generation quality due to their different contexts retrieved, thus we perform a deduplication that merges those and averages their Mi values. Thus, all unique hyperparameter configurations in the archive would correspond to a single generation quality (θi,Mi).

Step 2-Probabilistic seeding: To select which hyperparameter configuration θi of the generator from previous cycles to seed, we rank each configuration, ri, according to its (average) generation quality. For all J top 50% configurations, we assign a probability to each based on reciprocal of its rank:

p(θi) indicates the likelihood of θi to be used for seeding. This ensures that better performing θi would have a higher probability to seed, while there is also a chance of choosing slightly worse ones, hence ensuring diversity.

Recall that N is the cap for the evaluation counts to optimize the generator, we set the number of configurations to be seeded as N2 , within which N4 is seeded from the archive of previous cycles, while the remaining N4 are randomly generated configurations. This ensures that the seeds can guide the hyperparameter optimization of the generator without trapping at local optima. Note that if N4 is not an integer, we prefer the random seeds over the archived seeds, e.g., N = 10 means that 5 would be the seeds, in which 2 come from the archive.

Step 3-Contextual re-evaluation: Since the generation quality of generator configurations in the archive might be evaluated with a retriever hyperparameter configuration differs from the ϕ∗t at the current cycle t, all seeds for the generator should be re-evaluated with contexts C ∗ t retrieved under ϕ∗t .

5 Experiment Setup

5.1 Experiment Subjects

Benchmarks and Knowledge Bases. As the common practices for RAG, we use diverse/widely-used benchmarks [Guo et al., 2025; Stefano et al., 2024; Yang et al., 2018]:

• Agriculture and Biography [Guo et al., 2025]: Two domain-specific benchmarks requiring precise entity retrieval. We send all 100 and 180 queries to RAG.
• BioASQ [Krithara et al., 2023]: Biomedical questions/answers with complex terminology. We randomly sample 100 queries per run for sending to RAG.
• HotpotQA [Yang et al., 2018]: A multi-hop reasoning benchmark for high-fidelity retrieval. Again, we randomly sample 100 queries per run for sending to RAG.

For each benchmark, we consolidate the contextual information of source corpora for the queries into a knowledge base, which is then merged into a unified vector database.

RAG System. Like prior work [Fu et al., 2024; Lewis et al., 2020; Orbach et al., 2025; Guo et al., 2025], we focus on a refined backbone common to most RAG systems to ensure findings are generalizable without the confounding effects of auxiliary modules like re-rankers. The joint space Φ×Θ in Table 1 has up to 12 hyperparameters-a much more complex RAG case compared with the 2-7 hyperparameters used in existing work [Fu et al., 2024; Barker et al., 2025]-including discrete architectural choices and continuous options. This leads to a search space of at least 1.3 × 1018 , if discretized.

LLMs. We employ Llama-3.1-8B [Meta, 2024] and Qwen3-8B [Alibaba, 2025] as the generator for RAG due to their robust instruction-following capabilities. The prompt for RAG in evaluation can be found at Appendix B.1.

5.2 Compared Approaches andMetric

Compared Approaches. We compare CDS4RAG against several types of hyperparameter optimization algorithms:

• Baseline: We use Random Search (Random) and Greedy Search (Greedy) [Orbach et al., 2025] as the baselines.
• General algorithms: We also assess the state-of-theart general algorithm for hyperparameter optimization, including classic BO [Snoek et al., 2012], TPE [Bergstra et al., 2011], the advanced HEBO [Cowen-Rivers et al., 2022]. They jointly optimize all RAG hyperparameters.
• RAG-specific algorithms: We compare two state-of-theart algorithms designed for optimizing RAG hyperparameters: RayTune [Liaw et al., 2018] and AutoRAG-HP [Fu et al., 2024]: the former is a tailored TPE for RAG, while the latter relies on hierarchical multi-armed bandit. Again, they do not distinguish the hyperparameters of the retriever and generator in the optimization.

Despite being relevant, we have had to omit some algorithms due to different reasons: (1) Prompt-only algorithms (e.g., MIPROv2 [Opsahl-Ong et al., 2024]) and single-stage approaches (e.g., AdaptiveRAG [Jeong et al., 2024]) are omitted, as it is unfair to compare with them and they are orthogonal to this work. (2) Multi-fidelity algorithms (e.g., BOHB [Falkner et al., 2018] and DEHB [Awad et al., 2021]) are also ruled out, because multi-fidelity is an additional feature that can easily be paired with CDS4RAG under its cyclic dual-sequential optimization. (3) Since this work focuses on single objective optimization, we do not consider multi-objective RAG optimization algorithms such as METIS [Ray et al., 2025].

Evaluation Metric. To evaluate the generation quality of optimized RAG, we focus on the token-level F1-score (kindly see Appendix B.2) based on the ground truth answers of the given queries of a benchmark as they are sent to RAG, which is also the main optimization objective. F1-score is chosen because it is robust to data imbalance and has been widely used for RAG evaluation [Ray et al., 2025; Rajpurkar et al., 2016; Yu et al., 2024]. Since RAG follows the zero-shot learning paradigm without the need for any (re-)training on the given queries when changing hyperparameters, the above can evaluate/test the generalizability of RAG under optimized hyperparameters, resembling the common way of using a validation loss, which is both an optimization objective and evaluation metric, in classic AutoML [Santu et al., 2022]-the same is also applied in AutoRAG-HP [Fu et al., 2024]. Having said that, we have indeed additionally evaluated the optimized configurations on holdout queries from BioASQ and HotpotQA, which have sufficient samples (Appendix C.4).

5.3 Implementation and Setting Details

The RAG is implemented via LangChain 3 , with OpenBox [Jiang et al., 2024] for CDS4RAG and other hyperparameter optimization algorithms, or otherwise we directly use the code provided by their authors. All experiments run on a cluster of 10 machines, each equipped with an Intel CPU of 24 cores and 32 threads, 128GB RAM, and a NVIDIA RTX 5090 GPU. We set the hyperparameters of CDS4RAG as the aforementioned defaults, and those of the other algorithms as what used by their authors. The details can be found in Appendix B.3. To complete the experiment within a reasonable time while ensuring realism, for each algorithm/run, we set a budget of one hour wall-clock time (where most algorithms have already converged reasonably) and repeat 10 independent runs. In total, the entire experiment took ≈ 1, 000 CPU/GPU hours.

6 Evaluation Results

6.1 Benefits of Cyclic Dual-Sequential Optimization

To assess the benefits of cyclic dual-sequential optimization, we pair CDS4RAG with HEBO, BO, and TPE while comparing them against their respective single-stage optimization (jointly optimizing retriever and generator). As shown in Table 2, CDS4RAG yields considerably better results than its counterparts in 21/24 cases. For both LLMs, the improvement of CDS4RAG appears to be more significant on HotpotQA than on others, due to the queries therein being more complex. This makes sense, since if the queries in a benchmark are easy, then the room to be improved is limited. We see that CDS4RAG paired with HEBO has the best result than other variants, since HEBO perform better than BO and TPE in general. On holdout queries, CDS4RAG also lead to significant improvements overall (see Appendix C.4). All those confirm the benefits of distinguishing the hyperparameters of retriever and generator in CDS4RAG and its algorithm-agnostic capability.

3 https://github.com/langchain-ai/langchain

6.2 Comparison with State-of-the-Art Algorithms

We further evaluate CDS4RAG paired with HEBO against four baselines/state-of-the-art algorithms. As shown in Figure 5, remarkably, CDS4RAG achieves the significantly better results on all cases. Compared with RayTune, CDS4RAG shows a clear advantage, reaching 0.759 F1-score on HotpotQA for Qwen-3-8B versus the 0.639 of RayTune. The superiority over RAGspecific algorithms like Autorag-HP is even more pronounced; for example, on BioASQ with Llama-3.1-8B, CDS4RAG attains 0.293 compared with 0.259; on HotpotQA with Qwen-3-8B, CDS4RAG improves Autorag-HP by 1.54×. Most importantly, after the early phase of retriever optimization, the trajectory of CDS4RAG improves much steeper than those of the others with a faster and better convergence/speedup in a matter of a few minutes, showing its superiority even under a limited budget. On holdout queries, the conclusion remains unchanged: over all LLMs/datasets, CDS4RAG (with HEBO) achieves 0.4180 F1-score compared with the 0.3306-0.3853 of the other stateof-the-art algorithms (10%-25% better); see Appendix C.4.

6.3 Ablation Analysis

To confirm the contributions of individual design in CDS4RAG, we replace the within-cycle budget provisioning with fixed, equally dividing budgets to retriever and generator, denoted as w/o pro; we also remove the cross-cycle generator seeding, denoted as w/o seeding. As shown in Table 3, we see that removing either of the above mechanisms is harmful in general.

Notably, interactions exist between the two mechanisms: the budget provision determines the number of possible cycles, which then influences the cross-cycle seeding that impacts the convergence speed of RAG hyperparameter optimization.

6.4 ACase Study

For the case in Figure 6, CDS4RAG completes 6 cycles with ≈ 17% budgets spent on retrieval evaluations only. In less than 10 minutes, it obtains an excellent generation quality already, followed by F1-score increases at a slow, but steady pace: the marginal improvements of the overall lexical precision for the retriever always help to yield better/more promising overall F1-score for the generator-well-aligned with our hypothesis.

7 Conclusion

This paper presents CDS4RAG, a framework that uniquely distinguishes the hyperparameters of the retriever and generator, optimizing full hyperparameters in a sequential and cyclic manner. Such a paradigm is agnostic to the underlying general algorithm for the generator, enabling finer-grained budget provision within each cycle and cross-cycle generator seeding that can expedite the optimization via reusing promising hyperparameter configurations. Experiments on four benchmarks and two LLMs show that CDS4RAG can boost vanilla algorithms in general while outperforming state-of-the-art algorithms in all cases with up to 1, 54× improvement and better speedup. In future work, we seek to extend CDS4RAG with more optional stages of RAG and consider multiple optimization objectives.

Acknowledgments

This work was supported by a NSFC Grant (62372084).