Exploring Internal Emphasis for Robust Noisy RAG

1 Introduction

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) in knowledge-intensive question answering by incorporating external evidence. However, retrieved passages frequently contain irrelevant or misleading information (Noisy RAG) despite the continuous improvement of retrievers. In such cases, the LLM's reasoning can be disrupted: the model may temporarily form correct internal estimates yet still produce incorrect answers [Wu et al., 2024; Sun et al., 2024]. These errors, as an instance of hallucination, do not arise from a lack of knowledge for relevance identification, but can be attributed to the inconsistency of relevance signals inside the model [Orgad et al., 2024], i.e., those signals can be confounded or overwritten by complex interactions inside the LLM.

∗Corresponding authors.

Existing studies typically adopt external emphasis to enhance the denoising ability of the LLM [Wei et al., 2025; Wang et al., 2025; Chang et al., 2025], prompting the LLM to first identify helpful passages and then answer based on that identification. Although straightforward, such identifications are themselves susceptible to errors due to the same inconsistency: only a small subset of attention heads (retrieval heads), and only early feed-forward modules are responsible for relevance detection inside the LLM [Geva et al., 2023; Wu et al., 2024; Farahani and Johansson, 2024]. In Noisy RAG, we also observe that only around 10% feature units in the LLM function as passage denoising (details in §6.4). The relevance signals from these sparse modules can be easily confounded by the majority of other modules [Sun et al., 2024], leading to unreliable external emphasis.

Our key insight is that relevance identification is handled by only a small subset of modules within the LLM, and therefore the emphasis should be applied directly at these modules to mitigate the inconsistency rather than only at the text level. Motivated by state-editing techniques such as knowledge editing [Ilharco et al., 2023], we introduce internal emphasis, a selective intervention directly applied to the LLM's internal states.

Concretely, we treat the post-module intermediate states of attention and feed-forward modules as relevance signals. These signals exhibit two structural dimensions: per-token features corresponding to input token representations, and per-channel features corresponding to atomic feature units within a module. We design a lightweight signal amplifier that emphasizes these signals via token-channel gating during standard RAG fine-tuning with the base LLM frozen. Token gating prioritizes question-relevant tokens, steering internal attention toward key evidence, while channel gating amplifies channels responsible for relevance identification. Unlike mechanistic tools such as activation patching or probing [Zhang and Nanda, 2024; Belinkov, 2022], this approach requires no additional data or controlled experiments. The emphasis magnitude is bounded by variance-calibrated initialization to avoid overwhelming original representations. Empirically, our method achieves strong performance across datasets and model scales, while remaining robust and interpretable by consistently identifying retrieval-related channels and key evidence tokens. The code is available on https: //github.com/xiaolai2333/InternalEmphasisForNoisyRAG.

Contributions.

• To our knowledge, we are the first to study Noisy RAG via selective internal emphasis due to the observed inconsistency of relevance signals inside LLMs under Noisy RAG.
• We propose a lightweight signal amplifier featuring two-axis gating with variance-calibrated initialization, which enhances key signals without overwhelming them, and is trained within an RAG fine-tuning pipeline while keeping the base LLM frozen.
• Across benchmarks and model scales, our approach consistently surpasses strong baselines and maintains superior performance in cross-dataset and low-data volume settings. Interpretability analyses further demonstrate that the amplifier enables LLMs to more reliably route and filter evidence in the presence of noisy passages.

2 RelatedWork

2.1 Retrieval Augmented Generation

Retrieval-Augmented Generation (RAG) has been commonly used to ground LLMs in external knowledge [Lewis et al., 2020; Guu et al., 2020; Karpukhin et al., 2020]. Early work has focused on improving retrievers for recall and precision gains [Xiao et al., 2025; Li et al., 2024; Sohn et al., 2024; Yang et al., 2025b; Zhang et al., 2024]. However, recent findings indicate that retrievers remain imperfect and cannot completely filter out irrelevant passages, leading to the Noisy RAG setting. To address this, recent methods introduce external emphasis strategies that reduce the influence of noisy passages during generation. For example, [Wei et al., 2025; Wang et al., 2025] leverage LLM-generated rationales or judgments to perform passage selection before answer generation. MAIN-RAG [Chang et al., 2025] aggregates votes from multiple LLMs to filter retrieved documents without additional training. FilCo [Wang et al., 2023] learns a supervised passage filter, and [Zhu et al., 2024] formulates passage filtering under an information-bottleneck framework.

Despite their differences, these approaches consistently suggest that passage relevance signals are already encoded implicitly in LLMs. Yet, because these signals originate from a small subset of relevance-identification modules, they can be easily confounded by unrelated modules when surfaced in text form. This observation motivates our approach: instead of relying solely on external methods, we aim to extract and amplify these internal signals directly at the modules responsible for relevance identification.

2.2 Locating key retrieval modules

It is important to locate key modules for relevance identification before conducting the emphasis. Prior studies often employ mechanistic-interpretability techniques to locate LLM modules with specific functionality. Activation patching methods score candidate modules by measuring functional changes under patched activations [Zhang and Nanda, 2024; Heimersheim and Nanda, 2024]. ROME, as a knowledge-editing approach, finds where factual associations lie in mid-layer FFNs through causal tracing, then uses low-rank updates to change these associations [Meng et al., 2022]. Patchscopes probe hidden representations by inserting them into diagnostic prompts to elicit natural-language interpretations [Ghandeharioun et al., 2024]. For long-context LLMs, other studies analyze retrieval-aware attention or jointly train retrieval-specific components to suppress irrelevant content during decoding [Qiu et al., 2025]. In the RAG setting, ReDeEP [Sun et al., 2024] decomposes retrieval failures into those stemming from "knowledge FFNs" versus "copying heads" informed by retrieved evidence.

Although these techniques are novel and insightful, they typically require additional datasets, controlled interventions, or specialized diagnostic pipelines. Since experiments of these methods are usually not designed for Noisy RAG, these methods can perform poorly in our setting. Furthermore, iterative knowledge editing has been shown to cause forgetting [Gupta et al., 2024], while probing methods can fail to transfer across datasets [Belinkov, 2022].

Motivated by these limitations, rather than explicitly locating or editing modules via external experiments, we directly learn the relevant feature units inside the LLM within a single RAG fine-tuning pipeline, while keeping the LLM frozen. This allows us to obtain a low-cost, data-driven, and accurate localization of module functionality.

3 Preliminaries

3.1 LLM Internal States

Large decoder-only LLMs consist of a stack of Transformer blocks, each with two primary submodules: multi-head selfattention (MHSA) and a feed-forward network (FFN). Each module transforms representations of tokens layer by layer with different functionalities. Recent work attempts to interpret the functionality of different modules in LLMs rather than treating them as black boxes [Ferrando et al., 2024], called mechanistic analysis.

Attention module. The MHSAmodule uses queries, keys, and values to attend to preceding tokens; each attention head operates in parallel and may specialize in different types of information. Recent work in RAG settings finds that certain retrieval heads consistently attend to and copy relevant external content. Ablating these heads degrades retrieval quality and increases hallucinations, underscoring their causal role in relevance identification [Wu et al., 2024; Sun et al., 2024].

FFN module. FFNs implement key-value memories that store latent patterns capable of shaping the model's output distribution, offering insight into how factual knowledge is written into and read out of hidden states during generation [Geva et al., 2021], i.e., knowledge storage. Besides, FFN blocks also contribute to determining contextual relevance and encoding factual associations. Causal-tracing and activation-patching studies identify strong early-layer FFN effects on relevance decisions, suggesting that both attention and FFN modules encode signals essential for filtering noisy passages [Zhang and Nanda, 2024; Geva et al., 2023]. These FFN-stored signals may contribute directly to relevance decisions [Farahani and Johansson, 2024].

3.2 ProblemFormulation

Following prior work that uses the Noisy RAG framework [Wei et al., 2025; Wang et al., 2025; Chang et al., 2025], our objective is to improve LLMs' question-answering ability given noisy passages. Formally, let Q = {q1, q2, . . . , qN} be a collection of N queries. Given a query qi ∈ Q, a retriever produces K candidate passages:

where some of the ci,j may be noisy. We then condition an LLM fθ on (qi, Ci) to produce an output response ˆ yi:

The response should contain the answer ti to the query qi, despite irrelevant/noisy passages in Ci.

Terminology We use token to refer to a position of the input sequence (length T ). We use channel, also known as a feature unit, to refer to a single unit of the hidden vector (width H). Therefore, each post-module hidden state hl ∈ RT×H can be represented as a matrix with rows indexed by tokens and columns indexed by channels. In attention blocks, a channel corresponds to one dimension of the hidden state after the output projection. In FFN blocks, a channel denotes one neuron's activation after the down projection. The channel gating serves to identify and select retrieval-related modules, similar in spirit to mechanistic analysis, but it accomplishes this automatically within a single RAG fine-tuning process.

4 Method

As discussed, relevance identification signals can be confounded by other non-retrieval modules when such signals are surfaced as text and used as external emphasis [Wei et al., 2025; Wang et al., 2025; Chang et al., 2025]. We therefore move the emphasis inside the LLM by operating directly on post-module hidden states, treating them as the relevance signals. We then apply selective emphasis via token-channel gating and write back a small injection. The injection magnitude is constrained by a bottlenecked linear-nonlinear mapping, preventing the internal emphasis from distorting the original distribution. Finally, we analyze why token-channel gating leads to robust learning of passage relevance patterns in the gradient view. The lightweight signal amplifier enables low-cost training. An overview is shown in Figure 2.

4.1 Internal Emphasis

Auditable intervention To realize the internal emphasis on relevance signals, we first need to consider where the signals appear. We define the signals as the post-module, pre-residual output (e.g., attention or FFN output before being added to the residual stream). This isolates a module's contribution prior to mixing with other modules. Then, the signals are emphasized and written back at the same location, making the emphasis auditable and interpretable: we can trace how emphasis on each module influences the QA performance, without extra mechanistic experiments. Formally, hl,m ∈ RT×H are the post-module signals from a module m ∈ {attn, ffn} at layer l. The emphasis is computed from hl,m and forms as a small injection rl,m ∈ RT×H . Then we perform internal emphasis with the adding rl,m to the original signals:

The process can be viewed as masked, low-rank residual modulation of hl,m, which directly alters the original states, distinguishing it from structure-based methods, such as LoRA-like methods [Hu et al., 2022] and adapters [Houlsby et al., 2019]. Having localized the post-module signals hl,m, we next specify where and how to emphasize them.

Token-channel gating As an intervention, an effective emphasis should enhance rather than overwhelm the model's original signals. Intuitively, the emphasis should be placed at the most appropriate positions. We consider two dimensions of hl,m, i.e., T for tokens and H for channels. In Noisy RAG, question-relevant tokens in passages should be prioritized; due to the sparsity of retrieval computations, only a subset of channels, i.e., feature units, should be amplified. Naturally, we introduce rank-1 token-channel gating. The token gate stk ∈ [0, 1]T is designed to filter out tokens that are irrelevant to the question and shared across modules, ensuring consistent token emphasis. The channel gate sch l,m ∈ [0, 1]Hf is specific to module m and employed to select specific channels in m for retrieval (the reason for the dimension Hf is introduced next). Formally, the rank-1 mask is defined as: Ml,m = stk (sch l,m)⊤ ∈ [0, 1]T×Hf . To ensure low computational cost and easy training, we compute stk via a lightweight cross-attention, with mean question representation ¯hl,m,q as the query and passage token representations hl,m,p as the key and value; we leverage the linear projection with sigmoid as sch l,m. The emphasis is expected to concentrate on the most related tokens and channels for retrieval with the mask, which further differs from input-agnostic and globalupdating methods inside the LLM [Mahabadi et al., 2021; Ben Zaken et al., 2022; Liu et al., 2022].

Magnitude constraint While rank-1 gating localizes where to emphasize, we still need to control the injection magnitude to prevent the overwhelming of original signals. We therefore introduce a bottleneck with variance-calibrated initialization to bound the injection magnitude. We first project hl,m ∈ RT×H into a carrier space with dimension Hf < H, inspired by [Du et al., 2024], which also enables feature refinement and reduces computational cost. Denoting the projected signals as zl,m ∈ RT×Hf , we apply the selective emphasis in the carrier space:

where ⊙ is the element-wise product. Then, we obtain the injection with the bottleneck network:

where ϕ is GELU activation, W dn ∈ RHf×d and W up ∈ Rd×H are projections with a bottleneck size d ≪ H. Intuitively, a narrow bottleneck d limits the effective degrees of freedom of the update, and the spread from d to H by a linear projection keeps a small per-channel injection.

We further propose to bound rl,m via variance calibration for bottleneck initialization. Denoting Lϕ ≈ 1 as the Lipschitz constant of ϕ, the injection admits the operator-Frobenius bound 1 :

We initialize the bottleneck with variance-calibrated Gassian to keep ∥W up∥2∥W dn∥2 small:

By standard spectral concentration for Gaussian matrices, this yields E∥W dn∥2 ≈ αdn and E∥W up∥2 ≈ αup, thereby, the expected amplification of the bottleneck satisfies:

where αdnαup := Cinit is a dimension-agnostic constant. In addition, ∥Ul,m∥F is bounded by the two-axis gating that has values between [0, 1]. Together, we have ∥rl,m∥F with safe and trainable magnitude, which is more robust than setting upper bounds manually.

4.2 First-Order Gradient Alignment

Since the amplifier is trained within the Noisy RAG pipeline, we leverage the standard cross-entropy as the training loss:

With the base LLM frozen, L backpropagates to each emphasis point hl,m. The loss gradient Gl,m = ∇hl,m L ∈ RT×H quantifies emphasizing on which tokens and channels at module m are most sensitive to error. Consider a small injection rl,m, a first-order Taylor expansion gives L(hl,m + rl,m) ≈ L(hl,m) +Gl,m · rl,m. Then, we have:

where ⟨A,B⟩ = ∑t,j At,jBt,j is the Frobenius inner product. We denote G′l,m := W up⊤(Gl,m ⊙ zl,m)W dn as the effective gradient, which integrates the actual loss sensitivity for the emphasis on all tokens and channels. Since Ml,m = stk (sch l,m)⊤, the row-wise norms ∥G′l,m(t, :)∥2 drive token-gate updates, and the column-wise norms ∥G′l,m(: , j)∥2 drive channel-gate updates:

Therefore, the emphasis is aligned with the directions that most reduce cross-entropy, which means the amplifier learns passage-relevance patterns by tuning the gates. That explains why our method has better cross-dataset and low-data-volume robustness (see §6.2)

1 We bound the two linear projections with spectral norms to measure worst-case amplification.

5 Experiment Settings

We focus on denoising under standard open-domain QA and evaluate on four relevant benchmarks: PopQA [Mallen et al., 2023], TriviaQA [Joshi et al., 2017], Natural Questions (NQ) [Kwiatkowski et al., 2019], and ASQA [Stelmakh et al., 2022]. We adopt the off-the-shelf datasets from [Wei et al., 2025]. Each example comprises a question, a groundtruth answer, and five top-ranked Wikipedia passages retrieved with retrievers such as GTR [Ni et al., 2022] and DPR [Karpukhin et al., 2020]). Dataset statistics are summarized in Table 1. Following prior works [Asai et al., 2024; Wei et al., 2025], we report the official correctness metric (str-em) for ASQA and accuracy (acc), i.e., whether the gold answer appears in the generation, for other datasets.

We consider three LLMs of different scales: Llama3.2-1B-Instruct [Grattafiori et al., 2024], Qwen3-4BInstruct-2507 [Yang et al., 2025a], and Llama-3-8B-Instruct [Grattafiori et al., 2024], denoted Llama-1B, Qwen-4B, and Llama-8B respectively. Full-parameter FT, LoRA [Hu et al., 2022], IA3 [Liu et al., 2022], FILCO [Wang et al., 2023], Self-RAG [Asai et al., 2024], RetRobust [Yoran et al., 2023], and InstructRAG [Wei et al., 2025] are baselines. FILCO, Self-RAG, and InstructRAG are implemented via LoRA at q, k, v, o projections as a similar PEFT regime for comparison. IA3 is operated at k, v, and down projections in all attention and FFN modules. We reproduce baselines using official code releases and report their best performance after hyperparameter search. We generate the answer using Greedy Decode for all methods. We set Hf=1024/1280/1536 and d=256/384/512 for Llama-1B, Qwen-4B, and Llama-8B, and the trainable parameter count is around 2.5% of the base LLM. We set αup = αdn = 1. Following [Wei et al., 2025], we prompt LLMs to generate answers with the form: <Retrieved Passages>. Based on the provided information and your own knowledge, answer the question: <Question>. Ours+Rationale is used in all secondary experiments, where rationale is supervision text indicating which retrieved passages are relevant [Wei et al., 2025].

6 Experimental Results

6.1 Main Results

The main results are summarized in Table 2. Across three LLM backbones and four datasets, our method achieves stable state-of-the-art performance. Ours+Rationale attains the best score in 9 out of 12 settings, while Ours♠ achieves the top results in the remaining three. These outcomes suggest that our approach is both accurate and robust.

The signal amplifier trained only on ASQA (Ours♠) generalizes well to other datasets. On PopQA, TriviaQA, and NQ, the ASQA-trained amplifier consistently outperforms perdataset training, with average gains of 3.3, 5.9, and 3.4 points across the three LLMs. We attribute this behavior to the supervision signal: ASQA provides long-form answers whose semantics align with passage relevance, enabling the amplifier to learn general relevance patterns rather than datasetspecific cues. In contrast, datasets with short labels offer weaker guidance.

To examine this explanation, we incorporate explicit rationales during training, forming Ours+Rationale. The results show that adding rationales usually brings further improvements over Ours♠ and gives the strongest overall performance. This supports our claim that the signal amplifier benefits most from supervision signals that highlight relevant passages. The improvement on ASQA is smaller, as its longform labels already provide similar relevance supervision.

6.2 Robustness Results

Intuitively, learning general relevance patterns should transfer better across Noisy RAG datasets. Figure 3 shows the cross-dataset results. Our method displays strong and steady robustness across datasets. Although out-of-domain scores are naturally lower than each dataset's in-domain baseline, the off-diagonal average reaches 57.4%, which is 10.2 points higher than InstructRAG-LoRA. This improvement appears consistently across all out-of-domain settings, suggesting that our approach is less affected by dataset-specific quirks. Learning relevance patterns also requires less training data than learning dataset-specific templates. To check this, we measure performance under different training data sizes, as shown in Figure 4. Across all four datasets, Ours+Rationale keeps strong performance even with only 30% of the data and increases slowly as more data is added. In comparison, InstructRAG-LoRA shows a steeper growth curve but starts from a much lower point. At 30% data, our method already leads by a clear margin, and it maintains this advantage even when using the full dataset. The two experiments show that our method mainly learns passage relevance instead of dataset-specific patterns, leading to strong performance, and data-efficiency under Noisy RAG.

6.3 Ablation Study

Ablation results in Table 3 show that using both components yields the strongest performance, suggesting that channel gating and token-level attention complement each other. This also indicates that interventions on LLM internal states must be applied in a selective manner; otherwise, they may distort the model's original activation distribution. We also find that the effect of channel gating becomes more pronounced as model size increases. For larger LLMs such as Llama-8B, removing channel gating leads to a noticeably larger performance drop compared with smaller models. This aligns with the view that larger models develop more specialized internal functions and with prior findings that conditional gating mechanisms help improve capacity and generalization at scale [Shazeer et al., 2017; Fedus et al., 2022].

6.4 Interpretablity Analysis

Module Consistency To check whether our method truly identifies the modules responsible for filtering relevant passages, we run a cross-dataset consistency experiment with a simple idea: if these modules are tied to an intrinsic relevance-identification function rather than dataset-specific cues, then the Top-k important channels should largely overlap across different Noisy RAG datasets. For each module in Llama-1B, we compute channel importance on each dataset, measure the Jaccard similarity between the Top-k channel sets, and then average the results across modules (Figure 5). The left panel shows that the dataset-wise Jaccard similarity at Top-16 ranges from 0.10 to 0.14 across all dataset pairs. This is 25×-35× higher than the random baseline (around 0.0039 for H=2048, k=16, with an expected random overlap of ≈ 0.125 channels). The right panel shows the per-module viewpoint: the most stable modules reach a Jaccard of 0.158, which corresponds to about four overlapping channels out of sixteen on average 2 .

Channel Intervention We then intervene in the channels and test the effectiveness of channel selection. Given the trained amplifier, we record its selection of channels of 1k samples in the training set, conduct different interventions, and test it on the testing set. The results are presented in Table 4. From the results, we can find that disabling top channels does reduce the performance, even only the top 10%, indicating that our method enables accurate channel selection. Additionally, if we Switch Top-Bottom 10% channels, the performance drops sharply, which further indicates the importance of selective emphasis on channel dimension. Notably, the difference between disabling 10% and 50% channels is not very large, illustrating the sparsity of effective channels for retrieval. This finding aligns with the conclusion that LLMs' retrieval modules are sparse.

Comparison with CMA To further assess whether our channel-selection method is reliable, we compare it against a standard mechanistic-interpretability approach: activation patching with causal mediation analysis (CMA). In brief, CMA works by feeding both a normal input and a counterfactual input to the model, collecting the intermediate representations for the counterfactual, and then replacing the corresponding states in the normal run at a chosen layer. The resulting "corrupted" output is then compared with the original output to estimate the causal influence of that layer. In our setup, we prompt the model to select helpful passages, run CMA to compare the selections under normal and corrupted conditions, and derive the contribution of each attention and feed-forward module. We only remove the sigmoid activation in our amplifier and get rid of explicit channel gating for a fair comparison (the parameter volume in the amplifier remains the same). Figure 6 shows the results. They confirm that our channel-selection method identifies the important components accurately, and they also highlight a limitation of CMA: layer-level analysis tends to be coarse-grained and cannot pinpoint the finer functional structure that channel selection can capture.

2 For two Top-k sets of equal size, Jaccard = o/(2k - o), so o≈32 · J/(1 + J); with k=16 and J = 0.158, this gives o≈4.3.

Token-level interpretability We examine whether the relevance scores produced by the token-level attention actually highlight tokens that serve as evidence for the correct answer (Figure 7). In the example cases, the signal amplifier consistently points to key tokens-often overlapping directly with the answer spans. Although the highest-scoring token is not always what a human would consider semantically important (e.g., punctuation or articles may occasionally receive high scores), the overall score distribution still concentrates around the answer-bearing region. This trend aligns with the bar plot: approximately 68-76% of the top-scoring tokens fall inside the target passages across different models. These results suggest that placing more emphasis on question-relevant tokens helps the LLM locate and use the correct evidence under Noisy RAG settings.

7 Conclusion

Noisy retrieved passages can lead to retrieval-signal inconsistency inside the LLM. Unlike existing methods that rely on external emphasis, we instead emphasize these inconsistent signals directly where they occur, a strategy we call internal emphasis. We introduce a lightweight signal amplifier, based on token-channel gating and variance-calibrated initialization, which selectively strengthens relevance-related internal signals. Comparison results denote the robust performance of our method. Interpretability analyses further confirm that our approach consistently identifies meaningful retrieval patterns and amplifies internal signals aligned with question-relevant content. Future work will explore richer internal representations and extensions to more challenging settings, such as explicit multi-hop QA and noisier retrieval regimes.

Acknowledgements

The Science and Technology Development Fund, Macau SAR (0126/2024/RIA2, 001/2024/SKL, 0002/2025/EQP), CG-2026, GDST (2020B1212030003, 2023A0505030013), Nansha District (2023ZD001), MYRG-GRG2023-00186FST-UMDF, MYRG-GRG2025-00234-FST.