Exploring Internal Emphasis for Robust Noisy RAG

Abstract

Retrieval‑augmented generation (RAG) improves knowledge‑intensive QA tasks by incorporating external evidence, yet retrieval remains imperfect and often returns irrelevant or misleading passages. We refer to this as Noisy RAG, where LLMs suffer from inconsistent relevance signals due to the fact that relevance identification is carried out by only a small subset of internal modules (e.g., retrieval heads) and can easily be confounded by other modules.

Recent work attempts to enhance the denoising ability of the LLM through external emphasis, prompting the LLM to highlight helpful passages as the emphasis before answering. However, these textual identifications themselves depend on the same inconsistent signals and thus become less reliable.

Our key insight is that, because relevance signals reside in only a few modules, emphasis should be applied directly within these modules rather than only at the text level. We therefore propose \textit{selective internal emphasis} to amplify the relevance signals, implemented via a lightweight, plug‑and‑play signal amplifier that operates inside the LLM. The amplifier performs token‑ and channel‑level selective emphasis during a standard RAG fine‑tuning pipeline, with the base LLM frozen.

Across four QA benchmarks and three LLM scales, our method consistently improves accuracy and robustness. Furthermore, generalization and interpretability analyses show that the amplifier captures retrieval‑related patterns rather than only dataset‑specific patterns, enabling more reliable passage-denoising.