Beyond Hard Writes and Rigid Preservation: Soft Recursive Least-Squares for Lifelong LLM Editing

1 Introduction

Despite the large amount of knowledge they store within their parameters, large language models (LLMs) [Yang and others, 2024; OpenAI, 2023; DeepSeek-AI and others, 2025] inevitably contain outdated, incomplete, or incorrect knowledge [De Cao et al., 2021; Mitchell et al., 2022] when statically deployed without re-training or access to external knowledge bases. Due to the high computational cost of retraining from scratch, many applications require updating models through edits to a subset of parameters, with the goal of integrating new facts or rules while preserving general model behavior [Meng et al., 2022]. While early model editors largely focused on single or small-batch updates, practical deployments are inherently sequential: edits arrive continuously, and the editor must remain reliable after each edit [Hartvigsen et al., 2023; Gupta et al., 2024].

The many-edits regime presents a dilemma. To remain useful over long streams, an editor must both memorize incoming information and preserve knowledge acquired from earlier edits. In practice, failures often appear as two coupled forms of forgetting:

1. Retroactive Edit Forgetting: future edits can overwrite edits applied in the past.
2. General-Ability Degradation: edits can deteriorate out-ofscope reasoning and language understanding.

Existing sequential editors typically fail for complementary reasons. Locate-then-edit approaches [Meng et al., 2022; Meng et al., 2023] perform hard writes, forcing the model to learn new associations with limited control over previously stored knowledge. As the number of edits accumulates, these updates may interfere with one another, leading to instability and retroactive forgetting of earlier facts. Conversely, null-space editors [Fang et al., 2025; Sun et al., 2025; Lyu et al., 2025] project updates into a feasible subspace conditioned on an anchor mapping, thereby preserving the associations explicitly constrained by that mapping. However, as edits are repeatedly applied, maintaining complete preservation would require a growing set of constraints, while loosening these constraints may still allow degradation in general abilities or retention of previous facts. As a consequence, neither paradigm is sufficient for long sequential editing.

We focus on parameter-editing methods. This distinction is mechanism-based rather than dataset-based. [Wang et al., 2025] Alternatively, sequential parameter editing can be viewed as online regularized least squares on a layer-wise key-value surrogate [Sayed, 2003]. Each edit contributes a quadratic fitting term, while preservation is controlled by two explicit deviation terms: one penalizing deviation from the initial model, and another penalizing deviation from a designated anchor mapping. This gives a soft-constraint formulation in which learning new edits and preserving previous knowledge are optimized within a single objective.

From this view, we propose RLSEdit, a recursive leastsquares editor for long sequential editing. RLSEdit formulates editing as a quadratic objective and derives an efficient online recursion via the Woodbury identity [Sherman and Morrison, 1950; Woodbury, 1950; Hager, 1989]. Its per-edit cost is independent of the number of previous edits and instead scales with the current edit rank/size. We further provide theoretical deviation bounds and an asymptotic characterization of the edit-preservation trade-off under long edit streams.

To summarize, our main contributions are:

• We formulate lifelong editing as online regularized least squares with explicit parameter-deviation and anchordeviation controls, providing a soft-constraint alternative between hard writing and hard preservation.
• We derive a Woodbury-based online update whose per-edit cost is independent of the number of past edits.
• We provide deviation bounds and an asymptotic characterization of the adherence-preservation trade-off under long edit streams.
• We evaluate RLSEdit on Llama-3 [Dubey and others, 2024] and Qwen2.5 [Yang and others, 2024] after 10K edits, showing stronger edit success, improved early-edit retention, and better preservation of general capabilities on held-out reasoning and code benchmarks.

2 RelatedWork

We review model editing methods through a soft versus hard constraint lens. We explain how different editing methods balance between making new edits and preserving previous edits and knowledge, and why long-sequential editing is challenging. In particular, we consider layer-wise input-output pairs, where we edit a single linear map in layer ℓ (e.g., an attention projection). Given an input prompt, we run a forward pass and collect a set of module-level input-output feature pairs (k, v) at selected token positions, where k ∈ Rdk is the input activation to the edited map and v ∈ Rdv is the corresponding output activation. For the t-th edit, we stack ut such pairs to formKt ∈ Rut×dk and Vt ∈ Rut×dv .

The Writing and Preservation Trade-off. The goal of editing is to perform targeted updates to the model parameters to learn new key-value associations. With soft or no direct constraint on how this changes the parameters, this can be expressed as a constrained LS problem:

Here (K, V ) denotes the current key-value associations contained within the model parameters, and (K⋆ ,V ⋆ ) denotes the edit constraints defined by the new set of edits.

Alternatively, one can attempt to preserve greater amounts of existing knowledge by restricting updates to a feasible subspace so that performance on a designated preservation set (e.g., an anchor/background mapping) remains unchanged. This can be expressed as

This leads to a soft fit to the new edits under a hard preservation constraint. In ALPHAEDIT [Fang et al., 2025], Kpres,t is typically fixed to a chosen anchor/background set, preserving only what is explicitly constrained. LANGEDIT [Sun et al., 2025] retains the same principle but updates multilingual preservation statistics online, so that Kpres,t and the induced projector evolve over time.

Moving to Longer Edit Streams. When only a single batch of edits needs to be applied, existing editing methods have shown strong performance. However, such settings are not fully representative of real-world LLM use cases. Models often exist in environments where they are deployed for long periods of time and where the number of required edits can grow continuously. In such longer edit streams, existing methods can suffer for several reasons: hard satisfaction of each batch can induce growing interference between incoming edits and prior knowledge, leading to retroactive forgetting and degradation in out-of-scope performance, while attempting to preserve too much prior knowledge can lead to insufficient learning of new associations.

Soft Updates with Preservation. MEMIT [Meng et al., 2023] uses a soft LS objective over a batch of past and new associations, but is not formulated as a long-stream sequential objective. To address the long-edit stream setting, we enforce soft adherence and preservation within a quadratic objective that encompasses both hard regimes as limiting cases.

Relation to Continual Learning. RLSEdit is also related to continual learning methods such as EWC [Kirkpatrick et al., 2017] and online Laplace [Ritter et al., 2018], which use quadratic penalties to restrict parameter drift from previously learned solutions. In our objective, the termλ2 ∥WW0∥2F plays a similar regularizing role. However, our setting is different: RLSEdit targets long sequential LLM editing through layer-wise key-value constraints, combines the drift penalty with an anchor-mapping term, and admits an exact Woodbury-based recursive update whose per-edit cost is independent of the edit history length.

Beyond Direct Parameter Writes. Several recent lines differ from direct streaming parameter updates mainly in editing mechanism. ANYEDIT [Jiang et al., 2025] broadens the scope of editable knowledge beyond simple factual statements, while UNKE [Deng et al., 2025] targets unstructured knowledge. For lifelong settings, WISE [Wang et al., 2024] separates edited knowledge from pre-trained knowledge via dual memories and routing, and RECIPE [Chen et al., 2024] externalizes updates as retrieval-augmented continuous prompts with gating. These approaches can be evaluated on related editing tasks, but they intervene through additional components such as memory, retrieval, or prompting. They are therefore complementary to our focus on an efficient, single-objective streaming parameter editor. Model editing is also related to continual learning, especially quadraticregularization methods such as EWC and online Laplace approximations [Kirkpatrick et al., 2017; Ritter et al., 2018]. However, RLSEdit is designed for post-training LLM editing and couples cumulative key-value least squares, an anchormapping penalty, and a Woodbury recursion with historyindependent per-edit cost.

3 Methodology

We present our editing framework in three parts. We first introduce a recursive least-squares (RLS) formulation that accumulates all editing residuals in a single quadratic objective, with penalties on both deviation from the initial model parameters and deviation from an anchor mapping in Section 3.1. We then analyze the computational complexity of the resulting updates and compare them with existing sequential editors under long edit streams in Section 3.2. Finally, we contrast soft and hard constraint designs, showing how existing editors can be viewed as limiting cases of our formulation in Section 3.3.

3.1 A recursive least squares editor

Setup We consider editing a single linear map W ∈ Rdk×dv , such as a projection matrix in a transformer layer. Each edit provides a set of key-value constraints (Kt,Vt), where Kt ∈ Rut×dk and Vt ∈ Rut×dv , with ut being the number of contexts collected for the t-th edit. Layer-wise edit adherence is measured by the residual ∥KtW -Vt∥F . The goal of our method is two-fold. First, it should incorporate all edits up to time t. Second, it should control deviation from the initial weights W0 and from a set of anchor pairs (K0,V0). We therefore introduce two regularization terms for these two purposes.

Regularized Least-Squares Equation

At time t, our objective is to obtain the optimal weight W ∗ t that minimizes:

where λ and µ are hyperparameters. The first term fits all edit pairs observed so far. The second term penalizes parameter drift from the initial weight W0, and the third term encourages the edited weight to preserve the anchor mapping K0W ≈ V0.

To rewrite Equation 3 as a standard matrix least-squares problem, define

Then Equation 3 becomes

To make the closed-form solution explicit, consider one output column at a time. For a column w of W and the corresponding column bt of Bt, Equation 5 reduces to

Taking the derivative and setting it to zero gives the normal equation

Stacking the solutions for all output columns gives the matrix normal equation

Let

When λ > 0, St ≻ 0, so the minimizer is unique and is given by

This solution jointly fits all edit pairs up to time t, while the two regularizers keep the edited weights close to W0 and preserve the anchor mapping K0W ≈ V0.

Efficient Recursion via Normal Equations

Direct computation of Equation 8 requires inverting St, which is expensive when repeated over a long edit stream. We therefore derive an efficient recursive update. From Equation 8, the minimizer W ∗ t satisfies

Let Ct := S-1 t . Using Equation 6, we have

Algorithm 1 RLS-Woodbury Editing

Require: Initial weightW0; anchor pair (K0,V0); penalties (λ, µ); edit stream {(Kt, Vt)}Tt=1 .

Ensure: Edited weight W ∗ T .

• 1: S0 ← λ2Idk +µ2K⊤ 0 K0

2: T0 ← λ2W0 +µ2K⊤

• 4: C0 ←S-1 0
• 3: S0 =R⊤ 0 R0
• 5: W ∗ 0 ←C0T0
• 6: for t = 1, 2, . . . , T do Covariance update
• 7: Ft ←Ct-1K⊤ t
• 8: Ht ← Iut +KtFt
• 9: Ht =R⊤ t Rt
• 11: Ct ←Ct-1 -YtY ⊤ t Weight update
• 10: Yt ←FtR-1 t
• 12: Et ←Vt -KtW ∗ t-1
• 13: Gt ←FtH-1 t
• 14: W ∗ t ←W ∗ t-1 + GtEt
• 15: end for
• 16: returnW ∗ T

0 V0

▷ Cholesky factorization

▷ Computed via triangular solves

▷ Cholesky factorization

▷ Triangular solve

▷ Gain matrix

Next, define

By the Sherman-Morrison-Woodbury identity,

In implementation, we factorize Ht = R⊤ t Rt by Cholesky and use triangular solves, avoiding explicit matrix inversion.

From Equation 7, we also have

Combining this with the covariance recursion gives the weight update

Equivalently, since CtK⊤ t = FtH-1 t , the update only requires solving the small ut × ut system in Ht. Letting Et := Vt -KtW ∗ t-1 , each edit requires only updating Ct via Equation 11 and then updating W ∗ t via Equation 12.

3.2 Complexity analysis

We report the per-edit cost at step t. MultiplyingM ∈ Rm×n and N ∈ Rn×p costs O(mnp), and solving a dense n × n linear system costs O(n3 ).

RLS-Woodbury Updates

RLSEdit maintains Ct = S-1 t ∈ Rdk×dk and updates it via Woodbury using

The covariance-state update is dominated by forming these products, solving the resulting ut × ut system, and applying the low-rank correction:

For the weight update, we reuse the same ut × ut solve to apply the gain Gt = CtK⊤ t = FtH-1 t ∈ Rdk×ut and update Wt using the residual Et. This step is dominated by the key-value multiplication against dv outputs:

Overall, the per-edit runtime is therefore

which simplifies to O(d2k ut + dkdvut ) when ut ≪ dk, dv.

Comparison to other sequential editors

For a fair long-sequential comparison, we focus on existing sequential editors. ALPHAEDIT introduces hard preservation by projecting the weight update onto the null space of a fixed preserved-knowledge set. In sequential editing, it also regularizes against disrupting previously updated knowledge represented by accumulated key-value pairs. A direct sequential implementation therefore involves history-dependent dense matrices over the feature dimension and repeated projection or factorization steps. Let mt-1 denote the number of previously updated pairs and let ut denote the number of pairs in the current edit. The dominant cost can include a dense dk × dk factorization, together with Gram matrix construction over the accumulated pairs:

Even with more careful incremental implementations, the cost remains tied to the accumulated preservation set or the dimension of the remaining feasible subspace. In contrast, RLSEdit avoids an O(d3k ) factorization during the edit stream by maintaining Ct = (A⊤t At)-1 and using a Woodbury recursion. Each edit only solves a ut × ut system, where typically ut ≪ dk, making the update more efficient in long-edit settings, as shown in 3.

3.3 Hard versus Soft Constraints

RLSEdit can be viewed as a soft-constraint alternative to two common hard regimes. Locate-then-edit methods such as ROME enforce the new association as a hard write while softly penalizing background deviation. Null-space methods such as ALPHAEDIT instead enforce hard preservation by restricting updates to a feasible subspace. In contrast, RLSEdit keeps both edit adherence and preservation soft through the cumulative objective

As µ → ∞, the anchor mapping becomes a hard constraint. Similarly, assigning infinite weight to selected past fitting terms recovers hard preservation through the standard penalty method. Thus, RLSEdit interpolates between hard-write and hard-preserve behavior while remaining in a soft-soft regime for finite (λ, µ).

4 Theoretical Analysis

We provide deviation bounds in terms of (λ, µ): λ controls global parameter deviation from W0, and µ controls deviation of the anchor mapping K0W.

Theorem4.1 (Global deviation bounds). LetW ∗ t be theminimizer of Jt(W) and define Rt := Vt - KtW ∗ t-1 . Let σmin(K) denote the smallest singular value of K.

• (i) (Parameter deviation) If λ > 0, then for any T ≥ 1,

• (ii) (Anchor-map deviation) If µ > 0, then for any T ≥ 1,

Theorem 4.1 shows that λ and µ control different types of deviation. A larger λ keeps the solution closer to the original weights W0, while a larger µ better preserves the anchor mapping K0W ≈ V0. The residual Rt measures how well the current edit is satisfied before the t-th update. Thus, increasing λ or µ makes the update more conservative, but may also increase the edit residual when the new edit conflicts with the preservation constraints.

4.1 Asymptotic Scaling

To connect (λ, µ) to the many-edits regime, we view Equation 3 as a ridge-type estimator for a layer-wise linear mapping. We use the statistical model

where Ei captures approximation error from other layers, context variability, and mismatch in the linearized output.

Sequential edits do not need to be i.i.d. We only assume long-run stability of the empirical second moments, namely that there exist matrices Σk and Σkv such that

s.t. Σk ≻ 0 on the relevant subspace.

Allow λ = λt and µ = µt to depend on t and define

For asymptotic analysis, it is convenient to work with the normalized objective ˜ Jt(W) := Jt(W)/t, which has the same minimizer as Jt for each fixed t:

Proposition 4.2 (Asymptotic behavior of the RLS editor). Assume Equation 13, and the moment convergence in Equation 14. Also assume supi E∥Ki∥4F < ∞ and supi E∥Vi∥4F < ∞. Let W ∗ t be the minimizer of Jt(W), with αt = λ2t /t and βt = µ2t /t. Suppose that αt → α and βt→β for some α, β∈[0,∞) as t→∞. Define the limiting quadratic risk R(W) through the limiting second moments in Equation 14. Then:

• (i) The normalized objectives ˜ Jt converge pointwise to the regularized limiting risk

• (ii) The function Rridge is strictly convex and admits a unique minimizer W † .
• (iii) The RLS editor converges to this minimizer:

Remark 4.3. Proposition 4.2 is an asymptotic characterization for long-run stable edit streams, not a universal guarantee for arbitrary non-stationary deployments. It does not assume that edits are i.i.d.; the key requirement is the convergence of the empirical second moments in Equation 14. If these limits do not exist, then the editor should not be interpreted as converging to a single limiting risk minimizer. In that case, the relevant guarantee is the finite-time deviation bound in Proposition 4.1.

If α = β = 0 (e.g., when λt, µt are held fixed), then W † =W ⋆ and W ∗ t converges to the least-squares limiting minimizer. If α > 0 and/or β > 0, then W † interpolates between the data-driven optimum W ⋆ and the anchor constraints encoded by (W0,K0,V0): larger α shrinks W † towardW0, and larger β enforces K0W † ≈V0 even as t→∞.

Thus, increasing λ and µ makes the update more conservative. This improves preservation, but can reduce edit adherence when the new edit conflicts with the original mapping. A common policy is to set a deviation budget for the associated penalties, then tune (λ, µ) to satisfy both bounds.

The limits µ, λ → ∞ yield hard anchoring (K0W =V0) and frozen parameters (W ∗ t →W0).

5 Experiments and Results

5.1 Experimental Setup

Models and Baselines. We conduct experiments with two backbone models, Llama3-8B and Qwen2.5-7B, against ALPHAEDIT [Fang et al., 2025], ROME [Meng et al., 2022], MEMIT [Meng et al., 2023], and fine-tuning (FT) [Zhu et al., 2020].

Datasets and Metrics. Following prior work, we mainly use the CounterFact dataset [Meng et al., 2022]. We report Efficacy (rewrite success), Generalization (paraphrase success), Specificity (neighborhood success), Fluency (generation entropy), and Consistency (reference score). To further test whether the trend holds beyond CounterFact, we also evaluate on ZsRE.

5.2 Main Results

Editing Results. Table 1 reports performance after 10K edits with batch size 100 on CounterFact. On Llama3-8B, RLSEdit achieves the best scores across all five metrics. In particular, it obtains a clear improvement in Efficacy over the second-best method (89.94 vs. 74.76), while also improving Generalization, Specificity, Fluency, and Consistency.

On Qwen2.5-7B, the results are more nuanced. RLSEdit achieves the best Efficacy and Fluency, while ALPHAEDIT gives slightly higher Generalization and Specificity, and ROME gives the best Consistency. This difference suggests that the editing-preservation trade-off is modeldependent rather than evidence of instability in RLSEdit. Overall, RLSEdit remains competitive with ALPHAEDIT on Qwen2.5-7B and substantially outperforms ROME, MEMIT, and FT on most metrics.

ZsRE Results. We further test RLSEdit on ZsRE [Levy et al., 2017]. As shown in Table 2, RLSEdit achieves 81.20 Efficacy on Llama3-8B, outperforming ALPHAEDIT, MEMIT, and ROME. This supports the generality of the empirical trend across different editing datasets.

General Capability Results. To assess how well editing methods preserve the pre-edited model's general abilities, we evaluate five tasks from GLUE [Wang et al., 2018], together with additional benchmarks for general knowledge (MMLU), math reasoning (GSM8K), and coding ability (HumanEval, MBPP). We conduct the evaluation on multiple editing checkpoints of the Llama3-8B model, using 10K total edits with a batch size of 100.

Figure 3 summarizes the general capability evaluations. Across the evaluated language understanding, knowledge, math, and coding benchmarks, RLSEdit better preserves the model's general capabilities throughout the editing trajectory. This is especially important in the long sequential setting, where many edits may accumulate interference over time. In contrast, MEMIT, ROME, and FT show stronger degradation as edits accumulate. ALPHAEDIT performs competitively in the early stages, but drops more noticeably after a large number of edits. These results suggest that RLSEdit provides a better balance between edit reliability and general capability preservation.

5.3 Analysis and Discussion

Early Edits Comparison. To examine how well RLSEdit and the baselines preserve earlier edits in a sequential editing setting, we re-evaluate the first N edited cases, where N ∈ {500, 1K, 2K, 4K}, after performing 10K sequential edits with batch size 100 on Llama3-8B. As shown in Figure 4, RLSEdit consistently achieves the best retention across all N: its Rewrite scores range from 71.22 at N=500 to 81.28 at N=4K, and its Paraphrase scores range from 60.49 to 66.98. In contrast, the baselines remain noticeably lower, suggesting weaker preservation of previously edited knowledge under long sequential editing.

Effect of Regularization. The two regularization parameters play different roles. The parameter λ controls the drift from the original weights W0, while µ controls the anchor mapping K0W. We find that the effect of λ is modeldependent. On Llama3-8B, setting λ = 0 while keeping µ = 12000 reduces the Efficacy score to 87.0, with the other metrics also slightly lower. This suggests that the parameterdrift penalty improves the editing-preservation trade-off for this backbone. In contrast, Qwen2.5-7B performs best with λ = 0, indicating that this model is more robust to direct weight changes.

The anchor parameter µ is important in practice. Removing it can make the linear system ill-conditioned in long sequential editing. For a new backbone, we therefore tune (λ, µ) using a small pilot sweep and monitor the three objective terms: edit fitting, parameter drift, and anchor-mapping error.

Speed-upAnalysis. Table 3 reports the update computation time for RLSEdit and ALPHAEDIT across two model backbones and three batch sizes. Across all six configurations, RLSEdit runs faster, reducing update time by 1.37×-1.79× relative to ALPHAEDIT. This is consistent with the theoretical time-complexity analysis in Section 3.2.

6 Conclusion

Existing model editing methods often degrade when the number of edits becomes large. We propose RLSEdit, a recursive least-squares framework for long sequential editing with soft editing and soft preservation constraints. RLSEdit introduces two regularization terms to control deviation from the pretrained weights and from an anchor mapping, allowing it to balance edit adherence and preservation. The resulting objective admits an efficient Woodbury-based recursion whose per-edit update cost is independent of the edit history length.

Empirically, RLSEdit scales stably to 10K edits on Llama-3 and Qwen2.5, achieves strong results on CounterFact and ZsRE, and better preserves general capabilities across language understanding, knowledge, math, and coding benchmarks. These results support RLSEdit as a practical approach for continuous model editing. A remaining limitation is that our experiments focus on 7B-8B backbones; evaluating larger models and more failure cases is an important direction for future work.

Ethical Statement

While our work centers on model-editing methods, it is important to acknowledge that such techniques can also be misused to inject undesirable knowledge or behavioral traits into a model. These risks merit careful consideration and discussion.

Contribution Statement

Xinyu Wang, Sicheng Lyu, and Yu Gu contributed equally to this work. Xiao-Wen Chang is the corresponding author.