BetaEdit: Null-Space Constrained Sequential Model Editing

1 Introduction

Model editing seeks to locally modify a pre-trained model's behavior-e.g., updating a factual belief or correcting a misconception-without retraining from scratch or compromising its general capabilities [Jiang et al., 2025b; Liu et al., 2025b; Xie et al., 2025; Yang et al., 2025; Liu et al., 2025c]. A central challenge in this task is balancing two competing objectives: (1) faithfully incorporating new knowledge, and (2) preserving the vast pre-trained knowledge that underpins the model's general capability. Early approaches often rely on explicit regularization to protect pre-existing knowledge [Meng et al., 2022; Meng et al., 2023; Gupta et al., 2024b], but such methods struggle to simultaneously achieve high edit accuracy and strong edit locality [Fang et al., 2025a].

∗Corresponding author.

To circumvent this trade-off, null-space-based editing methods [Fang et al., 2025a] propose a theoretically elegant solution: perform edits exclusively within the null space defined by pre-existing knowledge space. By construction, such edits are intended to induce zero change on the model's outputs for unedited inputs, thereby preserving pre-trained knowledge perfectly without explicit regularization. However, we observe that this ideal behavior exists only in theory and fails in practice. We identify the root cause: the "null space" used in practice is not exact, but an approximate (or pseudo) null space obtained via truncated singular value decomposition. This approximation inevitably introduces knowledge leakage, i.e., small but non-zero perturbations to pre-trained knowledge, which accumulates over sequential edits and eventually corrupts both editing fidelity and the model's general capabilities.

Recent history-aware editing strategy [Fang et al., 2025a; Fang et al., 2025b], which explicitly regularizes new updates based on past edits, has been shown to significantly mitigate this degradation. As shown in Figure 1, history-aware regularization substantially improves edit efficacy and preserves the model's general capability during sequential editing. The gain in edit efficacy is expected due to the protection of past edits; the concurrent preservation of the model's general capability, however, is counterintuitive. How the history-aware regularization-a single mechanism-achieves both effects during sequential editing remains unclear.

In this paper, we bridge these gaps. First, we formally identify and quantify the knowledge leakage inherent in current null-space methods (Section 3). Second, we provide a principled theoretical analysis (Section 4) showing that historyaware updates implicitly regularize the edit trajectory by reducing interference among sequential edits and constraining overall weight perturbation, thereby better preserving both editing fidelity and general capabilities. Building on these insights, we propose BETAEDIT (Section 5), a refined framework that (1) mitigates knowledge leakage by reintroducing a penalty term specifically designed to compensate for the imperfection of the pseudo-null space, and (2) seamlessly integrates history-aware updates into the null-space paradigm. Experiments on three large language models and two standard editing benchmarks demonstrate that BETAEDIT remains effective even after 10,000 sequential edits, whereas existing editors typically fail, severely corrupting the model's general capabilities and achieving near-zero editing efficacy.

2 RelatedWork

Model editing techniques are broadly classified according to whether they modify the original parameters of the pretrained model.

Parameter-preserving editing methods aim to update or correct model behavior without altering the base weights. These approaches generally fall into two categories. The first introduces auxiliary modules to inject or override knowledge dynamically. For example, SERAC [Mitchell et al., 2022b] employs an external memory store together with a lightweight corrective model; GRACE [Hartvigsen et al., 2023] stores knowledge in discrete codebooks; while methods like CaliNet [Dong et al., 2022], T-Patcher [Huang et al., 2023], MELO [Yu et al., 2024], and WISE [Wang et al., 2024] incorporate trainable modules (e.g., LoRA adapters, or side memory) to store new knowledge. The second category relies on contextual prompting strategies-such as IKE [Zheng et al., 2023], Deck [Bi et al., 2025], and DecKER [Wang et al., 2025]-which steer the model toward desired outputs at inference time through carefully designed prompts.

In contrast, parameter-modifying editing directly updates the model's internal weights. These methods typically follow one of two paradigms. The first leverages metalearning frameworks, where a hypernetwork is trained to predict localized parameter adjustments. Notable examples include MEND [Mitchell et al., 2022a], MALMEN [Tan et al., 2024], InstructEdit [Zhang et al., 2024], and RLEdit [Li et al., 2025]. The second adopts a "locate-then-edit" strategy: it first pinpoints the specific activations or parameter subsets responsible for a target fact-often using gradientbased attribution or causal tracing [Zhou et al., 2025]-and then applies precise modifications. Representative works include ROME [Meng et al., 2022], MEMIT [Meng et al., 2023], EMMET [Gupta et al., 2024b], and AlphaEdit [Fang et al., 2025a]. Recent efforts within this paradigm also focus on mitigating model degradation during sequential editing. Techniques along this line include post-editing weight regularization [Gu et al., 2024; Ma et al., 2025; Li and Chu, 2025; Liu et al., 2025a; Qiao et al., 2025] and history-aware regularization strategies that explicitly account for past edits [Fang et al., 2025a; Fang et al., 2025b; Guo et al., 2025].

3 Knowledge Leakage in Null-Space Constrained Editing

We consider the locate-then-edit paradigm for model editing [Meng et al., 2022], which proceeds in two steps: (1) identifying a critical layer that stores factual knowledge-commonly the second linear layer in the MLP of early transformer blocks-and (2) updating its weight matrix W to incorporate new information while preserving pre-existing knowledge. This is formalized as a regularized least-squares problem [Meng et al., 2023]:

where (K0,V0) denotes pre-trained knowledge (satisfying WK0 = V0 by construction), and (K1,V1) represents the new factual update to be injected. The regularization parameter λ1 > 0 controls the trade-off between edit accuracy and preservation of pre-trained knowledge. The closed-form solution to Eq. (1) is given by:

with R1 = V1 -WK1 denoting the residual of the new knowledge.

To avoid compromising pre-trained knowledge, AlphaEdit [Fang et al., 2025a] reparameterizes the update as ˜∆P, where P is a projection matrix onto the null space of K0, i.e., PK0 = 0. With this constraint, the second term in Eq. (1) vanishes identically, yielding the simplified objective:

This yields the following closed-form update:

where λ2 > 0 ensures numerical invertibility.

In theory, the null-space formulation guarantees perfect preservation of pre-trained knowledge: for any sequence of edits {∆(t) }Tt=1 , the cumulative update satisfies (W + ∑Tt=1 ∆(t) )K0 = V0, since each ∆(t)K0 = 0. In practice, however, we observe a violation of this equality. To quantify the deviation, we define knowledge leakage as:

which measures unintended perturbations to pre-trained knowledge after edit. Figure 2 shows the KL with respect to the cumulative update ∑Tt=1 ∆(t) over T = 2000 sequential edits. Contrary to theoretical expectations, both null-space methods, AlphaEdit and AlphaEdit-H, exhibit non-zero and growing knowledge leakage, often exceeding that of baseline methods such as MEMIT [Meng et al., 2023]. This directly contradicts the core promise of null-space editing and explains the catastrophic model corruption observed in longhorizon settings (cf. Figure 1).

The root cause lies in the approximate nature of the employed null space. In practice, K0K⊤0 is typically fullrank [Gupta et al., 2025], which implies that all singular values of K0K⊤0 are positive, rendering the true null space trivial (i.e., {0}). To address this, prior work treat singular values below a threshold (e.g., 0.02) as 0 [Fang et al., 2025a]. This approximation implies that PK0 ̸= 0, causing each edit to introduce a small leakage that accumulates over time.

4 Theoretical Analysis: History-Aware Updates Reduce Weight Perturbation

Naively applying the update rules in Eq. 2 or Eq. 4 during sequential editing often leads to catastrophic degradation-both in editing efficacy and in the model's general capabilities [Gu et al., 2024]. Recent work by [Fang et al., 2025a; Fang et al., 2025b] empirically demonstrates that this degradation can be substantially mitigated by explicitly preserving previously edited facts when performing new edits.

Concretely, under a history-aware editing mechanism, the MEMIT-style weight update at step t takes the form:

where the key vectors from previous edits, k1, . . . , kt-1, are explicitly incorporated into the t-th update. In contrast, the vanilla (history-agnostic) approach uses only:

Compared to Eq. (2), we use lowercase rt and kt here to denote vectors (rather than matrices), and the subscript t emphasizes the sequential editing setting.

While the improvement in editing accuracy is intuitive-since past edits are directly protected-it is less obvious why general capabilities are simultaneously preserved. In what follows, we provide a theoretical analysis showing that the history-aware mechanism not only safeguards previously edited knowledge but also better preserves the model's general capabilities during sequential editing, in contrast to the vanilla (history-agnostic) approach.

To ensure a clean and meaningful comparison, we first exclude certain trivial cases. For instance, if the residual vector rt is zero, no actual edit occurs; if two edits introduce conflicting updates (e.g., "the capital of France is Paris" vs. "the capital of France is London"), they may cancel each other out, potentially restoring the model to its initial state without degrading general capabilities. Such scenarios do not reflect the steady growth of knowledge leakage observed in Figure 2. Therefore, we assume that the sequence of weight perturbations {∆t} is non-conflicting, i.e., ⟨∆i,∆j⟩F ≥ 0 for all i, j, which ensures non-decreasing knowledge leakage over time (see the supplementary material for details).

Under this editing scenario, the following theorem formalizes the advantage of the history-aware approach:

Theorem 1 (Reduced Weight Perturbation). Let ∆(A) t and ∆(B) t denote the weight perturbations at the t-th editing step for the vanilla (history-agnostic) method and the historyaware method, respectively. For any T > 1, the cumulative weight perturbation of the history-aware method is smaller in Frobenius norm than that of the vanilla method:

Proof sketch. The key insight is that the history-aware update ∆(B) t explicitly incorporates previously edited directions {ks}s<t via a Gram matrix that includes ∑t-1 s=1 ksk⊤ s . This effectively projects the new update onto a subspace orthogonal to past edit directions, thereby reducing interference with earlier updates, i.e., ⟨∆(B) t ,∆(B) s ⟩F ≤ ⟨∆(A) t ,∆(A) s ⟩F for all s < t, and yielding a smaller cumulative weight perturbation. Full details are provided in the supplementary material.

Figure 3(a) illustrates the evolution over time of the Frobenius norm of the cumulative weight update at each editing step. The history-aware method consistently suppresses the magnitude of perturbation more effectively than the historyagnostic method. This suppression becomes increasingly pronounced as the number of edits grows, since the likelihood of overlap between new and past edits increases. The historyaware method is able to orthogonalize these overlapping components, whereas the history-agnostic method conducts independent updates, leading to an accumulation of interference and an eventual explosion in cumulative weight perturbation, which corrupts the model's general capabilities (cf. Figure 1).

To further demonstrate the effectiveness of history-aware editing in suppressing weight perturbation, we evaluate several history-agnostic variants that attempt to reduce per-step weight perturbation:

• MEMIT-30000: Increases the regularization strength λ1 from 15,000 to 30,000, thereby shrinking the magnitude of each update ∆t.
• MEMIT-R: Replaces the historical keys {k1, . . . , kt-1} in Eq. (5) with random keys {kr 1 , . . . , kr t-1 }, which

strengthens the regularization but lacks meaningful alignment with historical edits. 1

• RECT [Gu et al., 2024]: Applies a sparsification strategy that zeroes out small-magnitude entries in the update matrix ∆t, retaining only the largest components.

The results, shown in Figure 3(b), indicate that while these strategies somewhat alleviate cumulative weight perturbation, they incur a notable trade-off in terms of editing performance.

5 Method

As revealed in Section 3, existing null-space-based editing methods suffer from non-negligible knowledge leakage, which leads to cumulative corruption of the pretrained knowledge over sequential edits. To address this, we propose to explicitly account for the leakage during weight update. Specifically, instead of discarding the preservation term as in Eq. (3), we reintroduce it into the objective optimization:

which yields the following closed-form solution:

where λ2 > 0 ensures numerical stability (set to 10 following [Fang et al., 2025a]), and λ1 > 0 balances edit accuracy against knowledge leakage control.

Extending this to the sequential editing setting, we incorporate both pre-trained knowledge and all previously edited facts into the protection set. The update at step t is given by:

where Rt = Vt - WKt and C(t) = λ1K0K⊤0 + ∑t-1 i=1 KiK⊤i aggregates the representations of all protected knowledge up to step t, yielding a history-aware update.

1 Since historical keys are derived from the subject token of each historical edit [Fang et al., 2025a], the random keys are sampled from subject tokens of randomly selected edits.

Motivated by the theoretical analysis in Section 4-which shows that preserving editing history protects both editing performance and the model's general capabilities-we extend the null-space protection beyond the original knowledge K0 to also cover all previously edited knowledge {Ki}t-1 i=1 . That is, instead of constructing the projector based solely on K0K⊤0 as in prior work [Fang et al., 2025a], we define Pt as the orthogonal projector onto the null space of C(t) .

Concretely, we compute the eigendecomposition of C(t) = UΛU⊤ and select the columns of U corresponding to eigenvalues smaller than a threshold ϵ (e.g., ϵ = 0.02). Denoting this subset of eigenvectors as Usmall, we form the projection matrix as Pt = UsmallU⊤small . By construction, Pt suppresses directions in which the model has already encoded factual knowledge, ensuring that new updates minimally perturb both pre-trained and edited facts. However, applying the null-space constraint to the editing history necessitates recomputing Pt at each edit step, which incurs substantial computational overhead. To balance efficiency and protection, we adopt a periodic refresh strategy: Pt is updated only every τ edits (e.g., τ = 1, 000). In the limiting case where τ → ∞, our method reduces to standard null-space editing that protects only the pretrained knowledge.

6 Experiments

6.1 Experimental Setup

LLMs, Datasets, and Evaluation Metrics. We evaluate our approach on three decoder-only language models: Qwen3-4B-Instruct-2507, GPT-J-6B, and LLaMA3-8BInstruct. Following prior practice [Fang et al., 2025a], we apply edits to layers {4,5,6,7,8} for LLaMA3 and Qwen3, and to layers {3,4,5,6,7,8} for GPT-J.

We assess editing performance on two standard factual editing benchmarks: COUNTERFACT [Meng et al., 2022] and ZSRE [Levy et al., 2017]. Performance is measured using three standard metrics: Efficacy (Eff.), which measures whether the edited model assigns the highest probability to the target output y for the input x (i.e., y = argmaxy′ P(y′ | x)); Generality (Gen.), which evaluates whether the edited model generalizes to paraphrased inputs xp (i.e., y = argmaxy′ P(y′ | xp)); and Specificity (Spe.), which measures whether the edited model preserves unrelated facts. Formally, for an unrelated input xs and its corresponding output ys , it requires that ys = argmaxy′ P(y′ | xs).

ComparedBaselines. We compare against a range of stateof-the-art model editing methods, categorized as follows: (i) fine-tuning with LoRA (FT) [Hu et al., 2022]; (ii) hypernetwork-based editing (RLEDIT) [Li et al., 2025]; (iii) locate-then-edit approaches, including MEMIT [Meng et al., 2023], PMET [Li et al., 2024], EMMET [Gupta et al., 2024b], and ALPHAEDIT [Fang et al., 2025a]; and (iv) posthoc adjustment techniques for per-step weight update, including RECT [Gu et al., 2024], PRUNE [Ma et al., 2025], ADAEDIT [Li and Chu, 2025], and SIMIE [Guo et al., 2025]. All locate-then-edit methods are adapted to the sequential editing setting using history-aware regularization.

6.2 Editing Performance

We evaluate editing performance at three scales-300, 5,000, and 10,000 sequential edits-to more comprehensively assess the scaling behavior of existing editors. Unlike some baselines that use large batch sizes (e.g., 20 or 100 [Li et al., 2025; Fang et al., 2025a]), we adopt the most challenging setting with a batch size of 1. Table 1 summarizes editing performance across three models and two datasets. We group editors into two categories: (1) history-agnostic editors, which perform each edit independently (e.g., FT, RECT and AdaEdit); and (2) history-aware editors, which explicitly preserve the editing history during new updates (e.g., SimIE, EMMET and BETAEDIT).

As shown in Table 1, history-aware editors consistently outperform the history-agnostic baselines. Notably, BETAEDIT achieves the best overall performance, particularly under large-scale sequential editing. Most baselines degrade catastrophically beyond 5,000 edits, whereas BETAEDIT remains effective even at 10,000 edits. We attribute this robustness to BETAEDIT's superior preservation of the model's general capabilities. Once these language capabilities are compromised (i.e., models are corrupted), subsequent editing becomes ineffective or even infeasible [Gupta et al., 2024a]. Figure 4 presents three key metrics as the number of edits increases from 0 to 10,000: (i) knowledge leakage, (ii) cumulative weight perturbation, and (iii) MMLU performance (Macro-F1 score). BETAEDIT effectively suppresses knowledge leakage and limits cumulative parameter drift, thereby preserving the model's general capabilities.

Among the two recipes in BETAEDIT, knowledge leakage control is specifically designed for null-space-based editing methods, while null-space-based history-aware regularization is a general-purpose technique that effectively minimizes interference with historical edits during new updates, thereby better preserving both editing performance and the model's general capabilities-as established by our theoretical analysis in Section 4. To further evaluate its effectiveness, we apply this technique to the baseline editors EMMET, PMET, and MEMIT. As shown in Figure 5, when enhanced with nullspace-based history-aware regularization, all editing methods remain effective under 10,000 sequential edits without corrupting the model.

6.3 Ablation Study and Hyperparameter Analysis

Ablation Study. We evaluate BETAEDIT under two ablated settings: (1) without the knowledge leakage (KL) penalty term, and (2) without the null-space (NS) constraint for preserving prior edits (denoted as w/o KL and w/o NS, respectively). The ablation results are shown in Table 2 (top: 5k edits; bottom: 10k edits). Two key observations emerge:

First, penalizing knowledge leakage is essential across both editing scales (5k and 10k), serving as a fundamental mechanism to protect pre-trained knowledge. Second, the null-space constraint on historical edits significantly enhances editing efficacy and preserves the model's general capabilities-particularly at 10k edits, where KL-only variants (w/o NS) typically achieve near-zero editing efficacy and corrupt the model's general capabilities. This underscores the critical role of protecting prior edits: it not only mitigates interference with editing history but also substantially curbs cumulative parameter perturbation-a key factor, as shown in Section 4, for preserving the model's general functionality during long sequences of edits.

Hyperparameter Analysis. We study two key hyperparameters in BETAEDIT: the knowledge leakage penalty coefficient λ1 and the null-space projector refresh period τ .

The top panel of Figure 6 shows the effect of varying λ1 ∈ {0, 10, 100, 500, 3000, 10000} on LLAMA3 under 5,000 sequential CounterFact edits. Small λ1 (e.g., 0 or 10) leads to severe knowledge leakage, collapsing both efficacy and specificity near zero. Larger values better preserve pretrained knowledge and improve both metrics, but excessively large λ1 (e.g., 10,000) over-constrains updates, limiting editability and degrading efficacy despite high specificity-highlighting a trade-off between editability and knowledge retention.

The bottom panel of Figure 6 examines the refresh interval τ, i.e., how often the null-space projector Pt is recomputed. With τ = 5000 (no refresh), BETAEDIT relies only on standard history-aware regularization (w/o NS) and fails to sustain performance: both metrics drop near zero. Since recomputing Pt is costly, a moderate τ (e.g., 1,000) suffices to maintain strong editing performance while avoiding unnecessary computational overhead.

7 Limitations

Existing editing methods typically perform edits at the subject token, which aligns well with standard editing benchmarks such as CounterFact and ZsRE-datasets where each edit involves a distinct subject. In this setting, edits are generally non-conflicting, and our theoretical analysis of historyaware regularization holds reasonably well. However, the non-conflicting assumption can be significantly violated in two practical scenarios: (1) when edits are not anchored at the subject token, or (2) when multiple edits share the same subject. As shown in Table 3, anchoring edits at the last token drastically reduces the effectiveness of history-aware regularization. On LLaMA3, edit efficacy already drops to 54.7% after just 300 sequential edits, and by 2,000 edits the model is effectively corrupted-far below the 10,000-edit capacity achievable with subject-token anchoring. This is because the last token (e.g., "is") frequently appears across many edit prompts; anchoring knowledge update at this position introduces substantial interference among edits.

This reveals a key limitation of our approach: its dependence on subject-token anchoring imposes practical constraints. Specifically, it struggles in settings involving (i) multiple edits targeting the same subject [Dong et al., 2025], or (ii) edits with complex or long-form prompts that deviate from simple subject-relation-object triplets and may involve multiple subjects [Jiang et al., 2025a; Zeng et al., 2025].

8 Conclusion

We have uncovered a fundamental gap between the theoretical promise and practical behavior of null-space-based model editing: despite being designed to preserve pre-trained knowledge perfectly, these methods suffer from knowledge leakage due to the use of approximate null spaces, leading to catastrophic failure in long-horizon sequential editing. Beyond identifying this issue, we provide a principled theoretical analysis showing that history-aware regularization serves a dual role-it not only protects previously edited facts but also implicitly constrains overall parameter drift, thereby preserving the model's general capabilities.

Building on these insights, we propose BETAEDIT, a refined editing framework that explicitly mitigates knowledge leakage by reintroducing a dedicated penalty term, while seamlessly integrating history-aware updates into the nullspace paradigm. Experiments across multiple large language models and standard benchmarks show that BETAEDIT maintains high editing performance and effectively preserves the model's general capabilities-even under up to 10,000 sequential edits, a regime in which prior methods typically collapse-demonstrating its effectiveness for long-horizon model editing.

Acknowledgments

This work is supported by the National Key Research and Development Program of China under grant 2024YFC3307900; the National Natural Science Foundation of China under grants 62436003, 62206102, 62376103 and 62302184 ; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Science and Technology Talent Service Project under grant 2024DJC078; and Ant Group through CCF-Ant Research Fund. The computation is completed in the HPC Platform of Huazhong University of Science and Technology.