Detoxifying Large Language Models via Localized Feature Editing with Sparse Autoencoders

1 Introduction

Large Language Models (LLMs) have achieved significant milestones in natural language processing (NLP) [Brown et al., 2020], becoming essential foundations for a variety of AI applications [Ren et al., 2025]. Despite their promising prospect, the training of LLMs on vast amounts of uncurated internet corpora introduces potential security risks, highlighting the need for careful safeguards. These concerns have driven increasing attention to the issue of language model detoxification [Ko et al., 2025; Shang et al., 2026].

∗Corresponding author.

Efforts on reducing the toxicity of LLMs have attracted widespread attention. Traditional data-driven methods [Lu et al., 2022] train models to generate text conditioned on non-toxic attributes or apply style transfer to remove toxicity. Decoding-control methods [Feng et al., 2024] regulate the generation process without changing model parameters, typically by adjusting output probabilities or incorporating auxiliary modules. While promising, these safety measures often yield superficial shortcuts rather than fundamental modifications [Goyal et al., 2025], making them vulnerable to circumvention through relatively simple techniques such as strategic prompting and fine-tuning. To address these challenges, recent works have attempted model-editing methods [Zhang et al., 2025] that directly edit models via task vectors. Due to the polysemantic nature of many neurons in LLMs, such methods may entangle seemingly unrelated concepts [Yang et al., 2026], thereby unintentionally perturbing non-target semantic dimensions and degrading generation quality.

Sparse Autoencoders (SAEs) [Cunningham et al., 2023] have recently emerged as a promising technique for understanding LLMs. SAEs are a tool that decompose model's internal activations into meaningful features, providing dual benefits of interpretability and the ability to perform targeted steering along the dimension of the chosen feature. We conducted an empirical study on the RTP-Extreme dataset to examine how identifying and manipulating specific features via SAE affects the model outputs, with results presented in Figure 1. We identify three activation patterns (left) and then perform bidirectional interventions on the top-5 toxicity-related features (right). The results reveal a consistent effect: scaling up toxic feature activations markedly increases the generation of toxic content, whereas ablating these activations mitigates toxicity at the cost of a nontrivial degradation in generation quality. Indiscriminately ablating toxic features to reconstruct the model activations reduces toxic outputs by 76.4%, but causes more than a twofold increase in fluency degradation for originally non-toxic generations. This inspires us to treat SAE features as interpretable and fundamental units of manipulation, laying the groundwork for effective detoxification while exposing challenges in preserving generation quality.

Despite its potential, achieving this goal entails significant challenges. First, how to accurately locate features in the SAE space that encode toxicity-related semantics? Due to the sparse nature of SAE features, a large number of dimensions exhibit low activation density or correspond to irrelevant features. Besides, the inherent weak correlation and semantic dispersion among SAE features make it difficult to accurately locate toxicity-related features. Second, once toxicity-related features are activated, how to steer LLMs away from the toxic property while not straying too far from the original space? Naively applying simple transformations to toxic feature distributions often yields unstable mappings in the latent space, which in turn degrades the fluency of the generated text.

To address the aforementioned issues, we develop DeLFE (Detoxifying Large Language Models via Localized Feature Editing with Sparse Autoencoders), a lightweight detoxification approach that enables localized feature transformations in SAE space. Using toxic/non-toxic sample pairs, we first employ SAEs to decompose model activations into a disentangled feature space and isolate two toxicity-irrelevant activation patterns. By introducing a contrastive loss to draw the separation rule between the toxic/non-toxic feature clusters, we learn a subspace that reflects toxicities, enabling finer-grained discrimination. Building on this, we learn a flow-matching guided feature transformation constrained to this subspace, optimized to redistribute the toxic feature distribution into its detoxified counterpart. During inference, DeLFE integrates the above procedure into three feature intervention strategies, where salient toxicity-related SAE features in the learned subspace are adopted to identify risks, and smooth transformations over these features are then employed to detoxify. The key contributions are as follows:

• We propose DeLFE, a lightweight steering method that learns a toxicity subspace to constrain interventions to sparse feature-level dimensions, enhancing both steering interpretability and detoxification efficiency.
• DeLFE learns a flow-matching guided feature transformation to edit internal model activations via three intervention strategies, addressing the latent instability issues of naive feature manipulation.
• Our results demonstrate that DeLFE consistently outperforms state-of-the-art baselines across a diverse suite of LLMs and against a wide range of datasets.

2 RelatedWork

Detoxification of Base Models. Traditional data-driven detoxification trains models on sanitized attributes [Zhang and Song, 2022; Lu et al., 2025] or applies style transfer [Dale et al., 2021], but these approaches depend on curated datasets and incur heavy computational costs [Tang et al., 2024]. Decoding-control methods operate during inference by suppressing the probability of toxic tokens [Kwak et al., 2023; Liang et al., 2024; Liu et al., 2021]. While avoiding changes to model parameters, such methods often compromise contextual semantic coherence, particularly under strict attribute constraints. Model-editing methods aim to control generation by neural activations [Wang et al., 2024; Xiao et al., 2025; Suau et al., 2024]. These methods modify the model internal representations [Leong et al., 2023; Yi et al., 2025] or weights [Uppaal et al., 2024]. Due to the polysemantic nature of many neurons in LLMs, editing methods entangle unrelated concepts, unintentionally perturb non-target semantics, and degrade generation quality.

Sparse Autoencoders. Recent advances in mechanistic interpretability have shown that SAEs can decompose internal activations of LLMs into sparse, interpretable features by learning sets of sparsely activating features [Gao et al., 2024; Wang et al., 2025]. Moreover, [Kissane et al., 2024] applied SAEs to reconstruct attention layer outputs, providing deeper insights into the semantic organization of neural circuits in LLMs. [Marks et al., 2024] utilized SAE-discovered sparse circuits to reduce gender bias in classifiers, whereas [O'Brien et al., 2024] revealed that SAEs can be used to steer model refusal to harmful prompts. [Goyal et al., 2025] adopted SAEs to decompose activations to identify toxic dimensions and perform targeted interventions. Like [Goyal et al., 2025], it tends to perturbed the features of SAEs indiscriminately, which may have a counterproductive effect on high-quality outputs. Besides, it performs an offset in the opposite direction of the identified toxic features, failing to accurately control the degree of manipulation (as reported in Figure 1). Our work distinguishes itself by learning a subspace that identifies toxicity-triggering risks, within which we perform finegrained feature distribution adjustments, achieving effective detoxification while preserving generation quality.

Our work makes two key distinctions compared to prior work: (1) Motivation: DeLFE constrains interventions from activation-level to feature-level subspace and accurately controls the degree of manipulation, whereas prior methods operate in dense, entangled activation spaces that fail to capture fine-grained semantics. (2) Implementation: Operating in the learned toxicity subspace, DeLFE learns smooth transformation to guide localized feature editing, avoiding unstable detoxification behaviors under distributional shift.

3 Formalization and Preliminaries

Problem Formulation. Given a prefix p consisting of T tokens, a language model autoregressively generates an output sequence when prompted with the input context. The objective of language detoxification is to reduce the occurrence of toxic content while maintaining generation quality.

Sparse Autoencoders (SAEs). Given an input activation x ∈ Rd corresponding to an input prefix at a chosen layer, the SAE aims to decompose complex activations into a more interpretable and high-dimensional representation. It maps x into a sparse feature activation vector z ∈ Rm (where m≫ d) and reconstructs it as ˆx:

where W(e) ∈ Rd×m, b(e) ∈ Rm are encoder parameters, W(d) ∈ Rm×d , b(d) ∈ Rd are decoder parameters. The parameters are learned by minimizing two competing goals: the reconstruction error, measured by the squared L2 norm ∥x- ˆx∥22 , and the sparsity of the latent code, encouraged by an L1 penalty ∥z∥1 weighted by a hyperparameter λspar:

The L1 penalty forces most elements of the latent code z to be zero. This encourages the SAE to learn localized features, where each active dimension in z ideally corresponds to a single, interpretable feature.

4 Approach

In this section, we introduce DeLFE, which comprises Stable Feature Selection (§4.1), Toxicity Subspace Learning and Detection (§4.2), Subspace-aware Feature Transformation (§4.3), and Feature Intervention Strategies (§4.4). An overview of the framework is shown in Figure 2.

4.1 Stable Feature Selection

Feature Collection. To identify the informative features in the SAE space for detoxification task, we use the ParaDetox dataset [Logacheva et al., 2022] containing pairwise "toxic" and "non-toxic" prefixes generated using paraphrasing, preserving the same meaning. Given sample pairs {(pi, yi)}ni=1 , where yi ∈ {0, 1} denotes the toxicity label, we process each prefix pi through a frozen pretrained LLM and extract residual stream activations xi at a target layer as described in section 3. These activations are passed through a pre-trained SAE encoder fenc to obtain sparse latent features zi.

Irrelevant Feature Filter. For a preliminary analysis of the SAE latent space, we visualize and compare the average activation intensities of representative SAE features between toxic and non-toxic samples (see Figure 1). We find that a majority of features display low activation pattern, firing only infrequently due to the L1 regularization in SAE training promoting sparsity by penalizing feature activations during training. Moreover, among the features triggered by high activation density, there exist non-semantic tokens, such as stop words, or punctuation, which are unrelated to detoxification, which are collectively referred to as general language pattern. To filter toxic-irrelevant features, we compute a tstatistic measuring the difference in activation intensities between toxic and non-toxic samples for each feature j:

where µ(·) and σ(·) denote the mean and standard deviation of activation intensities, and Ntoxic and Nnon-toxic are the total number of toxic and non-toxic samples, respectively. We rank all features by ttoxic scores and select the top-k to constitute the stable toxicity feature subset Ztox ∈ Rk , which tends to activate within toxicity-related contexts.

4.2 Toxicity Subspace Learning and Detection

While the above feature filtering isolates two prominent noise activation patterns, inherently weak correlations and semantic dispersion among SAE features make it not simple to characterize complex toxicity patterns. To this end, we propose to learn a subspace that serves as a compact discriminative region within the toxicity feature subset Ztox, upon which a classifier is built to assess the toxicity-triggering risk on the feature level. Ideally, the subspace learner should be lightweight and fast to update, since it will be used together in the feature transformation process to steer base LLM generation. Formally, for labeled feature pairs (z(t) , z(n) ) ∈ Ztox, the loss function can be written as:

where z(o) is a candidate feature from Ztox, τ is the temperature parameter, m is the separation margin, and I[·] is the indicator function. Lce is the cross-entropy loss. Intuitively, Lcl serves to carve out a toxicity-specific subspace in toxicity feature subset Ztox, restricting toxic signals to a small and controllable set of dimensions, and Lce captures the toxicitytriggering risk. Next, we describe how to accommodate the subspace information in the feature transformation process and steer the generation based on the learned subspace.

4.3 Subspace-aware Feature Transformation

Building on the learned toxicity subspace, we propose a smooth transformation that edits toxic feature distributions toward detoxified ones, steering generation away from this subspace. Given toxic/non-toxic feature pairs ztox, zntox obtained from toxicity subspace, we have two goals when designing the feature transformation mapping ϕ : Rk′ →Rk′ over candidate features: (1) detoxification: we want ϕ(ztox) to approach zntox along detoxified directions and (2) utility: we want ϕ(ztox) to stay close to original semantic space z.

Feature transformation in a one-shot manner is hard to learn. Therefore, we model ϕ using continuous normalizing flows (CNFs) [Chen et al., 2018], which induce smooth transformations that steer features away from the toxic subspace:

where zt is the intermediate feature during the transformation from ztox to zntox, vt(·; θ) is implemented as a locally Lipschitz MLP that parameterizes the flow inducing the mapping ϕ. Accordingly, the mapping ϕ transforms ztox into zt i.e., zt = ϕt(ztox). To approach this, we constrain the evolution trajectory of intermediate detoxification states, by a class-conditional Gaussian distribution N .

We adopt the derivative of the mean path in Eq. (6) as target velocity ut to guide the learning of the flow function vt.

We train vt(zt; θ) by optimizing a loss that aligns the learned features zt with the non-toxic targets zntox under the joint distribution of intermediate feature states:

When the toxicity risk is triggered (§4.2), it applies a corrective shift ∆zt to ztox, adjusting it toward a detoxification direction via an Ns-step numerical integration on [0, 1]:

The detoxified feature zdtox = ϕ(ztox) reconstructs to replace the original activations of the target model, steering its internal activation toward generating non-toxic outputs.

4.4 Feature Intervention Strategies

During the inference stage, intervention strategies use the SAEs features z to dynamically decide when (§4.2) and how (§4.3) to intervene. Therefore, this module offers three complementary intervention strategies: (i) coarse-grained intervention at the prefix level, (ii) fine-grained and (iii) adaptive interventions throughout the generation process.

(i) Coarse-grained Intervention: This strategy performs minimal intervention by applying modifications solely to the latent features of the last token at each generation timestep, rather than across the whole input sequence. Let z(i) denote the SAE feature at the first step. The transformation is applied as: ˜z(i) = ϕ(z(i) ) , and the reconstructed activation is passed to the remaining network layers.

(ii) Fine-grained Intervention: The fine-grained strategy decides whether steering should continue at the token level. For the i-th token, ˜z(i) = ϕ(z(i) ) , i = 1, . . . , T. This allows the model to continuously monitor the evolving toxicity risk throughout the entire generation process.

(iii) Adaptive Intervention: This strategy utilizes a dynamically adjusts intervention mechanism with threshold γt. Let λi = e-γt(i-1) be the decay-controlled intervention probability for the i-th token. During generation, the transformation is applied to z(i) with probability λi, and otherwise the original feature is retained. The updated feature is given by:

This strategy enforces stronger detoxification during early generation and progressively relaxes intervention over time.

5 Experiments

5.1 Experiment Setup

Datasets. We utilize the RealToxicityPrompts[Gehman et al., 2020] dataset comprising 100K prefixes with toxicity scores. We craft two evaluation sets: RTP-Broad, consisting of 10k prefixes sampled to broadly test toxicity mitigation, with an average toxicity score of 0.449; and RTP-Extreme, comprising 5k toxic prefixes to focus on extreme toxicity conditions, with a substantially higher average toxicity score of 0.737.

Evaluation Metrics. All methods are evaluated from two perspectives. (i) Toxicity Mitigation (Tox.). The Expected Maximum Toxicity (EMT) computes the average maximum toxicity over k = 25 generations, while the Toxicity Probability (TP) estimates the empirical probability of a generation with (TOXICITY ≥ 0.5) for at least once over k = 25 generations. Following prior work, we restrict the generations up to 20 tokens or below. (ii) Language Generation Ability. We categorize the metrics into two aspects: Quality includes Fluency (Flu.), assessed via perplexity (PPL) computed by LLaMA2-7B using the prefix combined with the generation. Diversity (Div.): the average number of distinct (Dist-n) unigrams, bigrams, and trigrams, across the 25 generations.

Models and Baselines. We evaluate DeLFE on three base LLMs: Gemma-2-2B, Gemma-2-9B, and Mistral-7B. We adopt 12 algorithms as baselines, including DEXPERTS[Liu et al., 2021], DisCup[Zhang and Song, 2022], AURA[Suau et al., 2024], DATG-P[Liang et al., 2024], CMD[Tang et al., 2024], DivDetox[Zhao et al., 2025], SASA[Ko et al., 2025], TRACE[Yidou-Weng et al., 2025], UniDetox[Lu et al., 2025], CP[Klein and Nabi, 2025], ARGRE[Xiao et al., 2025] and SAE-Steer [Goyal et al., 2025].

Implementation Details. We use pre-trained SAEs gemma2-2b-res-16k and gemma-2-9b-res-16k from Gemma Scope [Lieberum et al., 2024], and mistral-7b-res-wg from SAE Lens [Bloom et al., 2024]. For target model layers, we experiment with layer 20 for Gemma, layer 16 for Mistral. For (§4.1), we set the number of toxicity feature subset dimensions to k = 256 to ensure rich semantic coverage and sample the ParaDetox dataset to obtain 10k toxic/non-toxic sentence pairs. For (§4.2), we implement the classifier as a twolayer MLP operating on toxicity feature subset. For (§4.3), we implement the CNF using a fixed-step Euler solver with Ns = 8 steps over [0, 1], which incurs an average wall-clock overhead of approximately ∼3.9% compared to standard decoding. For (§4.4), we set the intervention decay factor γt = 0.6. For base LLMs and baselines, we apply nucleus sampling strategy with top-p=0.9 and temperature=1.0. All the main experiments are conducted on a Linux platform with 8 NVIDIA RTX 3090 (24GB) GPUs.

5.2 Overall Performance

Detoxification of Gemma-2-2B. The upper part of Table 1 reports the detoxification results for Gemma-2-2B. The analysis reveals that our method achieves satisfactory detoxification performance on both datasets while maintaining a balanced trade-off between generation quality and diversity compared with baselines. Taking RTP-Broad datasets as an example. Compared to the strong baseline TRACE, DeLFE (Adaptive) achieves superior performance in both toxicity mitigation by 23.40% and 62.58% on EMT and TP metrics and generation quality by 3.81% on Fluency aspects. Several baselines improve detoxification at the cost of degraded fluency and diversity. This effect is especially noticeable for methods that fail to accurately control the degree of manipulation, such as SAE-Steer and DEXPERTS.

Detoxification Across Models. We additionally evaluate two mainstream LLM variants: Gemma-2-9B, and Mistral-7B. The middle part of the Table 1 reports results across multiple base LLMs. Owing to space restrictions, we provide results for several representative baselines. Interestingly, our results indicate that DeLFE enables larger base LLMs to achieve effective detoxification without sacrificing text quality. This result validates that DeLFE maintains strong generalization performance across diverse model architectures.

Ablation Study. To elucidate the impact of each component, we conduct ablation studies by removing the subspace learning (SL) and subspace-aware feature transformation (SaFT) modules. Table 1 shows that when SL module is removed, a noticeable drop in detoxification metric is observed. This indicates that the SL module compensates for the shortcomings of DeLFE in terms of feature understanding. When the SaFT module is removed, the model degrades to directly enforcing guidance on SAE features, resulting in a pronounced decline in both generation quality and detoxification effectiveness.

5.3 Distribution Shift in Toxicity and Quality

To provide a more intuitive view, we contrast the distributional shifts between the base LLM with its detoxified version. For this analysis, 300 samples were randomly selected for each base LLM from both evaluation datasets, with the outcomes summarized in Figure 3. Among all base LLMs, DeLFE achieves a substantially lower average EMT score than the vanilla base LLM, while maintaining comparable PPL, with particularly pronounced improvements on RTP-Extreme dataset. These analytical findings underscore that the detoxified model significantly reduces toxicity while aligning more closely with the desired attribute distribution.

5.4 Toxicity Concept Separability Analysis

To further investigate whether our method can distinguish activation patterns between toxic and non-toxic samples, Figure 4 visualizes the 20th-layer activation patterns of Gemma-22B under different representations and reports recall and F1 scores for the toxicity detection task. We have two key observations: (1) Raw activations exhibit highly entangled and diffuse patterns across toxic and non-toxic samples, while SAE features via stable feature selection reveal sparse dimensions with clear class-specific activation differences. This indicates that SAE decomposes the low-dimensional activations into a sparse basis that enhances class separability. (2) By incorporating toxicity subspace learning, the result further reveals that the method can effectively capture and identify semantic intersections that are indicative of toxic risk. The resulting high-accuracy toxicity risk detection enables targeted interventions that mitigate toxic behavior, achieving precise edits.

5.5 Hyper-parameter Study

We investigate the effects of two key hyperparameters, as illustrated in Figure 5. Ns controls the granularity of feature transformation by decomposing the detoxification trajectory into multiple steps. When Ns = 1, the process degenerates into an overly aggressive one-step transformation, inducing a pronounced distributional shift and degraded generation quality. Conversely, excessively large Ns weakens per-step intervention strength and causes the detoxification effect to saturate or even degrade. Overall, Ns = 8 achieves the best trade-off. We further analyze the effect of the decay factor γt, which controls how quickly the intervention strength diminishes over time. A small γt causes insufficient decay, leading to persistent over-suppression and degraded generation quality. A large γt allows interventions to decay too rapidly, which results in toxicity accumulation and reduced detoxification effectiveness. Based on these results, we set γt = 0.6.

5.6 DownstreamTask Performance

We utilize the MMLU benchmark [Hendrycks et al., 2021] to measure the impact of detoxification on downstream task performance. We do 5-shot evaluation of the Gemma-2-9B model on MMLU benchmark and report the average score in Figure 6. DeLFE delivers competitive performance, showing that it preserves the task-solving capabilities of LLMs.

5.7 Inference Time

We conduct an empirical analysis of inference time across DeLFE and Gemma-2-9B on a single NVIDIA RTX 3090 GPU, processing 50 prefixes per batch. Table 2 reports the additional batch inference time incurred by DeLFE variants compared to the base LLM. To assess the trade-off between efficiency and effectiveness, we further report EMT metric. Notably, DeLFE strikes a strong balance between inference time and performance, significantly reducing EMT with only a modest increase in inference time compared to base LLM.

5.8 Interpretability Case Analysis

To better illustrate the detoxification behavior of DeLFE, we present a case study (see Figure 7) that compares the original and detoxified model outputs and interprets the top-3 most changed toxicity-related features after feature transformation, using semantic explanations from Neuronpedia [Lin, 2023]. By constraining interventions to interpretable feature-level dimensions, DeLFE effectively suppresses toxic activations while preserving unrelated features, demonstrating strong interpretability and high detoxification efficiency.

6 Conclusion

Leveraging SAE encoded features, we present DeLFE, a novel method that detoxifies language models by identifying robust toxicity-related features and editing the localized features of the target model when responding to toxic inputs. This is achieved by replacing the original hidden states of the target LLM with transformed detoxified features, which guide the activation toward generating contextually rich and non-toxic outputs. Extensive evaluations on three base models show that DeLFE consistently outperforms baseline methods in detoxification performance, and preserves model capabilities with minimal degradation.

Ethical Statement

Toxic content in pre-training data can lead LLMs to produce harmful or offensive outputs. This work aims to mitigate such risks through detoxification. Although some prompts and outputs may contain offensive content, they are used solely for research and evaluation purposes.

Acknowledgments

We would like to thank the National Natural Science Foundation of China under Grants No. 62576046, No. 62301066, No. 62406028, and No. 62372051, the Beijing Academy of Artificial Intelligence under Grant No. Z251100008125041, the Key Project of Philosophy and Social Sciences Research, Ministry of Education, China, under Grant No. 24JZD040, and the Graduate Education and Teaching Reform Research Fund of BUPT under Grant No. 2025YZ010.