Interpreting Pretrained Language Models via Concept Bottlenecks (Extended Abstract)

1 Introduction

Although Pretrained Language Models (PLMs) like BERT [Devlin et al., 2018] have achieved remarkable success in various NLP tasks [Zhu et al., 2020], they are frequently regarded as black boxes, posing significant obstacles to their responsible deployment in real-world scenarios, particularly in critical domains such as healthcare [Koh et al., 2020]. To date, many existing works [Madsen et al., 2022] leverage attention weights extracted from the self-attention layers to provide token-level or phrase-level importance. These low-level explanations are found unfaithful [Yin and Neubig, 2022] and lack readability and intuitiveness [Losch et al., 2019], leading to unstable or even unreasonable explanations. To address these limitations, we seek to explain via human-comprehensible concepts that use more abstract features (e.g., general notions) as opposed to raw input features at the token level [Zarlenga et al., 2022]. The foundation of this work is the Concept Bottleneck Models (CBMs) [Koh et al., 2020] that interpret deep models (e.g., ResNet [He et al., 2016]) for image classification tasks using high-level concepts (e.g., shape). For NLP tasks such as sentiment analysis, concepts can be Food, Ambiance, and Service as shown in Figure 1, where each concept corresponds to a neuron in the concept bottleneck layer. The final decision layer is then a linear function of these concepts. Using concepts greatly improves the readability and intuitiveness of the explanations compared to low-level features such as "lobster".

We propose to study Concept-Bottleneck-Enabled Pretrained Language Models (CBE-PLMs). There are two key challenges: ❶ First, existing CBMs [Koh et al., 2020; Zarlenga et al., 2022] require human-annotated concepts. This can be challenging for natural language since the annotator may need to read through the entire text to understand the context and label one concept [N´emeth et al., 2020]. This limits the practical usage and scalability of CBE-PLMs. ❷ Second, many studies have identified the trade-off between interpretability and task accuracy using CBMs since the predetermined concepts may leave out important information for target task prediction [Zarlenga et al., 2022]. Therefore, it is crucial to improve both interpretability and task performance to achieve optimal interpretability-utility trade-off.

❶ To tackle the first challenge, we propose leveraging Large Language Models (LLMs) trained on extensive humangenerated corpora and feedbacks, such as ChatGPT [OpenAI, 2023], to identify novel concepts in text and generate pseudo-labels (via prompting) for unlabeled concepts. Recent studies [OpenAI, 2023] exhibit that these LLMs encapsulate significant amounts of human common sense knowledge. By augmenting the small set of human-specified concepts with machine-generated concepts, we increase concept diversity and useful information for prediction. In addition, generated pseudo-labels offer us a large set of instances with noisy concept labels, complementing the smaller set of instances with clean labels. ❷ To further improve interpretability-utility trade-off (second challenge), we propose to learn from noisy concept labels and incorporate a concept-level MixUp mechanism [Zhang et al., 2017] that allows CBE-PLMs to cooperatively learn from both noisy and clean concept sets. We name our framework for training CBE-PLMs as ChatGPT-guided Concept augmentation with Concept-level Mixup (C3M). In summary, our contributions include:

• We provide the first investigation of utilizing CBMs for interpreting PLMs.
• We propose C3M, which leverages LLMs and MixUp to help PLMs learn from human-annotated and machinegenerated concepts. C3M liberates CBMs from predefined concepts and enhances the interpretability-utility trade-off.
• We demonstrate the effectiveness and robustness of testtime concept intervention for the learned CBE-PLMs for common text classification tasks.

2 RelatedWork

2.1 Interpreting Pretrained Language Models

PLMs such as Word2Vec [Mikolov et al., 2013], BERT [Devlin et al., 2018], and the more recent GPT series [OpenAI, 2023] have demonstrated impressive performance in various NLP tasks. However, their opaque nature poses a challenge in comprehending how PLMs work internally [Diao et al., 2022]. In order to improve the interpretability and transparency of PLMs, researchers have explored different approaches, such as visualizing attention weights [Galassi et al., 2020], probing feature representations [Bills et al., 2023], and using counterfactuals [Ross et al., 2021], among others, to provide explanations at the local token-level, instance-level, or neuron-level. However, these methods often lack faithfulness and intuitiveness, and are of poor readability, which undermines their trustworthiness [Madsen et al., 2022].

Recently, researchers have turned to global concept-level explanations that are naturally understandable to humans. Although this level of interpretability has been less explored in NLP compared to computer vision [Kim et al., 2018], it has gained attention. For instance, a study [Vig et al., 2020] investigates gender classification bias by examining the association of occupation words such as "nurse" with gender. In addition, the CBMs [Koh et al., 2020] have emerged as novel frameworks for achieving concept-level interpretability in lightweight image classification systems. CBMs typically involve a layer preceding the final fully connected classifier, where each neuron corresponds to a concept that can be interpreted by humans. CBMs also show advantages in improving accuracy through human intervention during testing. Yet, the application of CBMs to larger-scale PLMs interpretation is under-explored. Implementing CBMs necessitates human involvement in defining the concept set and annotating the concept labels. Such requirements are challenging for natural language as humans may need to read through the entire text to understand the context and label one concept [N´emeth et al., 2020].

2.2 Learning fromNoisy Labels

Addressing inaccurately labeled or misclassified data in realworld scenarios is the goal of learning from noisy labels, with techniques including noise transition matrix estimation [Liu et al., 2022], robust risk minimization [Englesson and Azizpour, 2021], and more. Recently, the resilience of semisupervised learning methods like MixMatch [Berthelot et al., 2019] and FixMatch [Sohn et al., 2020] to label noise has been discovered by using pseudo-labels for unlabeled data. Inspired by them, we porpose to utilize an LLM (ChatGPT) as a fixed-label guesser, generating noisy intermediate concept labels to potentially predict task labels.

Notably, CBMs specialize in the interpretation and interactability of deep models for general classification tasks. While Multi-Aspect Sentiment Analysis [Zhang et al., 2022] (MASA) shares similar goals when using aspects as concepts, it differs as concepts are not confined to fine-grained aspectual features and can be abstract ideas or broader notions throughout entire contexts. Aspect labels in MASA, primarily used for prediction accuracy, are not always mandatory. To summarize, this study pioneers the comprehensive exploration of utilizing concepts for interpreting large-scale PLMs, and provids a robust framework for harnessing the noisy signals from LLMs to achieve interpretable outcomes from lighter-weight PLMs, which can be easily understood by users.

3 Concept-Bottleneck-Enabled PLMs (CBE-PLMs) with C 3M

3.1 CBE-PLMArchitecture

We adapt CBMs for PLMs by introducing a projector layer pψ after the PLM encoder fθ. This layer maps the PLM's latent representation z = fθ(x) to a concept activation vector ˆ c = pψ(z), where each dimension corresponds to a concept. A final predictor gϕ then maps these concept activations to the task label ˆ y = gϕ(ˆ c). The model structure is x → z → ˆ c → ˆ y. Concepts can be multi-class (e.g., positive/negative/unknown).

3.2 The C 3M Framework

C 3M enables training CBE-PLMs effectively even with limited human-annotated concepts (Ds) and abundant unlabeled data (Du). It involves two main stages (illustrated conceptually in Figure 2 of the original paper [Tan et al., 2024b]):

1. ChatGPT-guided Concept Augmentation:

• Concept Set Augmentation: We prompt ChatGPT, using human-specified concepts (Cs) as examples (in-context learning), to generate additional relevant concepts (Ca). This expands the concept space.
• Noisy Concept Label Annotation: We use ChatGPT with few-shot prompting to generate pseudo-labels (˜ csa or ˜ cu) for all concepts across both Ds and Du. This creates an augmented dataset ˜D with noisy but comprehensive concept labels.
• Concept-level MixUp (CM): Directly training on ˜D treats clean and noisy labels equally, potentially harming performance. CM addresses this by encouraging linear behavior between examples. It interpolates latent representations, concept labels, and task labels between pairs of instances sampled from ˜Dsa (containing original human labels cs) and the shuffled full dataset W = Shuffle( ˜D). The interpolated values (ˆ z(i,j) , ˆ c(i,j) , ˆ y(i,j) ) are calculated using a mixing coefficient ˆλ = max(λ, 1 - λ) where λ ∼ Beta(α, α). This generates mixed instances for training. The final loss LjointMixUp combines the standard joint CBM loss applied to these mixed instances, weighted by a factor τ. This allows the model to learn robustly from the noisy signals provided by the LLM while leveraging high-quality human annotations.

4 Experimental Highlights

We evaluated CBE-PLMs trained with C 3M on sentiment classification tasks using the CEBAB dataset and a curated IMDB-C dataset (based on IMDB), using PLM backbones like BERT, RoBERTa, and GPT2. We compared against standard PLMs and baseline CBE-PLMs trained without CM.

Key findings (conceptualized in Table 1 based on [Tan et al., 2024b]):

• Interpretability with High Utility: CBE-PLMs provide concept-level interpretability with competitive task performance compared to standard PLMs. Smaller models like LSTM even showed improved task accuracy, suggesting the trade-off is not necessary.
• Effectiveness of C 3M: Our framework (CBE-PLMCM) consistently achieved the best concept prediction

accuracy (interpretability). It significantly boosted performance, especially on the small IMDB-C dataset, by leveraging noisy labels effectively. C 3M maintained or improved task accuracy compared to standard PLMs, demonstrating an excellent interpretability-utility tradeoff. Concept-level MixUp (CM) proved essential for robustness against noisy labels, preventing performance degradation seen when naively using augmented data.

• Explainable Predictions: CBE-PLMs allow visualizing concept contributions to the final prediction, offering intuitive insights as shown in Figure 2).
• Test-time Intervention: Users can correct mispredicted concept activations at test time to potentially improve task accuracy. C 3M significantly enhanced the effectiveness and robustness of this intervention, mitigating negative impacts from potential incorrect human corrections (see Figure 3).

5 Conclusion

This work introduce Concept-Bottleneck-Enabled PLMs (CBE-PLMs) as a way to bring concept-level interpretability to complex language models. We propose the C 3M framework to effectively train these models by leveraging LLMs for concept augmentation and pseudo-labeling, combined with a concept-level MixUp strategy for noise robustness. Our approach yields models that are not only more interpretable through concept activations and visualizations but also maintain high task performance and benefit from test-time intervention. Our follow-up works include discussions on providing both local and global explanations [Tan et al., 2024a], enabling autonomous test-time interventions [Tan et al., 2025a], the faithfulness of post-hoc explanations [Tan et al., 2025b], and the intrinsic barriers to explanations [Tan and Liu, 2025]. We hope our methods offer a practical path towards building more transparent, trustworthy, and interactive PLMs by effectively utilizing both limited human knowledge and large-scale AI capabilities.

Acknowledgements

This work is supported by the National Science Foundation (NSF) under grants IIS-2229461.