Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

1 Introduction

Contrastive vision-language models (VLMs) such as CLIP Radford et al. [2021] demonstrate remarkable capabilities in learning joint representations of images and text [Barraco et al., 2022]. These representations are adopted in various applications, including image retrieval [Baldrati et al., 2022], captioning [Barraco et al., 2022], and zero-shot classification [Radford et al., 2021]. However, their evaluation is still dominated by task-level performance metrics such as classification accuracy that provide little insight into how concepts are internally organized and related. In particular, understanding the semantic hierarchies of these representations has not been thoroughly explored [Lee et al., 2025].

1 Find supplementary material at: https://arxiv.org/abs/2603.26798

2 Find code at: https://github.com/gesina/ontological-commitment

Semantic hierarchies capture how concepts relate via commonalities and differences, and are central to human categorization and symbolic knowledge representation via ontologies [Confalonieri and Guizzardi, 2025; Guarino, 1998]. Extracting the hierarchy induced by a VLM is crucial for explainability [Bhalla et al., 2024; Alper and Averbuch-Elor, 2025; Lee et al., 2025; Wan et al., 2020]. It further enables identifying unintuitive or biased groupings (e.g., grouping man and woman primarily via hair rather than under person) and aligning learned similarities with human knowledge [Ribeiro and Leite, 2021; Confalonieri and Guizzardi, 2025]. Lastly, hierarchies provide an inductive bias for more robust and informative classification [Bertinetto et al., 2020; Dhall et al., 2020; Ge et al., 2023], such as explainable hierarchical tree-traversal [Wan et al., 2020].

Meanwhile, key desiderata formulated for VLM embedding spaces are text-to-image alignment, and that classes can be discriminated via zero-shot classification. We complement this by two new desiderata: Text and image encoders should also align in their induced hierarchies; and they should be plausible, i.e., the hierarchy should match human taxonomies. Note that plausibility and discriminability may diverge, since embedding similarities may accurately discriminate class members, but still violate human expectations.

Into that direction, prior work has explored the interpretability of VLM embeddings [Bhalla et al., 2024] (Which concepts are encoded?) and found and reinforced pre-defined semantic hierarchies in them [Alper and Averbuch-Elor, 2025] (Is a given hierarchy encoded?). However, there is a lack of systematic methods to answer the following questions for a trained VLM encoder, along which we organize our pipeline (cf. Figure 1):

• Q1: Can we explain which semantic hierarchy is induced by an encoder for a given set of leaf classes? How faithful and reliable are local explanations via explainable hierarchical inference? (Secs. 4.1, 4.2 and 5.2)
• Q2: Can we verify the plausibility of induced hierarchies relative to reference ontologies? How do they vary across modalities, models, and hierarchy depths? (Secs. 4.3 and 5.3)
• Q3: Can we post-hoc align an induced hierarchy to a target similarity structure without sacrificing zero-shot accuracy? (Secs. 4.4 and 5.4)

Contributions. We demonstrate the utility of semantic hierarchy extraction:

• We propose a post-hoc pipeline to explain, verify, and align semantic hierarchies in VLM embedding spaces. We extract a hierarchy via hierarchical clustering of leaf classes and dictionary-based parent naming, verify it with efficient ontology-based comparison metrics, and align it using UMAP-guided target neighborhoods and a lightweight embedding transformation.
• We empirically study 13 pretrained VLMs across 4 datasets and multiple common-sense ontologies, enabling systematic comparison of encoder semantics.
• We find a consistent modality gap: image encoders yield higher zero-shot accuracy, while text encoders induce more plausible hierarchies; and enriching hierarchical inference with UAES improves semantic correctness, bridging the explainability-faithfulness trade-off.

2 RelatedWork

Explaining Learned Concepts. The field of concept-based XAI (c-XAI) has evolved around the goal to associate vectors in the latent representations with semantic concepts [Lee et al., 2025]. Such associations have found manifold applications in the inspection of a DNN's learned ontology, such as: which concepts from a given ontology are learned? [Ribeiro and Leite, 2021], how similar are pairs of concepts? [Fong and Vedaldi, 2018; Schwalbe, 2021], and how are concepts from different layers or models correlated? [Kim et al., 2018; Mikriukov et al., 2023]. Notably, we here build upon the approach by Yuksekgonul et al. [2022] to name parent embeddings. They utilize the text-to-image alignment in VLMs to match vectors to the closest text embeddings from a dictionary. To go from concepts to hierarchical concept relations, Wan et al. [2020] employed hierarchical clustering on latent representations, but unlike our approach restricted to the last layer of a classifier. And despite rising interest in verification and steering of DNNs against ontologies [Ribeiro and Leite, 2021; Ponomarev and Agafonov, 2022; Badreddine et al., 2022; Alper and Averbuch-Elor, 2025], it remains largely unexplored how to post-hoc explain and control the learned semantic similarity structure within DNN embedding spaces [Confalonieri and Guizzardi, 2025].

Understanding VLM Embedding Spaces. The CLIPbased learning scheme [Radford et al., 2021; Jia et al., 2021] of concurrent VLMs have been shown to emerge deep concept understanding [Bhalla et al., 2024; Zhao et al., 2025], and even class hierarchies [Alper and Averbuch-Elor, 2025; Ge et al., 2023]-though existing hierarchy extraction techniques rely on costly parent supervision and don't align with ontology concept dictionaries. Moreover, previous findings that integrating hierarchy information into the inference process makes classification more explainable [Wan et al., 2020] and reliable [Bertinetto et al., 2020; Dhall et al., 2020], were shown to transfer to VLMs [Novack et al., 2023]. We therefore adapt the hierarchy tree-traversal approach fromWan et al. [2020] for VLM classification, newly adding early stopping. Despite these impressive properties, the modality alignment in CLIP embedding spaces remains suboptimal, mapping text and image to distinct latent regions [Liang et al., 2022; Eslami and de Melo, 2024], and requiring prompt finetuning to boost zero-shot performance [Radford et al., 2021; Ma et al., 2025; Ge et al., 2023]. Other than prior studies, we here for the first time investigate the text-to-image alignment with respect to the primarily learned semantic hierarchies of the encoders.

Ontologies and OntologyMatching. An ontology reflects a perspective on a domain [Guarino, 1998], formalized as concepts, relations, and assumptions thereon [Guarino, 1998]. By now a variety of both specialized and general domain ontologies are available [Confalonieri and Guizzardi, 2025; Mascardi et al., 2007]. The latter include closed-source ones [Lenat, 1989; Forbus and Hinrich, 2017], large-scale but less principled ones like ConceptNet [Speer et al., 2017] or DBPedia [Lehmann et al., 2015]. We here use the well-established, large, general domain ontologies SUMO [Niles and Pease, 2001], OpenCyc [Matuszek et al., 2006], and Yago [Mahdisoltani et al., 2015] as reference for human knowledge on plausible hierarchies. Ontology matching techniques aim to compare or integrate divergent ontologies [Otero-Cerdeira et al., 2015]. A common distance measure for graphs is edit distance, which transforms one instance into another via minimal edit operations [Zhang and Shasha, 1989; Akutsu, 2010]. For our special case of semantic hierarchy trees, tree edit distance measures can be applied [Zhang et al., 1992]. Aside from this global comparison, we here also aggregate local edge-wise scores based on matching lowest common ancestors (LCAs) [Bender et al., 2005]. However, existing works have not yet used ontology matching techniques to assess DNN-learned ontologies.

Figure 2: Running toy example on CIFAR-10 leaf classes.

3 Background

The goal of our approach is to discover a learned semantic hierarchy, constituted of latent embeddings of parent concepts. In the following we formalize semantic hierarchies, and recap the classical techniques for concept discovery as well as underlying key assumptions for hierarchy extraction.

3.1 Ontologies

A semantic hierarchy is a slice of a broader ontology.

Definition 1 (Ontology). An ontology O is a description of a domain by a set of class membership functions (the concepts), instance relations, and a logic theory of assumptions thereon. [Guarino, 1998]

Semantic similarity is an example for relations between instances. This paper only considers taxonomic is-a assumptions on concepts (hypernymy/subclass relations [Fellbaum, 2010]) for verification and alignment. We denote these as IsParentOf(P, C) := (∀v : C(v) ⇒ P(v))), meaning that instances of C are also instances of P (e.g., cat ⇒ animal). A semantic hierarchy is a tree formed by such relations; our goal is to extract such a hierarchy from a VLM's embedding space.

3.2 Assumed Properties of VLMLatent Spaces

We rely on a few high-level assumptions that are standard in concept-based interpretability [Lee et al., 2025], where often single vectors are used as prototypes for concepts [Kim et al., 2018; Fong and Vedaldi, 2018] (Ass. 1); and in the typical VLM training and zero-shot inference schemes which employ cosine similarity as a proxy for semantic similarity of latent representations (Ass. 2). These assumptions are formalized in the following.

First, we simplify the representation of a visual concept from its cluster of concept instance embeddings to a point in embedding space.

Assumption 1 (Concept Embeddability). A humaninterpretable visual concept C can be represented by a point e(C), its concept embedding, in the VLM embedding space [Kim et al., 2018; Fong and Vedaldi, 2018].

Notable work introducing and validating this modeling scheme of concepts as single vectors were the concept activation vectors by Kim et al. [2018] and concept embeddings by Fong et al. [2018]. More recently, Mikriukov et al. [2025] confirmed that despite the more distributed nature of concept instance representations, single vectors are usually a good approximation.

Second, cosine similarity between concept embeddings is a reasonable proxy for semantic relatedness, which is our key to find parent embeddings. This also is the same assumption used by zero-shot classification where we assign x to the closest concept embedding under cosine distance d(u, v) :=

1- u·vT

∥u∥·∥v∥ in embedding space [Radford et al., 2021].

Assumption 2 (Embedding Interpolatability). A concept C is more similar to C′ than to C′′ iff d(e(C), e(C′ )) ≤ d(e(C), e(C′′ )) [Fong and Vedaldi, 2018; Ghorbani et al., 2019; Posada-Moreno et al., 2024].

This interpolatability property was empirically confirmed for concept embeddings both in language [Mikolov et al., 2013] and vision model latent spaces [Fong and Vedaldi, 2018; Ghorbani et al., 2019; Posada-Moreno et al., 2024].

If cosine similarity is used to assign an instance's embedding v to a concept C (C(v) from above), a parent concept for given children concepts is the one most similar to all instances of its children simultaneously. Hence, a direct consequence of Ass. 2 is to assume that DNNs learn hierarchies in terms of similarity structures: Concept vectors of parent concepts can be approximated by aggregating their children's concept vectors (e.g., by averaging [Mikriukov et al., 2025; Wan et al., 2020]), which motivates our use of agglomerative clustering.

4 Methodology

This section introduces our pipeline to explain the semantic hierarchy induced by a VLM as binary tree of concepts (Sec. 4.1) and via enhanced explainable hierarchical classification (Sec. 4.2), which allows to verify the VLM plausibility against references (Sec. 4.3), and semantically align its embedding space to a target similarity structure (Sec. 4.4).

Example 1. To make the pipeline concrete, we use the CIFAR10 leaf classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck) as a running example throughout the paper. Figure 2a shows two hierarchies extracted for the same ten leaves from SigLIP, and foreshadows the modality-dependent differences we quantify later (Sec. 5.3).

Experimental Setup and Notation. We assume a trained VLM encoder F and a dataset of leaf classes (e.g., CIFAR-10 [Alex, 2009]) with train/test splits. Leaf concepts can be represented via images, text descriptions (e.g., LLM-generated prompts [Ma et al., 2025]), or both. We further assume a concept bank C of candidate parent concepts, also with text descriptions, that map to a given reference ontology O. For C we here consider the large-scale lexical database WordNet [Fellbaum, 2010], which conveniently maps to many general domain ontologies like SUMO [Niles and Pease, 2001].

4.1 Hierarchy Extraction

We aim to extract a labeled binary tree T that summarizes in a human-inspectable manner how the encoder groups the given leaf classes in embedding space. Leaves correspond to the leaf dataset classes; each internal node represents an extracted parent concept from the concept bank together with an embedding. Edges represent predicted IsParentOf relations (cf. Figure 2a).

Example 2 (Parent Relations). In Figure 2a, image-only leaf embeddings group bird and deer under avifaunal as captured by the extracted binary tree structure.

Approach. We build T in three steps, as shown in Algorithm 1. First, we compute a centroid embedding e(C) for each leaf class C from embeddings of train samples for C (image, text, or both). Second, we run agglomerative hierarchical clustering [Ward Jr., 1963] on the leaf centroids using cosine similarity and average linkage; each merge creates a parent node whose embedding is the normalized mean of its children's embeddings. Third, we name internal nodes by matching each parent embedding p to the closest concept-bank embedding e(˜P), using a one-to-one matching (linear sum assignment [Crouse, 2016], LSA) to optimally avoid duplicate names.

Rationale. Intuitively, each concept C is defined by the set of samples x to which C applies. Via F this translates to a cluster of embeddings F(x); for leaf concepts we approximate this cluster using the embeddings of their training samples. By definition, a parent concept P should cover the cluster of each of its children concepts C: If for all x holds C(x) =⇒ P(x), Algorithm 1 Hierarchy extraction and parent naming Require: encoder F, leaf concepts {Ci}ki=1 with specification sets Spec(Ci) (image, text, or both), concept-bank descriptions Descs, cosine distance d

{1. Obtain normalized leaf concept centroids}

for i = 1, . . . , k do

end for

{2. Recover hierarchy using cosine average linkage} T ← AgglomerativeClustering(

{e(Ci)}ki=1 , metric = d, linkage = average)

for each internal node P of T in bottom-up order do

end for

ParentEmbs ← {e(P) | P ∈ InternalNodes(T )}

{3. Name parent embeddings by one-to-one matching} e(desc) ← F (desc)/∥F (desc)∥ for each desc ∈ Descs assignment cost Lassign : (p, desc) ↦→ d(p, e(desc)) A ← LSA(ParentEmbs, Descs, Lassign)

return tree T with node embeddings e(·) and parent name assignment A

then {x | C(x)} ⊂ {x | P(x)}. This motivates iteratively merging the most similar clusters, as achieved by the agglomerative hierarchical clustering. Choosing cosine similarity reflects the objective used in CLIP training, differing from Euclidean-linkage choices in prior work on clustering concept embeddings in neural network latent spaces [Wan et al., 2020; Mikriukov et al., 2025]. Intuitively, a good parent embedding for a parent cluster should minimize the distances to all of its children equally; for cosine distance on normalized vectors, the mean of children centroids fulfills this, which we assume to be the children's concept embeddings (Ass. 2):

Averaging additionally robustifies the assignment of concept representatives and robustness increases with the number of samples in the leaf class datasets. Since clustering is applied to those robust means, it naturally becomes robust against fluctuations in the leaf datasets with more samples. Empirically we at most observed swaps between nearest and second-nearest siblings, as confirmed in qualitative UMAP visualizations (refer to supplementary for details).

4.2 Explainable and Faithful Tree-Traversal Inference with UAES

Standard zero-shot classification of a sample x directly picks the leaf class with the most similar embedding to F(x). Wan et al. [2020] instead suggest to use a hierarchy tree of embeddings for inference via tree-traversal: One hierarchically traverses the tree from root to leaf, at each node picking the child with the closest embedding to F(x). This results in a more explainable classification, since the full path of specialization is available (instead of just cat, "animal → mammal → cat"). Interestingly, predictions don't necessarily coincide with zero-shot classification: if a leaf lies at the boundary of its superclass P, the centroid of a sibling P′ may be closer to F(x), so traversal proceeds into the subtree of P′ (cf. Figure 2b). From a semantic perspective this can be desirable: assuming interpolatability, a parent covers the area between its children, thereby preserving abstraction for underrepresented superclasses.

Example 3. As illustrated in Figure 2b, in a hierarchy of many animal and few vehicle subclasses, tree-traversal can still correctly consider coach a vehicle instead of an outlier animal (horse).

Nevertheless, a deep traversal into less matching subclasses should be avoided. Conveniently, the calculated similarities at each node may serve as a certainty score that traversal down that node is reasonable. We suggest to stop early at a parent node if none of its children would be sufficiently certain, returning the more reliable parent as prediction instead of a wild guess of a leaf class, as specified in Algorithm 2.

Example 4 (UAES). In Figure 2b, vehicle should be returned when traversal is uncertain between subclasses motor vehicle and boat-like). Early stopping enables more reliable classification both of unknown classes (coach) as well as known but erroneously classified ones.

We estimate the cut-off threshold for a node at the pth quantile (here: 0.01) of similarities observed in the training set. The ideal stopping point is the first node where a "wrong turn" would be taken-the lowest common ancestor (LCA) of the ground truth and the leaf node predicted by vanilla tree-traversal. The average walking distance of a prediction to that LCA gives an intuitive measure how much reliability early stopping recovers, here abbreviated as last correct node distance.

Measuring Hierarchy Faithfulness. While vanilla treetraversal could theoretically surpass zero-shot performance, this is not achieved even on simpler embedding spaces [Wan et al., 2020]. Reasons can be: unfavorable multi-modal, scattered or overlapping concept representations in embedding space violating our assumptions [Mikriukov et al., 2025], or unequal covariances (≈ widths) of sibling class clusters as in unbalanced trees. Hence, we follow the XAI faithfulness notion used by Wan et al. [2020], which answers how well hierarchical-interpretation predictions coincide with vanilla zero-shot ones. They operationalize it by interpreting the ratio tree inference accuracy zero-shot accuracy as a proxy measure how faithfully the hierarchy captures the internal similarity structure. To capture how deeply rooted category confusions are, we suggest soft tree inference accuracy (the average fraction of the correct root-to-leaf path taken during traversal).

4.3 Verifying Plausibility of Induced Hierarchies

Since internal nodes are named and the structure is a tree, extracted hierarchies support both qualitative inspection and quantitative comparison to a reference hierarchy. We assume a

Algorithm 2 Tree-traversal with UAES

Require: encoder F, instance x, hierarchy tree T = (V,E, Proot), embeddings e(C) for all C ∈ V, node-wise UAES similarity thresholds (τP)P∈V, cosine distance d embedding v ← F (x) {Traverse while the best child remains sufficiently certain} current parent P ← Proot instanceHierarchy ← [Proot ] while P is no leaf do best child C∗ ← arg minC∈Children(P) d(v, e(C)) best child distance d∗child ← d(v, e(C∗ )) {Uncertainty-aware Early Stopping (UAES)} if d∗child > τP then break end if {Traverse down to best child} P ← C∗ append P to instanceHierarchy end while return prediction Ftree-traversed(x) = P, instanceHierarchy

target hierarchy T ′ represented as a directed acyclic graph, together with a mapping M from concept-bank entries to nodes in T ′ . The target T ′ can be a taxonomic subset of a humandefined ontology O or the hierarchy of a different encoder. We introduce two ontology-based verification metrics: (i) the distance to the closest valid tree extracted from the ontology (global fit), and (ii) a hierarchical consistency score Sonto that checks parent-child edges for matching ontology paths (local fit). Let dT be the integer walking distance between nodes in the underlying undirected graph of T .

Globally Verifying against the Closest Valid Trees. We consider a hierarchy plausible with respect to O, if it is a slice thereof. To quantify how far an extracted hierarchy T is from being plausible, we search for the best matching T ′ ⊂ O.

Definition 2. Let dtree be a tree comparison metric [Akutsu, 2010], like Tree Edit Distance [Zhang and Shasha, 1989] or Tree Alignment [Jiang et al., 1995]. The closest valid tree to T (without renaming) with respect to ontology O and dtree is the subtree

Tvalid can serve as a minimal target to optimize for plausibility, and dtree(T, Tvalid) as global (yet, costly to obtain) plausibility measure.

Hierarchical Consistency Score. This metric efficiently locally verifies that parent-child edges in the T respect the hypernymy constraints of the reference ontology O. We ground each parent node u to a representative ontology concept

For each edge u → v, let δ(ρ(u), ρ(v)) be the shortest directed path length in O. We define Sedge(u → v) = 0 if no path exists, else Sedge(u → v) = min(1, (γ · δ(ρ(u), ρ(v)))-1 ).

The factor γ = 0.5 is adapted from path-length normalization principles [Leacock, 1998] and allows to tolerate common onelevel skips. We then aggregate over edges E in T to obtain Hierarchical Consistency

Example 5 (Local Hierarchical Consistency). In Figure 2a, the edge animal → canine only skips mammal and receives high Sedge = 1, while Sedge(canine → equine) = 0.

4.4 Aligning Induced to Target Hierarchies

The last part of our pipeline aims to post-hoc align the observed similarity structure T of the VLM encoder to a target tree T ′ , like the closest valid tree or a different encoder's semantics. We here suggest an efficient post-hoc, model-agnostic approach that only learns a transformation for the embedding space. Thereby the loss function accounts for preserving the original similarity structure such that the transformation minimally affects zero-shot performance on the given leaf classes Ingredients are the tree structure and leaf labels of T ′ , and a training and test split of the leaf concept dataset. Then, a transformation is obtained in two steps:

1. Determine suitable target latent points to which the instances should be mapped, in order to fulfill the target similarity structure.
2. Train a transformation of the embedding space, here a 2layer DNN, on the pairs of original and targeted instance embeddings.

This modularization allows to use efficient manifold learning techniques like UMAP [McInnes et al., 2020] in the first step. Via an optimization procedure UMAP finds embeddings of a set of original points in a target space, such that the embeddings match targeted pairwise point distances. This is typically used to map points into a lower-dimensional space, whilst targeting the pairwise distances from the original space to preserve neighborhoods. We instead map points into the same space, and want to change neighborhoods to match those of T ′ . Hence, we initialize the UMAP mapping with the identity, and sum pairwise distances between points x, x′ of concepts C, C' that (αorig) anchor to the original cosine geometry, (βonto) drive pairwise embedding distances to align with the class distances in T ′ , and (γmidp) determine how much the original class distances should be reduced. Combining the original cosine geometry and the class distances prevents collapsing class representations, i.e.,

for cosine distance d, the undirected tree walking distance dT ′ in T ′ , the normalization constant Z :=maxdT ′/maxC1,C2 d(e(C1), e(C2)), and scalar parameters αorig, βonto and γmidp. It is a combination of maintaining as much as possible of the original distances between instances within a class representation, whilst driving the embedding distances to match the length of the shortest path between the classes in T ′ . During training, the second term is normalized by the maximum path distance and maximum class embedding distance in the dataset. Given these target distances demb(x, x′ ), UMAP creates a dataset of pairs each mapping an original embedding of a probing sample to a transformed counterpart embedding vector. The pairwise distances between the transformed embeddings are encouraged to match the target distances. Since the mapping is only valid on the probing samples, the second step then trains a continuous function to extend this mapping to the complete embedding space. This can be applied before zero-shot classification on the latent space to align representations closer with a target class hierarchy.

5 Experiments

We conduct an empirical study to investigate faithfulness and utility of the extracted hierarchies for tree-traversal (Sec. 5.2); compare plausibility of hierarchies across modalities and models (Sec. 5.3), and to measure how easily embedding spaces can be post-hoc aligned to target hierarchies (Sec. 5.4).

The supplementary provides further details on the setup, ablation study, tables with numerical results of all plots, hypothesis tests (p-values are < 10-5 if not reported).

5.1 Experimental Settings

A hyperparameter study selected WordNet-based over Llama3based or combined descriptions for generating concept bank embeddings as optimal (see supplementary). WordNet is used for the general, WordNet+Llama3 for the specialized CUB dataset. The following settings apply to the total of 156 main experiments.

Models. We consider in total 13 pre-trained foundational VLMs from 4 families [Miller et al., 2024]: CLIP (C) with 9 different backbones [Radford et al., 2021], ALIGN [Jia et al., 2021], FLAVA [Singh et al., 2022], and SigLIP (v2) with 224px and 384px backbones [Zhai et al., 2023].

Leaf Class Datasets. We use CIFAR-10/100 [Alex, 2009] and ImageNet [Deng et al., 2009] (10/100/1000 leaves) as standard datasets of increasing hierarchy complexity, and CUB [Wah et al., 2011] as a specialized 200-class bird dataset. The train split is used for hierarchy extraction and alignment training; the test split is used to assess faithfulness and plausibility.

Ontologies and Concept Bank. We use the WordNet lexical database [Fellbaum, 2010] as a concept bank and SUMO [Niles and Pease, 2001], OpenCyc [Matuszek et al., 2006], and Yago [Mahdisoltani et al., 2015] as reference general domain ontologies, linked via existing mappings. We include nouns and adjectives, and treat Subclass and InstanceOf as parent relations.

Metrics. We measure discriminability as zero-shot performance (ZS, ↑), faithfulness via our faithfulness ratio, (soft) tree-traversal inference (tree / tree soft, ↑) and distance to the last correct node (↓), plausibility using our hierarchical consistency score (↑), and normalized constrained Unordered Tree Edit Distance (nUTED, ↓) [Zhang, 1996] to obtain the closest valid tree. The latter is the minimal number of deletion/insertion/leaf rename operations to turn one tree into the other, normalized to [0, 1] by the operation number of the trivial delete-all-insert-all solution [Rico-Juan and Mic´o, 2003]. Two randomly constructed binary trees for 1000 leaf nodes have a nUTED of approximately 0.54 (see supplementary).

5.2 Explaining Hierarchies: Faithfulness

A qualitative global explanation of VLM hierarchies is demonstrated in Figure 2a. To quantitatively assess explanation quality in terms of faithfulness, we use the (relative) performance of local explanations via tree-traversal inference as a proxy. We conducted an ablation study across different leaf set complexities, VLMmodel architectures, and the chosen encoder (text, image, mean of both).

More Leaves Diminish Faithfulness. The dominating effect is that a higher number of leaf nodes considerably decreases faithfulness as shown in Table 1: average faithfulness drops from 94.2±6.8 on CIFAR-10 (10 classes) to 26±15 on ImageNet (1000 classes). Similar decrease in soft tree-traversal performance (89 to 38) indicates that tree-traversal inference takes a wrong turn more easily in deeper trees. While the effect is smaller on image-embedded leaves, it is still significant. As a first pathway, this motivates use of our suggested early stopping.

Early Stopping Improves Hierarchical Inference. We evaluate whether uncertainty-aware early stopping (UAES) can correctly capture wrong turns in hierarchical inference. Table 2 reports the undirected tree distance between the predicted output (leaf or early-stopped internal node) and the last correct node, defined as the LCA of the ground-truth leaf and the leaf predicted by vanilla traversal. Early stopping reduces this distance on the larger label sets (CIFAR100/CUB/ImageNet), which means that it often returns semantically appropriate super-categories instead of committing to an incorrect leaf; on CIFAR10 the effect is negligible. Notably, measuring the distance to the ground truth node not in the tree but in the reference ontology SUMO has no significant effect. This validates that early stopping does not change the quality of the semantic alignment against a target ontology, which we inspect in detail next.

5.3 Agreement with Human Ontologies

We quantify plausibility by comparing extracted hierarchies against human reference ontologies (OpenCyc, SUMO, and Yago), using the hierarchical consistency metric introduced in Sec. 4.3. Our evaluation is driven along our desiderata: Can discriminability and plausibility be achieved simultaneously? And does the text-to-image alignment in CLIP training also induce semantically aligned hierarchies?

Plausibility-Discriminability Trade-off. When considering per-model metrics, zero-shot performance has little predictive value about the plausibility, as visualized in Figure 3. More interestingly zero-shot performance and faithfulness are both significantly negatively correlated with hierarchical consistency across all ontologies (for zero-shot and SUMO/OpenCyc/Yago: p-values < 0.006 and Pearson r < -0.45/-0.21/-0.29; for faithfulness: Pearson r< -0.62/-0.18/-0.61 and p-value < 0.023 for OpenCyc, else 0). Hence, there seems to be a consistent trade-off between discriminability of leaf classes, and plausibility of the induced semantic structure.

Modality Gap: Text versus Image Hierarchies. As shown in Figure 4, image leaf embeddings achieve significantly higher zero-shot and faithfulness scores compared to text leaf embeddings (+22 ± 13 percent points), which get instead consistently higher plausibility values (+0.015 ± 0.032, pvalue< 0.002), see Figure 4. This is confirmed by the significantly different trees, measured via average nUTED between trees when switching from image to text leaf embeddings (see Figure 5). Combining text and image information when embed-

• (b) Faithfulness metrics averaged over model, dataset, and encoder.

Figure 5: nUTED (↓) between trees induced by image and text encoders of each the same VLM; averaged over standard datasets. nUTED> 0.5 marks worse than random.

ding leaves offers a useful trade-off between discriminability and plausibility (cf. Figure 2a and Figure 4).

We then ask whether these trends hold consistently across model families and backbones.

Across-Model Comparison. Across VLM families, we observe substantial variation in both faithfulness and ontological plausibility. Figure 3 summarizes these trends and highlights that higher zero-shot accuracy does not necessarily imply a hierarchy that is more consistent with human ontologies.

In the next section we evaluate in how far our alignment procedure also confirms this trade-off between faithfulness and plausibility.

5.4 Post-hoc Alignment to Target Hierarchies

We here investigate whether learned ontologies can successfully be post-hoc corrected, without destroying zero-shot accuracy. For this we consider the following tasks:

• Validation: Swap two leaves in the semantic tree.
• Modality alignment: Turn a tree for image leaf embeddings into the corresponding one for text embeddings.
• Ontological commitment: Steer to the closest valid tree.

We measure proximity of a current to a target ontology by turning them into their distance matrices and taking the Frobenius norm of their difference. ∆O is defined as the relative change in distance to the target from after manipulation to before. ZS Orig text , ZS Orig midp , ZS UMAP text and ZS UMAP midp denote zero-shot accuracy before (Orig) and after manipulation (UMAP) and base on (transformed) text embeddings and class midpoints. For this study, we investigate four models, ALIGN, CLIP with ViTL14@336px backbone, SigLIP-224 and FLAVA (as specified before), on the validation task and run experiments on 2, 500 random training images and evaluate on 10, 000 test images of the CIFAR10 dataset. Averaged over three seeds, 10 random swaps and all four models, according to our hyperparameter grid-search the parameter setting αorig = 2.0, βonto = 1.0, γmidp = 2.0, NUMAP = 250, minimizes a combined evaluation score ∆O - ZS UMAP midp , which balances ontology alignment and zero-shot performance.

Results for the best parameter setting are presented in Table 3. The general finding is that our approach can cheaply initialize transformations from one latent ontology to another, being capable to fully align the original clip embedding with an embedding where automotive and frog have been swapped. There is, however, a trade-off between zero-shot accuracy and successful ontology matching, which reveals how much alignment and class discriminability compete. A comparison against a linear transformation baseline shows how far the true transformation is from being (simply) linear (see supplementary). Results for steering of class hierarchies towards their text embeddings and closest valid trees confirm our findings and can be found in the supplementary as well as the details on the hyperparameter study.

6 Discussion and Outlook

Our experiments revealed a remaining faithfulness gap of our explainable surrogates to the true model behavior. Future work should close this gap further, e.g., by shifting to distribution-based [Mikriukov et al., 2025; Lee et al., 2025] and location-aware [Fong and Vedaldi, 2018] concept representations, and removing any background bias in the probing images [Schwalbe et al., 2026]. Nevertheless, it should be noted that even imperfect XAI surrogates remain useful and are widely used for knowledge and test-case discovery [Ribeiro et al., 2016].

Interesting extensions of our technique would be concept combinations like black cat, e.g., using basis decomposition [Zhang et al., 2021]; towards arbitrary vision models, e.g., using locked-image text approaches [Zhai et al., 2022]; comparing against specialized ontologies; more types of assumptions [Speer et al., 2017] and arbitrary graph structures; and ante-hoc ontology insertion during training.

Beyond applications to bias removal and model selection, we expect that our technique can guide specialization of VLMs using domain specific ontologies, including search for appropriate context prompts for a specific domain. Furthermore, relaxing the strong explainability assumptions on concept distributions in latent space could not only inform concept entanglement analysis, but also improve UAES and thus semantic uncertainty-informed VLM zero-shot inference. Further applications could invert the here proposed analysis and use a VLM to validate human ontologies: E.g., comparing human-defined ontologies with respect to their coverage of VLM knowledge, and expanding or correcting human-defined ontologies from DNN-learned knowledge.

7 Conclusion

This paper presented a principled approach to extract the learned class hierarchy induced by a given set of leaf classes from a VLM. Furthermore, we introduced techniques to measure and improve faithfulness and plausibility, where the latter makes direct use of existing human-defined knowledge bases. We generally could uncover a mismatch between class hierarchies learned by the text and by the image encoders. Our suggested method to post-hoc train ontology manipulating transformations was able to improve the matching of a learned ontology towards a desired one, with only moderate loss on zero-shot accuracy. This suggests that human-interpretable knowledge can directly be used to align VLM representations with human expectations, and potentially help in closing the gap in text-to-image alignment.

Ethical Statement

There are no ethical issues.

Acknowledgments

G.S. and S.T. acknowledge support through the project "chAI" funded by the German Federal Ministry of Research, Technology and Space (BMFTR), grant no. 16IS24058. A.M. and M.R. gratefully acknowledge financial support from the BMFTR

within the project "REFRAME" (grant no. 16IS24073C). E.H., A.M. and M.R. also gratefully acknowledge support through the BMFTR junior research group "UnrEAL" (grant no. 16IS22069). J.H.L. was supported by the German Research Foundation (DFG), project no. 551629603. M.K. acknowledges support through the project "NXT GEN AI METHODS - Generative Methoden f¨ur Perzeption, Pr¨adiktion und Planung" funded by the German Federal Ministry for Economic Affairs and Energy. The authors are solely responsible for the content of this publication.

Contribution Statement

Authors Jae Hee Lee and Matthias Rottman contributed equally in advising.