---
title: "Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings"
authors:
  - "Gesina Schwalbe"
  - "Mert Keser"
  - "Moritz Bayerkuhnlein"
  - "Edgar Heinert"
  - "Annika Mütze"
  - "Marvin Keller"
  - "Sparsh Tiwari"
  - "Georgii Mikriukov"
  - "Diedrich Wolter"
  - "Jae Hee Lee"
  - "Matthias Rottmann"
doi: "10.24963/ijcai.2026/82"
url: "https://ijcai.org/proceedings/2026/82"
pdf: "https://ijcai.org/proceedings/2026/0082.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "724-734"
track: "Main Track"
keywords:
  - "Machine Learning: Clustering"
  - "AI Ethics, Trust, Fairnes: Explainability and interpretability"
  - "AI Ethics, Trust, Fairnes: Trustworthy AI"
  - "AI Ethics, Trust, Fairnes: Bias"
  - "Computer Vision: Structural and model-based approaches, knowledge representation and reasoning"
---

# Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 724-734. DOI: [10.24963/ijcai.2026/82](https://doi.org/10.24963/ijcai.2026/82)

## Abstract

Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image–text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.