Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

Schwalbe, Gesina; Keser, Mert; Bayerkuhnlein, Moritz; Heinert, Edgar; Mütze, Annika; Keller, Marvin; Tiwari, Sparsh; Mikriukov, Georgii; Wolter, Diedrich; Lee, Jae Hee; Rottmann, Matthias

doi:10.24963/ijcai.2026/82

Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 724-734.

https://doi.org/10.24963/ijcai.2026/82

Keywords

Machine Learning: Clustering
AI Ethics, Trust, Fairnes: Explainability and interpretability
AI Ethics, Trust, Fairnes: Trustworthy AI
AI Ethics, Trust, Fairnes: Bias
Computer Vision: Structural and model-based approaches, knowledge representation and reasoning

Abstract

Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image–text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.

Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika Mütze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, and Matthias Rottmann. 2026. Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings. In Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26. IJCAI, 724–734. https://doi.org/10.24963/ijcai.2026/82

@inproceedings{ijcai2026p82,
  title     = {Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings},
  author    = {Schwalbe, Gesina and Keser, Mert and Bayerkuhnlein, Moritz and Heinert, Edgar and Mütze, Annika and Keller, Marvin and Tiwari, Sparsh and Mikriukov, Georgii and Wolter, Diedrich and Lee, Jae Hee and Rottmann, Matthias},
  booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-26}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Diego Calvanese},
  pages     = {724--734},
  year      = {2026},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2026/82},
  url       = {https://doi.org/10.24963/ijcai.2026/82},
}

TY  - CPAPER
TI  - Explaining, Verifying and Aligning Semantic Hierarchies in Vision-Language Model Embeddings
AU  - Schwalbe, Gesina
AU  - Keser, Mert
AU  - Bayerkuhnlein, Moritz
AU  - Heinert, Edgar
AU  - Mütze, Annika
AU  - Keller, Marvin
AU  - Tiwari, Sparsh
AU  - Mikriukov, Georgii
AU  - Wolter, Diedrich
AU  - Lee, Jae Hee
AU  - Rottmann, Matthias
T2  - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2026
SP  - 724
EP  - 734
DO  - 10.24963/ijcai.2026/82
UR  - https://doi.org/10.24963/ijcai.2026/82
ER  -