Proceedings

NaVQA: Mitigating Silent Failures in Question Answering over Virtual Knowledge Graph

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 4088-4096.

https://doi.org/10.24963/ijcai.2026/455

Abstract

Virtual Knowledge Graphs (VKGs) provide unified access to legacy relational data sources through a high-level ontology modeling a domain of interest. The content of the ontology elements (classes and properties) is virtually mapped to underlying data sources through declarative mappings. The standard approach to interacting with a VKG system is to use SPARQL as the query language. However, the complexity of SPARQL creates a high entry barrier for normal users. In this paper, we study the VKG-QA task, which enables users to interact with the VKGs through a natural language (NL) interface by translating their questions into SPARQL queries. One critical challenge in such translation is silent failures, where a syntactically correct SPARQL query returns empty results because it involves ontology elements that lack mappings to the underlying data. To address this, we propose NaVQA (Navigation-based VKG Question Answering), a framework leveraging Large Language Models that (1) identifies and indexes the active ontology elements based on the mapping specifications, and (2) iteratively constructs a query graph from the NL question using the active ontology elements and their relations to synthesize SPARQL. Experiments on three real-world VKGs show that NaVQA significantly mitigates silent failures and outperforms existing baselines.

  1. [Allemang and Sequeda, 2024] Dean Allemang and Juan Se- queda. Increasing the Accuracy of LLM Question- Answering Systems with Ontologies. In Proc. of ISWC, page 324-339, 2024.
  2. [Banerjee et al., 2023] Debayan Banerjee, Pranav Nair, et al. The role of output vocabulary in T2T LMs for SPARQL semantic parsing. In Findings of ACL, pages 12219- 12228, 2023.
  3. [Calvanese et al., 2016] Diego Calvanese, Benjamin Cogrel, Sarah Komla-Ebri, Roman Kontchakov, Davide Lanti, Martin Rezk, Mariano Rodriguez-Muro, and Guohui Xiao. Ontop: Answering SPARQL queries over relational databases. Semantic Web, 8(3):471-487, 2016.
  4. [Calvanese et al., 2021] Diego Calvanese, Davide Lanti, Tarcisio Mendes De Farias, Alessandro Mosca, and Guo- hui Xiao. Accessing scientific data through knowledge graphs with Ontop. Patterns, 2(10), 2021.
  5. [Chen et al., 2025a] Boyu Chen, Zirui Guo, et al. PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Paths. arXiv preprint arXiv:2502.14902, 2025.
  6. [Chen et al., 2025b] Chao Chen, Pengfei Luo, Changkai Feng, et al. MAPS: A Multi-task Framework with An- chor Point Sampling for Zero-shot Entity Linking. Data Intelligence, 7(4):1085-1107, 2025.
  7. [Ding et al., 2020] Linfang Ding, Guohui Xiao, Diego Cal- vanese, and Liqiu Meng. A framework uniting ontology- based geodata integration and geovisual analytics. ISPRS Int. J. Geo-Inf., 9(8):474, 2020.
  8. [Dong et al., 2023] Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, et al. C3: Zero-shot Text-to-SQL with ChatGPT. arXiv preprint arXiv:2307.07306, 2023.
  9. [D'Abramo et al., 2025] Jacopo D'Abramo, Andrea Zu- garini, and Paolo Torroni. Investigating Large Language Models for Text-to-SPARQL Generation. In Proceedings of KALM, pages 66-80, 2025.
  10. [Gao et al., 2024] Dawei Gao, Haibin Wang, Yaliang Li, Xi- unei Sun, Yanyang Qian, Bolin Ding, and Jingren Zhou. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endow- ment, 17(11):3103-3116, 2024.
  11. [Gu et al., 2021] Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond I.I.D.: Three Levels of Generalization for Question An- swering on Knowledge Bases. In Proceedings of the Web Conference 2021. ACM, 2021.
  12. [Huang et al., 2025] Zhisheng Huang, Xudong Jia, Tao Chen, and Zhongwei Zhang. A General Scalable Ap- proach for Knowledge Selection based on Iterative Hybrid Encoding and Re-ranking. Data Intelligence, 7(1):124- 142, 2025.
  13. [Jiang et al., 2025] Jinhao Jiang, Kun Zhou, et al. KG- Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph. 2025. [Kovriguina et al., 2023]
  14. Liubov Kovriguina, Roman Teucher, Daniil Radyush, and Dmitry Mouromtsev. SPARQLGEN: One-Shot Prompt-based Approach for SPARQL Query Generation. In SEMANTICS, 2023.
  15. [Lanti et al., 2015] Davide Lanti, Martin Ignacio Rezk, Guo- hui Xiao, and Diego Calvanese. The NPD benchmark: Re- ality check for OBDA systems. In Proc. of EDBT, pages 617-628, 2015.
  16. [Lenzerini, 2011] Maurizio Lenzerini. Ontology-based data management. In Proc. of CIKM, pages 5-6, 2011.
  17. [Li et al., 2024] Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, et al. CodeS: Towards Building Open-Source Lan- guage Models for Text-to-SQL. 2024.
  18. [Liu et al., 2025a] Runxuan Liu, Bei Luo, Jiaqi Li, Baoxin Wang, Ming Liu, et al. Ontology-Guided Reverse Think- ing Makes Large Language Models Stronger on Knowl- edge Graph Question Answering. In Proc. of ACL, pages 15269-15284, July 2025.
  19. [Liu et al., 2025b] Xinyu Liu, Shuyu Shen, et al. A Survey of Text-to-SQL in the Era of LLMs: Where Are We, and Where Are We Going? IEEE Transactions on Knowledge and Data Engineering, 2025.
  20. [Luo et al., 2024] Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning. In Proc. of ICLR'2024, 2024.
  21. [Luz and Finger, 2018] Fabiano Ferreira Luz and Marcelo Finger. Semantic Parsing Natural Language into SPARQL: Improving Target Language Representation with Neural Attention. arXiv preprint arXiv:1803.04329, 2018.
  22. [Meloni et al., 2025] Antonello Meloni, Diego Refor- giato Recupero, et al. Exploring Large Language Models for Scientific Question Answering via Natural Language to SPARQL Translation. ACM Transactions on Intelligent Systems and Technology, 2025.
  23. [Motik et al., 2012] Boris Motik, Bernardo Cuenca Grau, Ian Horrocks, et al. OWL 2 Web Ontology Language: Profiles. W3C Recommendation, World Wide Web Con- sortium, 2012.
  24. [OpenAI, 2025] OpenAI. GPT-5 System Card. System card, OpenAI, 2025.
  25. [Pan et al., 2025] Xueli Pan, Victor de Boer, and Jacco van Ossenbruggen. FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs. 2025. [Pourreza and Rafiei, 2023]
  26. Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: Decomposed In-Context learn- ing of Text-to-SQL with Self-Correction. In NeurIPS, vol- ume 36, 2023.
  27. [Qin et al., 2022] Bowen Qin, Binyuan Hui, et al. A survey on Text-to-SQL parsing: Concepts, methods, and future directions. arXiv preprint arXiv:2208.13629, 2022.
  28. [Rodriguez-Muro et al., 2013] Mariano Rodriguez-Muro, Roman Kontchakov, and Michael Zakharyaschev. Ontology-based data access: Ontop of databases. In Proc. of ISWC'2013, pages 558-573. Springer, 2013.
  29. [Sequeda et al., 2011] Juan Sequeda, Syed Ahmed Tirmizi, Oscar Corcho, et al. Survey of directly mapping SQL databases to the semantic web. The Knowledge Engineer- ing Review, 26(4):445-486, 2011.
  30. [Sequeda et al., 2024] Juan Sequeda, Dean Allemang, and Bryon Jacob. A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases. In Proc. of GRADES-NDA, pages 1-12, 2024.
  31. [Sun et al., 2024a] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, et al. Think-on-Graph: Deep and Responsible Rea- soning of Large Language Model on Knowledge Graph. In Proc. of ICLR'2024, 2024.
  32. [Sun et al., 2024b] Lei Sun, Zhengwei Tao, Youdi Li, and Hiroshi Arakawa. ODA: Observation-Driven Agent for in- tegrating LLMs and Knowledge Graphs. In Proc. of ACL, pages 7417-7431, 2024.
  33. [Sun et al., 2024c] Ruoxi Sun, Sercan Omer Arik, Alexan- dre Muzio, Lesly Miculicich, et al. SQL-PaLM: Im- proved Large Language Model Adaptation for Text-to- SQL. Transactions on Machine Learning Research, 2024.
  34. [Tian et al., 2024] Shiyu Tian, Yangyang Luo, et al. KG- Adapter: Enabling Knowledge Graph Integration in Large Language Models through Parameter-Efficient Fine- Tuning. In ACL Findings, pages 3813-3828, 2024.
  35. [Walter and Bast, 2025] Sebastian Walter and Hannah Bast. GRASP: Generic Reasoning And SPARQL Generation across Knowledge Graphs. ISWC (Posters, Demos & In- dustry Tracks), 2025.
  36. [Wang et al., 2024] Yu Wang, Nedim Lipka, Ryan Anthony Rossi, et al. Knowledge Graph Prompting for Multi- Document Question Answering. In AAAI, volume 38, pages 19206-19214, 2024.
  37. [Wang et al., 2025] Bing Wang, Changyu Ren, et al. MAC- SQL: A Multi-Agent Collaborative Framework for Text- to-SQL. In Proc. of COLING, pages 540-557, 2025.
  38. [Wei et al., 2022] Jason Wei, Xuezhi Wang, et al. Chain-of- thought Prompting Elicits Reasoning in Large Language Models. Advances in neural information processing sys- tems, 35:24824-24837, 2022.
  39. [Xiao et al., 2018] Guohui Xiao, Diego Calvanese, Roman Kontchakov, et al. Ontology-Based Data Access: A Sur- vey. In Proc. of IJCAI'2018, 2018. IJCAI
  40. [Xiao et al., 2019] Guohui Xiao, Linfang Ding, Benjamin Cogrel, and Diego Calvanese. Virtual Knowledge Graphs: An Overview of Systems and Use Cases. Data Intelli- gence, 1(3):201-223, 2019.
  41. [Xiao et al., 2025] Guohui Xiao, Diluxan Sathalingam, Al- bulen Pano, Yu Feng, and Linfang Ding. ATKIS-DLM- KG: a unified knowledge graph framework for integrating and querying fragmented topographic-cartographic data in the German digital landscape model. International Jour- nal of Digital Earth, 18(1):2528703, 2025.
  42. [Yang et al., 2025] An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 Technical Report, 2025.
  43. [Zahera et al., 2024] Hamada Zahera, Manzoor Ali, Mo- hamed Ahmed Sherif, Diego Moussallem, and Axel- Cyrille Ngonga Ngomo. Generating SPARQL from Nat- ural Language Using Chain-of-Thoughts Prompting. In SEMANTICS, pages 353-368, 2024.
  44. [Zhang et al., 2024] Zhiqiang Zhang, Liqiang Wen, and Wen Zhao. A GAIL Fine-Tuned LLM Enhanced Framework for Low-Resource Knowledge Graph Question Answering. In Proc. of CIKM'2024, pages 3300-3309, 2024.
  45. [Zhang et al., 2025] Yanzhao Zhang, Mingxin Li, et al. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. arXiv preprint arXiv:2506.05176, 2025.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards