---
title: "Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence"
authors:
  - "Yuyin Lu"
  - "Ziran Liang"
  - "Yanghui Rao"
  - "Wenqi Fan"
  - "Fu Lee Wang"
  - "Qing Li"
doi: "10.24963/ijcai.2026/77"
url: "https://ijcai.org/proceedings/2026/77"
pdf: "https://ijcai.org/proceedings/2026/0077.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "679-687"
track: "Main Track"
keywords:
  - "Knowledge Representation and Reasoning: Learning and reasoning"
  - "Knowledge Representation and Reasoning: Applications"
  - "AI Ethics, Trust, Fairnes: Trustworthy AI"
  - "Uncertainty in AI: Applications"
---

# Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence

Yuyin Lu, Ziran Liang, Yanghui Rao, Wenqi Fan, Fu Lee Wang, Qing Li

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 679-687. DOI: [10.24963/ijcai.2026/77](https://doi.org/10.24963/ijcai.2026/77)

## Abstract

Reliable reasoning in Large Language Models (LLMs) is challenged by their propensity for hallucination. While augmenting LLMs with Knowledge Graphs (KGs) improves factual accuracy, existing KG-augmented methods fail to quantify epistemic uncertainty in both the retrieved evidence and LLMs' reasoning. To bridge this gap, we introduce DoublyCal, a framework built on a novel double‑calibration principle. DoublyCal employs a lightweight proxy model to first generate KG evidence alongside a calibrated evidence confidence. This calibrated supporting evidence then guides a black-box LLM, yielding final predictions that are not only more accurate but also well-calibrated, with confidence scores traceable to the uncertainty of the supporting evidence. Experiments on knowledge-intensive benchmarks show that DoublyCal significantly improves both the accuracy and confidence calibration of black-box LLMs while maintaining low token cost.