{"type":"ScholarlyArticle","title":"Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence","authors":[{"name":"Yuyin Lu","slug":"yuyin-lu","given":"Yuyin","family":"Lu","affiliation":""},{"name":"Ziran Liang","slug":"ziran-liang","given":"Ziran","family":"Liang","affiliation":""},{"name":"Yanghui Rao","slug":"yanghui-rao","given":"Yanghui","family":"Rao","affiliation":""},{"name":"Wenqi Fan","slug":"wenqi-fan","given":"Wenqi","family":"Fan","affiliation":""},{"name":"Fu Lee Wang","slug":"fu-lee-wang","given":"Fu Lee","family":"Wang","affiliation":""},{"name":"Qing Li","slug":"qing-li","given":"Qing","family":"Li","affiliation":""}],"abstract":"Reliable reasoning in Large Language Models (LLMs) is challenged by their propensity for hallucination. While augmenting LLMs with Knowledge Graphs (KGs) improves factual accuracy, existing KG-augmented methods fail to quantify epistemic uncertainty in both the retrieved evidence and LLMs' reasoning. To bridge this gap, we introduce DoublyCal, a framework built on a novel double‑calibration principle. DoublyCal employs a lightweight proxy model to first generate KG evidence alongside a calibrated evidence confidence. This calibrated supporting evidence then guides a black-box LLM, yielding final predictions that are not only more accurate but also well-calibrated, with confidence scores traceable to the uncertainty of the supporting evidence. Experiments on knowledge-intensive benchmarks show that DoublyCal significantly improves both the accuracy and confidence calibration of black-box LLMs while maintaining low token cost.","doi":"10.24963/ijcai.2026/77","url":"https://ijcai.org/proceedings/2026/77","pdf":"https://ijcai.org/proceedings/2026/0077.pdf","markdown":"https://ijcai.org/proceedings/2026/77.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/77","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"AI Ethics, Trust, Fairnes","track":"Main Track","number":77,"pages":{"first":679,"last":687},"keywords":["Knowledge Representation and Reasoning: Learning and reasoning","Knowledge Representation and Reasoning: Applications","AI Ethics, Trust, Fairnes: Trustworthy AI","Uncertainty in AI: Applications"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}