{"type":"ScholarlyArticle","title":"When Can We Trust Fairness Audits? Identifying Reliability Boundaries of Third-party Audit Conclusions","authors":[{"name":"Yuanhao Liu","slug":"yuanhao-liu","given":"Yuanhao","family":"Liu","affiliation":""},{"name":"Qi Cao","slug":"qi-cao","given":"Qi","family":"Cao","affiliation":""},{"name":"Huawei Shen","slug":"huawei-shen","given":"Huawei","family":"Shen","affiliation":""}],"abstract":"Fairness auditing aims to assess whether a model is fair, playing a critical role in identifying potential risks in deployed AI systems. In practice, due to limited access, third-party auditors often rely on self-collected datasets (e.g., via sock-puppets), which may differ from real-world deployment scenarios. Such discrepancy can lead to inconsistencies between audit conclusions on the collected data and those in actual deployment, raising concerns about the reliability of third-party audits. This motivates a critical question: When can we trust the fairness audit conclusion derived from third-party datasets? Answering this question is challenging, as the actual deployment distribution is typically inaccessible or unobservable.  To tackle this, we introduce the Consistency Radius, a metric that quantifies the maximum distribution shift under which an audit conclusion based on third-party dataset remain consistent. We further propose a convex relaxation optimization-based method to estimate the radius relying solely on model responses over the audit dataset. Leveraging this framework, third-party auditors can provide their datasets to model providers and request the magnitude of distributional discrepancy relative to the deployment distribution, enabling reliable audit conclusions without requiring any direct data access.","doi":"10.24963/ijcai.2026/814","url":"https://ijcai.org/proceedings/2026/814","pdf":"https://ijcai.org/proceedings/2026/0814.pdf","markdown":"https://ijcai.org/proceedings/2026/814.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/814","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"AI and Social Good","track":"AI and Social Good","number":814,"pages":{"first":7319,"last":7327},"keywords":["AI Ethics, Trust, Fairness: AI Ethics, Trust, Fairness","AI4G: AI Ethics, Trust, Fairness"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}