---
title: "Scalable Multi-Stage Influence Function for Large Language Models via Eigenvalue-Corrected Kronecker-Factored Parameterization"
authors:
  - "Yuntai Bao"
  - "Xuhong Zhang"
  - "Tianyu Du"
  - "Xinkui Zhao"
  - "Jiang Zong"
  - "Hao Peng"
  - "Jianwei Yin"
doi: "10.24963/ijcai.2025/892"
url: "https://ijcai.org/proceedings/2025/892"
pdf: "https://ijcai.org/proceedings/2025/0892.pdf"
venue: "Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence"
year: 2025
pages: "8022-8030"
track: "Main Track"
keywords:
  - "AI Ethics, Trust, Fairness: ETF: Explainability and interpretability"
  - "Natural Language Processing: NLP: Interpretability and analysis of models for NLP"
---

# Scalable Multi-Stage Influence Function for Large Language Models via Eigenvalue-Corrected Kronecker-Factored Parameterization

Yuntai Bao, Xuhong Zhang, Tianyu Du, Xinkui Zhao, Jiang Zong, Hao Peng, Jianwei Yin

*Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence*, pages 8022-8030. DOI: [10.24963/ijcai.2025/892](https://doi.org/10.24963/ijcai.2025/892)

## Abstract

Pre-trained large language models (LLMs) are commonly fine-tuned to adapt to downstream tasks. Since the majority of knowledge is acquired during pre-training, attributing the predictions of fine-tuned LLMs to their pre-training data may provide valuable insights. Influence functions have been proposed as a means to explain model predictions based on training data. However, existing approaches often fail to compute "multi-stage" influence and lack scalability to billion-scale LLMs.

In this paper, we propose multi-stage influence functions to attribute the downstream predictions of fine-tuned LLMs to pre-training data under the full-parameter fine-tuning paradigm. To enhance the efficiency and practicality of our multi-stage influence function, we leverage Eigenvalue-corrected Kronecker-Factored (EK-FAC) parameterization for efficient approximation.
Empirical results validate the superior scalability of EK-FAC approximation and the effectiveness of our multi-stage influence function. Additionally, case studies on a real-world LLM, dolly-v2-3b, demonstrate its interpretive power, with exemplars illustrating insights provided by multi-stage influence estimates.