{"type":"ScholarlyArticle","title":"Scalable Multi-Stage Influence Function for Large Language Models via Eigenvalue-Corrected Kronecker-Factored Parameterization","authors":[{"name":"Yuntai Bao","slug":"yuntai-bao","given":"Yuntai","family":"Bao","affiliation":""},{"name":"Xuhong Zhang","slug":"xuhong-zhang","given":"Xuhong","family":"Zhang","affiliation":""},{"name":"Tianyu Du","slug":"tianyu-du","given":"Tianyu","family":"Du","affiliation":""},{"name":"Xinkui Zhao","slug":"xinkui-zhao","given":"Xinkui","family":"Zhao","affiliation":""},{"name":"Jiang Zong","slug":"jiang-zong","given":"Jiang","family":"Zong","affiliation":""},{"name":"Hao Peng","slug":"hao-peng","given":"Hao","family":"Peng","affiliation":""},{"name":"Jianwei Yin","slug":"jianwei-yin","given":"Jianwei","family":"Yin","affiliation":""}],"abstract":"Pre-trained large language models (LLMs) are commonly fine-tuned to adapt to downstream tasks. Since the majority of knowledge is acquired during pre-training, attributing the predictions of fine-tuned LLMs to their pre-training data may provide valuable insights. Influence functions have been proposed as a means to explain model predictions based on training data. However, existing approaches often fail to compute \"multi-stage\" influence and lack scalability to billion-scale LLMs.\r\n\r\nIn this paper, we propose multi-stage influence functions to attribute the downstream predictions of fine-tuned LLMs to pre-training data under the full-parameter fine-tuning paradigm. To enhance the efficiency and practicality of our multi-stage influence function, we leverage Eigenvalue-corrected Kronecker-Factored (EK-FAC) parameterization for efficient approximation.\r\nEmpirical results validate the superior scalability of EK-FAC approximation and the effectiveness of our multi-stage influence function. Additionally, case studies on a real-world LLM, dolly-v2-3b, demonstrate its interpretive power, with exemplars illustrating insights provided by multi-stage influence estimates.","doi":"10.24963/ijcai.2025/892","url":"https://ijcai.org/proceedings/2025/892","pdf":"https://ijcai.org/proceedings/2025/0892.pdf","markdown":"https://ijcai.org/proceedings/2025/892.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/892","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Natural Language Processing","track":"Main Track","number":892,"pages":{"first":8022,"last":8030},"keywords":["AI Ethics, Trust, Fairness: ETF: Explainability and interpretability","Natural Language Processing: NLP: Interpretability and analysis of models for NLP"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}