{"type":"ScholarlyArticle","title":"Not All Layers of LLMs Are Necessary During Inference","authors":[{"name":"Siqi Fan","slug":"siqi-fan","given":"Siqi","family":"Fan","affiliation":""},{"name":"Xin Jiang","slug":"xin-jiang","given":"Xin","family":"Jiang","affiliation":""},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":""},{"name":"Xuying Meng","slug":"xuying-meng","given":"Xuying","family":"Meng","affiliation":""},{"name":"Peng Han","slug":"peng-han","given":"Peng","family":"Han","affiliation":""},{"name":"Shuo Shang","slug":"shuo-shang","given":"Shuo","family":"Shang","affiliation":""},{"name":"Aixin Sun","slug":"aixin-sun","given":"Aixin","family":"Sun","affiliation":""},{"name":"Yequan Wang","slug":"yequan-wang","given":"Yequan","family":"Wang","affiliation":""}],"abstract":"Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.","doi":"10.24963/ijcai.2025/566","url":"https://ijcai.org/proceedings/2025/566","pdf":"https://ijcai.org/proceedings/2025/0566.pdf","markdown":"https://ijcai.org/proceedings/2025/566.md","bibtex":"https://ijcai.org/proceedings/2025/bibtex/566","volume":{"slug":"2025","title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2025","url":"https://ijcai.org/proceedings/2025/","year":2025,"location":"Montreal, Canada","dates":"16-22 August 2025","editors":"James Kwok","isbn":"978-1-956792-06-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2025"},"section":"Machine Learning","track":"Main Track","number":566,"pages":{"first":5083,"last":5091},"keywords":["Natural Language Processing: NLP: Applications","Natural Language Processing: NLP: Question answering","Machine Learning: ML: Applications"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}