---
title: "Toward Preference-aligned Large Language Models via Residual-based Model Steering"
authors:
  - "Lucio La Cava"
  - "Andrea Tagarelli"
doi: "10.24963/ijcai.2026/642"
url: "https://ijcai.org/proceedings/2026/642"
pdf: "https://ijcai.org/proceedings/2026/0642.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "5766-5774"
track: "Main Track"
keywords:
  - "Natural Language Processing: Interpretability and analysis of models for NLP"
  - "Natural Language Processing: Other"
  - "Natural Language Processing: Language models"
---

# Toward Preference-aligned Large Language Models via Residual-based Model Steering

Lucio La Cava, Andrea Tagarelli

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 5766-5774. DOI: [10.24963/ijcai.2026/642](https://doi.org/10.24963/ijcai.2026/642)

## Abstract

Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences. Existing approaches such as Reinforcement Learning from Human Feedback or Direct Preference Optimization typically require curated data and expensive optimization   over billions of parameters, and eventually lead   to persistent task-specific models. In this work, we introduce Preference alignment of Large Language Models via Residual Steering (PaLRS), a training-free method that exploits preference signals encoded in the residual streams of LLMs. From as few as one hundred preference pairs, PaLRS extracts lightweight, plug-and-play steering vectors that can be applied at inference time to push models toward preferred behaviors. We evaluate PaLRS on various small-to-medium-scale  open-source LLMs, showing that PaLRS-aligned models achieve consistent gains on mathematical reasoning and code generation benchmarks while preserving baseline general-purpose performance. Moreover, when compared to models aligned with DPO and SimPO, they perform better with great time savings. Our findings highlight that PaLRS offers an effective, much more efficient and flexible alternative to standard preference optimization pipelines, offering a training-free, plug-and-play mechanism for alignment with minimal data.
Extended version with Suppl. Mat. is available at https://doi.org/10.48550/arXiv.2509.23982.