---
title: "Streamlining Long-Chain Reasoning via Differentiable Hierarchical Fusion"
authors:
  - "Chuangen Gao"
  - "Wenlun Zhang"
  - "Shang Wang"
  - "Shuyang Gu"
doi: "10.24963/ijcai.2026/635"
url: "https://ijcai.org/proceedings/2026/635"
pdf: "https://ijcai.org/proceedings/2026/0635.pdf"
venue: "Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence"
year: 2026
pages: "5702-5710"
track: "Main Track"
keywords:
  - "Natural Language Processing: Resources and evaluation"
  - "Natural Language Processing: Language models"
---

# Streamlining Long-Chain Reasoning via Differentiable Hierarchical Fusion

Chuangen Gao, Wenlun Zhang, Shang Wang, Shuyang Gu

*Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence*, pages 5702-5710. DOI: [10.24963/ijcai.2026/635](https://doi.org/10.24963/ijcai.2026/635)

## Abstract

While large-scale reasoning models have achieved remarkable performance gains by scaling test-time computation through extended Chain-of-Thought (CoT) sequences, their practical utility is severely constrained by protracted inference lengths and high computational latency. In this paper, we present Differentiable Hierarchical Fusion (DHF), a novel framework that merges reasoning models with efficient base models via differentiable optimization to produce concise, accurate outputs.  We introduce a dual-factor adaptive weighting mechanism to capture intra-block (Attention vs. MLP) variance and inter-block (shallow vs. deep layers) importance hierarchies, thereby addressing key limitations of static merging heuristics. Specifically, DHF optimizes fusion coefficients using gradient descent on a loss function that jointly minimizes cross-entropy (for accuracy) and response length (for conciseness). Furthermore, we enhance an outlier-aware initialization strategy to seed coefficients based on activation density and construct multi-calibration datasets to improve the model’s generalization ability. Comprehensive evaluations on Qwen and LLaMA models across six reasoning benchmarks show that DHF reduces the average response length by 55% while boosting accuracy by 2.8%–6% compared to existing state-of-the-art merging methods.