Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks

1 Introduction

Spiking Neural Networks (SNNs), often regarded as the third generation of neural networks, are biologically inspired models that communicate through discrete spikes rather than continuous-valued activations ([Maass, 1997]). By processing information in an event-driven manner with sparse spikes over time, SNNs offer the potential for high energy efficiency, especially when deployed on neuromorphic hardware ([Guo et al., 2023]). Nevertheless, training SNNs to achieve accuracy comparable to Artificial Neural Networks (ANNs) remains challenging, because useful evidence is accumulated progressively through spikes, making different timesteps unevenly informative during learning ([Bellec et al., 2018; Neftci et al., 2019]).

→Corresponding author.

To narrow the performance gap between ANNs and SNNs, knowledge distillation (KD) has become a widely adopted strategy ([Xu et al., 2023; Qiu et al., 2024]). Early distillation methods typically supervise only the final aggregated output by matching averaged logits, which may mix inconsistent temporal information during optimization ([Zhao et al., 2025; Deng et al., 2022]). More recent approaches move toward timestep-wise distillation by injecting teacher supervision at each timestep and aligning logits throughout the temporal dimension ([Yu et al., 2025b; Yu et al., 2025a]). In parallel, several methods further exploit the temporal structure of SNNs through self-distillation ([Ding et al., 2025b]) or temporal consistency regularization ([Zhao et al., 2025; Ding et al., 2025a]), encouraging predictions or features from different timesteps to be consistent in order to stabilize optimization and improve performance ([Ding et al., 2025b; Yu et al., 2025b]).

The consistency assumption is partially misaligned with the intrinsic properties of SNNs and their prediction mechanism. Due to membrane potential integration and reset, spike firing may induce abrupt changes in membrane states, making it difficult for SNNs to maintain identical predictions across all timesteps, as detailed in Appendix A. Meanwhile, since the final decision is determined by temporal accumulation rather than any single timestep, an incorrect intermediate prediction does not necessarily imply an incorrect final outcome. As illustrated in the toy case in Figure 1(a), some intermediate timesteps are misclassified while the final temporally aggregated prediction remains correct. Figure 1(b) further shows that per-timestep accuracy is consistently lower than temporally aggregated accuracy. Moreover, Figure 1(c) reveals that more than 12% of correctly classified samples are misclassified at some intermediate timesteps, and some samples are never correctly classified at any individual timestep but still become correct after temporal aggregation. These observations suggest that a timestep should not be judged solely by whether it is already correct, but by whether it contributes useful evidence to the final temporal accumulation.

However, existing timestep-wise distillation strategies do not explicitly account for this distinction. By encouraging each timestep to align with the teacher signal, they may overlook what corrective evidence is actually needed for improving temporal accumulation and which temporal sources can provide reliable support. This perspective leads to two key questions for timestep-wise distillation: what corrective evidence should be injected into a currently erroneous timestep, and from which temporal sources should it be drawn? For a currently erroneous timestep dominated by a confusing wrong class, directly transferring teacher preference over the full class distribution may dilute the needed correction and interfere with the adjustment of the most critical class relation, namely that between the ground-truth class and the wrongly favored class. Moreover, in temporal selfdistillation, not all source timesteps are equally reliable: some provide confident and compatible evidence, while others may introduce noisy or misleading temporal signals. To address these issues, we propose SeAl-KD, a selective distillation framework with two components. Error-aware Logit Alignment (ELA) refines the class evidence received by erroneous timesteps, while Selective Temporal Alignment (STA) emphasizes reliable and compatible source timesteps during alignment.

Our main contributions are summarized as follows: (1) We reveal that erroneous timesteps are prevalent in SNNs and show that intermediate misclassifications do not necessarily impair the final prediction under temporal aggregation, exposing the mismatch between consistency and temporal evidence accumulation. (2) We propose SeAl-KD, a selective KD framework that improves corrective supervision for erroneous timesteps through ELA and STA. We further provide theoretical analysis to justify the proposed selective alignment. (3) We conduct extensive experiments on both static and neuromorphic image datasets. The results show that SeAl-KD preserves richer temporal distributions across timesteps and consistently improves performance.

2 RelatedWorks

2.1 Knowledge Distillation for SNNs

KD improves SNN training by transferring final outputs, logits, or features from ANNs to narrow the gap with realvalued ANNs ([Xu et al., 2023; Hong et al., 2025]). Along this line, distillation strategies have evolved from global supervision on temporally aggregated outputs ([Zhang et al., 2025]) to timestep-wise supervision that aligns logits or representations at each timestep ([Yu et al., 2025b; Yu et al., 2025a]). In addition, self-distillation and temporal consistency regularization further exploit the student's temporal structure to align outputs across timesteps ([Qiu et al., 2024; Zuo et al., 2024]). Despite these advances, most existing methods impose supervision on different timesteps in a largely uniform manner, without distinguishing whether a timestep is currently erroneous, what corrective information it actually needs, or whether the temporal source used for supervision is reliable.

2.2 Temporal Discrepancy in SNNs

Temporal discrepancy is intrinsic to SNNs, as membrane integration and spike-triggered resets cause neuronal states and predictions to evolve across timesteps ([Bellec et al., 2018]). Prior work has improved temporal modeling and training stability through learnable membrane dynamics ([Fang et al., 2021]), temporal normalization ([Zheng et al., 2021; Duan et al., 2022]), and surrogate-gradient design ([Li et al., 2021a; Wang et al., 2023]). Some studies further exploit temporal structure via timestep-dependent weighting ([Deng et al., 2022]) or temporal consistency regularization across timesteps ([Zhao et al., 2025; Ding et al., 2025a]). However, these methods mostly regulate temporal discrepancy uniformly, without explicitly considering the role of each timestep in the final temporally accumulated prediction. Consequently, they do not distinguish what supervision a timestep needs or which temporal sources are reliable for providing it.

3 Preliminaries

Timestep-wise distillation. Consider an SNN unrolled over T timesteps. At timestep t, the student SNN model, denoted by the superscript S, produces a logit vector zSt → RC , where C denotes the number of classes. The final prediction is typically obtained by temporally averaging the logits over all timesteps, while each zSt can also be treated as an intermediate prediction. For the same input, the teacher ANN model, denoted by the superscript A, produces a timeinvariant logit vector zA. The corresponding categorical distributions are given by the temperature-scaled softmax functions pSt = softmax(zSt ◁ω) and pA = softmax(zA◁ω), where ω > 0 is the temperature. The probability of class i under pSt and pA is denoted by pSt↪i and pAi , respectively.

For each timestep t, the student is optimized with two objectives. The first objective LCLS applies cross-entropy (CE)

loss for classification supervision at each timestep:

where y → ¶0↪ 1♦C is the one-hot ground-truth label.

The second objective LKD aligns each temporal output with the same teacher distribution using the Kullback-Leibler (KL) divergence:

The overall training objective L is defined as

where ϑ controls the contribution of the distillation term.

4 Methodology

This section presents SeAl-KD, which consists of ELA and STA. As illustrated in Figure 2, ELA refines class-level correction at erroneous timesteps, while STA selects confident and compatible temporal sources for alignment.

4.1 Error-aware Logits Alignment Distillation

ELA performs timestep-wise distillation while accounting for prediction errors across timesteps in SNNs. Let y→ → ¶1↪ ▷ ▷ ▷ ↪C♦ denote the ground-truth class index corresponding to the one-hot label y, and let zS t↪c and zA t↪c denote the student and teacher logits for class c at timestep t, respectively.

The student prediction at timestep t is cpred t = argmaxc zS t↪c . When an intermediate timestep is erroneous, ELA relaxes distillation on its dominant confusion instead of directly enforcing the correct class ordering, avoiding misleading correction on the confusing class pair while preserving supervision on the remaining classes.

Logit modification. If cpred t ↔= y→ , we focus on the class pair formed by the ground-truth class y→ and the predicted false class cfalse t = cpred t . We equalize only the logits of this pair and keep the remaining classes unchanged. Specifically, the logits of y→ and cfalse t are both set to the minimum of their original values, which avoids introducing unnecessary absolute shifts. The modified student and teacher logits, denoted by ˜ zS t↪c and ˜ zA t↪c , are defined as

ELA objective. The ELA loss is then defined as

By removing the forced preference between the ground-truth class and the dominant false class at erroneous timesteps, ELA prevents distillation from reinforcing misleading intermediate ordering while retaining teacher guidance on the remaining classes.

4.2 Similarity-aware Temporal Alignment Distillation

Based on uniform temporal alignment (UTA) [Zhao et al., 2025], which enforces unweighted pairwise alignment across timesteps, STA adopts a weighted temporal distillation scheme that allows each timestep t to selectively learn from confident and compatiable timesteps. This guides weak timesteps toward better temporal states that support effective temporal evidence accumulation.

Timestep confidence. The reliability of timestep t is quantified by an entropy-based confidence score. The entropy Ht measures the uncertainty of the class distribution and is normalized by the maximum entropy log C:

A lower entropy leads to a larger Conft, indicating that the timestep provides a more reliable source.

Timestep compatibility. The compatibility between a target timestep t and a source timestep t↑ is measured by cosine similarity in the student logit space. This similarity reflects the consistency of their class-level preferences:

A larger Sim(t↪ t↑ ) indicates that the two timesteps exhibit more compatible class preferences.

Source weighting. For each target timestep t, an unnormalized source score st↪t↑ is computed for source timestep t↑ by combining its confidence with its compatibility to the target:

The final weights are obtained by applying a softmax over all source timesteps satisfying t↑ ↔= t, with wt↪t = 0:

Thus, each target timestep primarily learns from confident and compatible source timesteps.

STA objective. Based on the obtained weights, we encourage each target timestep to align with other source timesteps through a weighted KL divergence. The STA loss is defined as

This objective lets each timestep absorb selectively weighted temporal guidance, instead of being uniformly aligned to all other timesteps, thereby providing more effective support for weak timesteps and reducing interference on already reliable ones.

4.3 Selective Alignment Distillation

SeAl-KD combines ELA in Eq. (6) and STA in Eq. (11) to realize selective temporal supervision. Specifically, ELA regulates what class-level knowledge is injected at erroneous timesteps, while STA guides weak timesteps toward weighted better temporal states, so that temporal correction better supports the final evidence accumulation process. Together with the classification loss, the overall training objective is

where ϖ and ϱ control the contributions of ELA and STA, respectively.

4.4 Theoretical and Statistical Analysis

We analyze SeAl-KD from the perspective of the additional update introduced by distillation beyond the standard crossentropy objective. The analysis focuses on three representative timestep conditions: erroneous timesteps, weak timesteps, and already-correct timesteps. For space efficiency, in the main paper we select one timestep for each condition. The full results over additional sampled timesteps are provided in Appendix B. These diagnostics show that SeAlKD intervenes selectively: it localizes correction when the prediction is wrong, transfers temporal guidance from more reliable temporal states, and remains restrained when the prediction is already reliable.

Proposition 1 (Localized correction). At an erroneous timestep, ELA encourages the distillation update to concentrate on the error-relevant class subspace, rather than spreading the correction uniformly over all classes.

Consider an erroneous timestep t with ground-truth class index y→ and currently predicted wrong class cfalse t . Let ¯ zS t↪rest denote the mean student logit over the remaining classes outside ¶y→ ↪ cfalse t ♦, and let ↗ωl denote the gradient with respect to the parameters of layer l. To examine whether the ELA update is concentrated on the dominant confusion pair, we define three directional effects:

The pair-concentration score is then defined as

The inner products measure how strongly an update along the ELA direction acts on each logit. A larger PairSharelt means that the update is more concentrated on the confusing pair ¶y→ ↪ cfalse t ♦ than on the remaining classes. As shown in Figure 3(a), SeAl-KD gives a higher pair-share score than timestep-wise KD, suggesting that ELA yields more localized correction at erroneous timesteps.

Proposition 2 (Reference-guided correction). For a weak timestep, STA encourages the update to reduce its margin discrepancy to reliability-weighted temporal references, thereby providing guidance from confident and compatible states.

Let mt = zS t↪y→ ↑ zS t↪cfalse t denote the student margin at timestep t, where the margin measures the logit gap between the ground-truth class y→ and the dominant false class cfalse t . A larger margin indicates that the student assigns stronger relative preference to the ground-truth class over the confusing false class. Let mref↪t be the reference margin for timestep t, obtained by aggregating source-timestep margins according to the STA weights. We measure whether the STA update is aligned with the direction that reduces the discrepancy between mt and mref↪t:

A larger RefAlignlt indicates stronger alignment with the discrepancy-reduction direction, suggesting that STA guides weak timesteps toward reliability-weighted temporal references. Rather than forcing each intermediate timestep to be independently correct, STA promotes more consistent evidence accumulation across timesteps. As shown in Figure 3(b), SeAl-KD produces positive alignment, whereas timestep-wise KD shows weaker or negative alignment.

Proposition 3 (Restrained correction). When a timestep is already correctly classified, the distillation update remains small relative to the task gradient, reducing unnecessary interference with reliable predictions.

For already-correct timesteps, a large distillation update may perturb a reliable prediction. We therefore measure the relative strength of the distillation gradient against the task gradient:

A smaller KDRatiolt indicates weaker interference from the distillation objective. As shown in Figure 3(c), SeAl-

KD yields a much smaller KD-to-CLS gradient ratio than timestep-wise KD at the representative correct timestep, indicating that it remains restrained when the prediction is already reliable.

5 Experiments

5.1 Implementation Details

Datasets. We evaluate the proposed method on four benchmark datasets, including three static image datasets, namely CIFAR-10, CIFAR-100, and ImageNet, as well as one neuromorphic dataset, DVS-CIFAR10. This selection allows us to assess the effectiveness of the proposed approach on both frame-based vision tasks and event-based neuromorphic data. Dataset details are provided in the Appendix C.

Training Settings. All experiments are implemented in PyTorch and trained on NVIDIA A100 GPUs. The SNNs are built with leaky integrate-and-fire neurons and trained using surrogate gradient learning [Neftci et al., 2019]. All experiments are conducted three times and the average results are reported, except for ImageNet. For neuromorphic datasets, ANN training uses the average value of the event data as input. Training details are provided in the Appendix D.

5.2 Performance Comparison

The results on CIFAR-10, CIFAR-100, ImageNet, and DVSCIFAR10 are summarized in Tables 1, 2, and 3. Across all evaluated benchmarks, SeAl-KD consistently outperforms directly trained SNNs and achieves superior or competitive performance compared with representative logit-based SNN distillation methods and other distillation baselines under comparable architectures and inference timesteps. The improvements are observed on both static frame-based datasets and event-based neuromorphic datasets, indicating that the proposed method generalizes well across different data modalities. In addition, SeAl-KD only introduces lightweight training-time objectives, and a detailed energy analysis is provided in Appendix E.

5.3 Ablation Study

Component Ablation of SeAl-KD

We first evaluate the individual and joint contributions of ELA and STA in Table 4. Removing either module degrades performance, and removing both causes a larger drop. This indicates that SeAl-KD benefits from making alignment selective in both class and temporal dimensions: ELA reduces misleading supervision on erroneous classes, while STA provides temporal guidance frommore reliable states.

Ablation Study of ELA Variants

We analyze several ELA variants, with results reported in Table 5. ELA-S applies error-aware modification only to the student logits, ELA-A only to the teacher logits, ELA-AS introduces error awareness on the ANN teacher side and aligns the student accordingly, and ELA-Both extends the alignment to three classes when the teacher prediction is also incorrect. All variants outperform the baseline without ELA, showing that class-level selective correction is beneficial once alignment is no longer imposed uniformly. Our ELA achieves the best overall performance, and student-driven variants are consistently more effective than teacher-driven ones, because student errors directly identify the timesteps where naive alignment is most misleading. This supports that ELA should be selective, student-driven, and focused on correcting the dominant confusing class relation rather than enforcing broader per-timestep matching.

Ablation Study of STA Variants

We analyze several STA variants, with results reported in Table 6. STA-NoConf removes confidence weighting, STANoSim removes similarity filtering, STA-Dist replaces cosine similarity with cosine distance, and UTA enforces unweighted pairwise alignment across all timesteps. All variants outperform the baseline without STA, confirming the benefit of temporal guidance, while the full STA performs best overall. This suggests that temporal correction should not be propagated uniformly across timesteps, but should instead guide erroneous timesteps using temporal states that are both reliable and compatible. Confidence weighting suppresses noisy sources, similarity filtering avoids mismatched guidance, and their combination yields the most effective selective temporal correction.

5.4 Visualization

Temporal Discrepancy

To better understand how selective alignment affects temporal prediction dynamics, we visualize per-timestep class logits on DVS-CIFAR10 in Figure 6. Compared with Logit-SNN, which exhibits a largely time-invariant pattern where a subset of classes remains dominant across most timesteps, our method produces more structured temporal trajectories. The dominant class can shift over time, and certain classes show larger logit variation across timesteps. For example, given that the ground-truth label is C3, the logit of C0 transitions from negative to strongly positive. Overall, the visualization suggests that SeAI-KD redistributes evidence over time and encourages temporally differentiated class responses, rather than maintaining a fixed class preference across timesteps.

t-SNE

To better understand the effect of selective temporal alignment, we visualize the learned representations using t-SNE, as shown in Figure 5. Compared with direct training and previous distillation methods, our approach produces more compact class clusters with clearer separation. Notably, the circled regions exhibit less class overlap and fewer scattered samples. Overall, the visualization indicates that selective alignment leads to more discriminative and well-structured representations over time.

6 Conclusion

This work examines the limitations of uniform timestep-wise KD in SNNs and shows that treating all temporal states equally can impose overly rigid supervision, even though intermediate timesteps need not all be individually correct. Instead, erroneous timesteps should receive guidance that moves them toward the correct final outcome. We propose SeAl-KD, which selectively aligns class-level and temporal knowledge via ELA and STA. Our analysis shows that ELA and STA reduce the influence of erroneous, low-confidence, or incompatible timesteps during alignment, enabling more effective correction where it is needed. Experiments on both static and neuromorphic datasets further confirm that this selective alignment strategy consistently improves accuracy with only minimal computational overhead.

Ethical Statement

There are no ethical issues.

Acknowledgements

This work was supported by start-up funds with No. MSRI8001004 and No. MSRI9002005 and Monash eResearch capabilities, including M3.