TabKD: Tabular Knowledge Distillation Through Interaction Diversity of Learned Feature Bins

1 Introduction

Data-free knowledge distillation trains a student model to mimic a teacher model's output without the original training data. In many real-world scenarios, organizations possess high-performing tabular models but cannot share the underlying training data. A hospital may deploy a diagnostic model trained on sensitive patient records; a bank may operate a credit scoring system built from proprietary transaction histories. Data-free distillation can be useful in scenarios where an organization wants to train another model with only black-box access to the original model and without the original data. One of the use cases of knowledge distillation is to train a smaller student model for edge devices. However, conventional distillation [Hinton et al., 2015] still requires representative input samples to query the teacher. This constraint has motivated data-free knowledge distillation (DFKD), where the student learns only from synthetic queries [Fang et al., 2019; Chen et al., 2019; Lopes et al., 2017; Truong et al., 2021].

Tabular models are fundamentally different from their vision counterparts. While image classifiers detect local patterns through convolutions, tabular models succeed by learning which feature combinations predict outcomes. A credit risk model does not independently assess income or debt; it learns that the specific combination of these features signals risk, and neither feature alone suffices to predict the outcome. Unlike image data, where feature interactions are hierarchical and locally correlated, tabular data relies on sharp, non-linear interactions across heterogeneous features with no predefined structure. This fundamental difference explains why existing DFKD methods are developed mainly for vision tasks and cannot effectively capture feature interactions in tabular data. Vision methods rely on spatial inductive biases (convolutions, locality) that have no analog in tabular domains where features interact arbitrarily [Kang et al., 2025].

Early work like StealML [Tram`er et al., 2016] laid the foundation for black-box distillation. Subsequent work ,such as DualCF [Wang et al., 2022] uses counterfactual explanations for binary classification; Marich [Karmakar and Basu, 2023] improves query efficiency through active sampling; TabExtractor [Jin et al., 2025] handles feature heterogeneity through entropy-guided generation. Despite these advances, a fundamental bottleneck remains: existing approaches frequently exhibit mode collapse, failing to explore the full decision manifold, and resulting in students that miss critical decision rules [Shin and Choi, 2024].

We hypothesize that effective tabular model distillation requires systematically exploring the interaction space rather than individual features. This insight is inspired by t-way combinatorial testing [Kuhn et al., 2013], a software engineering strategy where covering all t-way parameter interactions with minimal samples suffices to expose system behaviors. The key observation is that significant behaviors of complex systems are rarely triggered by all parameters simultaneously, but rather by interactions among a small subset. As in [Khadka et al., 2024; Khadka et al., 2026], tabular model decisions similarly depend on interactions among small feature subsets. However, combinatorial testing tries to cover every interaction at least once. This is typically sufficient for fault detection, but not for knowledge distillation. This is because the latter aims to learn the decision patterns of the teacher model. Learning a decision pattern requires observing how the teacher behaves across multiple related samples. Our approach therefore, attempts to have uniform pairwise interaction distributions, ensuring that each decision pattern receives equal representation during training rather than merely guaranteeing minimum coverage. Consider a credit risk model that learns the rule (age > 50 ∧ debt-to-income < 0.3) → low risk. Covering all pairwise feature interactions enables the student to observe different decision patterns in a uniform manner.

Our approach is designed to generate samples that achieve interaction diversity. Specifically, the generated samples will cover the space of pairwise interactions in a uniform manner, where each interaction is typically covered multiple times by different samples. Such uniform coverage of pairwise interactions can help expose diverse decision patterns, including those involving higher-order interactions. To operationalize this, we propose TabKD, which discretizes each feature into K learned bins aligned with the teacher's decision boundaries, reducing the infinite input space to a finite set of pairwise interactions that can be systematically covered. Our framework operates in two stages. First, we learn adaptive bin boundaries that partition each feature into semantically meaningful regions where teacher predictions are consistent. Second, we train a generator with an interaction diversity loss that maximizes entropy over pairwise bin combinations, pushing toward uniform coverage rather than modecollapsing into limited regions. Random sampling may never query specific rare interactions, and entropy-based methods may locate decision boundaries but cannot certify which interactions have been tested. TabKD provides both systematic coverage and boundary-aware sampling.

Experiments across 4 benchmark datasets and 4 teacher architectures (neural networks, XGBoost, Random Forest, TabTransformer) demonstrate that TabKD achieves highest student-teacher agreement in 14 of 16 configurations, outperforming 5 state-of-the-art baselines. We further show that interaction coverage strongly correlates with distillation quality, validating our core hypothesis. Our contributions are:

1. Approach: We propose TabKD, a data-free knowledge distillation framework built on the insight that interaction diversity, the systematic coverage of feature combinations, is essential for effective tabular distillation. TabKD operationalizes this through dynamic bin learning aligned with teacher decision boundaries and a diversity loss that ensures uniform pairwise interaction coverage.
2. Evaluation: We perform comprehensive experiments across 4 datasets and 4 teacher architectures, demonstrating that TabKD achieves highest agreement in 14 of 16 configurations over 5 baselines, and that interaction coverage strongly correlates with distillation quality.
3. Tool: We release our implementation as an open-source tool. 1

2 RelatedWork

Model extraction and distillation research spans query-based methods using auxiliary data and data-free approaches using generators. While knowledge distillation compresses a teachers knowledge into a smaller student, model extraction seeks to steal a model using similar techniques. Both are predominantly developed for vision domains and poorly suited to the heterogeneous feature interactions of tabular data. Related works in this domain are discussed below.

1 Hosted at https://github.com/PereiraMavs/int div

2.1 Model Extraction

Query-Based Extraction. Model extraction attacks [Tram`er et al., 2016] infer the behavior of a victim model through black-box API queries. Early work focused on equation-solving approaches [Tram`er et al., 2016], requiring exponential queries for high-dimensional inputs. Active learning strategies [Papernot et al., 2017; Orekondy et al., 2019] reduce query budgets by selecting informative samples based on uncertainty or gradient approximations. However, these methods assume access to auxiliary unlabeled data from public domain and do not provide any mechanism to assemble most meaningful sample set.

Data-Free Extraction. Recent work on data-free extraction employs generative models to synthesize training data. DFAD [Fang et al., 2019] uses adversarial training where a generator maximizes student-teacher disagreement. DAFL [Chen et al., 2019] adds activation matching to preserve intermediate representations. These approaches excel in vision domains but may not be sufficient for tabular data due to heterogeneous feature spaces and non-differentiable teachers (tree ensembles). TabExtractor [Jin et al., 2025] addresses non-differentiability through entropy maximization but lacks systematic interaction coverage, leading to mode collapse in high-dimensional feature spaces.

2.2 Knowledge Distillation

Standard Distillation. Knowledge distillation [Hinton et al., 2015] transfers knowledge from teacher to student via soft probability labels, preserving "dark knowledge" about class relationships. Variants explore attention transfer [Zagoruyko and Komodakis, 2016], intermediate representation matching [Romero, 2014], and relational knowledge [Park et al., 2019]. All assume access to original training data or indistribution samples.

Data-Free Distillation. DFKD methods synthesize training data through generative models. Zero-Shot KD [Wang, 2021] uses metadata and soft label smoothing. DeepInversion [Yin et al., 2020] inverts batch normalization statistics for vision models. CMI [Fang et al., 2021] uses contrastive learning for mode diversity. These methods rely on visionspecific inductive biases (spatial locality, batch norm statistics) absent in tabular domains.

2.3 Tabular Data Synthesis

Generative models for tabular data face unique challenges due to mixed feature types and complex dependencies. Among existing works the most prominent few are, CTGAN [Xu et al., 2019], which uses mode-specific normalization and conditional generators. TVAE [Ishfaq et al., 2018] employs variational autoencoders with tailored loss functions. However, these methods optimize data realism, not teacher fidelity-generating plausible samples that may not expose student weaknesses or cover decision boundaries.

Gap in PriorWork. No existingwork addresses the fundamental challenge of systematic interaction coverage in datafree tabular model extraction. Vision-domain methods ignore heterogeneity and interactions; tabular synthesis methods optimize realism over informativeness; active learning assumes auxiliary data. Our work fills this gap with a principled interaction-focused framework.

3 Background

3.1 Data-free Knowledge Distillation

In the absence of real data, Data-free Knowledge Distillation(DFKD) methods typically employ an adversarial framework to synthesize training samples. The Generator aims to synthesize inputs that maximize the divergence between the Teacher's and the Student's outputs (i.e., "hard" samples), while the Student tries to minimize this divergence. This adversarial game forces the Student to align its decision boundaries with the Teacher's, using the synthetic data as a proxy for the unavailable real data. The student does not need to know the real data, it only need to know how the teacher reacts to particular data-points [Lopes et al., 2017; Chen et al., 2019; Fang et al., 2019].

3.2 Combinatorial Testing

T-way combinatorial testing is grounded in the empirical interaction fault hypothesis, which suggests that significant behaviors in highly complex systems are rarely triggered by interactions of all parameters simultaneously, but rather by interactions among a relatively small number (t) of input parameters [Kuhn et al., 2004]. Based on this hypothesis, tway testing aims to generate a minimal, efficient set of samples that guarantees coverage for every possible combination of values across any subset of t parameters. The notion of interaction diversity is inspired by t-way combinatorial testing. In the context of generative modeling, interaction diversity offers a principled metric to encourage a generator to explore the full interaction space rather than mode-collapsing into limited regions. For example, suppose feature A has two bins: Bin 1 = [1-5] and Bin 2 = [6-10], and feature B has two bins: Bin 1 = [OLD] and Bin 2 = [NEW]. This creates four pairwise bin combinations that need to be covered: (A-Bin1, B-Bin1), (A-Bin1, B-Bin2), (A-Bin2, B-Bin1), and (A-Bin2, B-Bin2). To achieve uniform 2-way coverage, the generator produces samples such that the number of samples that land in each combination is approximately the same.

4 Approach

Effective tabular distillation requires the student to see samples covering all meaningful feature interactions, not just random points in the input space. Our approach operationalizes this insight in three stages: (1) learn bins aligned with teacher decision boundaries, (2) generate samples with uniform pairwise bin coverage, and (3) train the student on these diverse, challenging samples.

Problem Setup. Let T : RF → [0, 1]C denote a pretrained teacher mapping F-dimensional inputs to C class probabilities, and S : RF → [0, 1]C a lightweight student. We assume no access to the original training data.

4.1 Approach Overview

TabKD consists of four components working together (Figure 1):

• Generator G(z): Maps noise z ∼ N(0, I) to synthetic samples. Trained to maximize both interaction diversity and student-teacher disagreement.
• Bin Learner B(x): Learns K adaptive boundaries per feature, partitioning each into regions where teacher predictions are consistent. Outputs soft membership vectors m ∈ [0, 1]K .
• Teacher T(x): Teacher is a target frozen pre-trained model providing ground-truth soft labels.
• Student S(x): Student model is a smaller network trained to mimic the teacher. See section 5.3

Training proceeds in three phases to ensure stable convergence:

1. Warmup: Pre-train student on uniform random samples; populate replay buffer (used later to prevent catastrophic forgetting during adversarial training [Mnih et al., 2015]).
2. Bin Learning: Train the bin learner to find decisionboundary-aligned discretizations; freeze upon convergence.
3. Adversarial Distillation: Generator and student alternate, generator maximizes interaction diversity and hardness; student minimizes divergence from teacher.

The staged approach prevents co-adaptation between generator and bin learner, ensuring coverage reflects genuine feature space exploration. The following subsections detail each component.

4.2 Dynamic Bin Learning

To achieve interaction diversity, we first need a effective discretization of each feature. Random or uniform bins waste coverage on homogeneous regions where the teacher's predictions are constant. Instead, we learn bins aligned with the teacher's decision boundaries. We aim to ensure that covering all bin combinations captures the majority of behaviorally distinct regions.

Intuition. The teacher's decision boundary implicitly partitions each feature into regions of consistent predictions. Consider a feature from a dataset where the boundary crosses once, separating values [1-5] (Class 0) from [6-10] (Class 1). These boundary crossings define natural bin boundaries. By learning bins aligned with these transitions, we ensure that covering all bin combinations means covering all behaviorally distinct regions.

Learning Objective. We train the bin learner to minimize intra-bin prediction variance (ensuring homogeneity within bins) while maximizing inter-bin prediction variance (ensuring bins capture different behaviors):

where M denotes soft bin memberships and PT the teacher's predictions. Soft membership is required to ensure the assignment is differentiable, which a an discrete assignment cannot provide [Frosst and Hinton, 2017]. The variance is calculated as the spread of probability of samples inside each bin weighted with soft membership for each sample.

Boundary-Focused Sampling. Learning effective bins requires samples near decision boundaries, where predictions transition between classes. During bin learning, we train a a generator to target these regions:

where Lclass-div ensures class balance and Lentropy targets highuncertainty regions via entropy maximization.

Why Staged Training? Jointly optimizing bins and coverage causes instability: the generator fills bins faster than boundaries stabilize. Our staged approach, freezing bins before adversarial training, prevents this co-adaptation. With bins frozen, we can now define systematic coverage.

4.3 Interaction Diversity Loss

This is the core contribution of TabKD: a loss function that ensures systematic coverage of feature combinations. With frozen bins providing a effective discretization, we can now formalize what it means to "cover" the interaction space.

Formalization. For pairwise (t = 2) coverage, we consider all (F2 ) feature pairs. For each pair (i, j), the joint bin assignment forms a K ×K matrix. Ideally, generated samples should uniformly populate all K2 cells, ensuring every combination of feature i's regions with feature j's regions is explored.

We compute the empirical joint distribution over bin com-

where N is batch size and m(i) k (x) is the soft membership of sample x in bin k of feature i.

Diversity Loss. Maximum entropy over this distribution corresponds to uniform coverage. Our diversity loss maximizes this entropy:

By minimizing Ldiversity (negative entropy), we push the generator toward uniform pairwise coverage, systematically exploring the interaction space rather than mode-collapsing.

4.4 Adversarial Sample Generation

Diversity alone is insufficient, we also want samples that challenge the student's current approximation. We combine interaction coverage with a hardness objective.

Hardness Loss. We measure student-teacher disagreement via KL divergence:

The negative sign means the generator is rewarded for finding samples where the student fails. KL divergence naturally upweights uncertain regions near decision boundaries, where T (x) ≈ 0.5.

Combined Objective. The generator minimizes:

This balances systematic exploration (covering all interactions, Ldiversity) with targeted exploitation (finding student weaknesses, Lhardness). The diversity term encourages interaction coverage; the hardness term ensures samples are informative for learning. L(2) gen in equation 6, used as the loss to the generator during adversarial distillation

4.5 Student Training

The student minimizes KL divergence from the teacher's soft predictions:

Training exclusively on adversarial samples risks forgetting previously learned knowledge. To mitigate this, we maintain a replay buffer populated during warmup with random samples. Each update uses 90% adversarial samples and 10% replay samples, balancing challenge with stability.

Summary. We have explained three key mechanisms: (1) dynamic bin learning to discretize features meaningfully, (2) interaction diversity loss to ensure systematic coverage, and (3) hardness-guided generation to target student weaknesses. Together, these operationalize our core insight that interaction diversity can play an important role for effective tabular model distillation.

5 Experimental Setup

5.1 Research Question

Our primary research questions are:

RQ1: Agreement Accuracy. How accurately does TabKD replicate teacher model behavior, as measured by teacherstudent agreement and test accuracy?

RQ2: Comparison. How does TabKD compare against state-of-the-art data-free distillation methods in extraction quality?

RQ3: Interaction Diversity vs Agreement. Does systematic feature interaction coverage correlate with improved distillation quality?

5.2 Datasets

We evaluate our framework on four benchmark classification datasets, previously used by multiple works [Jin et al., 2025; Ma et al., 2023; Arik and Pfister, 2021; Gao et al., 2024; Hollmann et al., 2022].

• Adult [Becker and Kohavi, 1996]: This dataset contains 48K samples with 14 features and predicts whether an individual's income exceeds $50K.
• Breast Cancer [Zwitter and Soklic, 1988]: This dataset contains 569 samples with 30 features derived from cell nuclei measurements to classify tumors as malignant or benign.
• Credit [Yeh, 2009]: This dataset contains 30K samples with 23 features and predicts credit card default.
• Mushroom [mus, 1981]: This dataset contains 8K samples with 22 features describing physical characteristics to classify mushrooms as edible or poisonous.

All datasets are binary classification and features are normalized using standard scaling, and we use an 80/20 train-test split with stratified sampling.

5.3 Model Architectures

Teacher Models. We consider four teacher architectures spanning neural, ensemble, and transformer-based methods:

• Neural Network (NN): The neural network consists of two hidden layers (128 → 64 units) with ReLU activations and dropout (p = 0.2).
• XGBoost [Chen and Guestrin, 2016]: The gradient boosting model uses 100 estimators with maximum depth 6 and learning rate 0.1.
• Random Forest [Breiman, 2001]: The random forest comprises 100 trees with maximum depth 10 and minimum samples per split of 5.
• TabTransformer [Huang et al., 2020]: The transformerbased model with multi-head self-attention for feature encoding in tabular data.

Student Model. A single-hidden-layer network (32 units, ReLU activation) that provides a uniform, lightweight approximation regardless of the teacher's architecture. Whether the teacher is a neural network, tree ensemble, or transformer, the student offers a consistent inference pipeline with predictable latency and differentiability, making it suitable for deployment on resource-constrained devices.

5.4 Baselines

We compare against five baselines representing distinct extraction paradigms: (1) StealML [Tram`er et al., 2016], the seminal sampling-based attack with adaptive boundary refinement; (2) TabExtractor [Jin et al., 2025], a datafree method using entropy-guided synthetic data generation; (3) CF [A¨ıvodji et al., 2020] and (4) DualCF [Wang et al., 2022], which leverage counterfactual explanations for extraction; and (5) DivT [Ma et al., 2023], an active learning approach employing divide-and-conquer query selection.

Implementation. We use official implementations for StealML and CF where available. For TabExtractor, DualCF, and DivT, we reimplemented them following the methodology described in the original papers, as no official code was released. Following TabExtractor [Jin et al., 2025], all baselines use a query budget of 9,600 samples.

5.5 Training Protocol

Training proceeds in three phases. In Phase 0 (Warmup), the student is pre-trained for 30 epochs(means update steps with each batch) on randomly sampled instances using standard KL-divergence distillation:

Phase 1 (Boundary Stabilization) spans 200 epochs during which the adaptive bin boundaries bf,k are learned. We apply linear temperature annealing:

In Phase 2 (Adversarial Distillation), training continues for 400 epochs with fixed bin boundaries, focusing on adversarial sample generation to maximize coverage of the input space.

5.6 Temperature Schedules

Ensemble teachers produce sharper probability distributions than neural networks, necessitating teacher-specific temperature schedules to preserve soft label information during distillation. Table 1 summarizes these configurations.

5.7 Hyperparameters

We use a batch size of 128 and optimize with Adam (learning rate 0.001) with cosine annealing. The coverage and hardlabel loss weights are set to λcov = 10.0 and λhard = 2.0, respectively. We discretize each feature into K = 8 bins.

5.8 EvaluationMetrics

We report four metrics to assess distillation quality and input space exploration.

Test Accuracy measures the proportion of correctly predicted instances on held-out test data.

F1 Score is the harmonic mean of precision and recall.

Teacher-Student Agreement captures the proportion of test instances where the student and teacher produce identical predictions.

Cumulative Coverage quantifies the fraction of pairwise feature-bin interactions visited during training.

These metrics are formally defined as:

where TP , TN, FP , and FN denote true positives, true negatives, false positives, and false negatives, respectively. N is the number of test instances, ˆ y(s) i and ˆ y(t) i are the student and teacher predictions for instance i, V is the set of visited feature-bin pairs, F is the number of features, and K is the number of bins per feature. The results are the average of 5 runs.

Efficiency: The parallelizable computation related to pairwise coverage is to compute the joint distribution over bin combinations, in equation 3, and the cost for N samples is O(F 2 · K2 · N).

6 Results

We evaluated our knowledge distillation framework across four benchmark datasets with four different teacher architectures: Neural Network (NN), XGBoost, Random Forest (RF), and TabTransformer. Table 2 presents the comprehensive performance metrics, including student-teacher agreement, test accuracy, F1 score, AUC score, and bin coverage.

6.1 Overall Performance

TabKD demonstrates strong and consistent performance across all four teacher architectures. Overall, TabKD achieves the best agreement in 14 out of 16 datasetteacher combinations, demonstrating robustness across different teacher architectures.

6.2 RQ1: Agreement Accuracy

Agreement accuracy is measured by the percentage of samples predicted by the student that match the teacher's predictions. Across all configurations, the Neural Network teacher achieves the highest agreement of 97.0% on the Credit dataset, while XGBoost has the lowest agreement of 81.1% on the Adult Income dataset. On average across the four datasets, Neural Network, XGBoost, Random Forest, and TabTransformer teachers achieve agreement rates of 95.0%, 87.3%, 88.6%, and 87.2% respectively. In phase 3, the use of interaction diversity pushes the generator to produce samples covering the vast majority of 2-way interaction combinations, thus creating diversity. Additionally, the teacher-student disagreement objective provides incentive to create samples that are hard for the student to predict.

6.3 RQ2: Comparison with Other Baselines

Compared to five other baselines tested on the same datasets, the overall performance of TabKD is consistently better. In terms of agreement accuracy, TabKD achieves the best results in 14 out of 16 dataset-teacher combinations. The exceptions are: (1) Adult with Neural Network, where DivT achieves slightly higher agreement (91.4% vs 91.0%), and (2) Mushroom with TabTransformer, where DivT outperforms on both accuracy (86.6% vs 84.2%) and agreement (88.3% vs 85.2%). For all other combinations, TabKD shows superior agreement accuracy. In terms of model accuracy, TabKD also demonstrates strong performance across the board. All results are presented in Table 2. Current works do not address systematic interaction coverage, which is particularly important for tabular data. As a result, our dynamic binning feature, which creates optimal bins, combined with interaction diversity, achieves better results than existing approaches.

6.4 RQ3: Interaction Diversity vs Agreement

One of our main contributions suggests that as t-way combinations of features are covered more thoroughly by the generator, we should observe better agreement accuracy. Figure 2 shows that for all three teacher models, average agreement accuracy increases as interaction diversity increases. Because of the well-formed bins and enforced interaction diversity, TabKD covers important 2-way interactions, and as coverage increases, the student learns from new samples and agreement improves accordingly.

7 Ablation Study

One of the important component of TabKD is the dynamic

binning module. It is important learn the bins to find out optimal boundaries where prediction distribution remains similar. To find out the importance we removed the dynamic binning module and calculated agreement accuracy with preset static bins. The static bins are of equal lengths covering the whole feature length. The results in table 3 shows that for complex and imbalanced datasets like Adult shows significant decrement of performance. But for simpler datasets like Mushroom, where we have categorical features with clear rules uniform bins perform similar to adjusted bins.

8 Conclusion

We introduced TabKD, a data-free knowledge distillation framework for tabular data that leverages feature interaction diversity, inspired by combinatorial testing, to generate informative samples for student models. It uses dynamic bin learning based on prediction similarities of feature ranges. Experiments across four datasets and four teacher architectures show TabKD achieves best agreement in 14/16 configurations over five baselines.

While TabKD demonstrates strong performance across diverse settings, two aspects warrant future exploration. First, the number of bins per feature is currently set as a static hyperparameter. It is possible that fewer or more bins may be more optimal for particular features; future work could extend the bin learner to adaptively determine the optimal granularity for each feature. Second, TabKD focuses on 2-way interaction coverage, which proves effective for the evaluated datasets. However, for more complex datasets, higher-order combinations may be beneficial. Exploring whether a particular t-way interaction strength improves performance for specific dataset characteristics remains an open direction.