Dual-Channel Semantic-Enhanced Combinatorial Medication Recommendation via Knowledge Distillation

1 Introduction

Combination medication recommendation systems aim to generate appropriate therapeutic prescriptions by comprehensively assessing a patient's health status. Such prescriptions currently rely heavily on physicians' manual experience. However, the expansion of medical knowledge and complexity in multimorbidity cases [Skou et al., 2022] pose great challenges to clinical decision-making. Fortunately, the widespread adoption of electronic health records (EHRs) [Cowie et al., 2017] has enabled large-scale digitization of clinical narratives, providing a crucial foundation for automated medication-recommendation systems. This vast clinical data repository has fueled robust research interest in AIdriven medication recommendation.

∗Co-corresponding authors.

The predominant approach in current medication recommendation systems [Choi et al., 2016; Wu et al., 2022; Tan et al., 2022] is ID-based. These methods primarily focus on architectural designs that attempt to better mimic the clinical decision-making process. However, they face three key limitations: (1) neglecting crucial external knowledge that guides real-world prescribing practices; (2) their reliance on identifiers limiting model generalizability; and (3) the inherent lack of interpretability, which hampers their utility in clinical decision support. Recent advances have explored various strategies to incorporate external knowledge, with three main directions: leveraging clinical entity structures [Wang et al., 2024; Li et al., 2025], integrating drug chemical properties [Yang et al., 2021; Yang et al., 2023], and utilizing knowledge graphs [Zhang et al., 2023b]. Although these approaches have achieved notable performance gains, they largely overlook the rich semantic information embedded in clinical narratives. More recently, some studies [Tan et al., 2024; Liu et al., 2024a; Zhang et al., 2025b] have attempted to address this gap by employing natural language models to extract semantic information from clinical texts. Nonetheless, these efforts remain limited to surface-level semantic encoding, failing to comprehensively capture the complex relationships between medications and patient health status within the textual context.

In this paper, we propose a Dual-Channel SemanticEnhanced Network (DCSENet 1 ) that explores semantic associations between patient health status and medications. As illustrated in Figure 1, our framework employs LMs to extract meaningful semantic relationships from medical narratives. These relationships are modeled into attention maps that explicitly capture and highlight the semantic associations, serving as knowledge bridges. This facilitates the transfer of rich semantic representations to conventional recommendation systems. Consequently, the enhanced system leverages these semantic insights to generate prescriptions based on identifiers. Specifically, DCSENet fine-tunes domain-adapted LMs to effectively capture the semantic relationships between patient health status and medication texts. A dual-channel module further models hierarchical semantic information through disease- and visit-level pathways, fully revealing the semantic associations between patients' health condition texts and medication texts. Subsequently, attention map-based knowledge distillation transfers these semantic features to traditional ID-based recommendation models. This distillation approach transforms the knowledge of clinical texts to ID-based representations, while also reducing the computational overhead associated with LMs. We summarize our main contributions as follows:

• We propose to leverage semantic correlations in texts to enhance drug recommendation systems. Specifically, we fine-tune LMs to effectively extract semantic features from clinical texts and design a dual-channel module for comprehensive modeling of the semantic relationships between patient health conditions and medications.
• We introduce an attention-map-based knowledge distillation method to alleviate the semantic mismatch between clinical narratives and discrete identifiers. This approach effectively transforms the knowledge of textual narratives to ID-based representations.
• Extensive experiments on the MIMIC-III and MIMICIV datasets demonstrate that DCSENet outperforms state-of-the-art baselines in recommendation accuracy, while maintaining competitive computational efficiency.

2 RelatedWork

This section reviews work on medication recommendation and knowledge distillation for language models.

1 https://github.com/ResearchGroupHdZhang/DCSENet

2.1 ID-BasedMedication Recommendation

Existing medication recommendation approaches are predominantly ID-based, focusing on optimizing the recommendation process to improve performance. Many methods formulate this task as a multi-label classification problem. For instance, RETAIN [Choi et al., 2016] employs bidirectional querying; GAMENet [Shang et al., 2019] leverages memory augmentation; DPR-AG [Zheng et al., 2021] uses interactionaware graph induction; 4SDrug [Tan et al., 2022] and SSPNet [Zhang et al., 2025a] adopt a set-to-set paradigm; and DrugRec [Sun et al., 2022] provides causal explanations. Other approaches treat medication recommendation as a sequential decision-making task. For example, LEAP [Zhang et al., 2017] employs a recurrent decoder for sequential drug recommendations, while COGNet [Wu et al., 2022] and VITA [Kim et al., 2024] utilize Transformers [Vaswani et al., 2017] to model sequential medication decisions. Despite their impressive performance, these methods largely neglect external knowledge integration, resulting in suboptimal recommendations with limited generalizability and poor explainability.

2.2 External-Knowledge-EnhancedMedication Recommendation

External knowledge is fundamentally important for achieving optimal medication recommendation. Existing approaches incorporate various forms of external information: SafeDrug [Yang et al., 2021], KE-HMFNet [Zhang et al., 2023a], and MoleRec [Yang et al., 2023] use chemical-structure information; KG-MIML [Shang et al., 2018], TOKEPER [Wang et al., 2024], and MR-DTR leverage clinical entity structures; MedRec [Zhang et al., 2023b] employs knowledge graphs to enhance patient and medication representations. Recently, approaches such as NLA-MMR [Tan et al., 2024] leverage LMs to encode textual information. LEADER [Liu et al., 2024a] and KEDRec-LM [Zhang et al., 2025b] employ LMs to process textual data and utilize knowledge distillation from hidden states to transfer semantic information into smaller models. However, these approaches primarily exploit surface-level semantic information in clinical texts, limiting their ability to comprehensively leverage deep semantic relationships. To better leverage deep semantic relationships, we propose a dual-channel framework that explicitly mines and utilizes rich associations in clinical narratives.

2.3 Knowledge Distillation for LMs

Language models (LMs) exhibit powerful semantic encoding abilities that benefit various downstream tasks [Sun et al., 2020; Xu et al., 2024].Nevertheless, their heavy parameter scale and high computational cost hinder real-world deployment. To tackle this issue, numerous knowledge distillation techniques dedicated to LMs have been explored [Hinton et al., 2015]. For example, DistilledBERT [Tang et al., 2019] transfers BERT knowledge to task-specific models, while TinyBERT [Jiao et al., 2019] uses attention-map distillation to derive a condensed BERT. Recent methods like TransKD [Liu et al., 2024b] further improve distillation via a crossinteraction mechanism to fuse attention maps for cross-stage feature transfer. Motivated by these advances, we propose an attention-based distillation approach to bridge the semantic gap between textual information and identifier embeddings.

3 ProblemFormulation

This section formally defines electronic health records, clinical texts, and the medication-recommendation task.

3.1 Electronic Health Records (EHR)

For a given patient, the health information in the EHR is represented as a sequence of all historical visits, denoted as:

where each visit v(i) encapsulates the clinical data recorded at the t-th encounter and T is the total number of visits. Let Dc denote diagnosis codes, Pc procedure codes, Mc medication codes, Dtxt diagnostic textual descriptions, Ptxt procedural textual descriptions, and Mtxt medication textual descriptions. Each visit v(i) is a tuple comprising diagnosis codes, procedure codes, medication codes, and their associated textual descriptions.

where v(i) dc ∈ Dc, v(i) pc ∈ Pc, v(i) mc ∈ Mc, v(i) dtxt ∈ Dtxt, v(i) ptxt ∈ Ptxt, v(i) mtxt ∈ Mtxt. We denote the entire EHR as V and the set of all patients as P, and define |V| = |P| to fully leverage the EHR information.

3.2 Medication Recommendation Problem

For patient p ∈ P with multiple visits V , provides the diagnoses v(t) d = {d1, d2, ..., dk}, the procedure v(t) p = {p1, p2, ..., pl} and the previously prescribed medications mtp = {m1,m2, ...,mr}. Our objective is to learn a prescription-prediction function f(·) : 2D × 2P ↦→ 2M, that generates a prescription for the given visit,

where ˆm(t) ⊆M is encoded into multi-hot vector.

4 Methodology

Our model operates in two distinct stages, as illustrated in Figure 2. In the first stage, we fine-tune a language model to extract semantic information from clinical narratives, utilizing dual channels at the disease and visit levels to fully leverage the semantic relationships between patient health status and medications. In the second stage, we enhance the outputlayer hidden state distillation with attention-map-based distillation, transferring semantic knowledge from the LM-based framework to DCSENet.

4.1 LMs-Based Framework

The rich semantic information captured by LMs can effectively enhance downstream tasks [Qiu et al., 2020; Han et al., 2021]. Accordingly, we first fine-tune a language model to adapt it to the medication recommendation task and capture the semantic associations between patient health status and medications narratives. Then, we design a dual-channel module that separately models semantic information at both the disease and visit levels.

Clinical textual description

Drug-disease associations in clinical texts offer prior knowledge for medication recommendation.We extracted drug indication knowledge from DrugCentral [Avram et al., 2023] and built direct links among drugs, diagnoses, and procedures. Using prompt templates, we further organized patients' historical visit records to supply personalized features for medication recommendation. The templates of prompt [Liu et al., 2024a] for constructing the text is shown in Figure. 3. Given the EHR textual description Vtxt of patient p, we partition the former into disease-level and visit-level segments to fully exploit the semantic information related to the patient's health status. At the disease level, we feed the patient's current diagnosis names vtdtxt and procedure names vtptxt . At the visit level, we utilize all of the patient's visits. For i-th visit v(i) , we concatenate the diagnostic text v(i) dtxt , procedural text v(i) ptxt , and medication text v(i) mtxt into a single textual description v(i) txt , noting that the current visit vt does not include any medication text. Then, we can obtain the textual descriptions at the visit level of healthcare vtxt = [v(1) txt , v(2) txt , ..., v(t) txt ].

LMs-based Clinical Narratives Encoder Module

For domain adaptation, we fine-tune the LMs by unfreezing only the final layer while keeping other parameters fixed. The adapted LMs then encode textual clinical data into dense representations. Given a text sequence Text, we obtain its representation Etxt by encoding it through the LMs.

Given the disease-level patient texts v(t) dtxt and v(t) ptxt , the visitlevel text Vtxt, and the textual descriptions vmtxt of all medications, we encode them via the LM to obtain the corresponding feature vectors E(t) dtxt , E(t) ptxt , Evtxt, and Emtxt.

Dual-Channel Semantic Fusion Module

The disease-level textual feature vector contains universal associations between drugs, diagnoses, and surgeries derived from the real world, while the visit-level textual feature vector incorporates patient-specific personalized information. To avoid conflicts between personalized conditions and universal knowledge, we first employ two separate cross-attention modules for modeling, respectively, and then utilize two MLPs (Multi-Layer Perceptrons) to project the information into an appropriate space, thereby resolving discrepancies and fusing the information. Our dual-channel module consists of a disease cross-attention and a visit cross-attention. In the disease channel, the textual medication features Emtxt serve as the query, while the current diagnostic textual features Edtxt and the procedural textual features Eptxt are provided as two separate key-value pairs. In the visit channel, the same Emtxt functions as the query, paired with the visit textual features Evtxt provided as the key-value pair. as the key-value pair. Both channels use a Transformer Encoder (TransEec) [Vaswani et al., 2017] to refine the medication representations Emtxt.

…;

> …;

e3>, …;

e3>, …;

e3> , ….

Figure 3: Prompt for medication, diagnose, procedure and visit text.

Subsequently, we project Zdise and Zvisit into an appropriate space and concatenate them to obtain the hidden state of the output layer in the LM-based framework:

4.2 ID-Based Framework: DCSENet

Visit text prompt In the i visit, the patient had diagnosis: <Diag_Name1>, <Diag_Name2>, <Diag_Name3>, …; the patient had procedures: <Proc_Name1>, <Proc_Name2>, <Proc_Name3>, …; the patient had medications: While encoding patient health status and medication texts with LMs effectively captures rich semantic information, this approach also introduces significant redundancy in both data storage and computational complexity. Consequently, deploying such a medication recommendation system in realworld scenarios becomes more challenging. To mitigate this limitation, we design an identifier-based model , DCSENet, and transfer the semantic knowledge from LMs to DCSENet through knowledge distillation.

• <Med_Name1>, <Med_Name2> , <Med_Name3> , …. ID-based encoding

Unlike LM-based frameworks that utilize raw text inputs, DCSENet processes coded clinical information, including diagnoses, procedures, and medications. To represent these codes, we construct three embedding tables: Edc, Epc, and Emc, where each embedding vector corresponds to a diagnosis, procedure, or medication, respectively. For the i-th visit of a patient, we map the associated codes into their respective feature vectors, resulting in diagnosis matrix D(i) c , procedure matrix P(i) c , and medication matrix M(i) c , respectively. Repeating this process across all visits, we obtain sequences {D(1) c ,D(2) c , . . . , D(t) c }, {P(1) c , P(2) c , . . . , P(t) c } and {M(1) c ,M(2) c , . . . ,M(t-1) c } , noting that the final visit does not include medication codes. To align with the visit-level textual input used in LM-based frameworks, we aggregate each feature matrix D(i) c , P (i) c , andM (i) c via summation,

The resulting vectors h(i) diag , h(i) proc and h(i) med are concatenated and passed through a feed-forward network:

where the FF1: R3h → Rh . By aggregating the per-visit representations, we form the visits feature matrix Vc. Subsequently, the sequences D(i) c , P (i) c , Vc and Mc are encoded via a Transformer encoder to obtain the enhanced representations: D(i)′ c , P (i)′ c , V ′ c andM′c .

Dual-Channel Fusion

For effective knowledge transfer, we employ a module architecture similar to the dual-channel textual semantic model module that was used in modeling the textual-information. Then, we can get M ′dise and M ′visit . To fuse the feature information from the two channels, we employ two mapping functions to project the medication features from the two channels into a shared feature space.

where D-Proj and V-Proj denote two distinct feed-forward networks. Subsequently, we concatenate the Zd and Zv:

4.3 Prescription Generation

Given the drug hidden state representation refined from patient health information H we apply a linear projection to H: R2h → R to obtain a multi-hot vector ˆ o:

where FF2 is a feed-forward network. Medications with predicted probabilities exceeding threshold δ in the multi-hot vector ˆmconstitute the final prescription.

4.4 Loss Function

Following prior work, we employ a combined loss to optimize medication recommendation accuracy. For knowledge distillation, we utilize the Kullback-Leibler divergence (KL) [Van Erven and Harremos, 2014] to measure the divergence between attention maps, and cosine similarity to align the hidden states of the output layer.

Medication Recommendation Loss

Consistent with prior work [Yang et al., 2023], we adopt multi-label loss Lbec and cross-entropy loss Lmulti as the loss functions,

where the superscript index (i) represents the i-th component of the vector. We combine these two losses as the overall loss for the medication recommendation task:

where the α is a hyperparameter. Note that only the loss LMR is first used to train the LM-based framework.

Knowledge Distillation Loss

Existing methods [Liu et al., 2024a] perform distillation directly on the hidden states of the medication recommendation model.

where Wh is an alignment projection matrix. However, the semantic representations derived from LM-based medication recommendation models reside in a feature space that is distinct from that of ID-based models. To address this issue, we introduce an attention-based distillation loss. Empirical studies [Clark et al., 2019] have demonstrated that attention weights encode rich linguistic knowledge when processing textual data. The attention-based distillation loss function can be defined as:

where KL(· ∥ ·) denotes the Kullback-Leibler divergence, and n is the number of attention heads. ASi and ATi represent the attention maps from the LM-based framework and DCSENet models, respectively, for the i-th attention head. We perform knowledge distillation on the attention maps within the cross-attention modules of the TransDec for multiple channels, resulting in losses Ldattn (for medicationdiagnose attention), Lpattn (for medication-procedure attention), and Lvattn (for medication-visit attention). These individual channel losses are combined through weighted summation to obtain the overall attention distillation loss:

Merged Loss

The final training objective for the DCSENet is a weighted sum of the LMR and the LKD:

where β is a hyperparameter that balances the two loss.

5 Experiments

We conducted extensive experiments to demonstrate the effectiveness of DCSENet.

5.1 Datasets

We conducted experiments on the MIMIC-III [Johnson et al., 2016] and MIMIC-IV [Johnson et al., 2023]. Following the setting in [Yang et al., 2023; Yang et al., 2021], we split the dataset into training, validation and test as 4 : 1 : 1. The statistical information of the preprocessed dataset is shown in Appendix A.

5.2 Settings

This section details the experimental setup, including hyperparameter configurations, baselines, and evaluation metrics.

Hyperparameter Configuration

Our proposed method is implemented using PyTorch 2.6.0 (with CUDA 12.4 support), which is built on the Python 3.10.16 environment. All experiments are conducted on an Intel Xeon Platinum 8260 server, equipped with 24 CPU cores, 188 GB of RAM, and a single NVIDIA L40 GPU with 46 GB of VRAM. In our model, the embedding dimension for LM-based framework is set to 768. For the compressed DCSENet, this dimension is configured to 64. The optimal hyper-parameters weight α = 0.95, β = 0.9, and the threshold δ = 0.5 are determined based on their performance on the validation set. We train our model with the Adam optimizer [Kingma and Ba, 2014] at an initial learning rate of 1×10-4 .

Baselines and Metrics

We select a diverse set of medication recommendation models as baselines to comprehensively evaluate both variants of DCSENet. These include:

• Classical models: Logistic Regression (LR) and Ensemble Classifier Chain (ECC) [Read et al., 2011].
• Traditional ID-based models: GAMENet [Shang et al., 2019], COGNet [Wu et al., 2022] and VITA [Kim et al., 2024].
• Extra-information-enhanced model: SafeDrug [Yang et al., 2021], MoleRec [Yang et al., 2023], NLA-MMR [Tan et al., 2024] and MR-DTR [Li et al., 2025].

We employ three widely-used metrics: Jaccard similarity, F1 score and PRAUC, to verify the accuracy and effectiveness of these models. Due to space constraints, details of the baselines and metrics are provided in the Appendix B.

5.3 Main Result

We compare the performance of DCSENet against the baseline models; the results are presented in Table 1. Overall, DCSENet outperforms all baseline models across three evaluation metrics, Jaccard, F1-score, and PRAUC. Traditional machine learning models such as LR and ECC fail to effectively leverage patients' historical visit information, resulting in lower performance. LEAP exhibits suboptimal performance due to its difficulty in modeling long sequences of patients' visit records. Although GAMENet incorporates a memory module, while COGNet and VITA apply Transformers that are more adept at modeling long sequences, these methods all overlook the critical role of external knowledge, resulting in suboptimal performance. SafeDrug and MoleRec incorporate the molecular structural information of medications, while MR-DTR introduces clinical entity structure information, leading to improved performance. NLA-MMR leverages language models to extract clinical textual information, which substantially enhances its performance; however, it lacks effective utilization of the semantic relationships between patient health status and medications, resulting in only moderate improvements. In contrast, the LM-based framework uncovers semantic associations between patient health status and medications at multiple levels and effectively transfers this knowledge to DCSENet through knowledge distillation. With enriched textual semantics, DCSENet achieves the best overall performance.

5.4 Ablation Study

To validate the contribution of each component within our model, we conducted a comprehensive set of ablation experiments. The results are summarized in Table 2. Initially, we ablated the diagnostic and procedural information separately; the observed performance degradation confirms that both types of information are crucial for effective medication recommendation. Next, to evaluate the importance of our dual-channel architecture, we ablated the disease channel and the visit channel individually. The results reveal that the disease channel plays a more dominant role, while the visit channel further enhances overall performance. This aligns with the intuition that medication recommendations primarily target the patient's current health status, whereas historical visit records serve as auxiliary information. To assess the effectiveness of knowledge transfer, we first removed the entire knowledge distillation module, which resulted in a significant decline in performance. Additionally, we ablated the individual loss components, Lattn and Lhidn, separately. The results demonstrate that both losses are beneficial to the knowledge transfer process.

5.5 Impact of Different LMs on Performance

To assess how various textual encoding capabilities of different language models (LMs) affect the performance of DCSENet, we conducted experiments using six distinct LMs: BERT-base [Devlin et al., 2019], RoBERTa-base [Warstadt et al., 2020], ClinicalBERT [Huang et al., 2019], BioBERT [Lee et al., 2020], PubMedBERT [Gu et al., 2021], and BlueBERT [Peng et al., 2019]. As shown in Table 3, all variants outperform the baseline model without any LM. BERTbase and RoBERTa-base are general-purpose language models, whereas ClinicalBERT, despite its clinical text adaptation, exhibits relatively lower performance due to its insufficient coverage of extensive biomedical knowledge beyond standard clinical corpora. In contrast, BioBERT, BlueBERT, and PubMedBERT, all pre-trained on large-scale biomedical literature, outperform others in medication recommendation, highlighting the advantage of domain-specific pretraining. Of these domain-adapted models, PubMedBERT delivers optimal performance and is thus chosen as our foundational LM.

5.6 Effectiveness Analysis of Knowledge Transfer

To evaluate the effectiveness of knowledge transfer, we conducted the following experiments:

Attention Map Analysis

To illustrate the effectiveness of our attention-map-based knowledge transfer for medication recommendation, we visualize the attention maps of a representative patient during the distillation process, as shown in Figure 4. LM-based framework effectively captures the associations between diagnoses and medications. For example, diagnosis 568.81 (acute peritonitis) is characterized by a triad of symptoms: severe abdominal pain, signs of peritoneal irritation, and systemic inflammatory response. Correspondingly, the medications-A06A (Laxatives), N02A (Opioids), and N05B (Anxiolytics)-target the primary complications such as gut dysmotility, severe pain, and anxiety/stress. These medication selections represent a common combination in the comprehensive management of acute peritonitis. After knowledge transfer, DCSENet successfully acquires this semantic understanding, with its attention maps showing a high degree of

similarity to LM-base framework. In contrast, the attention maps of the model without knowledge transfer display significant discrepancies, highlighting the effectiveness of the semantic transfer facilitated by our approach.

Parameter Count and Efficiency Analysis

As previously noted, employing LMs as encoders introduces significant computational overhead. The LM-based framework contains 510.22 MB of parameters and requires 453.29

A06A

B05C

A12A

A12C

N02A

N05B

LMs DCSENet No Distillation

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

NDC : Drug A06A : Laxatives; B05C : Intravenous Solutions;

A12A : Calcium; A12C : Other Mineral Supplements;

N02A : Opioids; N05B : Anxiolytics

ICD9 : Diagnose

573.8 : Other specified disorders of liver;

568.81 : Hemoperitoneum; 584.9 : Acute kidney

failure, unspecified; 458.9 : Hypotension, unspecified

Figure 4: Visualization of attention maps. The horizontal axis dis-

plays ICD-9 diagnosis codes, the vertical axis shows NDC medi-

cation codes, with checkmarks indicating correct recommendations

and crosses indicating incorrect ones.

seconds for inference. We compare several strong baseline models. As shown in Figure 5, MoleRec has the lowest computational overhead but exhibits subpar accuracy. COGNet and VITA, which rely on sequential decision processes, incur longer inference times. MR-DTR efficiently encodes clinical entity structures with relatively moderate computational cost while yielding suboptimal performance. NLA-MMR, which heavily leverages LM encoding, introduces substantial additional computational burden. In contrast, our DCSENet transfers LM-based semantic knowledge to an ID-based model, achieving superior accuracy while maintaining low computational costs, demonstrating an optimal balance between performance and efficiency.

6 Conclusion

In this paper, we propose DCSENet, a dual-channel semantics-enhanced framework for medication recommendation. DCSENet leverages language models to capture the semantic information embedded in patient health records and medication-related texts. The disease channel establishes drug-symptom associations, while the visit channel captures heterogeneous patient information; together, they enable accurate personalized medication recommendations. We further adopt a knowledge distillation strategy based on attention maps, which achieves superior knowledge transfer efficacy.Extensive experimental validations on the MIMIC-III and MIMIC-IV datasets consistently demonstrate the superiority of the proposed DCSENet over existing approaches.

Ethical Statement

This study employs the HIPAA-compliant, deidentified MIMIC dataset; its use does not constitute human subjects research under HIPAA rules. Ethical precautions were strictly followed to safeguard participant privacy and rights.