RDPD: Rich Data Helps Poor Data via Imitation

1 Introduction

Many rich-data environments encompass multiple data modalities. For example, multiple motion sensors in a lab can collect activity signals from various locations of a human body where signals generated from each location can be viewed as one modality. Multiple leads for Electrocardiogram (ECG) signals in hospital are used for diagnosing heart diseases, of which each lead is considered a modality. Multiple physiological signals are measured in Intensive Care Units (ICU) where each type of measure is a modality. A series of recent studies have confirmed that finding patterns among rich multimodal data can increase the accuracy of diagnosis, prediction, and overall performance of the deep learning models [Xiao et al., 2018; Hong et al., 2017].

Despite the promises that rich multimodal data bring us, in practice we have more poor-data environments with data from fewer modalities of limited quality. For example, unlike in a rich-data environment such as hospitals where patients place multiple electrons to collect 12-lead ECG signals, in everyday home monitoring devices often only measure lead I ECG signal from arms. Although deep learning models often performwell in rich-data environment, their performance on poor-data environment is less impressive due to limited data modality and lower quality [Salehinejad et al., 2018].

We argue that given both rich- and poor-data from similar contexts, the models built on rich multi-modal data can help improve the other model built on poor data with fewer modalities or even a single modality. For example, a heart disease detection model trained on 12 ECG channels in a hospital can help improve a similar heart disease detection model trained on ECG signals from a single-channel at home.

The recent development of mimic learning or knowledge distillation [Hinton et al., 2015; Ba and Caruana, 2014; Lopez-Paz et al., 2015] has provided a way of transferring information from a complex model (teacher model) to a simpler model (student model). Knowledge distillation or mimic learning essentially compresses the knowledge learned from a complex model into a simpler model that is much easier to deploy. However they often require the same data for teacher and student models. Domain adaptation techniques address the problem of learning models on some source data distribution that generalize to a different target distribution. Deep learning based domain adaptation methods have focused mainly on learning domain-invariant representations [Glorot et al., 2011; Chen et al., 2012; Bousmalis et al., 2016]. However they often need to be trained jointly on source and target domain data and are therefore unappealing to the settings when the target data source is unavailable during training.

In this paper, we propose RDPD (Rich Data to Poor Data) to build accurate and efficient models for poor data with the help of rich data. In particular, RDPD transfers knowledge from a teacher model trained on rich data to a student model operating on poor data by directly leveraging multimodal data in the training process. Given a teacher model along with attention weights learned from multimodal data, RDPD is trained endto-end for the student model operating on poor data to imitate the behavior (attention imitation) and performance (target imitation) of the teacher model.

In particular, RDPD jointly optimize the combined loss of attention imitation and target imitation. The loss of target imitation can utilize both hard labels from the data and soft labels provided by the teacher model. Here are the main contributions of this work:

• We formally define the learning task from rich data to poor data, which has many real-world applications including healthcare.
• We propose RDPD algorithm based on mimic learning, which takes a joint optimization approach to transfer knowledge learned by a teacher model using rich data to help improving a student model trained only on poor data. The resulting model is also much lightweight than the original teacher model and can be more easily deployed.
• We show that RDPD consistently outperformed all baselines across multiple datasets and achieve the greatest performance improvement over the Direct model trained on common features between rich and poor data by 24.56% on PR-AUC and 12.21% on ROC-AUC, and over the standard distillation model in [Hinton et al., 2015] by 5.91% on PR-AUC and 4.44% on ROC-AUC.

2 Method

In this section, we will first describe the task, and then introduce the design of RDPD (shown in Figure 1).

2.1 Task Description

Consider data collected via continuous time series, given a teacher model trained from rich data environment, we want to teach a student model running on only poor data. And we hope the student model could benefit from the information contained in rich data via the teacher model, by imitating the teacher model in terms of learning outcome and the learning process. In this work, we call the former objective as target imitation, and the latter one as behavior imitation. Target imitation can be achieved by imitating the final predictions (i.e., soft labels) of the teacher model while behavior imitation can be achieved by imitating its attention weights over temporal time series.

Mathematically, denote Xr as the multi-modal rich data with Dr modalities that is available in training phase, and Xp as the poor data with Dp modalities that is available in both training and testing phases. Here the modalities in Xp are a subset of Xr, and Dp < Dr; Xp and Xr share the same labels Y . Our task is to build a student model Fp which only takes Xp as input, and will benefit from knowledge transferred fromXr.

Overview. For RDPD, the student model trained on poor data will imitate teacher model trained on rich data and hard labels in both intermediate learning behavior and final learning performance. The imitation of learning behavior is achieved by optimizing information loss from distribution of attention in student model to distribution of attention in teacher model while the performance imitation is done by jointly optimizing hard label, soft label and and their trainable combination. In the following we will detail each step of RDPD.

2.2 Training Teacher Model

Although RDPD can be applied on time series in general, in this paper we only consider regularly sampled continuous time series Xr (e.g., sensor data). Assume a patient has time series from Dr modalities, for time series in each modality with length l, we split Xr ∈ Rl×Dr into M segments at length S, thus l = M×S. We denote multi-modal segmented input time series as Sr ∈ RM×S×Dr .

We applied stacked 1-D convolutional neural networks (CNN) on each segment and recurrent neural networks (RNN) across segments. Such a design has been demonstrated to be effective in many previous studies on multivariate time series modeling [Ord´o˜nez and Roggen, 2016; Choi et al., 2016]. In detail, we apply 1-D CNN with mean pooling on each segment s(j) r ∈ RS×Dr , j = 1, · · · ,M as given by Eq. 1. Parameters including number of filters, filter size and stride in CNN are shared among segments s(1) r , · · · , s(M) r , and vary across different datasets. Details are shown in the Experiment Setup section.

Then, we concatenate all convolved and pooled segments to get Hr = [h(1) r , · · · , h(M) r ]T ∈ RM×Kr , where Kr is the number of filters in CNN1D. Next we applied an RNN layer on Hr and denote the output as Qr such that Qr = RNN(Hr). And Qr ∈ RM×Ur , where Ur is the number of hidden units in RNN layer. Here we use the widely-applied self-attention mechanism [Lin et al., 2017] as it is a natural choice to get better results by taking advantage of the correlations or importance of segments. It also generates attention weights Ar that could represent teacher's behaviors on each segment. The attention weights are calculated by Eq. 2.

where W ∈ RUr×1 , Ar ∈ RM×1 . We then multiplied the RNN output Qr with corresponding attention weights Ar.

The weighted output Gr is given by Eq. 3.

where Gr ∈ R1×Ur . Finally, the weighted output Gr is further transformed by a dense layer with weights Wd ∈ RUr×C to output logits Or ∈ R1×C .

For simplicity, we can summarize from Eq.1 to Eq.4 to represent the teacher model Fr as in Eq. 5: Fr takes Xr as inputs and outputs logits Or and attention weights Ar.

The objective function of the teacher model measures prediction accuracy, and also provides knowledge to student model. Typically, Or are transformed by softmax as final predicted probabilities, which can be used as distilled knowledge for student model to imitate. However, sharp distribution (e.g, hard labels) will be less informative. To alleviate this issue, we follow the idea in [Hinton et al., 2015] to produce more informative soft labels. Compared with hard label, the soft label imitation has much smoother probability distribution over classes, thus contains richer (larger entropy) informations. Concretely, we modify classic softmax to S(x, T) by dividing original logits Or with a predefined hyper-parameter T (larger than 1). T is usually referred to as Temperature. The modified softmax (shows ith soft probability) is given by Eq. 6 and the soft predictions are given by Eq. 7.

Finally, we use cross-entropy loss as prediction loss Lteacher (in Eq. 8) to measure the difference between soft predictions Pr,T ∈ R1×C and ground truth Y ∈ R1×C . We optimize teacher model via minimizing Lteacher.

2.3 Imitating Attentions and Targets

After training teacher model on rich data, we now describe the imitation process for the student model. For attention imitation, we mean to mimic attention weights. For target imitation, the student model imitates the following components: 1) soft label that is more informative, 2) hard label that could improve performance (according to [Hinton et al., 2015]), and 3) a trainable combination of both soft label and hard label. Again, we start with constructing the student model Fp using a CNN + RNN architecture, but with fewer filters in CNN and fewer hidden units in RNN. In our experiment, we roughly keep the proportion of hyper-parameters in teacher model to student model the same as the proportion of Dr to Dp using Kr/Kp ≈ Dr/Dp, where Kr and Kp is the number of filters of CNN in teacher model and student model. Also, Ur/Up ≈ Dr/Dp, where Ur and Up is the number of hidden units of RNN in teacher model and student model. Similar to Eq.5, Fp takes Xp as inputs and outputs logits Op and attention weights Ap as in Eq. 9.

Attention Imitation

In Eq.2 we define attention weights to represent the influence of different time segments to the final predictions. We assume that the attention behavior of student model should resemble that of teacher model, and formulate the attention imitation as below. Given Eq.5 and Eq.9, to enforce Ap and Ar to have similar distributions, we minimize the Kullback-Leibler (KL) divergence Latt given by Eq. 10 to measure the information loss from distribution of attention in student model Ap to distribution of attention in teacher model Ar.

Imitating Hard Labels

For hard label imitation, we optimize the student model by minimizing cross entropy loss Lhard (in Eq. 11) that measures the difference between predicted target values and ground truth values Y ∈ R1×C , where C is the number of target classes, Pp,1 = S(Op, 1).

Imitating Soft Labels

Given soft labels from Fr, we produce soft predictions Pp,T by the same temperature T on softmax in student model Fp. Then, we optimize a cross entropy loss Lsoft (in Eq. 12) that measures the differences between student and teacher.

Here, Pr,T is defined in Eq.7. Pp,T = S(Op, T). Since the magnitudes of gradients in Eq.12 is scaled by 1/T 2 as we divided logits by T , we should multiply the soft imitation loss by T 2 to keep comparable gradient during implementation.

Imitating Combined Label

While hard labels provide certain prediction outcomes and soft labels provide probabilistic predictions, the two labels may even be opposite. To resolve the gap between the two labels, a reasonable solution is to combine them to yield uncertain prediction (probabilities of each class). Besides, while hard label imitation helps student model learn more information from data, soft label imitation transfer more knowledge from the teacher model (smoother distribution), each will lead to either more bias (comes from data) or more variance (comes from model). To leverage their benefits and make them complement each other, we propose to minimize a linear combination of hard labels and soft labels, denoted as Pp,comb as the follows:

where w1,w2, b are learnable parameters. For the combined imitation, we also use cross entropy loss Lcomb (in Eq. 14) to define the loss between Pp,comb and ground truth Y .

2.4 Joint Optimization

Finally, for the student model to imitate attentions and targets simultaneously, we jointly optimize all loss functions above. Here, we simply summed them up to get the final objective function Lstudent given by Eq. 15. We summarize the RDPD method in Algorithm 1.

Algorithm 1 RDPD (Xr , Xp, Y , T )

• 1: Build teacher model Fr
• 2: Compute Ar,Or = Fr(Xr)

3:

Pr,T = S(Or, T)

• 4: Lteacher = CrossEntropy(Y , Pr,T )
• 5: while not convergence do
• 6: Update weights of Fr by optimizing Lteacher using back-propagation
• 7: end while
• 8: Build student model Fp
• 9: Compute Ap,Op = Fp(Xp)
• 10: Pp,T = S(Op, T ), Pp,1 = S(Op, 1)
• 11: Pp,comb = S(w1Pp,1 + w2Pp,T + b, 1)
• 12: Latt = DKL(Ap||Ar)

13:

Lhard = CrossEntropy(Y , Pp,1)

• 14: Lsof t = T 2CrossEntropy(Pr,T , Pp,T )

15:

Lcomb = CrossEntropy(Y , Pp,comb)

• 16: Lstudent = Latt + Lhard + Lsof t + Lcomb
• 17: while not convergence do
• 18: Update weights of Fp by optimizing Lstudent using back-propagation
• 19: end while

3 Experiments

3.1 Experiment Setup

Datasets

We used the following datasets in performance evaluation. Data statistics are summarized in Table 1.

PAMAP2 Physical Activity Monitoring Data Set (PAMAP2) [Reiss and Stricker, 2012] contains 52 channels of sensor signals of 9 subjects wearing 3 inertial measurement units (IMU, 100Hz) and a heart rate monitor (HR, 9Hz). The average length of each subject is about 42k points. We down-sample the signals to 50 Hz and choose S = 64 for experiment. We followed the "frame-by-frame analysis" in [Reiss and Stricker, 2012] to pre-process the time series with sliding windows of 5.12 seconds duration and 1 second stepping between adjacent windows. The task is to classify signals into one of the 12 different physical activities (e.g., walking, running, standing, etc.). In our experiment, we choose data of subject 105 for validation, subject 101 for testing, and others for training.

The PTB Diagnostic ECG Database (PTBDB) includes 15 channels of ECG signals collected from controls and patients of heart diseases [Bousseljot et al., 1995]. The database contains 549 records from 290 subjects. We down-sample the signals to 200 Hz and choose S = 500 for experiment. Similar to PAMAP2, we pre-processed the data using "frame-byframe analysis" with sliding windows of 10 seconds duration and 5 second stepping between adjacent windows. Our task is to classify signals into one of the 6 patient groups. In our experiment, we random divided the data into training (80%), validation (10%) and test (10%) sets by subjects.

The Medical Information Mart for Intensive Care (MIMIC-III) is collected on over 58, 000 ICU patients at the Beth Israel Deaconess Medical Center (BIDMC) from June 2001 to October 2012 [Johnson et al., 2016]. In our experiment, we focus on patients with following diseases: 1) acute myocardial infarction, 2) chronic ischemic heart disease, 3) heart failure, 4) intracerebral hemorrhage, 5) specified procedures complications, 6) lung diseases,7) endocardium diseases, and 8) septicaemia, in total 9, 488 subjects. In detail, we extract 6 vital sign time series of the first 48 hours including heart rate (HR), Respiratory Rate (RR), Blood Pressure mean, Blood Pressure systolic, Blood Pressure diastolic and SpO2. We resample the time series to 1 point per hour and choose S = 12 for experiment. Our task is to classify vital sign series into one of the 8 diseases. In our experiment, we random divided the data into training (80%), validation (10%) and test (10%) sets by patients.

Evaluations and Implementation Details

Performance was measured by the Area under the Receiver Operating Characteristic (ROC-AUC), Area under the Precision-Recall Curve (PR-AUC), and macro F1 score (macro-F1). ROC-AUC and PR-AUC are evaluated between predicted probabilities and ground truth. The PR-AUC is considered a better measure for imbalanced data with much more negative samples like our setting [Davis and Goadrich, 2006]. Macro-F1 is a commonly used with threshold 0.5, which determine whether a given probability is predicted as 1 (larger than threshold) or 0 (smaller than threshold).

Models are trained with the mini-batch of 128 samples for 200 iterations, which was a sufficient number of iterations for achieving the best performance for the classification task. The final model was selected using early stopping criteria on validation set. We then tested each model for 10 times using different random seeds, and report their mean values with standard deviation. All models were implemented in PyTorch version 0.5.0., and trained with a system equipped with 64GB RAM, 12 Intel Core i7-6850K 3.60GHz CPUs and Nvidia GeForce GTX 1080. All models were optimized using Adam [Kingma and Ba, 2014], with the learning rate set to 0.001. Our code is publicly available at https://github.com/hsd1503/RDPD.

Comparative Methods

We will compare following methods:

• Teacher: Teacher model is trained and tested on all channels. The model has better accuracy, a much heavier model architecture, and is only available for in-hospital setting where all channels of signals are available. It serves as an empirical upper bound of performance.
• Direct: Direct model is build on the partially observed data using RCNN, without attention imitation and soft label imitation. This model is equivalent to L = Lhard.

• Knowledge Distillation (KD): KD [Hinton et al., 2015] model is constructed on the partially observed data, with soft label imitation and hard label imitation. This model is equivalent to L = Lhard + Lsoft.
• Ours including RDPDr1: The reduced version of RDPD without attention imitation. And the objective function would be L = Lcomb + Lhard + Lsoft. RDPDr2: The reduced version of RDPD without combined labels. This model is equivalent to KD model with attention imitation. And the objective function would be L = Latt + Lhard+Lsoft. RDPD: Our whole model contains all proposed imitations. Using L = Latt + Lhard + Lsoft + Lcomb as objective function.

For all models, we use 1 layer 1-D CNN and 1 layer Bidirectional LSTM. In teacher model, for PAMAP2, the number of filters is set to 64, filter size is set to 8, stride is set to 4 and the number of hidden units is set to 32. For PTBDB, they are set to 128, 32, 8, 32 respectively. For MIMIC-III, they are set to 64, 4, 2, 32 respectively. In RDPD and compared baselines, since they have less input modalities, they have smaller number of CNN filters and RNN hidden units which is set proportionally as introduced before. However, the data length remains the same, so their filter size and stride keep unchanged. T is set to 5 for PAMAP2 and PTBDB, and set to 2.5 for MIMIC-III.

3.2 Results

Classification Performance

We compared the results of RDPD against other baselines and the reduced version of RDPD in Table 2 (PAMAP2 dataset), Table 3 (PTBDB dataset) and Table 4 (MIMIC-III dataset). RDPD outperformed other methods (except Teacher) in most cases and demonstrated the proposed attention imitation and target imitation successfully improved performance of student model. The teacher model performs best among all methods since it is trained using a full datasets with multiple modalities. It serves an empirical upper bound of the performance. In Table 3, RDPD works better than its reduced version in PR-AUC and F1-score but not ROC-AUC. The reason is that classes in PTBDB dataset is very imbalanced, some occasional samples in rare classes distort the final result.

Reduction of Model Complexity

We analyzed model complexity by comparing model size of the teacher model and RDPD. Table 5 shows that the model size of RDPD is only 6 - 7% of the model size of teacher model. According to experimental settings and previous results, other methods have comparable model size with our approach, but their performance are worse. In real world applications such as mobile health or ICU real-time modeling, it is very important that RDPD can achieve both lighter in model and better in performance.

Evaluation against Size of Rich Data

We evaluated the dependency of size of rich data. We used the same validation and test data, but scaled down the size of rich data in training. Figure 2 shows RDPD outperformed baselines even we have few rich data, and would perform better as we got more rich data. This demonstrated the efficacy of RDPD in extracting useful knowledge from rich data and teaching student even under rich-data insufficiency.

Evaluation for the Setting of Low Quality Poor-data

Here we also assess how much benefit the multi-modality data can bring us from low quality poor-data. We performs experiments by adding different level of noise to the entire single modality. The approach of adding noise is: x′ = x ⊕ amp ∗ random normal(-1, 1), where x is the original data and x′ is the noise interfered data, ⊕ is element-wise add, amp is the parameter to control the noise amplitude. From Figure 3, we can see with the increasing amplitude of noise, the performance of both Direct and RDPD decrease. However, RDPD still works better than Direct due to knowledge transfer from Teacher model.

4 RelatedWork

Domain adaptation Domain adaptation techniques address the problem of learning models on some source data distribution that generalize to a different target distribution. Deep learning based domain ad aptation methods have focused mainly on learning domain-invariant representations. For example, [Glorot et al., 2011] and [Chen et al., 2012] stacked Denoising Auto-encoders (SDA) and marginalized SDA respectively to extract meaningful representations. [Ganin et al., 2016] added a Gradient Reversal Layer that hinders the models ability to discriminate between domains. Moreover, [Zhou et al., 2016] transferred the source examples to the target domain and vice versa using BiTransferringDeepNeural Networks, while [Bousmalis et al., 2016] propose Domain Separation Networks. However they need to be trained jointly on source and target domain data and are therefore unappealing to the settings where both data are available.

KnowledgeDistillation KnowledgeDistillation [Hinton et al., 2015] or mimic learning [Ba and Caruana, 2014] are a family of approaches that aim to transfer the predictive power from more accurate deep models ("teacher model") to smaller models ("student model") like shallow neural networks [Hinton et al., 2015], soft decision tree [Frosst and Hinton, 2017] via training smaller models on soft labels learned from deep models. It has been widely used in model compression [Sau and Balasubramanian, 2016], omni-supervised learning [Radosavovic et al., 2017], fast optimization, network minimization and transfer learning [Yim et al., 2017]. Extensions of knowledge distillation unifies distillation and privileged information into generalized distillation framework to learn frommultiple machines and data representations [Lopez-Paz et al., 2015]. The performance of distilled shallow neural networks are often better than models that are directly built on training data. The biggest difference between our approach and knowledge distillation is that, knowledge distillation focus on transfer powerful predictions ability of teacher to student model, while our approach is designed to transfer both behaviors and predictions from rich data modalities to poor data (a single modality).

Attention Transfer Attention mechanism [Bahdanau et al., 2015] was proposed to improve performance of machine translation by paying more attention on relevant parts of the data. Recently, there are several works studying attention transfer [Zagoruyko and Komodakis, 2017; Huang and Wang, 2017] to enhance shallow neural networks. The goal was achieved by learning similar attention models in smaller neural networks, then defining attention as gradient with respect to the input [Zagoruyko and Komodakis, 2017] or use regularization term [Huang and Wang, 2017] to make two models have similar attention weights. Attention transfer has been used in video recognition from web images [Li et al., 2017], cross-domain sentiment classification [Li et al., 2018] and so on. The biggest difference between our approach and attention transfer is that attention transfer is used for model compression on one dataset, while our approach is used to transfer across datasets of very different data modalities.

5 Conclusion

In this paper we proposed to leverage the power of rich data to improve the learning from poor data with RDPD. RDPD learns end-to-end for the student model built on poor data to imitate the behavior (attention imitation) and performance (target imitation) of teacher model by jointly optimizing the combined loss of attention imitation and target imitation. We evaluated RDPD across multiple datasets and demonstrated its promising utility and efficacy. Future extension of RDPD includes considering modeling static meta information as one modality, and learning from less labels.

Acknowledgements

This work was supported by the National Science Foundation, award IIS-1418511, CCF-1533768 and IIS-1838042, the National Institute of Health award 1R01MD011682-01 and R56HL138415.