An Iterative Multi-Source Mutual Knowledge Transfer Framework for Machine Reading Comprehension

1 Introduction

As one of the core abilities of artificial intelligence, Machine Reading Comprehension (MRC) attempts to enable machines to answer questions after reading a passage. Due to its valuable industry applications such as search engines, it has always been one of the research focuses in natural language processing [Seo et al., 2017; Hu et al., 2018; Yu et al., 2018; Devlin et al., 2019; Liu et al., 2020]. As the basis for MRC studies, many datasets such as SQUAD [Rajpurkar et al., 2016] and NEWSQA [Trischler et al., 2017] have been developed in recent years. However, some domain-specific datasets are limited, and thus they are unable to individually train a domain-specific MRC model with satisfying performance. Therefore, how to overcome the shortage of domainspecific training data has become one of the important research directions in MRC.

∗These authors contributed equally to this work.

†Corresponding author

To this end, one direction is to mix multiple domains of data and train a unified MRC model [Xu et al., 2019] and another line of work introduces silver data by automatically generating questions [Duan et al., 2017; Song et al., 2018] . However, such approaches ignore the differences between these domains and introduce noises, resulting in the underutilization of these data. Recently, researchers have explored several transfer learning methods for MRC. Min et al. [2017] and Chung et al. [2018] first introduced fine-tuning into MRC. Furthermore, Talmor and Berant [2019] investigated the effect of domain-level similarities on fine-tuning. Despite their successes, fine-tuning still has some drawbacks. First, it is a knowledge transfer method with one-to-one correspondence, which also cannot effectively utilize multiple source-domain data with different impacts. Second, the significant difference between the source-domain and the targetdomain often makes the one-pass transfer procedure of finetuning fail to fully exploit domain-shared knowledge. It can be said how to fully exploit the training data in different domains to construct domain-specific MRC models still remains unresolved.

In this work, we propose a novel iterative multi-source mutual knowledge transfer framework for MRC. The intuition behind our method includes two aspects. First, each domain-specific MRC model can be enhanced by the training data and models of other domains, whose effects depend on their domain-level similarities to the target domain. Second, a process of iterative mutual reinforcement can achieve better knowledge transfer, where domain-shared knowledge among training data in different domains can be fully exploited to benefit all domain-specific MRC models.

Figure 1 shows our framework for MRC. Significantly different from previous approaches, the execution process of our framework involves synchronous executions of multiple many-to-one knowledge transfers in an iterative manner. More specifically, as shown in the blue dashed part of Figure 1, we conduct a many-to-one transfer involving two steps to update a target-domain MRC model: First, we apply knowledge distillation [Hinton et al., 2015; Tan et al., 2019; Clark et al., 2019] to establish a multi-domain MRC model, which is trained to simultaneously fit the training data and match the outputs of other domain-specific MRC models. Particularly, we introduce domain-level similarities to control the impacts from the other domains on the construction of this multi-domain MRC model. Second, after the initialization with the multi-domain MRC model, we employ knowledge distillation to fine-tune the target-domain MRC model, supervised by both its training data and the outputs of its previous best model.

Compared with previous methods [Min et al., 2017; Chung et al., 2018; Talmor and Berant, 2019], in this work, we make the following contributions: (1) We extend the existing knowledge transfer with one-to-one correspondence into a new setting of many-to-many mutual knowledge transfer for MRC; (2) We exploit multi-source mutual knowledge transfer for MRC, based on knowledge distillation, to particularly leverage the similarity between each domain and the target domain for better knowledge transfer; and (3) We propose to perform multi-source mutual knowledge transfer in an iterative manner, so that domain-shared knowledge can be fully exploited to benefit all domain-specific MRC models.

We conduct experiments on several commonly-used datasets. Experimental results and in-depth analyses show that our framework achieves significantly better performance than the dominant knowledge transfer approaches for MRC.

2 RelatedWork

Transfer learning for MRC. To address the lack of largescale domain-specific training data in MRC, previous studies use upstream datasets to enhance the performance of MRC models, including word embeddings [Pennington et al., 2014] and language models [Devlin et al., 2019]. Meanwhile, more attempts have been made to explore transfer learning approaches for MRC, where fine-tuning is the most common method achieving satisfying results on several datasets [Min et al., 2017]. Particularly, Golub et al. [2017] introduced a two-stage synthesis network to produce the target-domain synthetic dataset, which can be used to fine-tune the MRC model. Chung et al. [2018] demonstrated that transfer learning is helpful even in unsupervised scenarios. Xu et al. [2019] explored a multi-task learning framework to exploit different domains of datasets for MRC, where two re-weighting strategies are investigated. Talmor and Berant [2019] performed a thorough empirical investigation of generalization and transfer over different domains of MRC datasets.

Unlike these work, our framework iteratively transfers the knowledge among multiple MRC datasets, where each domain-specific MRC model and its training data can be iteratively and differentially exploited to enhance other models. To the best of our knowledge, such an approach has not been fully explored before in MRC.

Knowledge distillation based NLP. Recently, knowledge distillation has been successfully applied into many tasks, such as model compression [Kim and Rush, 2016] and knowledge transfer [Zeng et al., 2019; Tan et al., 2019].

Similar to our first many-to-one knowledge distillation, Tan et al. [2019] presented a distillation-based approach to boost the performance of multilingual machine translation. Furlanello et al. [2018] proposed BAN that is a simple retraining procedure. During this procedure, after the teacher model coverges, they initialize a new student model and train it with the dual goals of predicting the correct labels and matching the output distribution of the teacher model. On this basis, Clark et al. [2019] extended BAN to BAM, which utilizes multiple task datasets simultaneously and thus can be considered as the multi-task version of BAN.

The most related work to ours is [Clark et al., 2019]. However, our work significantly differs from this work in

the following important aspects: (1) During the first step of knowledge distillation, Clark et al. [2019] applied knowledge distillation to transfer the knowledge of all single-task models to a unified multi-task model. By contrast, we distills all non-target single-domain models into a multi-domain model, which leverages domain-level similarities and thus is a domain-specific one, resulting in the effectiveness of the iterative execution of our framework. (2) During the second step of knowledge distillation, our teacher model is the current domain-specific model rather than the multi-task model adopted by [Clark et al., 2019]; (3) We focus on MRC rather than natural language inference in [Clark et al., 2019]. Note that experimental results reported in Table 1 verify the superiority of our framework over [Clark et al., 2019].

3 BERTQA

In this section, we briefly introduce BERTQA [Devlin et al., 2019] that is chosen as our basic MRC model. Figure 2 shows the architecture of BERTQA. Given a passage p = p1, p2...p|p| and a question q = q1, q2...q|q|, we first pack them into a single sequence s=[<CLS>, q, <SEP>, p, <SEP>], where <CLS> is the token used for classification, <SEP> is the token separating q and p, and the sequence length |s| = |p| + |q| + 3. For each token st, we construct its input representation as h(0) t = etok t + epos t + eseg t , where etok t , epos t , and eseg t are the token, position, and segment embeddings for st, respectively. Please note that all tokens of p share a same segment embedding, and all tokens of q use a same segment embedding.

Such input representations are then fed into a BERT encoder with successive L Transformer [Vaswani et al., 2017] layers, each of which is composed of two sub-layers with the layer normalization mechanism: a multi-head self-attention and a position-wise fully connected feed-forward neural network. Finally, on the top of the hidden states h(L) t at the L-th layer, we use a linear layer with softmax functions to produce the probability of each token st to be the beginning or ending position of the answer span:

Start/End Span

whereWs∗ are related parameters.

To train BERTQA, we define the training objective based on the log-likelihood of the true beginning and ending positions:

where the quadruple qu = (p, q, b, e) denotes an example of the training data D, with b and e as the answer beginning and ending position, respectively, and θ is the set of model parameters. At inference time, the span (b,e) where a ≤ b with the maximum P1b · P2e is chosen as the predicted answer.

4 Our Framework

In this section, we describe our framework in detail. As displayed in Figure 1, it is a multi-source multi-pass knowledge transfer process. To facilitate the understanding of our framework, we also summarize its training procedure in Algorithm 1. We first train all initial domain-specific MRC models {θ(0) i }Ni=1 using their own training data {Di}Ni=1 , respectively (Line 1). Then, we calculate the domain-level similarities between training corpora, denoted by a matrix S={si,j}, 1≤i, j≤N (Line 2), which can be subsequently exploited to distinguish the impacts of other domains on the construction of each multi-domain MRC model. Afterwards, we conduct K iterations of many-to-many knowledge transfer (Lines 3-13). Concretely, at the k-th iteration, we synchronously perform multiple many-to-one knowledge transfers from other domains to each target-domain model θ(k+1) i , 1≤j≤N (Lines 4-12). Such a transfer mainly consists of two substeps: (1) We construct a multi-domain MRC model φ(k) i using {Dj}j̸=i and {θ(k) j }j̸=i (Lines 5-6), and (2) We use φ(k) i , Di and θ(k) i to establish θ(k+1) i (Lines 7-11). Through this multi-pass procedure, different domain-specific MRC models can constantly absorb domain-shared knowledge to reinforce each other.

Clearly, the calculation of domain-level similarities S and these aforementioned two substeps are the most important steps in our framework. We describe them below.

Calculations of Domain-level Similarities S. To this end, for the training data Di of each domain, we first collect its all questions to form a large question document Qi, and then use the TF-IDF representation to represent the domain Qi. Finally, we directly define the domain-level similarity si,j between i-th and j-th domains as the cosine similarity based on their TF-IDF representations: si,j = cosine(TF-IDF(Qi),TF-IDF(Qj)). Then, we normalize this similarity as si,j = si,j ∑j′̸=i si,j′ .

Please note that we only use the question words to calculate the domain-level similarities. This is because the passages of different domains may be similar. For example, SQUAD and HOTPOTQA are both constructed from Wikipedia. By contrast, the question words are able to better reflect the domainlevel differences.

Substep 1: Constructing Multi-domain MRC Model φ(k) i . Inspired by [Tan et al., 2019; Clark et al., 2019], we employ knowledge distillation to construct multi-domain MRC model φ(k) i , where we train φ(k) i to fit the training data {Dj}j̸=i and match the outputs of individual domain-specific models {θj}j̸=i simultaneously according to their domainlevel similarities to Di.

Specifically, for the i-th domain, we mix training corpora {Dj}j̸=i from other domains to form a mixed training corpus Mi, where we conduct up-sampling on {Dj}j̸=i to make all domains have the same amount of training instances. For a sampled training example qu = (p, q, b, e) from Dj, two kinds of errors are involved:

1. Negative likelihood L1(qu; φ(k) i ): it is used to enable φ(k) i to fit training data Dj. Formally, it can be formulated as

where P1b (φ(k) i ) and P2e (φ(k) i ) are defined in Equation 1 and 2, respectively.

1. Prediction divergence L2(qu; θ(k) j ; φ(k) i ): it aims to keep the outputs of φki consistent with those of domainspecific MRC model θ(k) j , and is formally defined as themean squared error between outputs of θ(k) j and φ(k) i :

Thus, the joint error for the training instance qu is

Here we follow Clark et al. [2019] to linearly increase λmd from 0 to 1 throughout training. In this way, φ(k) i will absorb the guidance information from its teachers models {θ(k) j }j̸=i as much as possible during the early stage of training, and then rely more on the training data so it can learn to surpass its teachers.

Finally, the objective function over Mi becomes

Algorithm 1 Iterative Multi-source Mutual Knowledge Transfer Framework for MRC

Input: Training corpora {Di}Ni=1 , validation sets {Dv i }Ni=1 , the maximal iteration number K

Output: Domain-specific MRCmodels {θ(K) i }Ni=1

• 1: θ(0) i ←TrainModel(Di), i = 1, 2, ...,N
• 2: Compute the similarity matrix S={si,j}, i, j = 1, 2, ...,N, where si,j indicates the domain-level similarity between Di and Dj // Step 1
• 3: for k = 0, 1, 2, ...,K-1 do
• 4: for i = 1, 2, ..., N do
• 5: Initialize the multi-domain MRC model φ(k) i
• 6: φ(k) i ← TrainMultiDomainModel({Dj}j̸=i, {θ(k) j }j̸=i, S) //Step 2
• 7: Initialize θ(k+1) i with φ(k) i
• 8: θ(k+1) i ←FineTuneModel(Di, θ(k) i ) // Step 3
• 9: if EvalModel(θ(k+1) i , Dv i ) < EvalModel(θ(k) i ,Dv i ) then
• 10: θ(k+1) i ← θ(k) i
• 11: end if
• 12: end for
• 13: end for

Where Mi is the previously-mentioned mixed training corpus. Note that by introducing si,j, we can effectively differentiate the impacts of different domains on φ(k) i .

Substep 2: Constructing Domain-specific MRC Model θ(k+1) i . During this process, we first initialize θ(k+1) i with φ(k) i , and then also apply knowledge distillation [Tan et al., 2019; Clark et al., 2019] to update θ(k+1) i using its previous model θ(k) i and training corpus Di.

Similarly, the objective function of each training example qu fromDi involves two kinds of errors:

1. Negative likelihood L3(qu; θ(k+1) i ): it requires θ(k+1) i to fit the reference of Di:

1. Prediction divergence L4(qu; θ(k+1) i ): it makes the outputs of θ(k+1) i consistent with those of θ(k) i :

Finally, the objective function over Di is defined as

where

where the hyper-parameter λds is adjusted in a similar way as λmd. In particular, we compare the performance of θ(k+1) i and θ(k) i to retain the current best model (Lines 9-11).

As described before, we apply the above procedure to generate all domain-specific models {θ(k+1) i }Ni=1 in a parallel manner, which can effectively reduce the training time of our framework. Furthermore, we repeat this many-to-many knowledge transfer process, until the maximal number of iteration K is reached or all domain-specific models converge.

5 Experiments

5.1 Setup

In our experiments, we use the following datasets:

• SQUAD [Rajpurkar et al., 2016]. It is one of the most popular MRC datasets, which consists of more than 100K instances. Each example is a pair of context paragraph from Wikipedia and a question created by a human, and the answer is a span in the context. Particularly, we use SQUAD v1.0 in this work.
• NEWSQA [Trischler et al., 2017]. It contains about 120K question-answer pairs, all of which are related to CNN articles.
• TRIVIAQA [Joshi et al., 2017]. This dataset includes 95K question-answer pairs from Trivia Database, each of which is paired with a document collected by completing a web search of the question.
• HOTPOTQA [Yang et al., 2018]. Crowdsourcing workers were shown pairs of related Wikipedia paragraphs and asked to author questions that require multi-hop reasoning over the paragraphs. There are two versions of HOTPOTQA, and we use the second version, where 10 paragraphs retrieved by an information retrieval (IR) system are given.
• NQ [Kwiatkowski et al., 2019]. It is a question answering dataset where the numbers of its training, validation and test examples are 307K, 7.8K and 7.8K, respectively. Each example is comprised of a google.com query and a corresponding Wikipedia page.

We adopt the F1 score as our evaluation metric. We sample 300 examples from the training data of each domains as our validation sets. Finally, we evaluate the performance of different models on the official validation sets.

To enhance the performance of our MRC model, we initialize the parameters of the BERT encoding layer using the pretrained model officially released by Google∗ , and randomly initialize other trainable parameters. These models were pretrained on the concatenation of BooksCorpus (800M words) and Wikipedia (2,500M words) via joint modeling of tasks of masked language model and next sentence prediction. We use the uncased base model, which is case insensitive and contains 12 Transformer encoding blocks, each with 12 selfattention heads and 768 hidden units. Moreover, we use the Adam optimizer [Kingma and Ba, 2015] with a learning rate of 1.5×10-5 and a batch size of 12.

∗ https://github.com/google-research/bert

Table 1: Experimental results on various test sets. SAD = SQUAD, NQA = NEWSQA, HQA = HOTPOTQA, and TRIVIAQA = TQA AVE. = average score.

5.2 Baseline Models

We refer to our framework as IMM and compare it with the following baseline models:

• Single. The BERTQA model trained on a single domain training corpus.
• Mix. The BERTQAmodel trained on the mix of all training corpora.
• Fine-tuning (FT). We first train an MRC model on the mix of non-target domain training corpora until convergence, and then update its parameters on the targetdomain training corpus.
• Fine-tuning with the most similar training data (MSFT). A variant of FT, where we only exploit a sourcedomain data that is the most similar to the target-domain data when we using fine-tuning.
• Mixed fine-tuning (MFT) [Chu et al., 2017]. Unlike FT, we resume training the MRC model on a mix of all training data.
• Mixed fine-tuning with the most similar training data (MS-MFT). A variant of MFT, where only the sourcedomain data most similar to the target-domain data is leveraged via mixed fine-tuning.
• Born-Again Multi-task Network (BAM). It is the multi-task version of Born-Again Network (BAN) [Clark et al., 2019]. Using the same many-to-one knowledge distillation procedure as BAM, we first construct a unified multi-domain model using all equally considered domain-specific models and training data, then further fine-train the domain-specific models on individual datasets.

Effect of the Maximal Iteration Number K

From Figure 1, we observe that the maximal iteration number K is an important hyper-parameter, which directly determines the amount of the transferred knowledge into domainspecific MRC models. To investigate the effect of K, we report the average performance of domain-specific MRC models using different Ks on the validation sets. Note that we only keep the best models at each iteration.

Figure 3 illustrates the experimental results using different Ks. At first, the performance of almost all domain-specific MRC models has been improved with the increment of the iteration number. However, when the iteration number is greater than 3, all models tend to converge. Therefore, we directly used K=3 in all subsequent experiments.

Overall Performance

Table 1 shows the experimental results. Overall, the conventional approaches including FT, MS-FT, MFT, MS-MFT and BAM, achieve better performance than both Single and Mix, echoing the reported results in previous studies [Min et al., 2017; Chung et al., 2018; Xu et al., 2019]. Besides, under our framework, the performance of all domain-specific models is further significantly improved. Specifically, the average score of our framework is 77.94, which is +2.91, +2.05, +0.96, +2.23, +1.20, +1.83, +0.83 points higher than those of Single, Mix, FT, MS-FT, MFT, MS-MFT and BAM, respectively. In particular, in all datasets except SQUAD, our model achieves the best performance among all models. Even compared with BAM, our improvement is still remarkable. This result strongly demonstrates the effectiveness and generality of our framework.

Robustness inMRC

To inspect the impact of our framework on enhancing robustness, we follow Hu et al. [2018] to report the results on two adversarial SQUAD datasets, namely ADDONESENT and ADDSENT.

From Table 2, we can observe that the improvement on adversarial data is much higher than the one on original squad dataset. This result is reasonable, because our framework can better absorb the knowledge of other domains, boosting its ability in dealing with the much more confusing answers in adversarial datasets.

5.3 Ablation Study

The main highlights of our framework consist of the usages of domain-level similarities of training corpora to the targetdomain one, and two stages of distillation-based knowledge transfers. To evaluate their effects on our framework, we compared our framework with its following variants: (1) - SIMI. The variant of IMM without domain-level similarities, i.e., we directly set si,j of Equation 7 to 1; (2) -KD1.

The variant of IMMwithout the distillation-based knowledge transfer from {θ(k) j }j̸=i to φ(k) i , where λmd of Equation 6 is directly set to 0; and (3) -KD2. The variant of IMMwithout the distillation-based knowledge transfer from φ(k) i to θ(k+1) i , where λds of Equation 11 is directly set to 0.

From the experimental results shown in Table 3, we can observe that our framework achieves better performance than all its variants. These results show that domain-level similarities, two stages of distillation-based knowledge transfers are indeed beneficial for MRC.

6 Conclusion

In this work, we have presented a novel iterative multi-source mutual knowledge transfer framework for MRC, which enables all domain-specific MRC models to constantly reinforce each other by iteratively and differentially absorbing the domain-shared knowledge from others. Experimental results and in-depth analyses on several MRC datasets strongly demonstrate the effectiveness of our framework.

In the future, we will focus on how to effectively leverage the unlabeled data of different domains under our framework. Besides, we plan to apply our framework to other tasks, such as machine translation [Zeng et al., 2019]. Finally, inspired by studies of multi-domain NMT [Zeng et al., 2018; Su et al., 2019], we will explore a unified multi-domain model for MRC.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61672440), Beijing Advanced Innovation Center for Language Resources (No. TYR17002), and the Scientific Research Project of National Language Committee of China (No. YB135-49). We also thank the reviewers for their insightful comments.