Towards Reading Comprehension for Long Documents

1 Introduction

Reading comprehension of human languages is one of the most challenging tasks for machines. Collecting clues from passages or contexts, machines are expected to understand the natural language, locate the probable span of the passages, and answer various types of questions [Hermann et al., 2015]. Recently, machine reading comprehension has been successfully deployed in the E-commerce systems such as Alibaba to quickly respond to questions from customers based on the detailed introduction pages.

Studies on reading comprehension have made great progress in the past few years. Attention-based deep neural networks learn to read documents and answer complex questions with minimal prior knowledge of language structure [Hermann et al., 2015]. Besides, the memory network jointly considers inference components as well as a longterm memory component that acts as a dynamic knowledge base [Weston et al., 2014], which helps bridge the gap of requiring co-reference and involving more verbs and nouns. In addition, co-attention mechanisms have shown prominent performance for locating the proper span corresponding to the questions and predicting final answers on machine reading comprehension (MRC) [Cui et al., 2017; Seo et al., 2016; Wang and Jiang, 2017]. Recently, the SLQA+ model and the r-net+ [Wang et al., 2017] model, proposed by Alibaba iDST NLP and Microsoft Research Asia respectively, have exceeded human-level performance according to the SQuAD's exact match (EM) metric [Rajpurkar et al., 2016].

Tasks of reading comprehension are much more difficult in many real-world scenarios. Figure 1 shows an example of the long document reading comprehension (LDRC) task. A decent question and answer (QA) system is expected to provide the precise answers to various types of questions over an entire document, rather than a short paragraph of at most several hundreds of tokens. For instance, the context in SQuAD has at most 800 words in each paragraph, while a single article has 150 paragraphs with 17174 tokens at most. Concatenating all paragraphs to one continuous context would not be a wise solution, as it would be difficult to focus on the targeted information, and prone to being misguided in the attention calculation. Hierarchical structure [Li et al., 2015] has been proposed to build embeddings for various structural levels, such as document level, paragraph level and even sentence level. However, implementing a large number of zero paddings and constructing dynamic graph may degrade the precision and efficiency of the machine reading comprehension system for LDRC tasks.

Intuitively, it can be observed that human written documents can always be split into paragraphs deliberately, where each paragraph expresses relatively independent meaning among the whole document. Therefore, we adopt a hierarchical LSTMmodel [Li et al., 2015] to learn the representation for each paragraph. We also implement a bi-directional match-LSTM [Wang and Jiang, 2016] to learn a hidden representation of each paragraph in the document for reflecting its relation to the question. The relationship between each paragraph and the question can then be figured out, providing us with the evidence to choose the best paragraph to find the answer. Moreover, the Pointer Net model [Vinyals et al., 2015]

Context Query:

1. … were added to the new nonPar

metropolitan county of Cheshire …

1. ... the county palatine boundaries remain the same in 1850s… Par
2. ... In the 19th century, the county contained several mill towns … Par
3. ... the administrative county of Lancashire was created … Par
4. ... During the 20th century , it started to be increasingly urbanised Par

…

1. ... the county shared in the national tradition of balladry … Par
2. When did the county become more urbanised ?

Correct Paragraph:

• Paragraph 5

Answer span in paragraph:

• 29 -- 30

Answer span in document:

• 983 -- 984

Answer:

• 20th century

...

Figure 1: An example of long document reading comprehension task.

is leveraged to find the most potential paragraph, while the softmax outputs can also provide the top-K best paragraphs, similar to how a recommender system works [Resnick and Varian, 1997]. Then, we can apply the state-of-the-art models to focus on these short paragraphs and analyze the most appropriate span of answers. The SQuAD dataset provides up to 150 paragraphs for each article, which can be used for the LDRC task. Experiments on SQuAD show that our proposed model performs better than simply applying single-paragraph MRC models such as Match-LSTM with Pointer [Wang and Jiang, 2017], AoA network [Cui et al., 2017] with Pointer, BiDAF [Seo et al., 2016], as well as structural model, i.e., Hierarchical-LSTM [Li et al., 2015].

Our contributions can be summarized as follows.

• We leverage the characteristic that documents are usually divided to paragraphs, and combine the matchLSTM network and the hierarchical-LSTM network to find the best paragraph for answering queries related to long documents. Then, the LDRC task can be simplified to reading comprehension task on single-paragraph.
• The proposed MH-LSTM model has achieved an exact match score of 58.83% and an F1 score of 67.15% on the modified development set of SQuAD, where it correctly locates 80.91% of the original paragraphs of the answers. It outperforms existing approaches for reading comprehension on single-paragraph.

2 RelatedWork

Three Types of Reading Comprehension Tasks: The first type comes from multiple choice questions, where the models are asked to choose one answer from several candidates, such as MCT [Richardson et al., 2013]. The second type is cloze style comprehension, either with given candidate tokens such as CBT [Hill et al., 2016] or without any candidate such as CNN/Daily Mail [Hermann et al., 2015]. The third type includes analyzing the context and answering the questions without options. Dataset such as SQuAD [Rajpurkar et al., 2016] is usually taken as the standard for comparing the reading comprehension capabilities of machine and human. In this paper, we study the reading comprehension task for long documents and experiments are conducted over SQuAD.

Single-paragraph Reading Comprehension: Reading comprehension on single-paragraph has gained efforts from researchers. Existing work includes developing various arrangements of layers of neural networks such as recurrent neural network (RNN) [Sutskever et al., 2014], convolution neural network (CNN) [Hu et al., 2014], fully-connected layers [Botha et al., 2017], and designing different attention mechanisms [Hermann et al., 2015] and memory mechanisms [Weston et al., 2014], for providing better answers to the questions. In particular, the co-attention model on both context and query is one of the preeminent solutions to this task. For example, BiDAF [Seo et al., 2016] is a hierarchical multi-stage architecture for modelling the representation of the context paragraph at different levels of granularity. The model constructs the character embedding layer and the word embedding layer as inputs to map each word to a vector space. Then, the attention flow layer is set to couple the query and context vectors and produce query-aware feature vectors for each token in the context. The attention flow layer connects to multi-layer bi-LSTM and finally the model provides answers to the queries.

Hierarchical Structure: Regarding long documents with individual paragraphs, most models for single-paragraph comprehension fail to work well. Treating the whole document as one paragraph would increase the possibility of vanishing or exploding gradient during training and make it tough to calculate accurate attention. The hierarchical structure is proposed to learn the representation of different levels of the documents. Hierarchical-LSTM [Li et al., 2015] is a typical hierarchical structure to learn the representation for paragraphs and documents. Together with attention, the learned embeddings can reserve most of the context and approximately recover the origin sentences.

Natural Language Inference: With the help of the hierarchical representation, the potential span of the answer can be fixed to one or several short paragraphs. Hence, it is necessary to find the relationship between each paragraph and the corresponding query. Natural language inference (NLI) is the task of determining whether one can infer one hypothesis from another premise [MacCartney, 2009]. Several attention models have been verified to be promising in addressing this problem, such as match-LSTM [Wang and Jiang, 2016]. Match-LSTM processes the premise and the hypothesis using two LSTMs. Then the model calculates the attention from premise to hypothesis, and feeds them to another LSTM network. The state of the last cell of this LSTM network can be taken as the evidence for natural language inference.

3 Match-based Hierarchical AttentionModel

Finding the appropriate paragraph or several sentences as the evidence is the core of our approach for the LDRC task. Given a query and a corresponding document context, it is necessary to learn the hierarchical representation of different granularity and figure out the inference between each paragraph and the query. The structure of our model is illustrated in Fig. 2, abbreviated as MH-LSTM.

3.1 ProblemFormulation

The problem is formulated as follows. Given a full document ˜ P∗ with several paragraphs (which we refer to as a piece of context, and a query ˜ Q∗ related to the context), the preprocessing script on the dataset would label the paragraph ˜ a∗ where the answers locate. Therefore, the dataset can be represented by a tuple {(˜ Pn, ˜ Qn, ˜ an)}Nn=1 , where N is the number of tuples in the dataset.

3.2 Design of the Model

Token Embedding Layer

The token embedding layer maps each token in the document to a high-dimensional vector space. The words in the document are first tokenized into individual tokens. Specifically, we add a paragraph break "<par>" at the end of each paragraph. Then the tokens are transformed to the corresponding embeddings by two mechanisms. The first mechanism comes from word-level embedding. We use pre-trained word vectors, GloVe [Pennington et al., 2014], as the fixed word embedding of each word, where the missing words from GloVe are denoted by "<unk>". The second mechanism computes the character-level embedding. Characters of every word are embedded into vectors and fed into CNN as introduced by Kim [Kim, 2014]. The outputs of the CNN are max-pooled over the entire word as the fixed-size embedding. The wordlevel embedding and character-level embedding are then concatenated to represent the embedding of each token, which has a dimension of d.

Preprocessing Layer for Query

Let Q denote the embeddings of words in a query from token embedding layer with length |Q|, where Q =<

q1, . . . , q|Q| >∈ Rd×|Q| . We first obtain representation vectors of each word in the query as well as the whole query sentences by feeding the tokens to a one-layer bi-LSTM, i.e.,

where HQ = [yQ 1 , . . . , yQ |Q| ] ∈ RhQ×|Q| represents the matrix of the outputs of bi-LSTM, where each column represents the concatenation of forward and backward outputs of the corresponding word from the LSTM network. Besides, qrep concatenates the forward and the backward final states of bi-LSTM, and hQ is the hidden size of the bi-LSTM layer.

Preprocessing Layer for Context

Let P ′ denote the embeddings of each word in the document from token embedding layer with length |P ′ |, where P ′ =< p′ 1, . . . , p′ |P ′ | >∈ Rd×|P ′ | . To discover the relationship between each word in the document and the query, we concatenate qrep to the embeddings of each word in the document, i.e., P =< p1, . . . , p|P| >∈ R(d+hQ)×|P| , where pt = p′ t||qrep, t ∈ [1, |P|] and "||" represents the concatenation. As the length of a document is usually far longer than a paragraph, it would be much more difficult for training [Glorot and Bengio, 2010]. Here, we follow the approach proposed by Zilly [Zilly et al., 2017] and leverage a twolayer recurrent highway network (RHN) for the preprocessing of context. Specifically, let WPH,T,C ∈ RhP ×(d+hQ) represent the weight matrices of the nonlinear transformation. Let RPHl,Tl,Cl ∈ RhP ×hP and bPHl,Tl,Cl ∈ RhP ×hP denote the weights and biases of the transformation, and carry gates at layer l ∈ {1, 2}. Then the calculation of the recurrence process could be expressed by

where I{·} is the indicator function, and σ(·) represents the sigmoid function. The initial states sP0,t are set to pt-1. We collect the outputs of layer 2, i.e. sP2,t , and take them as the representations of each token in the context, i.e.,

Hierarchical Mask Layer

As we mainly focus on the representation of paragraph-level, we should extract the evident vectors that contain sufficient knowledge of the paragraphs. Hence, we set up a mask layer, which only reserves the outputs from the RHN whose input corresponds to the paragraph break "<par>". The vectors are then fed into a bi-LSTM network to obtain the representations of each paragraph, i.e.,

where |M| is the length of the reserved representations after being masked, and it is exactly the number of paragraphs in the document.

Attention Layer

This layer is designed to interpret the degree to which the network attends to every paragraph in the document. Inspired by the word-by-word attention mechanism, we concentrate on the attention from each paragraph to every token in the query. Specifically, the attention of the t-th paragraph on the i-th token in the query in the forward direction can be calculated by:

where -→α t is the normalized attention from paragraph t to every token in the query.

Match-LSTM Layer

We are expected to focus on the textual entailment by treating the query as a premise and the paragraphs as hypotheses. The match-LSTM layer is used for providing the evidence for the inference [Wang and Jiang, 2017], which is actually a onelayer LSTM network. Let --→ HO = [-→ yO 1, . . . , -→ yO |M|] denote the hidden outputs of the match-LSTM, satisfying:

where

In order to improve the attention between each paragraph and query, we take the outputs from the match-LSTM into account for Eqn.(8), i.e., modified to

Similarly, we build a match-LSTM in the backward direction, denoted as ←-- HO = [←- yO 1, . . . , ←- yO |M|], to obtain the precise representation based on the surrounding contextual paragraphs. Finally, the outputs of the bi-directional matchLSTM network would be the concatenation of the outputs of the forward match-LSTM and the outputs of the backward match-LSTM, i.e., HO = [-→ yO 1||←- yO 1, . . . , -→ yO |M|||←- yO |M|].

Output Pointer Layer

The final outputs should indicate the probability of a paragraph to be the most appropriate one for answering the query. To address this problem, we leverage the novel attention mechanism to model these probabilities, i.e., the outputs of the match-LSTM layer connect to a sequence-to-sequence model [Sutskever et al., 2014]:

where ye∗ denotes the hidden outputs of the encoder, and yd represents the hidden output of the decoder. Finally, we focus on the output distribution over each paragraph, i.e., for the t-th paragraph,

We then take softmax normalization s on the distribution to represent the probability of each paragraph to be the best candidate:

Answer to the Query

The most appropriate paragraph would be targeted by s. Then we could leverage the reading comprehension model for single-paragraph on the selected paragraph to generate the answer. In this paper, we use BiDAF that has been verified to be effective on the original SQuAD dataset.

Training

We define the training loss (to be minimized) as the sum of the negative log probabilities that the softmax outputs can correctly identify the short paragraphs where the original answers locate, averaged over all examples:

where θ represents the trainable variants in the model as mentioned in this section, and s˜ an|˜ Pn, ˜ Qn represents the˜ an-th softmax value from the output layer, given ˜ Pn, ˜ Qn as inputs.

4 Experiments

4.1 Dataset

We use the SQuAD v1.1 dataset to conduct the experiments. Passages in SQuAD come from 536 articles in Wikipedia covering a wide range of topics. Each article contains up to 150 paragraphs, and each paragraph contains up to 785 words. Each paragraph has around 5 queries with the corresponding answers, and the answer to each query has a prelabeled position in the paragraph. Totally, there are 107,785 queries.

We first merge all paragraphs of every article to substitute the original context. Specific paragraph breaks are added as mentioned in section 3, and the answer spans are also updated. Meanwhile, we label the original paragraph, which would be used in the LDRC task. After merging the paragraphs, there would be at most 17174 tokens in each passage. We split the training set into a training set (81,398 tuples) and a validation set (4,285 tuples). The development set of SQuAD is treated as the test set in our experiments.

4.2 Experiment Settings

We use the pre-trained GloVe [Pennington et al., 2014] parameters with 300-dimension embeddings to initialize the model. The word embeddings would not be changed during training. Besides, the hidden size of every recurrent network involved in our model is fixed as 150 in single direction. The labelled paragraphs where the original answers occur are transformed to one-hot vectors. The learning rate is dynamically updated according to Adam optimization [Kingma and Ba, 2014]. Besides, in order to avoid overfitting or out of memory, the gradient descent is updated by minibatch of 32 instances. We do not observe performance gains using dropout on this task.

We mainly focus on three metrics in the experiments: (1) the percentage of exact match (EM) with the ground truth answers, (2) the word-level F1 score when comparing the tokens in the predicted answers with the tokens in the ground truth answers, and (3) the proportion of identified paragraphs which are exactly the short paragraphs where the original answers locate (CP). The experiments run on two GTX1080Ti GPUs.

4.3 Results

We establish two categories of models, i.e., non-hierarchical models and hierarchical models, to examine their performances on the LDRC task. Both categories of models take the word-level and character-level embeddings as inputs.

For the non-hierarchical models, we set up one baseline model and select three reputable models from the SQuAD leader board. The one-layer vanilla LSTM that takes the tokens from an entire document as inputs is used as the baseline model, and then the outputs connect to a Pointer network to find the beginning and the end of the span of the answer. We examine the performance of match-LSTM with answer pointer [Wang and Jiang, 2017] on the LDRC dataset. We also modified the AoAmodel [Cui et al., 2017], i.e., replacing the dot product in the last step of attention-over-attention process by a fully-connected feedforward network that takes the concatenation of attentions as inputs and obtains 150 output neurons. Therefore, we would get one representative vector for each paragraph as its representation. Similarly, a Pointer network is leveraged to seek the answer. Meanwhile, we directly implement BiDAF [Seo et al., 2016] on this LDRC task.

For the hierarchical models, we first compare the performance of our model to the hierarchical-LSTM [Li et al., 2015]. Specifically, we calculate the paragraph-level representation rather than the sentence-level representation. A biLSTM is implemented, and the hidden state of the last cell is taken as the representation of the query. We use dot product to depict the relationship between each paragraph and the query. The paragraph with the highest probability are then treated as the context for BiDAF to extract the most relevant answer.

Our results are summarized in Table 1. Due to the fact that there are some tokens from the queries which only appear once in the document, it is straightforward to find the corresponding spans of answers to the queries that contain these categories of tokens. However, the long document increases the difficulty for LSTM to learn the answer to queries. The performance of BiDAF model for single-paragraph decreases by about 26%, where the EM decreases from 66% to 50%, and F1 decreases from 76% to 58%. We find that the hierarchical models have greater probability to find the correct paragraph than the non-hierarchical models. The hierarchicalLSTM could help increase the probability of choosing the correct paragraph to 71%, while our MH-LSTM could further improve the CP to nearly 81%. The EM reaches 59% by our proposed model, indicating that the MH-LSTM is effective for the LDRC task.

4.4 Further Analysis

Performance with Various Lengths of Document

We examine the performance of our MH-LSTM on documents with various lengths. As there are only 48 articles with 48 different lengths in the development set of SQuAD, we construct "sub-articles" for experiments by enumeration. Specifically, we randomly enumerate several sets of paragraphs (which must include the correct paragraph) from the same article, treated as the contexts for queries. Therefore, we have a new LDRC dataset with more context-queryanswer tuples. The sub-articles are divided into 7 groups by their lengths: less than 500 tokens, 500-1000, 1000-2000, 2000-4000, 4000-8000, 8000-16000 tokens, and more than 16000 tokens. We set the number of tuples of sub-articles in each group to 1000. We repeat the random enumeration 20 times and calculate the average value for each performance metric. The performance metrics, i.e., CP, EM and F1, on each group of data are plotted in Fig. 3. We observe that performance of our model decreases slightly, compared with that of directly using the BiDAF model for single-paragraph.

Performance with Various Numbers of Paragraphs

We also examine the performance of our MH-LSTM on documents with various numbers of potential paragraphs. We seek to analyze the contribution of the number of reserved paragraphs for comprehension. Intuitively, the more paragraphs we reserve, the greater probability we would have to reach the correct paragraph. However, due to the fact that longer documents would decrease the performance of comprehension model for single-paragraph, it is necessary to restrict the length of the reserved passages. We evaluate the performance of our model when we reserve several paragraphs with the highest values of the output neurons rather than reserve only one paragraph, as shown in Fig. 4. Both EM and F1 increase until two or three reserved paragraphs, and then they begin to drop as the number of paragraphs increase. Up to 3% increase of EM and F1 could be obtained when reserving one or two more paragraphs.

4.5 Case Study

We found certain instances that can support the importance of LDRC and the strength of our proposed model. We illustrate four typical instances in Fig. 5.

The first instance indicates the condition where our proposed MH-LSTM helps find the correct paragraph and then the single passage MRC model produces the expected answer. Without locating paragraph, the BiDAF model would fail to find the correct answer. However, in the second instance when both models fail to find the pre-labeled paragraph, the two models could still answer correctly due to the supplementary information in other paragraphs of the document. Similarly, the whole document would help single passage MRC model to solve some queries although the model may choose a "wrong" paragraph. For these queries, the model should have produced wrong answers in the pre-labeled paragraphs, such as the third instance. Meanwhile, the forth instance implies that the two models may fail to find the correct paragraph, or produce the expected answer, which requires future efforts to address this problem.

5 Conclusion

Reading comprehension is a challenging task for machines. Recently, existing work has focused on the reading comprehension task for single-paragraph or short passages. However, these approaches fail to work well on the long documents. In this paper, we study the reading comprehension task for long documents to find paragraph-level representation and natural language inference. We design a model that takes advantages from hierarchical-LSTM and match-LSTM to find the most appropriate paragraph for the answers. The SQuAD dataset is modified for experiments, where the original development set is treated as the test set. Results show that the proposed MH-LSTM can outperform some existing known comprehension models for single-paragraph by at least 20% regarding exact match (EM), F1 and the proportion of identified paragraphs which are exactly the short paragraphs where the original answers locate (CP).

Acknowledgments

This work is partially supported by the National Key Research and Development Program No. 2017YFB0803302, the National 973 Grant No. 2014CB340405, and the National Natural Science Foundation of China under Grant Nos. 61572051 and 61632017.