A Probabilistic Lexical Approach to Textual Entailment

1 Empirical Evaluation

Experimental Setup

Though empirical modeling of semantic inferences between texts is commonly done within application settings, there is no common dataset available to specifically evaluate a textual entailment system. In order to test our model we therefore needed an appropriate set of text-hypothesis pairs. We chose the information seeking setting, common in applications such as QA and IR, in which a hypothesis is given and it is necessary to identify texts that entail it. The evaluation criterion is application-independent based on human judgment of textual entailment.

Experiments were done on the Reuters Corpus Volume 1. An annotator chose 50 sentential hypotheses from the corpus sentences. We required that the hypotheses convey a reasonable information need in such a way that they might correspond to potential questions, semantic queries or IE relations. We created a set of candidate entailing texts for the given set of test hypotheses, by following common practice of morphological and WordNet-based expansion. Boolean search (with expanded words) was performed at the paragraph level over the full Reuters corpus. Paragraphs containing all original words of the hypothesis or their morphological derivations were excluded from the result set and selected a random set of 20 texts for each of the hypotheses.

The resulting dataset was given to two judges to be annotated for entailment. Corresponding to the notion of textual entailment, judges were asked to annotate a text-hypothesis pair as true if, given the text, they could infer with some confidence that the hypothesis is true. They were instructed to annotate the example as false if either they believed the hypothesis to be false given the text or if the text is unrelated to the hypothesis. A subset of 200 pairs was crossannotated for agreement, resulting in a moderate Kappa statistic of 0.6. Overall, the annotators deemed 48% of the text-hypothesis pairs as positive examples of entailment.

Empirical Results

We trained our model on the Reuters corpus, classified the text-hypothesis pairs of the dataset and compared the model's prediction with the human judgments. The resulting (macro) average accuracy was 70%. Since our dataset did not include texts containing all content words of the hypotheses, the baseline model would have predicted none of the pairs to be correct (i.e. the text entails the hypothesis) yielding an average accuracy of only 52%.

We also compared our system's ranking ability. The entailment confidence score was used to rank the various texts of each hypothesis. The average confidence weighted score (cws) was measured for each hypothesis. The resulting cws macro average was 0.54 compared to average cws of 0.49 for random ordering. For further comparison, a state of the art idf semantic overlap measure [Saggion et al., 2004; Monz and de Rijke, 2001] achieved a score of 0.51. Though our model performs just slightly better, the results are statistically significant at the 0.02 level.