Evaluating an NLG System using Post-Editing

1 Introduction

In this paper we describe an evaluation technique, which looks at how much humans need to post-edit texts generated by an NLG system before they are released to users. Postedit evaluations are common in machine translation [Hutchins and Somers, 1992], but we believe that ours is the first large-scale post-edit evaluation of an NLG system. Mitkov and An Ha [2003] reported a small scale post-edit evaluation of their NLG system.

The system being evaluated is SUMTIME-MOUSAM [Sripada et al, 2003], an NLG system, which generates weather forecasts from Numerical Weather Prediction (NWP) data. The forecasts are marine forecasts for offshore oilrigs. SUMTIME-MOUSAM is operational and is used by Weathernews (UK) Ltd to generate 150 draft forecasts per day, which are post-edited by Weathernews forecasters before being released to clients.

Table 1 shows a small extract from the NWP data for 1206-2002, and Table 2 shows part of the textual forecast that SumTime-Mousam generates from the NWP data.

Weathernews uses SumTime-Mousam to generate draft forecasts; we call them 'Pre-edit Texts'. The forecasters then post-edit them to produce 'Post-edit texts'. When the forecaster is done, the complete forecast is sent to the customer.

2 Post-Edit Evaluation

The evaluation was carried out on 2728 forecasts, collected during period June to August 2003. Each forecast was roughly of 400 words, so there are about one million words in all in the corpus.

For each forecast, we have the following data:

• Data: The final edited NWP data
• Pre-edit text: The draft forecast produced by SUMTIME-MOUSAM.
• Post-edit text: The manually post-edited forecast, which was sent to the client.
• Background information: includes date, location, and forecaster

The following procedure is performed automatically by a software tool:

• First, we break sentences up into phrases, where each phrase describes the weather at one point in time.
• The second step is to align phrases from the previous step as a preparation for comparison in the next step. Our alignment procedure first generates an exhaustive
• list of possible alignments and uses a scoring scheme to select aligned phrases.
• The third step is to compare aligned phrases and label each pre-edit/post-edit pair as match, replace, add, or delete.

For example, A and B of Figure 1 are analyzed as in Table 3 where phrases are shown separated by a shaded row.

A. Pre-edit Text: SW 20-25 backing SSW 28-33 by midday, then gradually increasing 34-39 by midnight.

1. Post-edit Text: SW 22-27 gradually increasing SSW 34-39.

Table 3. Detailed Edit Analysis

We processed 2728 forecast pairs (pre-edited and postedited). These were divided into 73041 phrases. Out of these, the alignment procedure failed to align 7608 (10%) phrases. Out of the successfully aligned phrases, 43914 (60%) are perfect matches, and the remaining 21519 (30%) are mismatches. Table 4 summarizes the mismatches and suggests that the major problem is ellipsis. Most (25235 out of 35874, 70%) of these errors are deletions, where the forecaster deletes words SumTime-Mousam's texts.

3 Discussion of Post-Edit Evaluations

We were attracted to post-edit evaluation because we believed that (A) people would only edit things that were clearly wrong; and (B) post-editing was an important usefulness metric from the perspective of our users.

Looking back, (B) was certainly true. The amount of postediting that generated texts require is a crucial component of the cost of using SUMTIME-MOUSAM, and hence of the attractiveness of the system to users.

(A) however was perhaps less true than we had hoped. Wagner [1998] also described post-edited texts in MT as at times noisy. During the development of SUMTIMEMOUSAM, our analysis of manually written forecasts [Reiter and Sripada, 2002] had highlighted a number of "noise" elements that made it more difficult to extract information from such corpora. While collecting the post-edit data, we assumed that people would only post-edit mistakes, where the generated text was wrong or sub-optimal, and hence post-edit data would be better for evaluation purposes than corpus comparisons.

In fact, however, there were many justifications for postedits. Some post-edits fixed problems in the generated texts (such as overuse of then); some post-edits refined/optimized the texts (such as using for a time); some post-edits reflected individual preferences (such as easing vs decreasing); and some post-edits were downstream consequences of earlier changes (such as introducing SSW before '34-39' in B, in the example of Section 2). We wanted to use our post-edit data to improve the system, not just to quantify its performance, and we discovered that we could not do this without attempting to analyze why post-edits were made. Probably the best way of doing this was to discuss post-edits with the forecasters.

4 Conclusion

We have used analysis of post-edits, a popular evaluation technique in machine translation, to evaluate SumTimeMousam, an NLG system that generates weather forecasts. While we encountered some problems, such as the need to identify why post-edits were made, on the whole we found post-editing to be a useful evaluation technique which gave us valuable insights as to how to improve our system.