Learning from the Report-writing Behavior of Individuals

1 Introduction

In this paper we describe a personalized learning-based approach to summarization that minimizes the need for learning-expert time and eliminates the need for expertgenerated evaluation materials such as a "gold standard" summary, since each user provides their own standard. While the work we describe is specific to summarization, we believe that the basic techniques are extensible to other learning situations that share the characteristic of repeated execution and the availability of unambiguous user feedback.

Of course this comes at a cost, which is the end-user time needed to teach the system how to produce satisfactory sum- Alexander I. Rudnicky Pittsburgh, USA

∗This work was done in part when the author was a student at CMU.

Carnegie Mellon University air@cs.cmu.edu

maries. We would however argue that end-user involvement is more likely to generate quality products that reflect both functional needs and user preferences and is indeed worth the effort. The current paper describes the application of this approach in the context of an engineering project class in which students were expected to produce weekly summaries of their work. For the course we worked with, a reporting requirement was already in place. We minimally modified the process (by augmenting and instrumenting the web-based reporting software already in use) to collect relevant data from users; apart from this their normal activities were not perturbed. The data collected was used to perform off-line learning experiments.

While each student produces their own logs, their team leader was additionally tasked with reading the logs and selecting log entries (referred as 'items' hereon) suitable for a summary of group activity. Learning is based on information about which items are selected by the team leader. We further collect data identifying the "features" deemed important by the leader creating the summary. This is done by asking the team leader to highlight key words/phrases in the selected summary items (the phrases that they believe led them to select that item). This might be thought of as having the users directly select Summary Content Units [Nenkova and Passonneau, 2004] or identify Rouge-like units (n-grams, word sequences) [Lin, 2004]. Although in this paper we focus on extractive summarization, we believe that this approach can be extended to prime information for abstractive summarization, one of our long-term goals for this work.

The plan of this paper is as follows. We describe relevant ideas from the literature. We then describe the domain and the report generation tasks performed by the class and give details of the data collected. The Learning System section outlines key details (feature representation, weighting, ranking, model selection etc.). After describing the basic Learning framework, we describe the Experiments and Results from the different settings of our system followed by the Conclusions.

2 RelatedWork

Although automatic text summarization has been explored for almost a half-century [Luhn, 1958], most current work focuses on generic newswire summaries [DUC, 2002 2003 2004 2005 2006]. [Radev and McKeown, 1998] provide a useful distinction between briefings and the general concept of summaries where they focus on generating multidocument newswire briefings. [Mani et al., 2000] also focus on generating briefings, however their approach contrasts with ours and is in a different domain. They assume that users are given an outline of the briefing and then try to populate the outline, whereas our system does not provide an initial structure (except for a generic template). On the other hand [Mani et al., 2000] extend their work to multimedia inputs. From our current perspective we believe that the system should focus on identifying important information, allowing the user to concentrate on its organization into a coherent presentation.

A few personalized interactive learning systems have also been proposed for summarization. [Amini, 2000] describes a query-relevant text summary system based on interactive learning. Learning is in the form of query expansion and sentence scoring by classification. [Leuski et al., 2003] have explored interactive multi-document summarization, where the interaction with the user was in terms of giving the user control over summary parameters, support rapid browsing of document set and alternative forms of organizing and displaying summaries. Their approach of 'content selection' to identify key concepts in unigrams, bigrams and trigrams based on the likelihood ratio [Dunning, 1993] is different from our statistical analysis and is of some interest. [Zhang et al., 2003] have proposed a personalized summarization system based on the user's annotation. They have presented a good case of the usefulness of user's annotations in getting personalized summaries. However their system differs from the current one in several respects. Their scenario is a single document newswire summary and is very different from a briefing. Also, their system is purely statistical and does not include the concept of a human-in-the-loop that improves performance.

[Elhadad et al., 2005] have applied personalized summarization in the medical domain where, given a patient profile and a set of documents from a search query, the system generates a personalized summary relevant to the patient. The active learning community [Raghavan et al., 2006] has also been moving towards using user feedback for identifying important features. Some other interesting domains in which summarization systems have been developed are technical chat [Zhou and Hovy, 2005], newsgroup conversations [Newman and Blitzer, 2002], email threads [Rambow et al., 2004], spoken dialogues [Zechner, 2001] as well as others.

[Garera and Rudnicky, 2005] describe a summarization system for a recurring weekly report-writing taking place in a research project. They found that knowledge-engineered features lead to the best performance, although this performance is close to that based on n-gram features. Given this, it would be desirable to have a procedure that leverages human knowledge to identify high-performance features but does not require the participation of experts in the process. We describe an approach to this problem below.

3 Target Domain

We identified a domain that while similar in its reporting structure to the one studied by [Garera and Rudnicky, 2005] differed in some significant respects. Specifically, the reportwriters were not experienced researchers who were asked to generate weekly reports but were students taking a course that already had a requirement for weekly report generation. The course was project-based and taught in a university engineering school. The students, who were divided into groups working on different projects were required to produce a weekly report of their activities, to be submitted to the instructors. Each group had well-defined roles, including that of a leader. Students in the class logged the time spent on the different activities related to the course. Each time-log entry included the following fields: date, category of activity, time spent and details of the activity. The category was selected from a predefined set that included Coding, Group Meeting, Research and others (all previously set by the instructors).

An example time-log entry is:"Visited the Voyager site. Spoke with the marine architects. Got a feel for the possibilities and scope of the project. 1 " The data in this domain is not as structured and grammatical as newswire documents. It is also different from chat-like/conversational corpora in its usage of words although it does contain a few spelling mistakes. One important characteristic of the data is that the content of the time-logs changes over time as the project progresses. So the word-based features that may indicate importance in the earlier weeks may not be useful in the later weeks.

The task of the team leader is to prepare a weekly report for the benefit of the instructor, using the time-log entries of the individual teammembers as rawmaterial. As the students were already using an on-line system to create their logs, it was relatively straightforward to augment this application to allow the creation of leader summaries. The augmented application provided an interface that allowed the leader to more easily prepare a report and was also instrumented to collect data about their behavior. Instrumentation included mouse and keyboard level events (we do not report any analysis of these data in this paper).

1 Italicized text shows the report-writer's highlighting for this time-log.

3.1 Data Collection Process

Following [Garera and Rudnicky, 2005], the leader selected items from a display of all items from the student reports. Figure 1 shows the interface used for the data collection. The leader was instructed to go through the items and select a subset for inclusion in the report. Selection was done by highlighting the "important" words/phrases in the items (described to the participant as being those words or phrases that led them to select that particular item for the report). The items with highlighted text automatically become the candidate briefing items. 2 The highlighted words and phrases subsequently were designated as custom user features and were used to train a model of the user's selection behavior. Examination of the data indicated that users followed these instructions and did not, for example, simply select entire items or just the first word.

3.2 Data Collected

We were able to collect a total of complete 61 group-weeks of data. One group-week includes the time logs written by the members of a particular group and the associated extractive summaries. The class consisted of two stages, design and implementation, lasting about 6 and 9 weeks respectively. The groups and roles were reconstituted after the first stage. This meant that after the first stage we had new "authors" and could not combine the first stage data with the second stage into a continuous sequence. There were 5 groups in the first stage with average 4.8 students per group and 5 groups in the second stage with average 3.6 students per group (several students dropped the course after the first few weeks). There were on average 4.5 weekly summaries per group for Stage 1 and 7.5 weekly summaries per group in Stage 2. As is evident from the averages, not all the groups submitted a summary every week. To provide consistent data for development, testing and analysis of our system, we selected those 3 groups from the later stage of the class that produced reports most consistently (these are described further in the Evaluation section below).

4 Learning System

We modeled our Learning process on the one described by [Garera and Rudnicky, 2005]; that is, models were rebuilt on a weekly basis, using all training data available to that point (i.e., from the previous weeks). This model was then used to predict the user's selections in the current week. For example, a model built on weeks 1 and 2 was tested on week 3. Then a model built on weeks, 1, 2 and 3 was tested on week 4, and so on.

Because the vocabulary varied significantly from week to week, we trained models using only those words (features) that were to be found in the raw data for the target week, since it would not be meaningful to train models on non-observed features.

2 The interface enforces a minimum (10%) and maximum (40%) number (as percentages of the total number of items for the particular week) of item selections to make sure some summary is reported.

The resulting model is used to classify each candidate item as belonging in the summary item or not. The confidence assigned to this classification was used to rank order the raw items and the top 5 items were designated as the (predicted) summary for that week. The following sections explain the learning system in more detail.

4.1 Classifier Settings

Features

The features used in the classifier are words (or unigrams), stemmed and with stop words removed. We experimented with three different classes of features: a) NEfalse: only unigrams b) NEclass: unigrams + abstracted named entities using only the NEclass label (i.e., person, organization etc.) eg. 'White House' is treated as a unique token ('Location') representing its class. 3 c) NEuniq: raw unigrams + each named entity substituted by a unique token eg. 'White House' is treated as a unique token ('Location-1'). We used BBN-Identifinder [BBN-Technologies, 2000] for extracting the Named Entities.

Feature Extraction

We extract the features in two different settings. Firstly, we use the entire sentences for extracting the unigram features: FRaw. Secondly, we combine the entire sentence features with the user-specific features:FUser (FUser ⊆ FRaw) which is similar to the idea of combining the 'context' and 'annotated keywords' described in [Zhang et al., 2003]. 4 The details of how the final scores were computed are given below.

System Description

Standard Information Retrieval (IR) metrics are used to score the features. We did not fix the parameters in the schemes as the characteristics of the data did not particularly recommend any particular setting. The tunable parameters in the schemes and their possible values are: a) Term weighing method - TF (term frequency), TF.IDF, Salton-Buckley(SB) [Yates, 1999]. b) Corpus for measuring IDF: For any word, the inverse document frequency can be obtained by considering either the documents in the training set or the test set or both. Therefore we have three different ways of calculating IDF. c) Normalization scheme for the various scoring functions: no normalization, L1 and L2.

Feature scoring in the first setting of extracting unigram features FRaw is straightforward using the above mentioned IR parameters (TF, TF.IDF or SB). For combining the scores under the second setting with the 'user-specific' features we used the following equation:

3 The idea being to capture user's preference wrt particular classes of NEs i.e. the user prefers to select an item where a person and organization are mentioned together.

4 We also experimented with using just the user-specific features in isolation but found these less useful than a combination of all features.

where α is the weight contribution for the user-specific features and Sfbase is the base score (TF or TF.IDF or SB). We empirically fixed α to '1' for the current study.

We tested the above mentioned variations of feature description, feature extraction and feature scoring using four learning schemes: Naive Bayes, Voted Perceptron, Support Vector and Logistic Regression. In the event, preliminary testing indicated that Support Vector and Logistic Regression were not suited for the problem at hand and so these were eliminated from further consideration. We used the Weka [Witten and Frank, 2000] package for developing the system.

4.2 Evaluation

The base performance metric is Recall, defined in terms of the items recommended by the system compared to the items ultimately selected by the user. 5 We justify this by noting that Recall can be directly linked to the expected time savings for the eventual users of a prospective summarization system based on the ideas developed in this study. The objective functions that we used for selecting the system model (built on the basis Recall) are:

1. Weighted mean recall (WMR): of the system across all weeks. The weeks are given linearly increasing weights (normalized) which captures the intuition that the performance in the later weeks is increasingly more important as they have consecutively more training data.

2. Slope of the phase-wise performance curve (Slope): We first calculate the three phase-wise recall performance values (normal average of the recall values) and then find out the slope of the curve for these three points.

Note that these metrics are used as a selection criterion only. Results in Figure 2 are stated in terms of the original Recall values averaged over the phase and across the three users. We compare these with the results for the random baseline. The random baseline is calculated by randomly selecting items over a large number (10000) of runs of the system and determining the mean performance value. 6

5 We are not doing a ROUGE-like [Lin, 2004] evaluation as we have modeled our problem as creating a draft summary in terms of the log items to be selected and not a final polished summary.

6 For our domain, other baseline methods which have been tradi-

5 Experiment and Results

We selected for experimentation the three groups that most consistently generated complete weekly datasets . These groups had 9, 8 and 8 complete weeks of data. Detailed statistics of the data are shown in Table 1. Since the groups had different number of weeks, we grouped the observations into three phases by combining the results from successive weeks to create a phase. Merging the data into three phases is also consistent with our intuition about the task, that is, an activity has an initial starting period, middle activities and then the closing activity. We combined the observations with a sliding window to form the three phases. The window size and the overlap were different for the groups, though the same within a group.

Since we treated modeling technique as a free variable, we ran experiments for all the possible combinations of feature and system dimensions described in the previous section. The overall best performing system based on the jointly optimized metrics of WMR and Slope was selected as the final model from which to generate the system predictions. The final model has the following system parameters:- Learning Scheme: Voted Perceptron, TermWeighing Scheme: SaltonBuckley, Document Frequency Set: Training set, Normalization scheme: None. The feature set that gave best performance was NEuniq.

Figure 2 shows the key results of our experiments with the above-mentionedmodel. The performance is averaged across the three groups.

Figure 2(a) shows that the performance obtained with generic n-gram features is comparable to the performance incorporating user-specific features. While one would have hoped that user-specific features would do better than generic ones, it's clear that users select consistent personal features.

Figure 2(b) shows that user-selected features are at least as good as those selected based on information gain, arguably an "optimal" way to select useful features. Table 2 shows the mean number of active features in each phase (that is, how many previously identified features occur in the test set). Here we see something interesting, which is that information gain (IG) selects very few features, while humans select a broader number, not only those that might directly predict item importance, but also features that could in the future be of use. It is also curious that there is little overlap between the IG selections and the human ones. One interpretation is that humans are selecting a broader and potentially more robust set of features. The latter is supported by the observation that the IG-based model fails to select any active feature 27% of the time, while the human model fails only 9% of the time for this reason.

tionally used such as the 'first' sentence of a document etc. are not suitable because of a characteristically different corpora.

Figure 2: Figure showing the various experiments. 'Raw N-grams' represents the entire n-grams in the items. 'User-combined' represents the group-wise (single user) selected features combined with Raw n-gram features. 'IG-Combined' is the Information Gain selected features combined with Raw n-grams. 'Cross-group' is the training data pooled across groups and the user's selections combined with Raw n-grams.

Figure 2(c) shows what happens if we pool user-selected features across any two groups and use these to train a single model for testing performance on the third group. Here we see something different: cross-group features do not perform as well as group-specific features. This suggests that the specific features do matter, as they reflect a particular group's activities (as well as perhaps the leader's emphases in his or her reports). Table 3 is comparable to Table 2 and indicates that, while there is better overlap, pooling data introduces significant error into the model. Together these results suggest that important features are both task-specific and user-specific. The latter interpretation is consistent with the results reported by [Garera and Rudnicky, 2005].

Table 4 shows the most frequent active features selected under the conditions of Single User selection, Information Gain and the Cross-group pooled data.

6 Conclusions

We show that consistent summarization models can be built from relatively sparse user-generated data and that these models are sufficiently consistent to predict, with increasing accuracy, the summarization behavior of these users. We also show that naive end-users are able to consistently select features at least as well as an automatic process. There is moreover evidence that the feature sets selected by humans may turn out to be more robust in the long run than automatic features (since predictive power is spread over a larger number of features). Although this in itself is a useful result, it also opens the possibility of understanding and eventually automating the apparently sophisticated feature selection process used by humans.

These results, together with the implicit-feedback paradigm embodied in the report-generation interface suggest that it may be possible to design viable learning systems that can be effectively trained by an end-user. Such an approach to customization may be more powerful than ones based on adaptation, where models and features sets may be pre-determined, and that may produce system behavior that more faithfully reflects the needs of individual users.

Acknowledgement

We would like to thank Daniel P. Siewiorek and Asim Smailagic for allowing us to do data collection in their class and Susan Finger and team for helping us in instrumenting the data collection. We would also like to thank Eric Nyberg, Eric Riebling and team for the Named Entity annotations. This work was supported by DARPA grant NBCHD030010. The content of the information in this publication does not necessarily reflect the position or the policy of the US Government, and no official endorsement should be inferred.