Towards Multi-paper Summarization Using Reference Information

1 Introduction

Recently, scientific information explosively increases, be­ cause of the increase of the number of researchers and the ramification of research domains, and it is difficult for researchers to read all the papers that they can get. In this situation, surveys of specific domains can make it easier to grasp the outlines of the domains. However, the number of surveys we can obtain is very small [Garvey, 1979], because it is quite time consuming to write them.

Now we are studying towards automatic survey gen­ eration. A survey can be considered to be a summary of multiple papers, and should describe their essence and differences among them. Furthermore, it is desirable that the author describes in the survey his viewpoint and what are needed to study in the domain. However, the task to generate such a survey automatically seems very difficult.

This paper presents a system to support writing a sur­ vey of a specific domain. The system utilizes reference information that consists of reference relationships be­ tween papers and the information derived from the de­ scription around citations. We think the following are inevitable for writing a survey : collecting papers of the specific domain, and understanding their essence and dif­ ferences among them. Therefore, we firstly extract frag­ ments of papers where the author describes the essence of a referred paper and the differences between his pa­ per and it(we call them reference areas). Then with the information of reference areas, we identify the types of reference relationships that indicate the reasons for cita­ tions. These types make it possible to collect papers in the same domain. The system can display the collection of the papers. It can also show abstracts and reference areas of the collected papers. With the system, we can understand the relationships between the collected pa­ pers.

In the following sections, we first explain the essen­ tial points of multi-paper summarization, and then we describe reference information and its usage for summa­ rization. We also report some experiments of reference information extraction and show the support system for writing a survey.

2 Multi-paper Summarization

2.1 Essential Points of Multi-paper Summarization

There have been several techniques proposed for summa­ rizing a technical paper [Kupiec et al., 1995; Teufel and Moens, 1997; Mani and Bloedorn, 1998]. However, in case of multi-paper summarization, we have to perform more tasks than just extracting important fragments from each paper. First, we should take into account how to collect the target papers for summarization. Second, a multi-paper summary should clearly describe the sim­ ilarity and differences among papers. Therefore, we can sum up the essential points of multi-paper summariza­ tion as in Figure 1.

2.2 Related Works

Kando proposed several rules using lexical cues to an­ alyze the functional structure of technical papers, and used the structure for paper retrieval [Kando, 1997]((a) in Figure 1). She also pointed out that sentences that are assigned particular categories( e.g. "Review of rele­ vant previous research") in the collected papers are use­ ful for grasping the outlines of the studies in a specific domain(b). Similarly, we use reference information for multi-paper summarization.

In the field of bibliometrics or citation analysis, there have been many related researches [Narin et al., 1994; Liu, 1993; White and Mccain, 1989]. Their basic idea is that the papers that jointly cite others are related to each other, and the purposes of these researches are to find an important citation from citation diagrams and to identify which papers should be included in a survey. This can be considered as the first step of our study to make the multi-paper summarization system. To real­ ize this process, we develop the method to decide the reference types automatically.

3 Reference Information in Multi-paper Summarization

3.1 Reference Information

Consider 5 sentences in Figure 2 that are extracted around the citation of the referred _eaper [Murata and Nagao, 1993] in the referring paper lBond, et al., 1996]. Both papers are on machine translation, and particularly deal with noun phrases including numerical expressions. Sentence (2) introduces the theme of the referred pa­ per. Sentence (3) points out the problem of the referred paper. Then, sentence (4) describes that the referring paper copes with the problem pointed out in sentence (3).

By reading sentences (2),(3),( 4), we can understand the relationships between [Murata and Nagao, 1993] and [Bond, et at., 1996]. We call the fragment reference area. With the information in it, we can also identify the reason for citation. We classify the reason for cita­ tion into the following three categories( we call these cat­ egories reference t,YPes ), based on 15 categories pro­ posed by WeinstocklWeinstock, 1971].

· type B

The references to base on other researchers' theories

• (a) introduction of previous research
• (.B) description about the problem of previous research
• { (-r) the purpose of the research in the referring paper

Figure 3: The information obtained from type C refer­ ence area

or methods.

• type C The references to compare with related works or to point out their problems.

· type 0

The references other than types B and C.

We think the references of type C are more important than others, because from reference areas of type C, we can obtain information shown in Figure 3. In case of the example in Figure 2, sentences (2), (3), and (4) corre­ spond to (a), (/3), and (1'), respectively. Here, (a) can be considered as a kind of summary of the referred paper from the author's viewpoint. (a) can also be regarded as a fragment that describes the similarity of research top­ ics between two papers. Reading sentence (2), we can understand that the research topic of both papers is on generating articles and possessive pronouns and deter­ mining countability and number in English sentences. On the other hand, the problem of previous work and the purpose of research are described in sentences (3) and (4) respectively. These sentences can be regarded to describe differences between two papers.

3.2 Using Reference Information for Multi-paper Summarization

Retrieval of Papers Using Reference Information

If we collect papers by tracing all reference relationships, many papers in other domains are also included in the collection. By tracing only type C reference relation­ ships, however, we can collect the papers in the same domain. By investigating 31 cases of type C reference relationships in the database we will explain later, we actually find that both referring and referred papers be­ long to the same domain in 29 cases(94%). Figure 4 illustrates the target collection of papers for summariza­ tion( we call the collection in the shadow oval reference graph).

Detection of Similarity and Difference among Papers

Making correspondence of the information in type C ref­ erence area (in Figure 3) to the essential points of multi­ paper summarization(in Figure 1), (a) corresponds to (b)- 1, 2, and (,6), ('y) correspond to (b)- 3. Therefore, extracting and displaying reference areas can be a good support for writing a survey.

4 Support System for Writing A Survey

In this section, we explain the method to realize the support system for writing a survey.

4.1 Analysis of Reference Relationships between Papers

We use as the database 450 papers in TEX style source on computational linguistics from e-Print archive 1 . Since TEX has commands to write the bibliography such as "\cite" or "\bibitem", by analyzing information in such commands, we can get the information of reference relationships among papers. For the database, we can obtain the relationships in the accuracy of 94%.

4.2 Extraction of Reference Areas

Reference areas can be considered as a succession of sen­ tences that have a connection with the sentence includ­ ing citation in the paragraph. Since we think that such a connection between sentences can be indicated by 6 kinds of cue words, we use those cue words for reference area extraction. Examples of cue words are shown in Table 1. The procedure to select cue words is as follows:

1 http:/ /xxx.lanl.gov /em p-Ig/

1. Create the reference area corpus by hand,
2. Apply n-word gram analysis to this corpus,
3. Select 86 cue words manually, by checking the list of frequently used expressions made in step 2.

The flow chart of reference area extraction is shown in Figure 5. Reference area extraction rules add to the reference area candidate one sentence before or after the candidate if it includes any cue words.

4.3 Identification of Reference Types

In a reference area, if a negative expression appears at the beginning ofthe sentence after the sentence including citation, the reference area can be considered as type C. Similarly if the expression like "we adopt" or "we use" appears in the sentence including citation, the reference area can be considered as type B. Therefore, we prepare a list of cue words and make a set of rules using them for reference type identification.

Biber explicated the difference of linguistic features among standard four-part organization in medical pa­ pers: Introduction, Methods, Results, and Discussion [Biber and Finegan, 1994]. In making a list of cue words, we also pay attention to the difference among the sec­ tions. Type C reference areas tend to occur in Introduc­ tion, Related Works and Discussion. On the other hand, type B reference areas tend to occur in Introduction and Experiment. Therefore, we make a list of cue words with the following procedure:

1. Collect sentences for type B and C from correspond­ ing sections,
2. Calculate n-word gram separately,
3. Apply cost criteria [Kita, et al., 1994], which tends to extract longer expressions, to the result of n-word gram statistics,
4. Select 76 cue words for type C and 84 for type B manually, by checking the list of frequently used ex­ pressions made in step 3.

Examples of cue words are shown in Table 2 and 3. Then we make 160 rules for reference type identification manually.

5 Experiments of Reference Information Extraction

We have conducted two experiments to evaluate the ef­ fectiveness of our methods.

5.1 Extraction of Reference Areas

For the experiment, we prepare 150 reference areas, that are manually identified from paragraphs including cita­ tion. We use 100 for making rules and 50 for evaluation. We use F-measpre[van Rijsbergen, 1979] in formula(l) for evaluation, where b=l.

where P and R are as follows:

The results are shown in Table 4. To compare with the performance of our method, we calculate two baselines.

If we consider the whole paragraph as a reference area, all the sentences which should be extracted are ex­ tracted. In this case, the score ofF-measure is 0.534(Re­ call/Precision: 100.0/36.4%). On the other hand, if we consider only the sentence including citation as a refer­ ence area, the extracted sentenc'e is always correct as a sentence in the reference area. In this case, the score of F-measure is 0.575(Recall/Precision: 40.4/100.0 %).

As can be seen in Table 4, the performance of our method is better than the baselines.

5.2 Identification of Reference Types

For the experiment, we prepared 382 reference areas whose reference types are manually identified. Then we use 282 for making rules and 100 for evaluation. The results are shown in Table 5. The sum of bold numbers in Table 5 shows the number of reference areas whose reference types are correctly identified by rules. There­ fore, we obtain the accuracy of 90.1(%) and 83.0(%) in the data for making rules and evaluation, respectively.

6 Using the Support System for Writing a Survey

We show in Figure 6 the process of using the support system. The process consists of two stages. One is the stage of paper retrieval, and the other is the stage of overviewing the similarity and difference among papers.

In the first stage, PRESRJ(Paper REtrieval System using Reference Information) provides 2 methods of pa­ per retrieval. One is retrieval by queries (using authors' names and/or terms in titles). The other is retrieval by using reference relationships between papers. If the pa­ per retrieved by queries has reference relationships with other papers in the database, the graph of the reference relationships around it can be displayed. By tracing this graph, we can retrieve other papers.

In the second stage, by clicking the button "show type C reference relationships" on the display, we can see the reference graph. By clicking icons of abstracts or refer­ ence areas, we can see abstracts and reference areas of the papers in the graph.

Figure 7 shows the display of the support system. The left window shows the reference relationships around [Murata and Nagao, 1993]{9405019). 4 papers refer it, and [Bond96] (9601008) is darkened because three pa­ pers other than it refer [Murata and Nagao, 1993] in type C.

The right window display-s three reference areas, where [Takeda, 1994](9407008), [Bond, et al., 1994](9511001), and [Bond, et al., 1996]{9608014) describe [Murata and Nagao, 1993]. In this way, by displaying several abstracts and reference areas, the sytem can assist our overview of similarity and differences among papers. Therefore we think it is useful for writing a survey. Now this system is fully implemented and can be used on World Wide Web 2 ·

7 Conclusion

In this paper, as the first step towards automatic survey generation, we developed a support system for writing a survey by utilizing reference information between papers. By using the system, we can collect papers in the same domain. Additionally, the system displays the important parts and the descriptions of difference between papers, which are extracted from the collection of papers. We are now studying towards automatic survey generation based on the method described in this paper.

2 http:/ /galaga.jaist.ac.jp:8000/pub/tools/sum

Acknowledgements

The authors would like to express our gratitude to E­ Print archive administrators for allowing us to use the paper resources, Dr. Noriko Kando of NACSIS for her valuable suggestions of statistical analysis of cue words, and anonymous reviewers for their suggestions to im­ prove our paper.