Challenge: How IJCAI 1999 can Prove Value of AI by Using AI

1 Introduction

In this paper we will formulate a challenge that should result in the design and implementation of a reviewer selection program to be used by the organizing commit­ tee of IJCAI 1999. The motivation behind this problem is partially rooted in a dissatisfaction with the assign­ ment process of papers to reviewers that was apparently used for IJCAI 1997. The challenge is that the process of assigning papers to reviewers should be automated and performed by a flexible and intelligent AI program. We will explain this motivation in more detail, based on a metaphor from the business of telephone system development. Then we will outline some of the major problems that are part of this challenge. Finally, we will recapitulate the importance of this challenge problem. Indeed, in our opinion, this challenge problem can well compete with any other challenge problem that is likely to be presented at IJCAI 1997.

2 Motivation

It is helpful to recount very briefly a personal experience of this author to create a clear picture of the motiva­ tion for this challenge problem. After graduating from college, this author looked for his first entry level job in software development. At the time and place where this occurred, the only game in town were the phone compa­ nies. There were four of them, let's call them AB, CD, EF, and YZ. The author was eventually hired by the AB phone company.

The reader might think that this should cause no prob­ lems. Unfortunately, a preamble to the list of Content Areas stated:

Please use the exact phrases below; do not make up new keywords, since they will be ig­ nored by the paper classification software.

What conclusions do we have to draw from this state­ ment? First of all, papers will be assigned (partially?) to reviewers by a program. Secondly, this program is so unintelligent that it cannot even deal with variant spellings, much less Content Areas that are not known to it. What else could the p:dmonishment to use "exact phrases" mean?

This sounds like a phone company that does not have a phone system itself.

3 Current Reviewer Assignment

After acceptance of this challenge paper, the au thor con­ tacted the I.JCAI 1997 program chair to find out more about the reviewer assignment program that had been used. The program was developed by Ramesh Patil at USC/ISI 1 for use by the National Conference on Artifi­ cial Intelligence, commonly known as "AAAI." Due to the different committee setup at IJCAI and AAAI, a con­ siderable amount of human post-processing was neces­ sary for IJCAI 1997 reviewer assignment [Pollack, 1997).

An inquiry at lSI revealed that the AAAI reviewer assignment program has been used for five years. It relies on a system of bidding for papers, in combination with stored profiles of expertise. Details of this program have not been published, as the author considers them "not algorithmically novel." However, future versions of the reviewer assignment program might be published [Patil, 1997).

4 The Challenge Problem

The following is a suggestion of what should appear in the Call for Papers of IJCAI 1999:

Authors must supply a list of Content Areas from any subarea of Artificial Intelligence that they consider appropriate. These Content Ar­ eas will be used by a flexible and intelligent program to assign papers to reviewers. Only in cases where this program reports insufficient confidence in its own results, will human editors make the assignment of papers to reviewers.

Actions speak louder than words. Having the major world-wide conference on Artificial Intelligence in 1997 issue a Call for Papers that admits that it relies on exact string matching for reviewer preselection speaks very loudly about the state-of-the-art of Artificial Intel­ ligence. We think that this is a statement that we as members of the AI community do not want to make.

1 University of Southern California - Information Systems Institute

What would be involved in the kind of processing re­ quired for an intelligent reviewer assignment program (which we will call REVIEWER-SELECT)? We need at least a Natural Language Processing system that is able to parse noun phrases. We need a knowledge base that associates in a flexible way content areas with reviewers . We need a second Natural Language Processing system that is powerful enough to extract possible content ar­ eas from given source text, as will be explained below. Finally, we need an expert system that selects reviewers, keeping in mind certain overall constraints which will be elaborated in more detail shortly.

To summarize, the challenge for IJCAI 1999 one more time succinctly:

Papers should be assigned to reviewers by an intelligent program that does not impose any limitations, except that the Content Areas supplied by paper authors need to be cogent to the eyes of a human expert.

5 Sources of Information

In order to create the same base-line conditions for all the teams that want to take on this challenge, the IJCAI organization is asked to do the following. Due to the electronic submission process of abstracts through the world-wide web, the paper abstracts of most IJCAI 1997 submissions will be available on-line. The relatively few abstracts that are not submitted electronically should be typed in by IJCAI volunteers. This set of text sources should be made available through the world-wide web to the world AI community immediately after the I.l­ CAI 1997 conference. The (second) Natural Language Processing system mentioned above should be used by research teams to extract candidate Content Areas from the stored abstracts. Some human filtering and/or aug­ mentation might still be necessary to create a compre­ hensive list of Content Areas.

An important issue to address in preparing this source of information is how to treat the abstracts of rejected I.JCAI 1997 papers. Copyright reasons as well as the large number of rejected papers make it difficult to deal with them as part of the source data. On the other hand, omitting the abstracts of rejected papers might bias the reviewer selection process towards papers in the areas that were accepted at IJCAI 1997. A solution to this problem needs to be found and publicized quickly, to create an even playing field for all future paper submit­ ters.

Secondly, after I.JCAI 1997, authors of papers should be asked to supply a list of five to ten individuals that they would have considered good reviewers for their own papers. Authors might also be asked to supply all the ref­ erences cited in their own 1997 papers in electronic form to the I.JCAI organization, adding an additional source of knowledge for the selection of reviewers. Another source of information is already there, and available to anybody with a browser, and that is the world-wide web itself. How the web could be used will be hinted at below.

One additional point should be made, although it is hopefully obvious at this stage. A response to this chal­ lenge will not consist of a Call for Papers containing a 10 page list of Content Areas. Rather, there might be no Content Areas listed at all. Anybody who is capa­ ble of submitting a paper to I.JCAI should be aware of what constitutes an acceptable Content Area of AI. On the other hand, it appears useful to make the knowledge base of REVIEWER-SELECT itself publicly accessible at some point in time. Whether this should be before or after the I.lCAI 1999 dead-line is open to discussion.

6 Problems

The problems posed by designing and implementing the REVIEWER-SELECT program are definitely non­ trivial. This includes all aspects of it, but we want to stress the overall control that has to be exercised by this program. For instance, the REVIEWER-SELECT pro­ gram should not come back with

1. Reviewers with the same affiliation as the author (conflict of interest);
2. The author of a paper as a suggested reviewer (ex­ treme conflict of interest) ;
3. The doctoral advisor of the author as a suggested reviewer (another extreme conflict of interest);
4. Reviewers that the author has recently written a paper with (potential conflict of interest);
5. Assignment of fifty papers to one reviewer, and one paper each to fifty other ·reviewers (uneven load for reviewers);
6. Assignment of thirty reviewers to one paper, and no reviewers to thirty other papers (uneven load for papers).

The problem of identifying conflicts of interest could be attacked by using the world-wide web. As many in­ stitutions now have home pages on the world-wide web, information about affiliations could be retrieved by ap­ propriate agents and softbots. Even potential conflicts of interest could be detected when authors have their vitae available on-line.

7 Limitations and Success Measurements

In order to make this challenge self-contained and its so­ lution feasible in the allotted time, we must allow for some Content Areas that will not be contained in the knowledge base of REVIEWER-SELECT. In such cases the system will need to ask a human editor for help. It will be a measure of the success of any program tak­ ing on this challenge, how many (or how few) requests for human intervention will be required in processing all I.lCAI 1999 submissions.

To ensure both fairness to I.JCAI 1999 submissions and an alternative evaluation process of several programs that might take on this challenge (assuming that there will be several) the reviewer assignments should be rated blindly by a committee of human experts. By computing precision and recall measures compared to the ratings of human experts, a numeric evaluation of different pro­ grams will be performed.

A second, more expensive, way to evaluate the suc­ cess of a REVIEWER-SELECT program would be to perform the reviewer selection process exactly as it was done in 1997, and to use the results of this process as the gold standard. This appears attractive for I.lCAI 1999 because confidence in a completely automated re­ view process might still be low in the AI community at large. If the evaluation is successful, then I.JCAI 2001 might opt for completely automatic reviewer selection.

A third way to assess the success of a REVIEWER­ SELECT program would be to compare its results with the names of reviewers suggested by the authors ofl.JCAI 1999 papers. 2 If this route is chosen, then the I.lCAI 1999 Call for Papers must state that authors need to propose reviewers for their papers at submission time.

Whatever choice is made for evaluation, two condi­ tions must be maintained. (1) If there are several teams taking on this challenge, then all of them must get the same starting conditions, including access to relevant materials collected by AAAI. (2) It must be assured that no REVIEWER-SELECT program has a built-in bias towards papers of its own designers. The author of this challenge paper will gladly serve as a coordinator of such evaluation activities throughout the whole life-time of this effort.

8 Acknowledgment of Debt to Previous Research

The idea of doing "reviewer selection driven research" is not new. Gio Wiederhold, a leading researcher in databases and other fields, and previously the editor of the most prestigious database journal, has mentioned the problem of automatically finding reviewers for his journal in conference presentations [Wiederhold, 1992]. However, we feel that in our formulation as a challenge problem, the difficulties that will be encountered are by an order of magnitude larger. On the other hand, this challenge problem appears still small enough and well defined enough to lead to real practical results within the given time frame.

2 This idea is due to one of the anonymous reviewers of this paper.

9 Why this Problem is Important

We have no doubt that there are many other challenge problems that make sense at the current state-of-the-art. Kitano's Grand Challenge problems have not been solved [Kitano et al., 1993]. Those problems are, however, in a different league, as opposed to the relatively small and well defined problems sought in this year's Challenge Pa­ per track. The Turing Test, in whichever guise, is still the grand challenge problem to all of AI, but there is little hope to see it solved by I.JCAI 1999.

Among problems in the same league as the reviewer assignment problem, we feel that our challenge is very important for a simple reason. It is the same reason why a phone company should first build a phone system for in-house use, before starting to sell it to other people. It looks very bad when an enterprise does not produce products that are good enough for its own needs. If I.JCAI 1999 will have to rely on string matching for as­ signing reviewers to papers, it might be better to return to manual assignments, and to never even mention the use of a program in the Call for Papers. On the other hand, any solution to this challenge problem would con­ siderably improve the credibility of AI as a whole.

10 The Real Challenge- A Final Remark

The challenge formulated in this paper is very limited, compared to the real challenge of making AI programs superior to average human practitioners in a field. From this point of view, why are authors of papers asked to supply Content Areas at all? Why can't an intelligent program "just read" the abstract of a paper and decide what the Content Areas are? A program that can as­ sign papers to reviewers without being given Content Areas and that can do this for the whole range of AI problems would be a supreme achievement, as very few human individuals can perform this task. Such a pro­ gram would probably build on many current subareas of AI, including data mining, machine learning, advanced natural language processing, knowledge representation, reasoning, etc. Research on summarizing text might be a valuable starting point. Maybe we will be able to pose this task as a challenge problem for I.TCAI 2009.

Acknowledgment

Mike Halper has improved the English of this paper.