An assessment of submissions made to the Predictive Toxicology Evaluation Challenge

1 Introduction

In his essay "Two conceptions of science" [Medawar, 1984], the distinguished biologist Peter Medawar de­ scribes the valuation of contributions to science thus:-

Here then are some of the criteria used by scientists when judging their colleagues' discoveries and the interpretations put upon them. Foremost is their explanatory value - their rank in the grand hier­ archy of explanations and . their power to establish new pedigrees of research and reasoning. A second is their clarifying power, the degree to which they resolve what has hitherto been perplexing ...

Explanations that reach this stage of inspection are usually understood to have achieved an acceptable level of accuracy, however measured. With the emergence of ML programs capable of constructing empirical gener­ alisations from scientific data, it is possible to examine the extent to which such machine-authored descriptions meet the criteria used to judge their human counterparts. This is of special interest if such programs are intended to act as genuine scientific assistants to experts.

One area for conducting such an examination was pro­ posed in the form of the Predictive Toxicology Eval­ uation Challenge (in IJCAI-97, see [Srinivasan et al., 1997]). The problem of predicting chemical carcinogen­ esis described there is particularly well-suited as a test­ bed for a number of reasons. Besides its undisputed hu­ manitarian value, principal reasons are that: (1) there is an urgent need for low-cost, accurate toxicity mod­ els that can reduce a reliance on slow, expensive rodent bioassays [Bristol et al., 1996] ; (2) there is much to be learnt about the molecular mechanisms underlying car­ cinogenic activity; and (3) there is a well-established sci­ entific programme within the U.S. National Toxicology Program (NTP) concerned with the comparative evalu­ ation of toxicity models (which may be of human origin, see: dir.niehs .nih.gov/dirlecmjpte2.htm 1 ). These pro­ vide machine-based "hypothesis constructors" the op­ portunity to construct accurate models, which may yield new insights and subject to review much in the manner described by Medawar.

1 All Internet sites mentioned in this paper are to be pre­ fixed with http:// unless otherwise indicated

This paper reports on the machine learning (ML) sub­ missions made to this IJCAI challenge from its inception in August, 1997 to December, 1998. The paper is organ­ ised as follows. Section 2 summarises the course of the challenge from 1997, and presents the models selected for further evaluation. Section 3 contains an assessment of the accuracies of the ML models in comparison to those developed under the guidance of expert toxicolo­ gists (this includes toxicology expert systems). Section 4 contains an appraisal of the explanatory value of the ML models 2 · Section 5 concludes this paper.

2 The IJCAI PTE Challenge: details and submissions

As part of the NTP, the National Institute of Envi­ ronmental Health Sciences (NIEHS) organises the Pre­ dictive Toxicology Evaluation (or PTE) project. The project [Bristol et al., 1996) is concerned with predict­ ing the outcome of rodent bioassays measuring the can­ cerous activity of a pre-specified set of compounds. In its simplest setting, predictions are restricted to either "POS" to denote carcinogenic, or "NEG" if otherwise. There is no restriction on the type of method used to construct the toxicity model. The PTE project accepted predictions until late 1996 for 30 compounds (collectively known as PTE2) undergoing bioassays within the NTP - the last of these assays being completed by June, 1998.

The relevance of the PTE project to programs con­ cerned with "knowledge discovery" directly led to the PTE Challenge in IJCAI-97. Here, it was proposed to collect submissions from AI techniques. Submis­ sions were to be made at a prescribed Internet site (www.comlab.ox.ac.uk/oucl/groups/machlearn/PTE) and consisted of two parts: (1) prediction: POS and NEG classification for the PTE2 compounds; and (2) description: details of the materials and methods used, and results obtained with the technique. The former was needed to assess model accuracy, and the latter for replicability of results and evaluations of model compre­

hensibilty.

The site accepted submissions from August 29, 1997 (one week after the challenge was announced at IJCAI97). Submissions received up to November 15, 1998 were eligible for assessments of chemical comprehensi­ bility. The challenge was regularly advertised at major AI conferences and in electronic newsgroups, and our records indicate that the data provided by the challenge site were retrieved over 100 times 3 . By November 15

2 Performed by one of the authors (D.W.B.), who is a toxicologist.

3 Clearly, only very limited conclusions can be drawn from this figure. Our records suggest that the data were extracted by groups with a wide range of research interests. However, the reader will note that the final submissions appear to be largely from those interested in Inductive Logic Programming (ILP). While it is possible that the emphasis on a descriptive

1998, 9 legal submissions were received (by legal here we mean that both "prediction" and "description" parts of the submission were in order). These are summarised in Figure 1. Space restrictions prevent us from providing a description here of each entry - the reader is directed to the Internet site under the "Description" column for complete details 4 .

At this point, it is worth noting an important point of difference between the submissions made to the NTP's PTE project, and those in Figure 1. All predictions by the former were made before true classifications on any chemical in PTE2 were known. The timing and duration of the IJCAI challenge has precluded the possibility of such a truly blind trial. We rely on submissions to abide by challenge regulations that prevent the use of PTE2 classifications in any way to direct model formation or selection.

3 Assessment of predictive accuracy

At the time of writing this paper, the classification of 23 of the 30 compounds had become available. Figure 2 tabulates the predictive accuracies achieved by the mod­ els described in the submissions in Figure 1 (henceforth called "ML-derived models").

Benigni [Benigni, 1998) provides a tabulation of the predictions made by several toxicity prediction methods on a subset of the PTE2 compounds. We concentrate here on those techniques that involve substantial input from experts. These include models devised directly by toxicologists or those that rely on the application of com­ pilations of such specialist knowledge (that is, expert systems). In [Benigni, 1998), there are 9 such "expert­ derived" models due to: Huff et al. (HUF, [Huff et al., 1996)), OncoLogic (ONC, [Woo et al., 1997)), Boot­ man (BOT, [Bootman, 1996)), Tennant et al. (TEN, [R.W. Tennant, 1996)), Ashby (ASH, [Ashby, 1996)), Be­ nigni et al. (BEN, [R.Benigni et al., 1996}), Purdy (PUR, (Purdy, 1996)), DEREK (DER, (Marchant, 1996)), and COMPACT/HAZARDEXPERT (COM, (Lewis et al., 1996}). Excluding missing entries, predictions are avail­ able from these methods for 18 PTE2 compounds. A comparative tabulation on this subset against the ML­ derived models is in Figure 3.

Comparisons based on predictive accuracy overlook an important practical concern, namely that the costs of different types of errors may be unequal. In toxi­ cology modelling, the cost of false negatives is usually higher than those of false positives. Borrowing from ter­ minology in signal-detection, "sensitivity" refers to the fraction of POS chemicals classified as POS by a model; and "specificity" refers to the fraction of NEG chemicals classified as NEG by a model. Figure 4 is a scatter­ diagram that shows the position of each model in this two-dimensional probability space.

component discouraged the use of methods like neural net­ works, we have no way of knowing why some ML researchers failed to respond to the challenge.

4 The reader should note that the submission OU2 was from two of the authors here (A.S. and R.D.K.). As far as we are aware, none of the submissions appear to have involved a toxicologist during model-development.

Complementary to sensitivity and specificity are: the fraction of POS predictions that are actually POS, and the fraction of NEG predictions that are actually NEG. Termed here as "positive predictivity" and "negative predictivity", these measure the accuracy of each type of prediction. Good models should exhibit high predic­ tivity values. Figure 5 shows the scatter-diagram of the models along these dimensions.

Keeping in mind the mandatory caution that must be exercised when interpreting figures derived from such small test-sets, Figures 2, 3, 4 and 5 appear to suggest that ML-derived models are able to at least match the performance attained by their expert counterparts. As is evident, 7 of the 9 ML-derived models achieve the predictive accuracy threshold set by the DEF model. This is in contrast to 3 of the 9 expert-derived mod­ els. 6 ML-derived models (LE2, LE3, LRD, LRG, OAI, OU2) achieve false-negative error rates of at most 0.25 with false-positive rates of no more than 0.50. This is matched by only 1 expert-derived model (ONC). 7 ML­ derived models (the previous 6 and OU1) also achieve positive and negative predictive rates of at least 0.50

(Figure 5) . This is in contrast to 4 expert-derived mod­ els. Elsewhere, we present a more detailed assessment of these trends based on a cost-sensitive technique termed ROC-analysis [Srinivasan et al., 1999]. Due to space restrictions, a summary has to suffice here. The analy­ sis shows the ML-derived models to be extremely com­ petitive, with LRG being the pick of the best across a range of reasonable error-costs and prior distributions over class values. LRG was obtained with a stochastic technique which resulted in rules that use, amongst oth­ ers, attributes encoding the results from ILP methods.

4 Assessment of explanatory value

At the outset of this section, it is worth emphasising that as submitted, none of the ML-derived models would be considered toxicology acceptable. This comment extends even to the most transparent submission like OU2, which presents a relatively simple (by ML standards) decision­ tree obtained from a well-known algorithm (C4.5). Much of this probably stems from a lack of toxicology expertise amongst the program users, and the lack of any "client specifications" in the statement of the challenge. We in­ tend to rectify the latter in future experiments (PTE-3, see [Srinivasan et al., 1999]). However, some attempt by all developers at improving clarity by including tables of names and structures identified by the rules, clear state­ ments of their reliability etc., would have greatly assisted the evaluation exercise. The application of ML tech­ niques to modelling toxicity endpoints is a relatively new research development in toxicology, and it is essential that descriptions of representations used and results ob­ tained are as thorough as possible (see for example, [Bris­ tol , 1995) for an appraisal of the requirements of mod­ els from both developer and prospective-client points of view). Nevertheless, the performance of the models have been sufficiently intriguing to foster further examination.

In performing an evaluation of the explanatory value provided by the ML-derived models, we have found it in­ structive to examine their contributions in the following categories:

1. Those that suggest any new lines of investigation for toxicology modelling;
2. Those that confirm, clarify or contribute to current ideas in toxicology; and
3. Those that are uninteresting or unlikely.

Our examination is restricted to the models that showed the most promise in the previous section, namely: LE2, LE3, LRG, OAI, OU2. Unfortunately, the most accu­ rate model (LRD) could not be considered, as no explicit model was provided. Further, it is not our intention to single out any one model as being the "best" -rather, it is to provide an overall assessment of the value of using ML methods in toxicology.

Of most interest is the frequent use of combinations, in models like LRG, of chemical structure and biologi­ cal tests. For some time, there has been vigourous de­ bate on how classical structure-activity modelling can be applied to toxicity problems. This form of mod­ elling relates chemical features to activity, and works well in-vitro. The extent to which these ideas transfer to toxicity modelling - which deals with the interaction of chemical factors with biological systems - is not evi­ dent. By using a combination of chemical features and biological test outcomes, the ML-derived models provide one possible method for dealing with the chemical ef­ fects in such "open" systems. If the accuracies obtained with such rules are borne out on larger datasets, then this would constitute a significant advance in structure­ activity modelling for toxicology. This is certainly worth further investigation and falls in Category A.

A ·number of aspects of some of the models can be categorised in Category B. As an example, OU2 se­ lects a combination of mouse lymphoma and Drosophilla tests as a strong indicator of carcinogenicity. Many tox­ icologists believe that relationships exist between gena­ toxicity and carcinogenicity. While the only accepted correlation involves the Salmonella assay, this rule sug­ gests a different combination of short-term tests could be equally, or more effective. Similar comments could be made on a number of other fronts: the presence of methoxy groups, sulphur compounds, and biphenyl groups are all identified in various ways as being re­ lated to toxicity. These are in line with what is currently known in toxicology.

Given the relative opaqueness of the output, it is hard to judge the extent to which the models have identified aspects in Category C. The general approach in toxicol­ ogy is to be wary of "explanations" that only pertain to a few or uninteresting chemical structures. These do occur in the models submitted, and appear to have been ignored (except in the case of OU2, where some editing attempt is undertaken). We do not enumerate examples of this here.

5 Conclusions

Toxicology is a young science that is primarily driven by intense health and industrial interests focused on spe­ cific chemical substances. A practicing toxicologist is regularly confronted with urgent requests to provide re­ liable information about the next substance of interest - whatever it might be. This situation demands that the toxicologist be able to call on, or develop, predic­ tive models that are not only accurate, but also cover an extremely wide range of noncongeneric dissimilar chemi­ cals. These range from pure organic and inorganic com­ pounds to polymers and complex mixtures. Predictions also need to be generated for a variety of toxicity end­ points [Bristol, 1995]. Aspiring assistants - human or otherwise - seeking to aid an expert toxicologist in this model-building endeavour, must be capable of suggesting robust solutions that are accurate and understandable. This forms the crux of the PTE Challenge - do AI pro­ grams meet these requirements when constrained to the task of predicting chemical carcinogenesis? The short answer, for the submissions participating in the chal­ lenge, is: "not yet." The qualifier is important though, as they do show considerable potential to achieve this goal. This opinion is based on the evidence that (a) models developed by these programs are clearly competitive on accuracy terms with those derived with significant expert assistance; and (b) even with almost no effort made to render the output chemically understandable the models have still suggested unusual ways to proceed with toxicology modelling. Whether such programs can make the transition from promising apprentices to valu­ able assistants will depend on whether their developers recognise the paramount importance of ensuring that the models are phrased in terms familiar to a toxicologist, and on continued good results with larger datasets.

Acknowledgements

The authors would like to acknowledge the significant effort made by the Machine Learning community in responding to the PTE Challenge. In particular, sincere thanks are due to the groups whose models we have used in the analysis here, namely those at Leuven (Belgium), LRI (France), OFAI (Austria), and the AI Lab, Taiwan. At Oxford, the model OUl was largely developed by Ngozi Dozie, a MSc student in Computation. A.S currently holds a Nuffield Trust Research Fellowship at Green College, Oxford. During the first six months of the PTE Challenge, he was supported by Smith­ Kline Beecham. A.S. and R.D.K would also like to thank Donald Michie, Stephen Muggleton, and Michael Sternberg for interesting and useful discussions concerning the use of machine learning for predicting biological activity. We thank the support staff of the Computing Laboratory, Oxford - in particular, Ian Collier - for his help with the Web pages related to the PTE Challenge.