The Predictive Toxicology Evaluation Challenge

1 Introduction

Programs developed under the umbrella of Machine Learning are increasingly being used for "knowledge dis- R.D. King* M.J .E. Sternberg Biomolecular Modelling Laboratory Imperial Cancer Research Fund 44 Lincoln's Inn Fields, London U.K.

*Now at:Department of Computer Science, The University of Wales Aberystwyth

covery" tasks. Early specialised programs (for exam­ ple, [Feigenbaum et al., 1971; Langley et al., 1983]) have e;iven way to more general-purpose ones (for example, lMuggleton, 1995; Muggleton and Feng, 1990]) which have been applied with some success in areas of bio­ chemistry ([King et al., 1996; 1992; Muggleton et al., 1992]). While the experimental studies reported are pre­ liminary, they have at least one commendable feature, namely, they constitute examples of AI programs par­ ticipating in true scientific discovery tasks. By "true" here, we mean problems where existing scientific knowl­ edge is incomplete, the descriptions found automatically were unknown to experts in the field, and have been ac­ knowledged by publication in peer-reviewed journals in the field. Given the promise shown by machine learning programs in biology and chemistry, this paper describes a challenging test-bed with the following desirable fea­ tures: (1) a widespread scientific interest in any new results; (2) the availability of a large database of chem­ icals with classifications certified by experts; (3) strong competition from methods developed by expert chemists; and ( 4) the opportunity to participate in true blind tri­ als.

The problem concerns obtaining a better understand­ ing of the molecular mechanisms of chemical carcino­ genesis. This is central to the prevention of many envi­ ronmentally induced cancers. One approach is to form Structure Activity Relationships (SARs) that empiri­ cally relate molecular structure with ability to cause can­ cer. This work has been greatly advanced by the long term carcinogenicity tests of compounds in rodents (util­ ising both genders of one rat and mouse strain) by the US National Toxicology Program (NTP) of the National In­ stitute of Environmental Health Sciences (NIEHS: [Huff and Haseman, 1991]). So far, the NTP tests have resulted in a database of more than 300 compounds that have been shown to be carcinogens or otherwise. The NIEHS Predictive Toxicology Evaluation (or PTE) project ([Bristol et al., 1996]) is closely associated with the NTP. The PTE project identifies a group of assays that are scheduled or ongoing in the NTP. These chem­ icals form the "test" set for researchers. Predictions for presence or absence of carcinogenicity activity are compared against true activity as observed in the ro­ dents. The first such blind trial, PTE-1, is now com­ plete. The second, PTE-2, is ongoing, and true activity levels will be available by June 1998. It is the prediction of, and reasons for, carcinogenic activity in chemicals constituting PTE-2 that we commend as a challenge for AI programs concerned with knowledge discovery from databases.

This paper is organised as follows. Section 2 presents the statement of the challenge. Section 3 summarises the data available in the NTP database, and the chemicals in PTE-1 and PTE-2. Section 4 sets out the evaluation criteria. Section 5 describes the results obtained using the Inductive Loe;ic Programming (ILP) system Progol [Muggleton, 1995j . These results are intended to provide initial benchmarks for future entries. Section 6 concludes this paper including information on submitting entries to the challenge.

2 The PTE Challenge

The aim is to obtain a theory for predicting the carcino­ genicity of 30 compounds currently undergoing rodent bioassays in the NTP (called PTE-2: see below). The performance of the theory is to be evaluated according to the criteria described in Section 4.

3 Materials

3.1 The NTP database

A compilation of 330 chemicals is available directly from the database of the National Cancer Institute and NTP ([Huff and Haseman, 1991], and via the Internet at http:/ /ntp-server. niehs. nih.gov/. These compounds represent all the organic chemicals that have completed NTP reports at the time of writing this paper. Of the 330 compounds, 182 (55%) are classified carcinogenic, and the remaining 148 non-carcinogenic. Carcinogenic­ ity is determined by analysis of long term rodent bioas­ says. For the purposes of this challenge, compounds classified by the NTP as equivocal are considered non­ carcinogenic, as this allows direct comparison with other SAR predictive methods. No analysis is made of differ­ ences in incidence between rat and mouse cancer, or the role of sex, or particular organ sites. 39 of the 330 com­ pounds in the NTP database formed the first of the blind trials (PTE-1) conducted by the PTE project . Results from the bioassays for these chemicals are now available, and show 22 (56%) to be carcinogenic, and the remain­ ing 17 to be non-carcinogenic. Further details of these compounds are available in [Bahler and Bristol, 1993]. The 330 chemicals make this database very large and diverse, making it a great challange to learn in.

The PTE-2 compounds

The second round of blind trials (PTE-2) consists of 30 com pounds (of which 5 are inorganic) . These are fully described in [Bristol et al., 1996], where the schedule of events suggest that all bioassay results will be available by July 1998.

3.2 Other information available

The NTP has recently made available a number of structural attributes (features) describing a large section of their database. These descriptions are available at http:/ /ntp-server.niehs.nih.gov/MainYages. The other information available is in the form of the atom and bond connectivity of the compounds (including those in PTE1,2). This is described further in Section 5.

4 Evaluation

In [Bristol et al., 1996], the goal of predicitive toxicol­ ogy (PT) is summarised as " ... the ultimate value and most important goal of PT research may lie in the de­ velopment of its potential to identify, characterise, and understand the various mechanisms or modes of action that determine the type and level of response observed when biological systems are exposed to chemicals ... ". Given this emphasis on understandability of models, we follow [M uggleton et al., 1996] in using the following def­ inition for comparing the performance of rival theories.

Definition 1 Performance comparison. If the pre­ dictive accuracies of two theories are statistically equiv­ alent then the theory with better explanatory power has better performance. Otherwise the one with higher accu­ racy has better performance.

We now elaborate further on the methods for evaluating predictive accuracy and "explanatory power".

4.1 Predictive accuracy

Predictive accuracy is taken to be the proportion of com­ pounds in PTE-2 whose predicted classification (carcino­ genic or non-carcinogenic) agrees with that rodent bioas­ says. Significant differences in predictive accuracy are best assessed by McNemar test for changes [Bland, 1989]. This test exploits the fact that the different prediction methods are applied to the same data and is based on counting the examples where the methods disagree about predictions. We suggest that differences be judged to be significant at least at P = 0.10.

4.2 Explanatory power

In the absence of an expert chemist to act as adjudica­ tor, we propose that the simple criterion that theories are judged to have "explanatory power" - a boolean prop­ erty - if some or all of it can be represented diagram­ matically as chemical structures. The intuition underly­ ing this is that such structural alerts form the preferred mode of discourse amongst chemists.

5 An experiment with the ILP system Progol

5.1 Progol

We refer the reader to [Muggleton, 1995] for complete de­ scriptions of the ILP system Progol. In the current context, Progol is provided with a set of carcinogenic ("pos­ itive") and non-carcinogenic ("negative") examples from the NTP database together with background knowledge B about these compounds (see Section 5.2). The aim is to generate a theory (expressed as a set of rules) which explains all the carcinogens in terms of the back­ ground knowledge whilst remaining consistent with the non-carcinogens. To achieve this Progol1) randomly se­ lects a positive example e;; 2) uses inverse entailment to construct the most specific hypothesis l.(B, e;) which explains e; in terms of B; 3) finds a rule D; which gen­ eralises l.(B, e;) and which maximally compresses a set of entailed example E;; and 4) adds D; to the theory H and repeats from 1) with examples not covered so far until no more compression is possible. Compression is here defined as the difference, in numbers of descriptors, between E; and D;.

5.2 Background knowledge

The generic atom/bond representation used in an ear­ lier study [King et al., 1996; Srinivasan et al., 1996] is used. This consists of two basic relations to repre­ sent structure: atom and bond. For example, the fact atom{127,127_1,c,ar_c_6_ring,-0.133) states that in com­ pound 127, atom no. 1 is of element carbon, and of type aromatic carbon in a 6 membered ring, and has a partial charge of -0.133. The type of the atom and its partial charge were taken from the molecular mod­ elling package QUANTA, although any similar mod­ elling package would have been suitable. Equivalently, bond(127,127_1,127-2, ar) states that in compound 127, atom no. 1 and atom no. 2 are connected by an aro­ matic bond. In QUANTA, a partial charge assignment is based on a specific molecular neighbourhood. This has the effect that a specific molecular sub- structure can be identified by an atom type and partial charge. This relational representation is completely general for chemical compounds and no special attributes need to be invented. The structural information of these com­ pounds was represented by~ 18,300 facts of background knowledge.

Information was also given about the results of Salmonella mutagenicity tests for each compound. The mutagenic compounds were represented by the relation Ames, e.g. ames{127) states that compound 127 is mutagenic. The Progol algorithm allows for the in­ clusion of complex background knowledge, either in the form of facts, or in the form of arbitrary Pro­ log programs. In this study we included the back­ ground knowledge of chemical groups defined in [Srini­ vasan et al., 1996], along with the structural alerts in [Ashby et al., 1989]. All the information used is available in Prolog form at a prescribed Internet site: http:/ /www.comlab. ox. a c. uk/oucl/groups/machlearn.

5.3 Results and discussion

The 39 compounds comprising PTE-1 were excluded and rules for carcinogenicity obtained using Progol. The re­ sulting theory consists of 18 rules. Figure 1 tabulates a comparative evaluation on PTE-1. More details on the rules obtained are available in [King and Srinivasan, 1996]. Figure 2 tabulates the predictions made by the Progol theory for compounds in PTE-2. The first three entries in Figure 1 have been marked out for special attention because they had access to additional infor­ mation in the form of short-term rodent (in-vivo) tests. The first two entries also require a degree of expert eval­ uation. The Ashby structural alerts are based on elec­ trophilic attack on DNA, which makes them statistically dependent on the Ames test. It is also worth noting that the TIPT and Benigni methods rely on structural alerts derived by the Ashby method for their explana­ tory component. CASE, TIPT and Progol are the only data-driven inductive methods, and Progol is the only automated method capable of identifying new structural alerts. With these comments in place, the results in Fig­ ure 1 offer significant encouragement for machine learn­ ing programs on the following counts. First, we point out that one PTE-1, the results of Progol, TIPT and CASE demonstrate performance that is competitive with the current state-of-the-art - Progol has marginally the highest accuracy of all methods that do not use rodent tests. Second, the relatively low accuracies of all meth­ ods is primarily due to the diversity of compounds in­ volved. It does however leave the door open for signifi­ cant improvement. Progol, for example, achieves its per­ formance with a very low-level atom/bond representa­ tion of compounds. Enriching this background informa­ tion with the new structural descriptors available from the NTP could significantly improve its accuracy. These comments are reinforced by early results of using the Progol theory to classify compounds in PTE-2. Figure 2 shows that tentative classifications are available from the NTP for 13 chemicals. Progol's theory has correctly classified 7 of these. This should be seen in context of the performance of other theories listed in [Bristol et al., 1996] which shows that most chemists and automated methods have not been able to better this count. This should provide further impetus for participation by other AI programs.

6 Conclusions

The field of toxicology is a rich source of difficult scien­ tific problems, and there is a pressing need for analysis methods that can advance our understanding of the is­ sues involved. We believe that the Predictive Toxicology Evaluation trials being conducted by the US National In­ stitute for Environmental Health Sciences afford AI pro­ grams a unique opportunity to participate in obtaining an improved understanding of the molecular mechanisms underlying chemical carcinogenesis. Should they be suc­ cessful, it would also constitute a noteworthy example of a realistic application of AI techniques.

Entering the PTE Challenge

Entries to the PTE Challenge can be submitted via http:/ jwww. comlab.ox.ac. ukjoucl/groupsjmachlearn/PTE.

Entries are accepted here for compounds in PTE-2 . A submission requires the following : (a) name of entry; (b) predictions for the compounds in PTE-2; (c) whether or not the theory has explanatory power; and (d) a short description of the technique used for prediction. This is sufficient to compute automatically the accuracy and performance ranking of each entry.

Acknowledgements

This research was supported partly by the Esprit Ba­ sic Research Action Project ILP II, the SERC project project 'Experimental Application and Development of ILP' and an SERC Advanced Research Fellowship held by Stephen Muggleton. Stephen Muggleton is a Re­ search Fellow of Wolfson College Oxford. R.D. King was at Imperial Cancer Research Fund during the course of much of the early work on this problem. We would also like to thank Professor Donald Michie and David Page for interesting and useful discussions concerning the use of ILP for predicting biological activity.