Trading-Off Cost of Deployment Versus Accuracy in Learning Predictive Models

1 Introduction

Many industries (e.g., retail, manufacturing, and medicine) are recognizing the advantages of using predictive models to make key decisions. They also understand that the cost of obtaining input measurements should be balanced with their effectiveness in prediction when choosing which model to deploy. This is especially challenging when the cost structure for an application is complicated. As an important example, consider the cost structure associated with deploying a predictive model in an Intensive Care Unit (ICU) (see the costdependency graph in Figure 1). In such a setting, the following hold: (i) costs may be defined for tests, measurements, or activities and these costs may be of different types (e.g., the

⇤This project was supported by NSF IIS-1418590 and the Johns Hopkins University IDIES Seed Funding Initiative.

Equal contribution.

Suchi Saria

Johns Hopkins University Department of Computer Science Baltimore, Maryland ssaria@cs.jhu.edu

financial cost of acquiring a blood test versus the staff time taken to draw blood); (ii) features are obtained using one or more measurements (e.g., lactate level or creatinine) which in turn are obtained by ordering a test; (iii) a test may consist of a single measurement (e.g., lactate level) or a panel of measurements (e.g., CBC panel); (iv) a measurement can be ordered via multiple tests (e.g., creatinine can be ordered on its own, as part of a basic or a comprehensive metabolic panel, each having a different financial cost); (v) multiple features can be derived from the same measurement (e.g., the heart rate variability and the heart rate trend can both be derived from the heart rate trace); and (vi) some features may require multiple measurements (e.g., shock index is derived from blood pressure and heart rate measurements). These aspects make the cost structure complicated.

Before expanding upon the challenges involved with addressing complex cost-structures such as the one above, we first introduce the mathematical setup for learning predictive models. This involves data that is formally represented by sets of pairs ¶(xi↪ yi)♦Ni=1 for some integer N, where xi 2 <n for some integer n and yi 2 <, for all 1  i  N. The vector xi denotes the ith input (feature) vector and yi the output (label) associated with the ith input vector xi. The goal is then to predict the unknown output associated with a newly obtained input vector by using the knowledge one learns from the data ¶(xi↪ yi)♦Ni=1 . A popular approach for performing this task is to build predictive models via empirical regularized-loss minimization [Vapnik, 1998]. The problems used take the form

where β 2 <n is the parameter vector to be learned, L is a loss function such as the logistic loss L(β; (xi↪ yi)) = log(1 + expyixTi β ), and R is a regularizer. The choice of regularizer amounts to giving preference to certain models, e.g., the `1-regularizer R(β) = λkβk1 for some λ > 0 prefers models defined by a sparse vector β. In practice, the regularizer should be chosen to reflect the preferred models, which are often driven by the costs associated with the application. For example, in compressed sensing, one wishes to find a sparse solution to a linear system of equations. Thus, the cost, i.e., the number of nonzeros in a prospective solution, is harmonious with the `1-regularizer, which promotes sparse solutions. Note that in this example, as well as many others, the costs are directly tied to the feature vectors themselves, i.e., they occur at the feature level.

How does one design an appropriate regularizer for problems with a complicated cost structure, such as for the ICU example above? We address that question in this paper.

Related work. Learning models in the presence of costs has received significant attention in recent years (e.g., [Xu et al., 2012; Ji and Carin, 2007; Weiss and Taskar, 2013; Xu et al., 2013; Raykar et al., 2010]). Existing work has primarily targeted applications where the cost of computation is the primary concern, and this cost is elicited at the feature level. Moreover, much of this work has focused on optimizing performance when information is acquired incrementally [Ji and Carin, 2007; Xu et al., 2012; Trapeznikov and Saligrama, 2013; Kanani and Melville, 2008; Kapoor and Horvitz, 2009]. In [Ji and Carin, 2007], they define the problem of cost-sensitive classification and use a partially observable Markov decision process to trade-off the cost of acquiring additional measurements with classification performance. While they apply their method to a medical diagnosis problem, their costs were approximated at the feature level. In [Xu et al., 2012], stage-wise regression is used to learn a collection of regression trees in a manner that ensures that classifiers built from more trees is more accurate, but more expensive. For the task of ranking web page documents, they showed improved speed and accuracy by accounting for feature costs-simple lookups (e.g., word occurrences) versus those needing more computation (e.g., a document-specific BM25 score for measuring relevance to a query). For structured prediction, [Weiss et al., 2013] proposed a two tier collection of models of varying costs and a model selector; for each new test example, their selector adaptively chooses a model. For vision applications (e.g., articulated pose estimation in videos), they showed gains in performance by adaptively selecting models of varying costs by using a histogram of gradient features at a fine (expensive) versus a coarse (cheap) resolution. These solutions focused on applications with no dependencies between the costs for the units reasoned over (i.e. feature or model costs are independent) and when they are provided upfront. As predictive models continue to find their way into many important applications, a means for incorporating rich cost structures is needed.

Returning to our example in healthcare, the challenge of incorporating costs arises from the dependencies between features, measurements, tests, and required activities. Measurements may be obtained from a singleton test or as part of a test that yields multiple measurements. Tests may have different resource costs associated with them, while features may be derived from more than one measurement. These dependencies between features, measurements and tests yield a complex dependence structure between the features. Moreover, various costs are specified at different levels of this hierarchy; therefore, the cost of a feature is not specified upfront, but rather is dependent on which other features, measurements, and tests are selected.

Cost imposed via a hierarchical dependency graph is reminiscent of past works utilizing structured sparsity penalties (see the survey [Wainwright, 2014] and [Bach et al., 2012]), especially those using tree-based regularizers [Kim and Xing, 2010] and penalties with overlapping groups and hierarchical structure [Zhao et al., 2006; Bach et al., 2012]. Different from these past works, a key challenge for our task is that the structure of the group regularizer is not given and its construction is not straightforward. We show that cost-dependency graphs are naturally captured via Boolean circuits-graphs where nodes share a combination of AND and OR connections with its parents. Only leaf nodes (i.e. feature nodes) of this circuit are included in the regularizer while the internal nodes (e.g., measurements needed to obtain features) induce dependency between the leaf nodes. The presence of mixed AND/OR relationships and the non-inclusion of internal nodes renders our application different from past works. Other regularizers such as OSCAR [Zhong and Kwok, 2012] and Laplacian Net [Huang et al., 2011] aim at discovering group structure when the features are highly correlated. In our setting, the groups are determined by the structure of the cost graph, not by the correlations between the features.

Our contributions. We develop a new framework for defining structured regularizers suitable for problems with complex cost structures by drawing upon a surprising connection to boolean circuits. In particular, we represent the problem costs as a boolean circuit, and then use properties of boolean circuits to define the exact cost penalty. Based on our exact cost penalty, any standard convex relaxation may be employed for the purpose of computational efficiency, and here we choose a standard `1-`1 relaxation. Our new regularizer may be used within an empirical risk minimization framework to tradeoff cost versus accuracy. We focus on the oneshot setting (i.e., when all measurements are obtained upfront), although our regularizer is also applicable in the incremental setting. Since the cost-structure of many real-life applications may be represented as a boolean circuit, the contribution of our work is substantial.

Our ideas are presented in the context of a challenging healthcare application-the development of a rapid screening tool for sepsis [Angus and van der Poll, 2013]-using data from patients in the ICU [Saeed et al., 2011]. In this setting, examples of potential users include patients, doctors, and administration. Our experiments show that our regularizer allows for a collection of models that are in harmony with a user's cost preferences. Numerical comparisons to a costsensitive `1-a natural competitor to our proposed regularizer that does not account for the complicated cost structureshows that models obtained with our regularizer have a better prediction/cost tradeoff. Compared to existing approaches in predictive modeling where cost preferences are often accounted for post hoc, our scheme provides a new way to account for complex cost preferences during model selection.

2 Regularizers for complex cost structures

Our scheme is general since it may be applied to any problem with a cost structure that may be represented as a finite-layer boolean circuit. However, for clarity of exposition, we first focus on a particular healthcare application that also serves as the basis for the numerical results presented.

2.1 An example from the intensive-care unit (ICU)

We formulate a structured regularizer for the cost structure associated with risk prediction applications for the in-hospital setting. These include problems such as prediction of those at risk for death, the likelihood of readmission, and the early detection of adverse events, e.g., shock and cardiac arrest.

Recall the cost dependency graph for the ICU example in Figure 1. The features are represented by nodes in layer-1, and their calculation requires a subset of measurements from layer 2, i.e., nodes in layer-1 share an AND or OR relationship with those in layer-2. Measurements can be obtained in a number of ways by performing various tests, which are represented at layer-3, i.e., nodes in layer-2 similarly share an AND or OR relationship with those in layer-3. The caregiver activities are represented at layer-4 and are performed when a test is needed that requires that action, i.e., layer-3 shares an AND relationship with layer-4. Every relationship in this boolean circuit is described using only logical AND and OR operations. Note that, without loss of generality, we include fictitious nodes "none-1" and "none-2" in layer-4 so that the collection of input nodes are in the same layer.

There are three relevant costs: the financial cost of ordering a test, the waiting time to obtain a test result, and the caregivers' time needed to perform the activities required for the tests. The ideal regularizer should account for the following: (i) obtaining a measurement may cost different amounts depending on which tests are ordered to obtain it; (ii) features share costs with other features derived from the same measurement; (iii) a feature may require multiple measurements so that its cost depends on more than one measurement; and (iv) caregiver time and financial costs are additive while wait time is the maximum of the separate wait times.

Our structured regularizer requires the following sets:

to be the sets of features (layer-1 nodes), measurements (layer-2 nodes), tests (layer-3 nodes), and caregiver activities (layer-4 nodes), with nf ↪ nm↪ nt↪ and na being the number of each, respectively. We use fi - mi to mean that there is a directed edge that links node mj to node fi. Thus, our specific boolean circuit allowus to interpret fi - mj, mj - tk, and tk - al tomean the ith feature requires the jthmeasurement, the jth measurement can be obtained by performing the kth test, and the kth test requires the lth activity.

We now define the set valued mappings m(fi) := ¶mj : fi - mj♦, t(mj) := ¶tk : mj - tk♦, and t(al) := ¶tk : tk - al♦, which represent the set of measurements required to obtain feature i, the set of tests that produce measurement j, and the set of tests that require action l. We note that we have overloaded the definition of the function t above, i.e., we have two different definitions for t(mj) and t(al). However, this should not lead to any confusion since the correct definition is always clear from the context.

Next, we resolve the fact that some features may be obtained in multiple ways by ordering various combinations of tests. If this is not considered, the cost of a feature may be over penalized by our regularizer. To address this issue, let wi denote the numbers of ways feature i can be obtained. Then, for the ith feature, we define ~ fi := [fi↪1↪ · · · ↪ fi↪wi ]T and ~ βi := [βi↪1↪ · · · ↪ βi↪wi ]T so that βi↪p represents the parameter associated with ordering feature fi in the pth way. This allows us to define the extended feature and parameter vectors ~ f := [ ~ f1↪ ▷ ▷ ▷ ↪ ~ fnf ]T and ~ β := [ ~ β1↪ ▷ ▷ ▷ ↪ ~ βnf ]T .

Modeling financial cost and caregiver time: To model financial cost, which is incurred at the test level, for each test tk and feature fi we define ~nk↪i := [nk↪i↪1↪ · · · ↪ nk↪i↪wi ]T with

for all 1  k  nt, 1  i  nf , and 1  p  wi. Given a financial cost C T k of ordering test k and a weighting parameter λ$, the exact structured regularizer for financial cost is

with the indicator function I satisfying I(0) = 0 and I(z) = 1 for all z 6= 0. It follows from (2) that a financial cost for test tk is incurred only when instructed to order some feature fi in the pth way (βi↪p 6= 0), and that pth way requires test tk (nk↪i↪p 6= 0). The regularizer (2) is not computationally friendly, so we instead use the relaxed structured regularizer

where we define for a set of vectors ¶zi♦ni=1 and a subset S = ¶i1↪ i2↪ ▷ ▷ ▷ ↪ ir♦ ✓ ¶1↪ 2↪ ▷ ▷ ▷ ↪ n♦ the quantities

Note that (3) is a sum of group `1-norms, which is supported by the software SPAMS.

To model the caregivers time cost in a similar way, we define ~nl↪i := [nl↪i↪1↪ · · · ↪ nl↪i↪wi ]T with

nl↪i↪p := ⇢1 if al is used when fi is ordered in its pth way, 0 otherwise,

for all 1  l  na, 1  i  nf , and 1  p  wi, where we have again overloaded notation. The regularizer associated with caregiver activity time then becomes

with CA l being the time cost associated with the lth activity and λtime > 0 a weighting parameter. Overall, our structured regularizer becomes

By varying λ$ and λtime we trade-off the financial and caregiver activity time costs, respectively.

Remark 1 If a scaled-`1-norm was used, the user chooses a weight for each feature by condensing the complex cost structure into a single number, necessarily in an ad-hoc way.

Remark 2 Consider the 3-layer boolean circuit where layer1 contain the nodes F, layer-2 contain the nodes Z := ¶fi↪p : 1  i  nf and 1  p  wi♦, and layer-3 contain the nodes A. Moreover, the gate functions at layer1 are given, for each fi, by gfi (Z) := OR 1pwi fi↪p for all 1  i  nf , and the gate functions at layer-2 are given, for each fi↪p, by gfi↪p (A) := AND ¶l:nl↪i↪p=1♦al for all 1  i  nf and 1  p  wi. In particular, only OR gate functions are used in layer-1 and only AND gate functions are used in layer-2. Moreover, the properties of this 3-layer gate allows us to conclude that for a given caregiver activity, say al, we have ∥∥∥Wnf i=1 ~nl↪i ⊙ ~ βi∥∥∥1 ⌘ ∥∥∥W(i↪p)2Sl ~ βi↪p∥∥∥1 with the index set Sl defind as Sl := ¶(i↪ p) : nl↪i↪p = 1♦ = ¶(i↪ p) : the output of gfi↪p (·) depends on al♦, so that the definition of our regularizer (4) follows from our knowledge of the 3-layer boolean circuit. In fact, the only properties of the circuit that we used were (i) layer-1 was the feature layer; (ii) layer3 contained the nodes whose costs we were modeling; (iii) layer-1 only contained OR gates; and (iv) layer-2 only had AND gates. This motivates the general case below.

Modeling testing wait time: We use a simpler approach to address the time needed to obtain test results. Note that the wait time for a set of test results is the maximum of the wait times for each individual test (assuming that tests can be ordered in parallel). Thus, for a given upper bound, say W, on the tolerated testing wait time, we only allow tests that have a wait time less than W to be used. This amounts to selecting a reduced boolean circuit containing only these allowed tests, the caregiver actions required to obtain these allowed tests, measurements that result from the allowed tests, and the features that may be calculated from the included measurements.

2.2 Structured regularizer: the general case

We now show how to define our regularizer for any problem whose cost structure may be represented as a finite r-layer boolean circuit; Figure 1 is an instance of such a circuit.

An r-layer boolean circuit consists of layers of finitely many nodes. The lowest layer (layer-1) consists of the set of output nodes, while the highest layer (layer-r) contains the input nodes. Additionally, we are given boolean functionsdefined on the basis B = ¶AND↪ OR↪ NOT♦-for all nodes. Formally, each boolean function performs the basic logical operations from B on one or more logical inputs from the previous layer, and produces a single logical output value. The healthcare example in Figure 1 is a 4-layer boolean circuit with the features corresponding to layer-1, the measurements to layer-2, the tests to layer-3, and the activities to layer-4.

Let Ni := ¶xi↪1↪ xi↪2↪ ▷ ▷ ▷ ↪ xi↪ni ♦ be the nodes in layer-i for some ni. By removing double negations, and using the laws of distribution and De Morgan's laws, the r-layer circuit may be reduced to a 3-layer boolean circuit in disjunctive normal form [Pfahringer, 2010; Zeng et al., 2012]. The nodes in the 3-layer circuit are then layer-3: ¶xr↪1↪ xr↪2↪ ▷ ▷ ▷ ↪ xr↪nr ♦, layer2: ¶z1↪ z2↪ ▷ ▷ ▷ ↪ zm♦, and layer-1: ¶x1↪1↪ x1↪2↪ ▷ ▷ ▷ ↪ x1↪n1 ♦ for some m and set ¶zi♦mi=1 of nodes for layer-2. Moreover, the only logical operations used by the boolean functions gzi (·) in layer-2 are AND and NOT operations, while the boolean functions gx1↪i (·) in layer-1 only use logical OR operations. (In Remark 2 we showed how a circuit of this form could be obtained for the healthcare example.) If we define the vectors ~z := [z1↪ z2↪ ▷ ▷ ▷ ↪ zm]T and ~ β := [β1↪ β2↪ ▷ ▷ ▷ ↪ βm]T , then we define our cost-driven structured regularizer as

with Sk := ¶j : gzj (·) depends on the logical value of xr↪k♦. When this regularizer is used in model prediction, an optimal value for the extended vector ~ β is obtained. Using this vector and the fact that layer-1 only has OR gates, we know that a node x1↪i in layer-1 (i.e., the feature layer for the healthcare application) has the logical value of 1 (i.e., the feature should be computed) if βj 6= 0 for some j 2 ¶k : gx1↪i (·) depends on the logical value of zk♦.

Remark 3 Although our exact penalty is approximated by an overlapping group regularizer, what is non-trivial is determining which features belong to which groups for complex cost graphs. By relating the cost graph to a Boolean circuit, we can use properties of Boolean circuits to define an extended feature set and overlapping structure that is correct for arbitrary cost graphs. Moreover, this connection allows for the use of widely used off-the-shelf software such as SymPy to convert an arbitrary graph to the 3-layer circuit in disjunctive normal form used to define our exact regularizer.

3 Numerical experiments

We focus on early detection of septic shock-an adverse event resulting from sepsis-since it is the 11th leading cause of patient mortality in the United States. (Mortality rates are between 30% and 50% for those who develop septic shock [Angus et al., 2001].) Although early treatment can reduce the patient mortality rate, less than one-third of patients receive appropriate therapy before onset. Therefore, an early warning system that accurately predicts a sepsis event allows for appropriate treatment and a higher quality of patient care. (See the references in [Ho et al., 2014] for recent work on sepsis detection; none have tackled the cost of deployment.) More broadly, this problem is an instance of cost-sensitive risk prediction for automated triage [Wilson et al., 1981].

We constructed the full cost-graph in collaboration with domain experts, which resulted in 119 nodes and 294 edges. (The full list of measurements and tests can be found in Figure 2.) We combine the logistic-regression function with our structured regularizer (5) to predict the probability that a patient will develop septic shock. We use MIMIC-II [Saeed et al., 2011], a large publicly available dataset of electronic health records from patients admitted to four different ICUs at the Beth Israel Deaconess Medical Center over a seven year period. Using the processing described in [Henry et al., 2015], 2↪291 positive patients and 12↪646 negative patient cases were obtained.

We answer two questions. Does our structured regularizer lead to diverse models, especially in terms of the various costs? How well does our new structured regularizer perform compared to existing available solutions? Natural comparisons include regularizers that account for cost but do not account for the cost-dependence structure, e.g., the `1-norm regularizer or the cost-sensitive `1-norm. A comparison with other structured sparsity penalties would also seem appropriate, but none exist that construct the penalty for complex cost graphs (see the discussion in the related work section). We do not include comparisons to stage-wise alternatives because they are suboptimal to the proposed cost-sensitive `1-norm, which yields a global optimum.

Experimental setup. We split the individuals into training (75%) and test (25%) sets. From the training set, we process the data using a sliding window to extract positive and negative samples consisting of the features observed at a given time, and an associated label that is positive if septic shock was occured within the following 48 hours and negative otherwise. Since the dataset is imbalanced, we subsample the negative pairs to obtain a balanced training set.

For our test set, we use the learned model to predict the risk of septic shock at each time point. This gives a trajectory of risk for septic shock over time for each individual. For a given threshold, an individual is identified as having shock if their risk trajectory rose above that threshold prior to shock onset. For this threshold, we calculate: (i) sensitivity as the fraction of patients who develop septic shock and are identified as having a high risk of septic shock; (ii) the false positive rate (FPR) as the fraction of patients who never develop septic shock but are identified as high risk patients by our model; and specificity as 1 - FPR. The receiver operating characteristic (ROC) curve and area under that curve (AUC) are obtained by varying the threshold value, with patients identified as at-risk if their predicted probability was above the threshold value. We use 10 bootstrapped samples to estimate confidence intervals for the AUC.

We used the MEXFISTGRAPH routine in SPAMS to minimize the sum of the logistic function and our structured regularizer (5). The maximum allowed iteration limit was set to 5↪000 and the termination tolerance (duality gap) to 10-3 .

Model diversity. Three costs were considered: (i) financial cost associated with ordering a test; (ii) nursing-staff's time needed to perform the activities required for the tests; and (iii) waiting time to obtain a test result. For a chosen maximum wait time and weighting parameters λ$ and λtime, our algorithm minimizes the sum of the logistic-regression function with the regularizer (5), which returns parameters for a model from which we may compute an associated ROC, AUC, financial cost, nurse-time, and test result wait time. By sweeping over a range of values for the maximum allowed wait time, λ$, and λtime, we obtain models with various costs that reflect preferences for different models. For our cost-dependency structure, there are three possible maximum wait times: 50 minutes, 10 minutes, and 0 minutes. For each of these scenarios, we select values for λ$ and λtime from an equally spaced grid over the interval [10-3 ↪ 10-7 ], which yields a collection of models at the cost-accuracy frontier. Four models-denoted as M1, M2, M3, and M4-are represented in Table 1 to illustrate the tradeoff achieved by our approach.

ModelM1 is the most cost-effective. It uses existing measurements that are routinely collected and therefore it neither incurs a financial cost nor the need for nursing-time to acquire new measurements. Since no additional tests are required, the wait time for the model is also zero minutes. The model achieves a relatively high AUC of 82▷79. The set of measurements that were found to be most predictive include: clinical history (on ventilator, on pacemaker, has cardiovascular complications); vitals (shock index, raw and derived features of the heart rate, SpO2, FiO2, blood pressure, respiratory rate); and time since first presentation of systemic inflammatory response syndrome (SIRS).

At the other extreme, model M4 has a financial cost of $170, requires a nurse-time of 30 minutes, and a total test result wait time of 50 minutes. It requires measurements attained from numerous additional tests such as the arterial blood gas, comprehensive metabolic panel, hematocrit, hemoglobin, and urine tests. By using these measurements, the accuracy increases to an AUC of 87▷21, and shows a clinically significant gain in sensitivity compared to model M1.

Models M2 andM3 have cost and performance intermediate to models M1 and M4. Also, it is interesting to see that M2 and M3 achieve similar performance in very different ways. Model M2 selects a urine measurement with a test result wait time of 10 minutes and 10 minutes of nurse time, while M3 does not require any nurse time, but needs 50 minutes of wait time to receive test results.

For the specificity level of 0▷85, the models vary significantly in terms of sensitivity. As expected, model M1 has the lowest sensitivity value of 0▷61, followed by model M3 with a value of 0▷65, then model M2 with a value of 0▷66, and finally model M4 with a value of 0▷72. Thus, with additional resources, M4 is significantly better at identifying patients that eventually did experience septic shock. The added sensitivity is useful for units with vulnerable populations.

In practice, a user can benefit from our structured regularizer in at least two ways. First, the user can obtain multiple predictive models by choosing a diverse set of values for the weighting parameter values λ$ and λtime. This brute force approach would provide a diverse landscape of models with very different cost distributions. A second approach involves the user making a sequence of decisions. In particular, the user would adjust the weighting parameter values after the results using their current values is obtained. Specifically, the user would adjust the parameter values so as to obtain a new model that is more aligned with their preferences.

Cost in dollars ($) versus area-under-the-curve (AUC) tradeoff.

Comparison with the `1 and scaled `1-norm. Simple regularizers (e.g., the `1-norm) can not capture the rich structure of the cost-dependencies in real-world domains such as healthcare. Figure 3 compares our structured group regularizer (Group) to the `1-norm (L1) and a scaled-`1-norm (L1scaled). The L1 method is a straightforward implementation of logistic regression plus `1-norm minimization. The L1scaled algorithm combines the logistic function with a scaled`1-norm given by R(β) := λkSβk1 for some diagonal scaling matrix S = diag(s1↪ ▷ ▷ ▷ ↪ sn) and weighting parameter λ > 0. In our tests, we defined si as the maximum of 1 and the minimum cost required to obtain the ith feature. Although this choice is reasonable, it is also ad-hoc, which is necessarily true for any choice of the scaling matrix S. This is a consequence of the fact that it takes a complicated cost structure and represents it by n numbers, which is too simplistic.

Figure 3 compares the tradeoff between financial cost and AUC values of Group, L1, and L1-scaled. (Similar plots could be constructed for test result time and nurse time.) The reported cost of a model is obtained by post-processing, whereby we sum the costs for the unique set of tests required. Each point in the plot represents a pair ($↪AUC) for some model. For algorithms L1 and L1-scaled, the points were obtained by varying the parameter λ over the interval [10-3 ↪ 10-7 ]. For algorithm Group based on the regularizer (5), we fixed λtime = 10-7 and let λ$ take on the same values as λ for algorithms L1 and L1-scaled; this placed different levels of emphasis on only the financial cost, which further illustrates the flexibility of our cost-driven structured regularizer. For all three algorithms we only use tests that have a maximum allowed wait time of 50 minutes.

First, observe that algorithm L1 performs the worst. In particular, the cheapest model recovered by algorithm L1 costs $129 and had an AUC of approximately 77▷5. At that same price-point, algorithms L1-scaled and Group were able to obtain AUC values of approximately 84▷6 and 86▷1. This is not surprising since the `1-regularizer used by algorithm L1 causes the most predictive features to be chosen first, without any regard to the resulting financial cost. This performance is not surprising and may be used to motivate algorithm L1scaled. In essence, L1-scaled incorporates a rough measure of the cost for each feature through the choice of si, as described above. Second, Figure 3 shows that our cost-sensitive regularizer significantly outperforms algorithm L1-scaled. Third, observe that a (perhaps) surprisingly high AUC value (approximately 84▷5) may be achieved for models without any financial cost by algorithms L1-scaled and Group. For the prediction of sepsis, this means that although expensive tests produce measurements that allow for better prediction accuracy, one may still do well without incurring any (additional) financial costs. This observation should be leveraged when implementing screening tools or assessing risk stratification.

4 Conclusions and discussion

We designed a structured regularizer that captures the complex cost structure of many applications. The feature, measurement, test, and caregiver activity hierarchy in healthcare was used as an example, but we showed how our method can be used anytime the cost structure can be represented as a finite-layer boolean circuit. By building a regularizer that was in harmony with user's application-specific cost preferences, our experiments produced a diverse collection of models. Moreover, our cost-sensitive regularizer achieved better prediction accuracy for the same (often lower) cost when compared to `1 or weighted-`1 norms commonly used. We comment that the design of our regularizer must only be done once up-front for each application, and then may be reused to answer a host of questions, e.g., through model prediction.

Beyond sepsis, our regularizer applies to many prediction problems in healthcare [Bates et al., 2014] including early detection of other potentially preventable conditions, e.g., pneumonia, c-diff, and renal failure [Fuller et al., 2009]. More broadly, our regularizer is applicable to cost-sensitive prediction problems whose cost-graphs may be represented with a logical AND and OR structure associated with boolean circuits. In traffic prediction, for example, features (e.g., mean and trend) of the traffic velocity can be computed from streams acquired from sources (e.g., querying crowdsourced GPS devices, pneumatic road tubes, piezo-electric sensors, cameras, and manual counting) at different locations including live event streams [Horvitz et al., 2012]. Considerations for choosing a model include the cost of acquiring and deploying the sensors, the staff time to maintain the sensors, and the recurring costs of acquiring traffic, weather and live event streaming data. Depending on the availability and cost of resources, one may wish to deploy different models in different regions.

Although our cost-sensitive regularizer may be used in many important applications, it has limitations. Its more accurate modeling of the cost-graph is achieved at the expense of requiring additional computation to construct. Converting a general r-layer Boolean circuit to a 3-layer Boolean circuit has complexity O(sfr ), where s is the number of nodes and f is the fan (the largest number of allowed gate inputs/outputs) of the circuit. However, most cost-graphs are highly structured, thus dramatically reducing the computational cost. For example, constructing the regularizer for the ICU application took approximately 10 seconds on a MacBook Air laptop (1.8 GHz Intel Core i5 processor with 4GB of RAM). This modest additional cost is a consequence of the structure of the costgraph: most nodes have relatively few connections to nodes in adjacent layers, and the logical gates mostly contain simple OR and AND constructs. Since these properties hold for many cost-graphs in practice, our approach is often practical.

Acknowledgments

We would like to thank Mu Wei for fruitful conversations that ultimately lead to the ideas presented here. We would also like to thank Katherine Henry for her help in obtaining and cleaning the data used in the healthcare example, and for many conversations in which we benefited from her expert knowledge of the data. Finally, we thank Dr. Harold Lehmann for helping us understand end-user preferences in the clinical environment.