Learning Groupwise Explanations for Black-Box Models

1 Introduction

The decision process of state-of-the-art machine learning models such as deep neural networks is often obfuscated by their intricate architectures. Despite the impressive prediction accuracy, lack of interpretability inevitably hinders the adoption of these models, especially when users need to understand the model behavior and ensure the models are correct and ethical [Ribeiro et al., 2016; Chu et al., 2018].

In recent years, providing explanations for black-box models has attracted increasing attention in the research community. Substantial efforts have been devoted to explaining the model prediction of an individual instance with high fidelity [Ribeiro et al., 2016; Elenberg et al., 2017; Lundberg and Lee, 2017; Dhurandhar et al., 2018; Guidotti et al., 2018; Plumb et al., 2018]. These methods have achieved great success in providing explanations that are both succinct and faithful. However, for machine learning practitioners, there is still a gap between understanding each instance well and gaining a clear and comprehensive understanding of the overall model behavior on most instances. We observe that during the subsequent exploitation of explanations in practice, fulfilling the following user demands are essential.

D1: Obtaining a faithful understanding of the overall model behavior with limited cognitive load. While instance-level explanations are insightful, it is infeasible to examine explanations for all instances to gain a overall understanding of the model. Existing methods solve this issue either by generating one global explanation over the entire input space [Ghorbani et al., 2019; Kim et al., 2018] or selecting representative local explanations [Ribeiro et al., 2016; Ramamurthy et al., 2020]. While global explanations often fail to provide a succinct or faithful explanation when the target model is complex [Ribeiro et al., 2018], selecting representative local explanations depends on heuristic assumptions about which explanations are the most representative. The latter methods lack a mechanism to directly optimize the fidelity of the selected explanations on all instances. As a result, there is no guarantee that the representative explanations they select are highly faithful over the entire input space.

∗Yasha Wang is the corresponding author

D2: Making accurate predictions about the model behavior on unseen instances. Researchers have found that to achieve high human precision [Ribeiro et al., 2018] or generalized fidelity [Ramamurthy et al., 2020], it is essential that we clearly define the region where an explanation applies. Otherwise, users may easily make mistaken predictions about the model behavior on unseen (test) instances by employing an incorrect explanation. Existing methods typically assume that explanations can be applied to instances that are similar according to a certain feature space [Plumb et al., 2018; Ramamurthy et al., 2020]. This biased assumption often yield sub-optimal generalized fidelity [Ribeiro et al., 2018]. To alleviate this issue, [Ribeiro et al., 2018] define the region an explanation applies by using association rules. However, this method fails to provide insights on the relative importance of each feature, and there is no guarantee that the association rules can be applied to a large percent of instances.

Satisfying the two user demands is importance as they enable us to take a more complete view of the human cognitive process when interpreting models (Fig. 1). According to cognitive psychology, the major sub-processes of human's information-processing model are encoding, storage, and retrieval [Lang, 2000]. While local explanation methods help interpret and encode a single piece of information about one instance faithfully (encoding), users still need to distinguish important pieces of encoded information so that they can be stored properly with limited cognitive resource (D1, storage). Then, users need to accurately reactivate the piece of relevant information for decision making (D2, retrieval).

In this paper, we propose a principled way to simultaneously fulfill D1 and D2. Given a post-hoc local explanation method, we study how groupwise explanations that faithfully reveal model behavior on multiple instances can be learned. In particular, our contributions are three folds.

First, we propose a unified GroupwIse Model-agnostic Explanation (GIME 1 ) framework, which jointly 1) learns a limited number of groupwise explanations that interpret the overall model behavior with high fidelity (D1) and 2) specifies the region each explanation applies (D2). Different from existing methods, which handle D1 or D2 separately with different heuristic assumptions, our framework treats the extraction of groupwise explanations as a learning task and fulfills the two inherently interconnected user demands simultaneously by directly optimizing the overall fidelity. As shown in Fig. 2, our method can automatically divide instances into groups with noticeable difference in model behaviors, clearly defines each group, and provides interesting insights.

Second, we show how to effectively formulate the region an explanation applies. In particular, we discuss the desirable properties of the region formulations and why straightforward solutions may be problematic. Based on the discussion, we introduce the von Mises-Fisher (vMF) distribution [Banerjee et al., 2005], which is an example distribution that satisfies the desirable properties and leads to good empirical results.

1 Source code: https://github.com/jygao97/GIME

Finally, we conduct both quantitative experiments and experiments with real users to demonstrate the effectiveness of our method. Codes are provided in the supplementary material to facilitate reproduction of the experimental results.

2 ProblemFormulation

We formulate our problem as follows.

Input. The input of our framework includes a dataset X, a target model f to be explained, and a cognitive budget K.

• The dataset X = {x1, ..., xN} consists of N instances, each denoted by a feature vector xi ∈ Rd .
• The target model is treated as a black-box function f : Rd → R. For a classification model, f(xi) denotes the predicted probability that xi belongs to certain class.
• The cognitive budget K is the maximum number of explanations that the user can examine.

Output. We find K groupwise explanations and the regions where they apply. The explanations can be formulated by following a given local explanation method. We use LIME [Ribeiro et al., 2016] as a guiding example and define an explanation as an interpretable surrogate model gk(x) = θTk x, where θk ∈ Rd reveals the feature importance.

3 Background

Our groupwise explanation framework is designed by extending existing local explanation methods, which interpret the model prediction of a single instance with high-fidelity. Given an instance x and a black-box model f, identifying important features that contribute to f(x) is usually achieved by finding casual structures in the model's response to input features. A common practice is to perturb the input and observe how f(x) changes accordingly. Following this spirit, LIME [Ribeiro et al., 2016] constructs the neighborhood N(x) of instance x by perturbing x and derives the explanation θ by approximating f in the neighborhood:

where L(f(x′ ), θ⊤k x′ ) = ω(x, x′ )(f(x′ ) - θ⊤k x′ )2 is a loss function that measures how well the surrogate model approximates the target model, ω(x, x′ ) = exp(-D(x, x′ )2 /σ2 ), D(x, x′ ) is the distance function (cosine distance for text and L2 distance for image or tabular data), and σ is the bandwidth. Ω(·) is a regularization term that punishes complex θ to make it sparse and ensure it can be easily understood. λ controls the trade-off between fidelity and interpretability.

4 Groupwise Explanation Framework

In this section, we introduce our GIME framework.

Vanilla group assignment. When the number of instances N is much larger than the cognitive budget K, it is infeasible for users to check local explanations for all instances. In this case, we need to learn groupwise explanations that can be applied to multiple instances. A straightforward method is to introduce a group assignment matrix O ∈ {0, 1}N×K , where oik is 1 if xi is assigned to the k-th group and is 0 otherwise. Then, Eq. (1) can be extended to a groupwise version:

When K = N, solving Eq. (2) is equivalent to generating local explanations via Eq. (1) for each instance. When N > K, optimizing Eq. (2) captures K larger patterns by directly optimizing the explanation fidelity over the entire dataset.

Although Eq. (2) provides a mechanism to generate groupwise explanations (D1), it fails to help users make accurate predictions about the model behavior on unseen instances (D2). Since each explanation is characterized by a potentially large number of training instances instead of a well-defined and easy to understand closed-form expression, it is difficult for users to determine which explanation could be used for an unseen instance: users either need to make heuristic assumptions (e.g., instances with certain common features share one explanation) or need to optimize Eq. (2) again. The former method leads to low human precision [Ribeiro et al., 2018] or decreased generalized fidelity [Ramamurthy et al., 2020] due to the potentially problematic assumptions. The latter method is not only computationally and cognitively expensive, but also has data leakage issues and contradicts the goal of improving generalized fidelity, since the explanation assignment can only be determined after we observe the model behavior (e.g., f(x)) on the unseen instance.

Bayesian assignment. We tackle the above problem by adopting a Bayesian framework [Bishop, 2006]. Let us consider the closed-form expression for the applied region of an explanation. Without loss of generality, we define the expression by using a probabilistic distribution p(x|ψk), where

ψk denotes the region parameters for the k-th groupwise explanation, and p(x|ψk) refers to the probability that the k-th explanation can be applied to a potentially unseen instance x. Choosing different types of distribution allows us to adjust how "hard" the region assignment is. If we use a distribution like Gaussian, then whether an instance belongs to a region is probabilistic ("soft"). When we adopt distributions like the Gumbel-Softmax [Jang et al., 2016], we can clearly determine whether an instance belongs to a region or not ("hard").

Let Ψ = [ψ1, ...,ψK] denote all region parameters. Following the topic assignment probability in topic models [Blei et al., 2003], we rewrite p(xi|ψk) as

Then the Bayes' Rule can be used to compute the posterior probability of oik:

where p(oik = 1) is the prior that an instance be assigned to the k-th explanation. In this paper, we assume that there is no prior knowledge for preferred regions, i.e., p(oik = 1) = p(oik′ = 1) for ∀k, k′ . In this scenario, Eq. (4) simply chooses explanations for an instance through normalization. When there exists prior knowledge about the preferred regions (e.g., when larger regions are considered better), we can also easily incorporate these priors with Eq. (4).

Then, we can rewrite Eq. (2) to consider the expected fidelity in all instances:

Eq. (5) joint learns the groupwise explanations (Θ) and the regions in which they apply (Ψ). Here, minimizing the first term in J(Θ,Ψ) is equal to maximizing the expected fidelity in all instances. Note that the weight for the infidelity ∑x′∈Ni L(f(x′ ), θ⊤k x′ ) is p(oik = 1|xi,Ψ), which satisfies ∑Kk=1 p(oik = 1|xi,Ψ) = 1 for ∀i. This means that the infidelity in terms of any single instance is punished. Moreover, we add Ω(Ψ), which is a regularization term that measures how easy it is for humans to understand the applied regions. Optimization. J(Θ,Ψ) can be optimized by using an alternative minimization algorithm. The key idea is to minimize the objective function with respect to either Θ or Ψ, while fixing the other parameter. We find that this alternative optimization method stably convergences to better solutions than directly applying gradient descent on both parameters. More specifically, we first initialize the instance groups by using K-means. We then initialize Θ by learning an explanation for each group. Next, we iteratively apply two steps. The first step is learning applied regions, in which we fix Θ and minimize J(Θ,Ψ) with respect to Ψ by using gradient descent. The second step is groupwise explanation generation, in which we fix Ψ and minimize the objective with respect to Θ. Following [Ribeiro et al., 2016], we set Ω(θk) to the L0 norm ||θk||0 and approximately optimize the objective by using the K-LASSO algorithm [Ribeiro et al., 2016].

5 Region Formulation

In this section, we introduce how to formulate the region that an explanation applies, i.e., how to determine the mathematical form of distribution p(xi|ψk). We start by discussing the desirable properties of the regions and why intuitive solutions like Gaussian may be problematic. Then, we introduce an example distribution that satisfies the desirable properties: the vMF distribution [Banerjee et al., 2005].

Desirable properties. Similar with the surrogate model gk, to achieve high human precision, we need to define the regions to ensure both interpretability and accuracy (fidelity):

• Interpretability requires that it is easy for humans to understand the regions. Thus, the formulation of the region cannot be too complicated (e.g., consists of multiple nonlinear neural layers). Moreover, the number of features involved should be limited and it is more desirable if the formulation allows for modeling of sparse features.
• Accuracy demands that the formulation of the region is expressive enough to discriminate different regions and accurately represent the covered instances of an explanation. For example, compared with a formulation that assumes independence between features, it is more desirable if relations between features can be modeled.

Issues of the Gaussian distribution. A straightforward choice of p(xi|ψk) is the Gaussian distribution, in which Ψ = [(µ1, Σ1), ..., (µK, ΣK)] and

Eq. (6) means that k-th explanation can be applied to instances with mean µk and variance Σk in the feature space. However, characterizing explanation region with Gaussian distribution has two issues. First, the interpretability of the regions is limited, because the Gaussian distribution cannot effectively constrain the number of involved features by modeling sparse features. When only part of the features are involved, Σk is not full rank, i.e., |Σk| = 0, and the Gaussian distribution degenerates and does not have a density. Second, the accuracy of the regions may not be desirable, since it is difficult for the Gaussian distribution to model relationships between features. If we consider Σk as a diagonal matrix with d parameters, then Eq. (6) fails to model how correlated features compose a pattern. Learning feature correlations requires that we learn the d2 parameters of Σk, which is quite computationally expensive, especially for natural language processing tasks with tens of thousands of words (features). Modeling with the vMF distribution. To address these issues, we introduce the von Mises-Fisher (vMF) distribution, which 1) is an example distribution that satisfies the two desirable properties and 2) achieves good empirical results. The vMF distribution is a probability distribution on a unit hypersphere in Rd . In particular, vMF is parameterized by a direction vector φ ∈ Rd with ||φ|| = 1 and a factor τ > 0 that determines the concentration of the distribution:

C(τ) = (2π)-d/2 τ d/2-1 Id/2-1(τ) is a normalization factor and Id/2-1(τ) is the modified Bessel function of the first kind.

After normalization ( xi ||xi|| ), all instances lie on the surface of a unit hypersphere. φk denotes the mean direction vector of the instances and τ represents how concentrate the distribution is. The sign of φkj illustrates whether the j-th feature should be positive or negative for instances in the region, and a large |φkj| means that the instances in region k should have a large absolute value in terms of the j-th feature and that the j-feature is important for determining whether an instance belongs to region k. τ is learned together with φk and larger τ denotes a more concentrated (narrow) distribution.

We can easily check that Eq. (7) is well-defined when some elements of φk are zeros. This means that vMF can model sparse features and leads to better region interpretability compared with the Gaussian distribution. Moreover, vMF is often used by topic models [Song et al., 2015] to represent the distribution of co-occurred words (features) in a topic, which illustrates its capability in modeling relationships between features and representing regions with good accuracy.

Based on Eqs. (3)(4)(7), we have:

Eq. (8) shows that p(oik = 1|xi, Ψ) can be computed without considering the complicated normalization factor C(τ).

6 Experiment

6.1 Experimental Settings

Datasets. We use six real-world benchmark datasets. The first three are textual datasets and the last three are tabular ones. Specifically, Polarity [Maas et al., 2011] contains highly polar movie reviews and the task is to classify their sentiment. Subjectivity [Pang and Lee, 2004] includes processed sentences that are labeled as either subjective or objective. 20 Newsgroup 2 is a collection of news articles. Following [Ribeiro et al., 2016], we focus on the task of determining whether an article is about Christianity or Atheism.

2 http://qwone.com/∼jason/20Newsgroups/

AutoMPG concerns predicting fuel consumption based on attributes of cars. Wine Quality predicts wine quality based on physicochemical tests. Communities enables predicting community crimes based on socio-economic data. Statistics of datasets are shown in Table 1.

6.2 Generalized Fidelity

Baselines. We compare GIME with five baselines. The first two baselines, LIME [Ribeiro et al., 2016] and MAPLE [Plumb et al., 2018], are widely-used local explanation methods. We use the submodular method in [Ribeiro et al., 2016] to pick representative explanations for them. The third baseline, MAME [Ramamurthy et al., 2020], selects multi-level representative explanations from pre-computed instance groups. We also design two other methods as baselines for groupwise explanations: CTE (Cluster-ThenExplain) that first determines instance groups by clustering with K-means and then learns an explanation for each group by optimizing Eq. (2), and ETC (Explain-Then-Cluster) that first learns local explanations and then clusters the local explanations to form instance groups. The groupwise explanations of ETC are average local explanations in each group.

Evaluation Metrics. Following [Plumb et al., 2018], we evaluate the fidelity of explanations by:

where x is a test instance, the neighborhood size |N(x)| is set to 10, and the neighborhood weight ω(x, x′ ) is the same as that in Eq. (1). To measure the difference between f and g, two types of M(·, ·) are considered: the root mean square error (RMSE) and the classification Accuracy that evaluates whether f(x′ ) and g(x′ ) give the same label. Lower RMSE and higher accuracy indicate better performance.

Implementation details. We train f and learn explanations on the training set, tune hyperparameters by using the validation set, and evaluate explanations on the test set. The hyperparameters of the baselines are initialized by following the corresponding paper and tuned to achieve the optimal performance. For fair comparison, all explanation methods use the same interpretable surrogate model as in [Plumb et al., 2018; Ribeiro et al., 2016], i.e., gk(x) = θ⊤k x. If not specifically mentioned, K is set to 20 for large datasets (Polarity and Subjectivity), 10 for middle-sized datasets (20 Newsgroup, Wine Quality, Communities), and 4 for small datasets (AutoMPG). We ensure that all explanations have the same number of nonzero features (5 for tabular data and 50 for textual datasets). Each experiment is repeated five times and we report the average and standard deviation of RMSE and Accuracy.

Following [Ramamurthy et al., 2020], we measure generalized fidelity by simulating the process that users apply explanations on unseen instances. Given an unseen instance xi and explanations g1, ..., gK, simulated users leverage the explanation with the maximum probability pik to predict model behaviors on xi. Though pik is well defined for our method (pik = p(oik = 1|xi,Ψ)) and MAPLE (pik is the local training distributions), other baselines lack the mechanism for determining whether an explanation can be applied to a test instance. For these baselines, we try three formulations of pik and use the one that results in the highest validation fidelity. The first and second formulations measure how many features in xi are "activated" by gk: pik ∝ esik , sik is ||θk ⊙xi||0 and ||θk ⊙ xi|| respectively, where ⊙ denotes elementwise product. In the third formulation, sik is the negative L2 distance between xi and xk (local methods) or the center of instance groups (others). We find that LIME performs the best with the first formulation and others with the third one.

Overall Performance. We train BERT [Devlin et al., 2018] and SVR [Awad and Khanna, 2015] on textual and tabular datasets, respectively. These models are considered as blackbox models to be explained. Table 2 and ?? show the explanation fidelity on unseen instances. The results of MAME on textual datasets are omitted due to memory issues (mentioned also in [Ramamurthy et al., 2020]): it cannot successfully run on a 64-bit server with 64G of memory for textual datasets.

We observe that GIME significantly outperforms other baselines in terms of fidelity. This indicates that GIME allows users to get more faithful understanding of the model within limited cognitive budget. We also find that MAPLE and MAME generally perform better than LIME. This is because they consider more instances when generating explanations, thus their explanations may generalize to a larger region. CTE and ETC achieve the second best performance on some datasets, illustrating the effectiveness of groupwise explanations. However, they heuristically determine instance assignment via clustering, thus yielding lower fidelity compared with GIME, which jointly learns groupwise explanations and the regions that they apply in a unified framework. Parameter Sensitivity Analysis. Fig. 3 shows how RMSE and accuracy change with the user budget K on 20 Newsgroups. We find that GIME stably performs better than baselines at different values of K, which demonstrates the robustness of our method. Moreover, the fidelity of GIME increases with increasing K at first, and then slowly decreases after K > 15. This shows that too many or too few explanations may hurt generalized fidelity, validating the necessity of learning a limited number of groupwise explanations.

We also evaluate how fidelity changes with different levels of coverage. An instance xi is marked as covered if ∃k, pik ≥ τ. We vary τ to obtain the results when the percentage of unseen instances that are covered is 20%, 40%, .., and 100%. As shown in Fig. 4, GIME consistently achieves better fidelity than other methods at different levels of coverage. Moreover, the fidelity of some baselines like MAPLE does not always increase with increasing pik (decreasing coverage), which indicates that the way they determine whether an explanation can be applied is not always reliable. This may due to the fact that these methods heuristically determine pik instead of learning it jointly with the explanations. In comparison, the fidelity of GIME stably increases with larger pik, indicating that our region formulation is effective in selecting the instances that can be best interpreted by the explanation.

6.3 User Study

This experiment evaluates whether the explanations are useful in helping real users predict the model behavior on unseen instances. Specifically, we hire five native English speakers through a vendor company. For each explanation method, users are asked to first investigate the provided explanations and the information about when the explanations may be applied (φk for GIME, instance center for CTE and MAME, and the examplar instance for MAPLE). Then, the users are given five instances from the test set and asked to 1) match each instance with an explanation and 2) predict the model output for the instance. For the regression task on AutoMPG, users are only required to choose the appropriate range: [<0.5, -0.5 to 0, 0 to 0.5, and >0.5]. The explanations are provided to the users in random order and the users do not know which explanation method they are labeling. Explanations generated by GIME are shown in Fig. 2.

Table 4 compares GIME with the three most competitive baselines. M-Acc denotes the portion of instances that are matched to the correct explanations and P-Acc represents the portion of instances that are predicted correctly. We can see that GIME achieves higher M-Acc than other methods on the two datasets. It demonstrates that modeling the region with an intuitive closed-form expression helps users better understand the applied regions of explanations, which is a prerequisite for successful prediction of model behaviors on unseen instances. GIME also achieves the highest P-Acc. This validates the effectiveness of our framework in faithfully interpreting the black-box model and enabling users to make accurate predictions with a limited number of explanations.

7 Conclusion

We study two user demands that are important for understanding black-box models: 1) obtaining a faithful overall understanding of the model with limited cognitive load and 2) making accurate predictions about the model on unseen instances. To fulfill the two demands, we propose a unified Groupwise Model-agnostic Explanation framework, which learns a limited number of groupwise explanations with high fidelity as well as the region where each explanation apply. Experiments demonstrate the effectiveness of our method.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (61772045) and the Project 2019BD005 supported by PKU-Baidu fund.