A Geometric View of Conjugate Priors

1 Introduction

In probabilistic modeling, a practitioner typically chooses a likelihood function (model) based on her knowledge of the problem domain. With limited training data, a simple maximum likelihood estimation (MLE) of the parameters of this model will lead to overfitting and poor generalization. One can regularize the model by adding a prior, but the fundamental question is: which prior? We give a turn-key answer to this problem by analyzing the underlying geometry of the likelihood model, and suggest choosing the unique prior with the same geometry as the likelihood. This unique prior turns out to be the conjugate prior, in the case of the exponential family. This provides justification beyond "computational convenience" for using the conjugate prior in machine learning and data mining applications.

In this work, we give a geometric understanding of the maximum likelihood estimation method and a geometric argument in the favor of using conjugate priors. Empirical evidence showing the effectiveness of the conjugate priors can be found in our earlier work [1]. In Section 4.1, first we formulate the MLE problem into a completely geometric problem with no explicit mention of probability distributions. We then show that this geometric problem carries a geometry that is inherent to the structure of the likelihood model. For reasons given in Sections 4.3 and 4.4, when considering the prior, it is important that one uses the same geometry as likelihood.

Hal Daum´e III

Department of Computer Science University of Maryland College Park, Maryland USA

hal@cs.umd.edu

Using the same geometry also gives the closed-form solution for the maximum-a-posteriori (MAP) problem. We then analyze the prior using concepts borrowed from the information geometry. We show that this geometry induces the Fisher information metric and 1-connection, which are respectively, the natural metric and connection for the exponential family (Section 5). One important outcome of this analysis is that it allows us to treat the hyperparameters of the conjugate prior as the effective sample points drawn from the distribution under consideration. We finally extend this geometric interpretation of conjugate priors to analyze the hybrid model given by [7] in a purely geometric setting, and justify the argument presented in [1] (i.e. a coupling prior should be conjugate) using a much simpler analysis (Section 6). Our analysis couples the discriminative and generative components of hybrid model using the Bregman divergence which reduces to the coupling prior given in [1]. This analysis avoids the explicit derivation of the hyperparameters, rather automatically gives the hyperparameters of the conjugate prior along with the expression.

2 Motivation

Our analysis is driven by the desire to understand the geometry of the conjugate priors for the exponential families. We motivate our analysis by asking ourselves the following question: Given a parametric model p(x; θ) for the data likelihood, and a prior on its parameters θ, p(θ; α, β); what should the hyperparameters α and β of the prior encode? We know that θ in the likelihood model is the estimation of the parameter using the given data points. In other words, the estimated parameter fits the model according to the given data while the prior on the parameter provides the generalization. This generalization is enforced by some prior belief encoded in the hyperparameters. Unfortunately, one does not know what is the likely value of the parameters; rather one might have some belief in what data points are likely to be sampled from the model. Now the question is: Do the hyperparameters encode this belief in the parameters in terms of the sampling points? Our analysis shows that the hyperparameters of the conjugate prior is nothing but the effective sampling points. In case of non-conjugate priors, the interpretation of hyperparameters is not clear.

A second motivation is the following geometric analysis. Before we go into the problem, consider two points in the Euclidean space which one would like to interpolate using a parameter γ ∈ [0, 1]. A natural way to do so is to interpolate them linearly i.e., connect two points using a straight line, and then find the interpolating point at the desired γ, as shown in Figure 1(a). This interpolation scheme does not change if we move to a non-Euclidean space. In other words, if we were to interpolate two points in a non-Euclidean space, we would find the interpolating point by connecting two points by a geodesic (an equivalent to the straight line in the nonEuclidean space) and then finding the point at the desired γ, shown in Figure 1(b).

This situation arises when one has two models, and wants to build a better model by interpolating them. This exact situation is encountered in [7] where the objective is to build a hybrid model by interpolating (or coupling) discriminative and generative models. Agarwal et.al. [1] couples these two models using the conjugate prior, and empirically shows using a conjugate prior for the coupling outperforms the original choice [7] of a Gaussian prior. In this work, we find the hybrid model by interpolating the two models using the inherent geometry 1 of the space (interpolate along the geodesic in the space defined by the inherent geometry) which automatically results in the conjugate prior along with its hyperparameters. Our analysis and the analysis of Agarwal et al. lead to the same result, but ours is much simpler and naturally extends to the cases where one wants to couple more than two models. One big advantage of our analysis is that unlike prior approaches [1], we need not know the expression and the hyperparameters of the prior in advance. They are automatically derived by the analysis. Our analysis only requires the inherent geometry of the models under consideration and the interpolation parameters. No explicit expression of the coupling prior is needed.

3 Exponential Family

In this section, we review the exponential family. The exponential family is a set of distributions, whose probability density function can be expressed in the following form:

1 In exponential family statistical manifold, inherent geometry is defined by the divergence function because it is the divergence function that induces the metric structure and connection of the manifold. Refer [2] for more details.

here φ(x) : Xm → Rd is a vector potentials or sufficient statistics and G(θ) is a normalization constant or logpartition function. With the potential functions φ(x) fixed, every θ induces a particular member p(x; θ) of the family. In our framework, we deal with exponential families that are regular and have the minimal representation[9].

One important property of the exponential family is the existence of conjugate priors. Given any member of the exponential family in (1), the conjugate prior is a distribution over its parameters with the following form:

here α and β are hyperparameters of the conjugate prior. Importantly, the function G(·) is the same between the exponential family member and its conjugate prior.

A second important property of exponential family member is that log-partition function G is convex and defined over the convex set Θ := {θ ∈ Rd : G(θ) < ∞}; and since it is convex over this set, it induces a Bregman divergence [3] 2 on the space Θ.

Another important property of the exponential family is the one-to-one mapping between the canonical parameters θ and the so-called "mean parameters" which we denote by μ. For each canonical parameter θ ∈ Θ, there exists a mean parameter μ, which belongs to the space M defined as:

It is easy to see that Θ and M are dual spaces, in the sense of Legendre (conjugate) duality because of the following relationship between the log-partition function G(θ) and the expected value of the sufficient statistics φ(x): ∇G(θ) = E(φ(x)) = μ. In Legendre duality, we know that two spaces Θ and M are dual of each other if for each θ ∈ Θ, ∇G(θ) = μ ∈ M. We call the function in the dual space M to be F i.e., F = G∗ . A pictorial representation of the duality between canonical parameter space Θ and mean parameter space M is given in Figure 2.

2 Two important points to note about Bregman divergence are: 1) For dual spaces F and G, BF (p‖q) = BG(q∗ ‖p∗ ), where p∗ and q∗ are the conjugate duals of p and q respectively. 2) Bregman divergence is not symmetric i.e., in general, BF (p‖q) ̸= BF (q‖p), therefore it is important what directions these divergences are measured in.

In our analysis, we will need the Bregman divergence over φ(x) which can be obtained by showing that an augmented M contains all possible φ(x). In order to define the Bregman divergence over all φ(x), we define a new set of mean parameters w.r.t. all probability distributions (not only w.r.t. exponential family distributions): M+ := {μ ∈ Rd : μ = ∫ φ(x)p(x) dx s.t. ∫ p(x) dx = 1}.

Note that M+ is the convex hull of φ(x) thus contains all φ(x). We know from (see Theorem 3.3, [10]) that M is the interior of M+. Now we augment M with the boundary of M+ and Θ with the canonical parameters (limiting distributions) that will generate the mean parameters corresponding to this boundary. We know (see Theorem 2, [9]) that such parameters exist. Call these new sets M+ and Θ+ respectively. We also know [9] that Θ+ and M+ are conjugate dual of each other (for boundary, duality exists in the limiting sense) i.e., Bregman divergence is defined over the entire M+.

In the following discussion, M and Θ will denote the closed sets i.e. M+ and Θ+ respectively.

4 Likelihood, Prior and Geometry

In this section, we first formulate the ML problem into a Bregman median problem (Section 4.1) and then show that corresponding MAP (maximum-a-posteriori) problem can also be converted into a Bregman median problem (Section 4.3). The MAP Bregman median problem consists of two parts: a likelihood model and a prior. We argue (Sections 4.3 and 4.4) that a Bregman median problem makes sense only when both of these parts have the same geometry. Having the same geometry amounts to having the same log-partition function leading to the property of conjugate priors.

4.1 Likelihood in the form of Bregman Divergence

Following [5], we can write the distributions belonging to the exponential family (1) in terms of Bregman divergence 3 :

This representation of likelihood in the form of Bregman divergence gives insight in the geometry of the likelihood function. Gaining the insight into the exponential family distributions, and establishing a meaningful relationship between likelihood and prior is the primary objective of this work.

In learning problems, one is interested in estimating the parameters θ of the model which results in low generalization error. Perhaps the most standard estimation method is maximum likelihood (ML). The ML estimate, ˆ θML, of a set of n i.i.d. training data points X = {x1, . . . xn} drawn from the exponential family is obtained by solving the following problem: ˆ θML = maxθ∈Θ log p(X; θ) = maxθ∈Θ ∑ni=1 log p(xi; θ).

Theorem 1. Let X be a set of n i.i.d. training data points drawn from the exponential family distribution with the log partition function G, F be the dual function of G, then dual of ML estimate ( ˆ θML) of X under the assumed exponential family model solves the following Bregman median problem: ˆμML = minμ∈M ∑ni=1 BF (xi‖μ).

3 For the simplicity of the notations we will use x instead of φ(x) assuming that x ∈ Rd . This does not change the analysis

Proof. Proof is straightforward. Using (4) in MLE problem maxθ∈Θ ∑ni=1 log p(xi; θ), and ignoring terms that do not depend on θ:

which using the expression ∇G(θ) = μ gives the desired result.

The above theorem converts the problem of maximizing the log likelihood log p(X; θ) into an equivalent problem of minimizing the corresponding Bregman divergences which is nothing but a Bregman median problem, the solution to which is given by ˆμML = ∑ni=1 xi. ML estimate ˆ θML can now be computed using the expression ∇G(θ) = μ, ˆ θML = (∇G)-1 (ˆμML).

Lemma 1. If x is the sufficient statistics of the exponential family with the log partition function G, and F is the dual function of G defined over the mean parameter space M then (1) x ∈ M; (2) there exists a θ ∈ Θ, such that x∗ = θ.

Proof. (1) By construction of M, we know x ∈M. (2) From duality of M and Θ, for every μ ∈ M, there exists a θ ∈ Θ such that θ = μ∗ , and since x ∈ M, which implies x∗ = θ.

Corollary 1 (ML as Bregman Median). Let G and X be defined as earlier, θi be the dual of xi, then ML estimation, ˆ θML of X solves the following optimization problem:

Proof. Proof directly follows from Lemma 1 and Theorem 1. From Lemma 1, we know that x∗i = θi. Using Theorem 1 and expression BF (xi‖μ) = BG(θ‖x∗i ) = BG(θ‖θi) gives the desired result.

The above expression requires us to find a θ so that divergence from θ to other θi is minimized. Now note that G is what defines this divergence and hence the geometry of the Θ space (as discussed earlier in Section 2). since G is the log partition function of an exponential family, it is the log-partition function that determines the geometry of the space. We emphasize that divergence is measured from the parameter being estimated to other parameters θi(s), as shown in Figure 3.

4.2 Conjugate Prior in the form of Bregman Divergence

We now give an expression similar to the likelihood for the conjugate prior (ignoring the term log m(α, β)):

which can be written in the form of Bregman divergence by a direct comparison to (1), replacing x with α/β.

The expression for the joint probability of data and parameters (combining all terms that do not depend on θ in const) is given by:

4.3 Geometric Interpretation of Conjugate Prior

In this section we give a geometric interpretation of the term BF (x‖μ) + βBF ( αβ ‖μ) from (9).

Theorem 2 (MAP as Bregman median). Given a set X of n i.i.d examples drawn from the exponential family distribution with the log partition function G and a conjugate prior as in (8), MAP estimation of parameters is ˆ θMAP = ˆμ∗MAP where ˆμMAP solves the following problem:

which admits the following solution: ˆμMAP = ∑ni=1 xi+α n+β .

Proof. Proof is a direct result of applying the definition of Bregman divergence on (9) for all n points.

The above solution gives a natural interpretation of MAP estimation. One can think of prior as β number of extra points at position α/β. β works as the effective sample size of the prior which is clear from the following expression of the dual of the ˆ θMAP :

The expression (10) is analogous to (5) in the sense that both are defined in the dual space M. One can convert (10) into an expression similar to (6) in the dual space which is again a Bregman median problem in the parameter space.

here ( αβ )∗ ∈ Θ is dual of αβ . The above problem is a Bregman median problem of n + β points, {θ1, . . . θn, (α/β)∗ , . . . , (α/β)∗ ︸ ︷︷ ︸ β times }, as shown in Figure 3 (left).

A geometric interpretation is also shown in Figure 3. When the prior is conjugate to the likelihood, they both have the same log-partition function (Figure 3, left). Therefore they induce the same Bregman divergence. Having the same divergence means that distances from θ to θi (in likelihood) and the distances from θ to (α/β)∗ are measured with the same divergence function, yielding the same geometry for both spaces.

It is easier to see using the median formulation of the MAP estimation problem that one must choose a prior that is conjugate. If one chooses a conjugate prior, then the distances among all points are measured using the same function. It is also clear from (11) that in the conjugate prior case, the point induced by the conjugate prior behaves as a sample point (α/β)∗ . A median problem over a space that have different geometries is an ill-formed problem, as discussed further in the next section.

4.4 Geometric Interpretation of Non-conjugate Prior

We derived expression (12) because we considered the prior conjugate to the likelihood function. Had we chosen a nonconjugate prior with log-partition function Q, we would have obtained:

Here G and Q are different functions defined over Θ. Since these are the functions that define the geometry of the space parameter, having G ̸= Q is equivalent to consider them as being defined over different (metric) spaces. Here, it should be noted that distance between the sample point (θi) and the parameter θ is measured using the Bregman divergence BG. On the other hand, the distance between the point induced by the prior (α/β)∗ and θ is measured using the divergence function BQ. This means that (α/β)∗ can not be treated as one of the sample points. This tells us that, unlike the conjugate case, belief in the non-conjugate prior can not be encoded in the form of the sample points.

Another problem with considering a non-conjugate prior is that the dual space of Θ under different functions would be different. Thus, one will not be able to find the alternate expression for (13) equivalent to (10), and therefore not be able to find the closed-form expression similar to (11). This tells us why non-conjugate does not give us a closed form solution for ˆ θMAP . A pictorial representation of this is also shown in Figure 3. Note that, unlike the conjugate case, in the non-conjugate case, the data likelihood and the prior both belong to different spaces. We emphasize that it is possible to find the solution of (13) that is, in practice, there is nothing that prohibits the use of non-conjugate prior, however, using the conjugate prior is intuitive, and allows one to treat the hyper-parameters as pseudo data points.

5 Information Geometric View

In this section, we show the appropriateness of the conjugate prior from the information geometric angle. In information geometry, Θ is a statistical manifold such that each θ ∈ Θ defines a probability distribution. This statistical manifold has an inherent geometry, given by a metric and an affine connection. One natural metric is the Fisher information metric because of its many attractive properties: it is Riemannian and is invariant under reparameterization (for more details refer [2]).

In exponential family distributions, the Fisher metric M(θ) is induced by the KL-divergence KL(·‖θ), which is equivalent to the Bregman divergence defined by the log-partition function. Thus, it is the log-partition function G that induces the Fisher metric, and therefore determines the natural geometry of the space. It justifies our earlier argument of choosing the log-partition function to define the geometry. Now if we were to treat the prior as a point on the statistical manifold defined by the likelihood model, the Fisher information metric on the point given by the prior must be same as the one defined on likelihood manifold. This means that the prior must have the same log-partition function as the likelihood i.e., it must be conjugate.

6 Hybrid model

In this section, we show an application of our analysis to a common supervised and semi-supervised learning framework. In particular, we consider a generative/discriminative hybrid model [1; 6; 7] that has been shown to be successful in many application. The hybrid model is a mixture of discriminative and generative models, each of which has its own separate set of parameters. These two sets of parameters (hence two models) are combined using a prior called the coupling prior. Let p(y|x, θd) be the discriminative component, p(x, y|θg) be the generative component and p(θd, θg) be the coupling prior, the joint likelihood of the data and parameters can be written as (combining all three):

The most important aspect of this model is the coupling prior p(θg, θd), which interpolates the hybrid model between two extremes: fully generative when the prior forces θd = θg, and fully discriminative when the prior renders θd and θg independent. In non-extreme cases, the goal of the coupling prior is to encourage the generative model and the discriminative model to have similar parameters. It is easy to see that this effect can be induced by many functions. One obvious way is to linearly interpolate them as done by [7; 6] using a Gaussian prior (or the Euclidean distance) of the following form:p(θg, θd) ∝ exp (-λ ||θg - θd||2 ) (15)

where, when λ = 0, model is purely discriminative while for λ = ∞, model is purely generative. Thus λ in the above expression is the interpolating parameter, and is same as the γ in Section 2. Note that the log of the prior is nothing but the squared Euclidean distance between two sets of parameters.

It has been noted multiple times [4; 1] that a Gaussian prior is not always appropriate, and the prior should instead be chosen according to models being considered. Agarwal et al. [1] suggested using a prior that is conjugate to the generative model. Their main argument for choosing the conjugate prior came from the fact that this provides a closed form solution for the generative parameters and therefore is mathematically convenient. We will show that it is more than convenience that makes conjugate prior appropriate. Moreover, our analysis does not assume anything about the expression and the hyperparameters of the prior beforehand, rather derive them automatically.

6.1 Generalized HybridModel

In order to see the effect of the geometry, we present the discriminative and generative models associated with the hybrid model in the Bregman divergence form and obtain their geometry. Following the expression used in [1], the generative model can be written as:

where T(·) is the potential function similar to φ in (1), now only defined on (x, y). Let G∗ be the dual function of G; the corresponding Bregman divergence is given by BG∗ ((x, y)‖∇G(θg)). Solving the generative model independently reduces to choosing a θg from the space of all generative parameters Θg which has a geometry defined by the log-partition function G. Similarly to the generative model, the exponential form of the discriminative model is given as:

Importantly, the sufficient statistics T are the same in the generative and discriminative models; such generative/discriminative pairs occur naturally: logistic regression/naive Bayes and hidden Markov models/conditional random fields are examples. However, observe that in the discriminative case, the log partition function M depends on both x and θd which makes the analysis of the discriminative model (and hence of hybrid model) harder.

6.2 Geometry of the HybridModel

We simplify the analysis of the hybrid model by rewriting the discriminative model in a a form that makes its underlying geometry obvious. Note that the only difference between the two models is that discriminative model models the conditional distribution while the generative model models the joint distribution. We can use this observation to write the discriminative model in the following alternate form using the expression p(y|x, θ) = p(y,x|θ) ∑y′ p(y′ x|θ) and (16):

Denote the space of parameters of the discriminative model by Θd. It is easy to see that geometry of Θd is defined by G since function G is defined over θd. This is same as the geometry of the parameter space of the generative model Θg. Now let us define a new space ΘH which is the affine combination of Θd and Θg. Now, ΘH will have the same geometry as Θd and Θg i.e., geometry defined by G. Now the goal of the hybrid model is to find a θ ∈ ΘH that maximizes the likelihood of the data under the hybrid model. These two spaces are shown pictorially in Figure 4.

6.3 Prior Selection

As mentioned earlier, the coupling prior is the most important part of the hybrid model, which controls the amount of coupling between the generative and discriminative models. There are many ways to do this, one of which is given by [7; 6]. By their choice of Gaussian prior as coupling prior, they implicitly couple the discriminative and generative parameters by the squared Euclidean distance. We suggest coupling these two models by a general prior, of which the Gaussian prior is a special case.

Bregman Divergence and Coupling Prior:

Let a general coupling be given by BS(θg‖θd). Notice the direction of the divergence. We have chosen this direction because the prior is induced on the generative parameters, and it is clear from (12) that parameters on which prior is induced, are placed in the first argument in the divergence function. The direction of the divergence is also shown in Figure 4.

Now we rewrite (8) replacing ∇G(θ) by θ∗ :

Now taking the α = λθ∗ d and β = λ, we get:

For the general coupling divergence function BS(θg‖θd), the corresponding coupling prior is given by:

The above relationship between the divergence function (left side of the expression) and coupling prior (right side of the expression) allows one to define a Bregman divergence for a given coupling prior and vise versa.

Coupling Prior for the Hybrid Model:

We now use (21) to derive the expression for the coupling prior using the geometry of the hybrid model which is given by the log partition function G of the generative model. This argument suggests to couple the hybrid model by the divergence BG(θg‖θd) which gives the coupling prior as:

where λ = [0,∞] is the interpolation parameter, interpolating between the discriminative and generative extremes. In dual form, the above expression can be written as:

Here exp(-λG(θd)) can be thought of as a prior on the discriminative parameters p(θd). In the above expression, exp(-λBG(θg‖θd)) = p(θg|θg)p(θd) behaves as a joint coupling prior P (θd, θg) as originally expected in the model (14). Note that hyperparameters of the prior α and β are naturally derived from the geometric view of the conjugate prior. Here α = λθ∗ d and β = λ.

Relation with Agarwal et al.:

The prior we derived in the previous section turns out to be the exactly same as that proposed by Agarwal et al. [1], even though theirs was not formally justified. In that work, the authors break the coupled prior p(θg, θd) into two parts: p(θd)

and p(θg|θd). They then derive an expression for the p(θg|θd) based on the intuition that the mode of p(θg|θd) should be θd. Our analysis takes a different approach by coupling two models with the Bregman divergence rather than prior, and results in the expression and hyperparameters for the prior same as in [1].

7 RelatedWork and Conclusion

To our knowledge, there have been no previous attempts to understand Bayesian priors from a geometric perspective. One related piece of work [8] uses the Bayesian framework to find the best prior for a given distribution. It is noted that, in that work, the authors use the δ-geometry for the data space and the α-geometry for the prior space, and then show the different cases for different values (δ, α). We emphasize that even though it is possible to use different geometry for the both spaces, it always makes more sense to use the same geometry. As mentioned in remark 1 in [8], useful cases are obtained only when we consider the same geometry.

We have shown that by considering the geometry induced by a likelihood function, the natural prior that results is exactly the conjugate prior. We have used this geometric understanding of conjugate prior to derive the coupling prior for the discriminative/generative hybrid model. Our derivation naturally gives us the expression and the hyperparameters of this coupling prior.