Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities

1 Introduction

Information-theoretic methods have become the workhorse of several impressive deep learning achievements over the past years, ranging from practical applications (e.g., the variational information bottleneck in representation learning [Alemi et al., 2017]) to theoretical investigations (e.g., the generalization bound induced by mutual information [Xu and Raginsky, 2017; Steinke and Zakynthinou, 2020]). Further, a few novel information-theoretic quantities (e.g., mutual information) estimators and learning principles have been developed and applied to different deep learning problems in fruitful ways. A recent example is the mutual information neural estimator (MINE) [Belghazi et al., 2018] and its application in representation learning [Hjelm et al., 2018] with the renowned information maximization (InfoMax) principle [Linsker, 1988].

Information theory requires the knowledge of the data probability density function (PDF), and in machine learning this is normally unknown. Therefore, the application of information theory to machine learning is predicated to the selection of PDF estimators or mathematical alternatives to compute from data the statistical quantities of entropy, mutual information or divergence. The theory of information-theoretic estimators is vast [Pardo, 2018; Gao et al., 2018], so it is often hard for practitioners to make a quick decision on suitable estimators. On the other hand, the rapid development of computer vision and natural language processing applications is likely to drive our attention to a particular class of estimators, neglecting other possible choices that may enjoy other favorable properties. Such restrictions will, in turn, impose constraints on the performance limit of those applications.

In this work, we first discuss popular information-theoretic quantities (i.e., entropy, mutual information and divergence) and their estimators, aiming at illustrating their inner connections and specific properties. We then introduce common information-theoretic learning principles (e.g., InfoMax and the information bottleneck (IB) approach [Tishby et al., 1999]) and their practical usages in the understanding and the design of DNNs. We finally demonstrate how information theory can be brought to bear on several challenging deep learning problems in unorthodox and fruitful ways. To conclude this survey, we provide a list of future directions that we consider, have potential to move the field forward.

2 Information-Theoretic Quantities and Estimators

Information-theoretic quantities provide useful descriptions of the underlying behavior of random variable (or process) and that this behavior is a key factor in developing and analyzing deep models. Shannon entropy and relative entropy (a.k.a., Kullback-Leibler or KL divergence) evidence a long track record of usefulness in information theory and machine learning. However, there is no reason to restrict ourselves to Shannon's measures of entropy and relative entropy as alternative quantities may have other properties advantageous for machine learning [Kapur, 1994].

Hence, we first introduce definitions and estimators of popular information-theoretic quantities. For conciseness, we focus our discussion on continuous random variables for which it is assumed the probability density function exits.

2.1 Shannon's Entropy, Mutual Information and KL divergence

For a continuous random variable X with probability density function (PDF) f(x) and support X, Shannon's differential entropy H(X) is given by:

Similarly, the joint entropy for a pair of random variables (X, Y ), with joint PDF f(x, y), is defined as:

A fundamental measure of dependence (or correlation) between two random variables is the mutual information (MI):

MI is symmetric and reduces to zero iff X and Y are statistically independent. Another important problem in information theory is measuring the dissimilarity between two probability distributions (or in our case PDFs) f(x) and g(x). The first principled approach that is based on the probability measure is the KL divergence (a.k.a., relative entropy), defined as:

MI can be expressed as the KL divergence between the joint distribution f(x, y) and the product of marginals f(x)f(y):

A natural way to estimate MI from samples {xi, yi}ni=1 is the plug-in approach that uses a non-parametric density estimator such as kernel density estimator (KDE) [Parzen, 1962] to approximate f(x, y), f(x) and f(y), and then uses the estimated densities to compute MI. However, density estimation is notoriously hard in high-dimensional space. Therefore, most of the commonly employed estimators for MI are based on nearest neighbour and graph properties [Kraskov et al., 2004]. One of the most recent proposals of this kind is ensemble dependency graph estimator (EDGE) [Noshad et al., 2019]. Despite fast convergence rate and low computational complexity of EDGE, one downside of all estimators based on neighborhoods or graphs is their lack of differentiabilty, and thus cannot be used for gradient-based optimization that is so prevalent in DNNs.

In an effort to devise estimators that scale to present-day machine learning problems, most recent work on estimating MI has focused on variational lower bounds that can be parameterized, for instance using neural networks [Belghazi et al., 2018]. These approaches do solve the differentiability issue; however, theoretical results have shown that such high confidence estimators based on the lower bound on MI require a sample size that is exponential in the MI of the data, making reliable estimation impractical in high entropy, high MI scenarios [McAllester and Stratos, 2020].

2.2 R´enyi's α-order Entropy and Related Quantities

A widely used generalization of Shannon entropy is R´enyi's α-entropy. For a continuous random variable X with PDF f(x) and support X, the α-entropy Hα(X) is defined as:

A big difference with respect to Shannon entropy is the interchange of the "log" with the integral, which has enormous implications for estimation [Principe, 2010]. R´enyi also extended the concept to divergence. The α-relative entropy) between random variables with PDFs f and g is given by:

In limit case where α → 1, both (6) and (7) reduce to Shannon's entropy and KL divergence, respectively. There are multiple ways to generalize MI in a similar way, as first done by R´enyi [R´enyi, 1961]. One of the most notable generalizations on α-MI was suggested by Arimoto [Arimoto, 1977]. For simplicity, one can also define the α-MI as the α-relative entropy between f(x, y) and f(x)f(y) [P´al et al., 2010], i.e.,

k-NN graph-based estimators have been proposed for α-order generalizations (i.e., Eqs. (6)-(8)) [Leonenko et al., 2008]. However, the consistency proof of these estimators (e.g., [P´al et al., 2010; P´oczos and Schneider, 2011]) are usually restricted to a narrow range of α ∈ (0, 1). Additionally, [Singh and P´oczos, 2014] established a rate of convergence for the simple KDE estimator of α-divergence.

A modified version of R´enyi's definition of α-relative entropy is given by Lutwak [Lutwak et al., 2005]:

For α = 2, expressions for entropy of f and relative entropy (Lutwak's definition) between f and g can be obtained as functions of inner products between PDFs:

Eq. (11) is also called the Cauchy-Schwarz (CS) divergence [Jenssen et al., 2006] and is defined based on the wellknown CS inequality:

Given Eq. (11), the quadratic mutual information (QMI) between X and Y can be defined as the CS divergence between f(x, y) and f(x)f(y):

Alternatively, one can use the Euclidean distance DED(f‖g) = ∫X (f(x) - g(x))2dx to define QMI as:

Note that definitions in Eq. (13) and Eq. (14) are not equivalent. The case of α = 2 is of special interest as it allows simple closed form expressions for KDE-based estimators [Principe, 2010, Chapter 2.10].

2.3 Matrix-based Entropy Functional

Information-theoretic quantities can be defined (or measured) over the eigenspectrum of symmetric positive semidefinite (SPS) matrix, avoiding the necessity of estimating the underlying density distributions of variables. For example, it is reasonable that the eigenvalues of sample covariance matrices of two collections of samples manifest their distributional divergence [Moskvina and Zhigljavsky, 2003]. More formally, given a strictly convex, differentiable function ϕ : S d+ → R that maps SPS matrices (of size d × d) to a real number, the Bregman matrix divergence from matrix ρ to matrix σ is defined as [Tsuda et al., 2005]:

where tr(A) denotes the trace of matrix A. Examples include ϕ(σ) = ‖σ‖2F , which leads to the squared Frobenius norm ‖σ - ρ‖2F .

When ϕ(σ) = tr(σ log σ - σ), where log σ is the matrix logarithm, the resulting Bregman matrix divergence is:

which is also referred to von Neumann divergence or quantum relative entropy [Nielsen and Chuang, 2010]. Another important matrix divergence arises by taking ϕ(σ) =

-logdet σ, in which the resulting Bregman matrix divergence reduces to:

and is commonly called the LogDet divergence.

Given two sets of observations SX = {xi}ni=1 ∼ f and SY = {yi}mi=1 ∼ g, the divergence from f to g can be simply measured by Dϕ,B on their respective sample covariance matrices Σx and Σy with Dϕ,B(Σx‖Σy). Taking the chain rule of KL divergence, Dϕ,B can also be used to quantify the divergence of two conditional distributions (i.e., D(f(y|x)‖g(y|x))) [Yu et al., 2020b].

Similarly, one is able to non-parametrically obtain measures of entropy and MI on the eigenspectrum of a Gram matrix of the projected data in a reproducing kernel Hilbert space (RKHS) [Sanchez Giraldo et al., 2014]. Specifically, the matrix-based R´enyi's α-order entropy for samples {xi}ni=1 (each xi can be a real-valued scalar or vector) is defined over the eigenspectrum of their normalized Gram matrix A of size n×n as follows:

where tr(Aα) = ∑ni=1 λi(A)α, andA = K/ tr(K) in which K = κ(xi, xj) is sample Gram matrix 1 and tr denotes matrix trace. λi(A) denotes the i-th eigenvalue of A.

Let B be another (normalized) Gram matrix from {yi}ni=1 , the joint entropy between X and Y is defined as:

where A ◦ B denotes the Hadamard product between A and B. Given Eqs. (18) and (19), one can define the matrix-based R´enyi's α-order MI, in analogy to Shannon MI, as:

and the matrix-based R´enyi's α-order conditional entropy,

1κ : χ × χ ↦→ R is a real valued positive definite kernel that is also infinitely divisible [Bhatia, 2006]. Usually, a Gaussian kernel is chosen [Sanchez Giraldo et al., 2014; Yu et al., 2019].

2.4 Other Popular Divergence and Dependence Measures

Apart from the above-mentioned divergences and their matrix-based counterparts, Maximum Mean Discrepancy (MMD) [Gretton et al., 2012] and Wasserstein distance or optimal transport (OT) are perhaps the most two popular classes of distances in DNNs. MMD quantifies distances between distributions as the distance between mean embeddings of features. Specifically, given distributions f and g over X, the MMD is defined by a feature map ϕ : X ↦→ H, where H is also called a reproducing kernel Hilbert space (RKHS). More formally, the MMD is defined as:

An unbiased U-statistic estimator for MMD on observations SX = {xi}ni=1 and SY = {yi}mi=1 is given by:

where Hij = κ(xi, xj ) + κ(yi, yj ) - κ(xi, yj ) - κ(yi, xj ). On the other hand, let Π(f, g) denotes the set of joint distributions π whose marginal distributions are f and g, the Wasserstein distance is defined as:

In recent years, alternative dependence measures, based on MMD and OT between probability joint and product of marginal distributions, have been proposed. Examples include the Hilbert Schmidt Independence Criterion (HSIC) [Gretton et al., 2007] based MMD and the Wasserstein's dependence measure (WDM) [Ozair et al., 2019]. Empirical results show that WDM overcomes to some extent the issue of measuring dependencies when MI is large. Nevertheless, the estimator of WDM requires optimizing the parameters of a Lipchitz-continuous neural network, a condition that remains challenging to be enforced in practice. A very recent development, is the concept of usable information under computational constraints [Xu et al., 2020]. In this work, it is assumed that limiting the expressiveness of the quantity describing information might be beneficial not only in terms of estimation but also in the context of learning. A characteristic feature of these measures of mutual information and mutual information-like quantities, including the matrix-based entropy and mutual information, is the incorporation of some inductive bias in the estimation process, which can be advantageous for machine learning [Tschannen et al., 2020].

Another popular variant of MI, also called the squared-loss MI (SMI) [Suzuki et al., 2009], defines the dependence of two random variables X and Y as the Pearson's χ2 divergence from f(x, y) to f(x)f(y):

SMI is usually estimated by least-square approaches. A recent proposal is SMI with OT [Liu et al., 2019].

3 Information-Theoretic Principles and Regularizations in DNNs

3.1 Information-Theoretic Loss Functions for Robust Deep Learning

Information-theoretic measures can be directly used as a loss function to train DNNs. The most notable example is the cross-entropy loss for classification. Another example is the aforementioned CS divergence, which has been used, as a loss function, in both deep clustering [Kampffmeyer et al., 2019] and classification [Janocha and Czarnecki, 2017]. In terms of clustering, one could model each cluster by its PDF and optimize cluster assignments such that the divergence between their PDFs is maximized. On the other hand, the fdivergence (a generalization of KL divergence) can be used to train deep energy-based models [Yu et al., 2020a].

However, recent research has demonstrated that DNNs trained with cross-entropy or mean squared error (MSE) can easily fit (corrupted or even randomly) labeled data [Zhang et al., 2017; Jiang et al., 2018; Feng et al., 2020], but with very poor generalization capacity on held out data, indicating that loss functions like cross-entropy or MSE are not a reliable indicator of generalization and robustness.

To improve the learning performance in non-Gaussian noises and outliers, a variety of (information-theoretic) nonMSE criteria have been proposed in the literature. Particularly in recent years, the maximum correntropy criterion (MCC) have found many successful applications in domains of machine learning and signal processing, which is very useful for the case where the signals are contaminated by heavytailed impulsive noises [Chen et al., 2019]. Under the MCC, an optimal model can be obtained by maximizing the correntropy between the desired output (e.g., class label) y and the estimated output ˆ y [Liu et al., 2007]:

where f ∗ is the optimal model, F stands for models hypothesis space, e = y - ˆ y is the prediction residual, E refers to mathematical expectation, and Vσ(ˆ y, y) = E(Gσ(e)) denotes the correntropy between ˆ y and y [Liu et al., 2007], with Gσ(e) being the Gaussian kernel function with width σ.

Compared to MSE (i.e., argminE(e2 )), MCC is robust to outliers because the correntropy criterion is closely related to M-estimators [Mandanas and Kotropoulos, 2016]. A popular alternative to MCC is the minimum error entropy (MEE) criterion argminH(e), in which H(e) refers to the entropy of prediction residual e [Erdogmus and Principe, 2002]. The robustness of MEE over MSE is elaborated in [Chen et al., 2016] and the references herein. As a replacement of MSE, both MCC and MEE have demonstrated improved robustness to non-Gaussian noises and outliers, when they are been deployed as a loss function to train DNNs [Qi et al., 2014; Yu et al., 2021a]. A most recent proposal along this line of research is the Determinant based Mutual Information (DMI) [Xu et al., 2019], with a loss of - log[DMI(ˆ y, y)].

Information-Theoretic measures also demonstrate great potential towards robust deep learning under domain shift [Qui˜nonero-Candela et al., 2009], i.e., the joint distribution of samples (x, y) in the training (or source) data Psource(x, y) is different to that in the test (or target) data Ptarget(x, y). Taking the covariate shift (i.e., Ptarget(y|x) = Psource(y|x), but Ptarget(x) ̸= Psource(x)) as an example, [Greenfeld and Shalit, 2020] suggests that a DNN is robust to covariate shift iff the distribution of the prediction residuals y - f(x) is statistically independent of the distribution of the input x. The authors use HSIC to measure the degree of independence. [Yu et al., 2021a] replaces HSIC with the matrix-based mutual information (i.e., Eq. (20)) and obtained favorable performance improvement. Note that, such independence criterion (for training regression models) was initially developed to identify causal direction among two variables. Feasible independence measures include HSIC [Mooij et al., 2009] and least-square mutual information [Yamada et al., 2014]. Interestingly, the independence criterion is also closely related to the aforementioned MEE [Yu et al., 2021a], which provides new insight on its robustness.

3.2 InfoMax Principle and its Practical Usage

The information maximization (InfoMax) is well studied in statistics as exemplified by Burg's power spectrum algorithm [Burg, 1974]. For machine learning, the InfoMax principle dates back to [Linsker, 1988], which pointed out that a linear or nonlinear network can be treated as an information channel. A principle to self-organize such networks is to transfer as much information as possible of given data through the network. The InfoMax principle has recently been used in many computer vision and natural language processing studies on self-supervised representation learning, with the objective of maximizing the MI I(X; T) between input X and latent representation T [Oord et al., 2018; Hjelm et al., 2018; Kong et al., 2020]. Since MI is computational intractable in high-dimensional space, most of existing studies turn to optimize a lower bound of MI. However, recent studies show that InfoMax principle may introduce excessive and noisy information, which could be adversarial. The IB principle, as will be introduced later, can mitigate this issue [Wang et al., 2021; Mahabadi et al., 2021].

Alongside the representation learning, the InfoMax principle also offers an appealing strategy to train DNNs in a layerwise manner without backpropagation. Traditionally, to learn a feedforward DNN in a supervised setting, one needs to train all components (or modules) of the network simultaneously using backpropagation (BP) since there is no explicit target for each hidden layer. By contrast, the InfoMax principle can help us to implement layer-wise training of DNNs. But to train a DNN, one needs to incorporate information about the desired output Y as well. Linsker's idea can be extended with the MI formalism to include not only the input information but also other sources of information which are usually characterized by desired outputs. From this perspective, different layers can be trained greedily [Hu and Principe, 2021]: the first hidden layer can be trained such that the MI I(Y ; T1) between the output of this layer (denote T1) and the desired output Y is maximized; then the second hidden layer can be trained in the same way by maximizing I(Y ; T2). This procedure is repeated until the last hidden layer (see Fig. 1(a)). Interested readers can refer to [Duan et al., 2020] for a comprehensive discussion on this topic.

(a) Information channel in InfoMax (b) Markov chain in deep IB

Figure 1: Given input X and desired output Y , denote T their latent representation. InfoMax can be implemented explicitly via maxI(X; T) or implicitly with maxI(Y ; T(X)). By contrast, deep IB approaches search T via max I(Y ; T ) - βI(X; T ).

3.3 Information Bottleneck (IB) in Deep Learning

The IB principle was proposed in [Tishby et al., 1999] as an information-theoretic framework for learning. It considers extracting information about a target signal Y through a correlated observable X. The extracted information is quantified by a variable T , which is (a possibly randomized) function of X, thus forming the Markov chain Y ↔ X ↔ T. Suppose we know the joint distribution p(X, Y ), the objective is to learn a representation T that maximizes its predictive power to Y subject to some constraints on the amount of information that it carries about X:

where I(·; ·) denotes the mutual information. β is a Lagrange multiplier that controls the trade-off between the sufficiency (the performance on the task, as quantified by I(Y ; T)) and the minimality (the complexity of the representation, as measured by I(X; T)). In this sense, the IB principle also provides a natural approximation of minimal sufficient statistic.

The IB principle has both practical and theoretical impacts to DNNs. Practically, it can be formulated as a learning objective (or loss function) for deep models. When parameterizing IB with a DNN, X denotes input variable, Y denotes the desired output (e.g., class labels), T refers to the latent representation of one hidden layer. Usually, this was done by optimizing the IB Lagrangian (i.e., Eq. (27)) via a classic cross-entropy loss (which amounts to maxI(Y ; T) [Achille and Soatto, 2018; Amjad and Geiger, 2019]) regularized by a differentiable mutual information term I(X; T). Depends on implementation details, I(X; T ) can be measured by variational approximation [Alemi et al., 2017; Kolchinsky et al., 2019], MINE [Elad et al., 2019] and the matrix-based entropy functional [Yu et al., 2021b]. According to [Kolchinsky et al., 2019], the IB curve in classification scenario is piecewise linear and becomes a flat line at I(Y ; T) = H(Y ) for I(X; T) ≥ H(Y ). We obtain both theoretical and empirical IB curve by training a three layer MLP with 256 units in the bottleneck layer on MNIST dataset, as shown in Fig. 2(a). Empirically, the IB objective was observed to improve model generation performance and robustness to adversarial attack.

Theoretically, it was argued that, even though the IB objective is not explicitly optimized, DNNs trained with crossentropy loss and stochastic gradient descent (SGD) inherently solve the IB compression-prediction trade-off [Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017]. The authors also posed the information plane (IP), i.e., the trajectory in R2 of the mutual information pair {I(X; T ), I(Y ; T )} across training epochs, as a lens to analyze dynamics of learning of DNNs (see Fig. 2(b)). According to [Shwartz-Ziv and Tishby, 2017], there are two training phases in the common SGD optimization: an early "fitting" phase, in which both I(X; T ) and I(T ; Y ) increase rapidly, and a later "compression" phase, in which there is a reversal such that I(X; T) continually decreases. This work attracted significant attention, culminating in many follow-up works that tested the proclaimed narrative and its accompanying empirical observations. So far, the "fitting-and-compression" phenomena of the layered representation T have been observed in other types of DNNs, including the multilayer perceptrons (e.g., [Chelombiev et al., 2019; Shwartz-Ziv and Tishby, 2017]), the AEs (e.g., [Yu and Principe, 2019]), and the CNNs (e.g., [Noshad et al., 2019; Yu et al., 2020c]). However, the IB theory is still a controversial issue, and different MI estimators may lead to different behaviors of curves in IP. We have to remember that not all the properties of the statistical definition of mutual information are transferred to the estimators [Yu and Principe, 2019]. We recommend interested readers to [Goldfeld and Polyanskiy, 2020; Zaidi et al., 2020] for comprehensive surveys on IB approach for deep learning.

3.4 Other Principles and Applications

There are other popular information-theoretic principles that have demonstrated great potential in their respective application fields. For example, in reinforcement learning, the maximum entropy regularization encourages exploration and avoids getting stuck in a local optima [Haarnoja et al., 2017; Ahmed et al., 2019]; the Stratonovich's value of information criterion [Stratonovich, 1965] can help agent to strike a balance between exploration and exploitation [Sledge and Pr´ıncipe, 2017]. Here, we additionally introduce two other learning principles that are relevant to InfoMax and IB.

CorEx and Representation Learning

The Cor-relation Ex-planation (CorEx) [Steeg and Galstyan, 2014] is an information-theoretic principle for learning representations that are maximally informative about the data. Specifically, let x = (x1; x2; · · · ; xd) be a d-dimensional random variable with PDF p(x), a measure of total dependence amongst each dimension of x is defined as (which is also known as total correlation [Watanabe, 1960]):

Let z be the latent variable we want to infer from x. The total dependence of x, after conditioning on z, becomes TC(x|z) = ∑di=1 H(xi|z) - H(x|z). A measure of informativeness of z about the dependence among the observed variables x can then be quantified as how total correlation is reduced after conditioning on z, i.e.,

TC(x; z) corresponds to the amount of dependence (in x) that is explained by z. Obviously, TC(x; z) is maximized (or TC(x|z) is zero) iff the conditional distribution p(x|z) factorizes, in which case we can interpret z as capturing the information about common causes across all xi. The CorEx principle is thus formulated as [Steeg and Galstyan, 2014]:

Recently, [Gao et al., 2019] constructed a variational lower bound of CorEx and optimized the bound with DNNs. Interestingly, the resulting objective (under mild assumptions) is the same to the evidence lower bound (ELBO) in variational autoencoder (VAE) [Kingma and Welling, 2014].

Principle of Relevant Information

The fixed point update underlying the IB can de extended to a single random variable X, as demonstrated in the Principle of Relevant Information (PRI) [Principe, 2010, Chapter 8]. PRI is an unsupervised principle that aims to perform mode decomposition of a random variable X with a known and fixed probability distribution g. Suppose we aim to obtain (from X) a reduced statistical representation characterized by another random variable Z with probability distribution f .

The PRI casts this problem as a trade-off between the entropy H(f) of Z and its descriptive power about X in terms of their divergence D(f‖g):

where β is a hyper-parameter controlling the amount of relevant information that Z can extract from X. The minimization of entropy can be viewed as a means of reducing uncertainty (or redundancy) and finding the statistical regularities in the outcomes, whereas the minimization of information divergence ensures that such regularities are closely related to X. The PRI is similar in spirit to the IB approach, but the formulation is different because PRI does not require a relevant auxiliary variable Y and the optimization is done directly on the random variable X (rather than the joint distribution p(X, Y ) as in the IB).

The choice of entropy and divergence estimators is application-specific and depends on the simplicity of optimization. An interesting scenario arises when we use the R´enyi's 2-order entropy (i.e., Eq. (10)) and the CS divergence (i.e., Eq. (11)), in which Eq. (31) has an elegant expression and provides multiscale representations of X controlled by increasing β, yielding clustering, principal curves (Fig. 3(b)) or surfaces (Fig. 3(c)), vector quantization and X itself in the limiting case (β → ∞). Recent applications of PRI include the extraction of spectral-spatial features for hyperspectral image classification [Wei et al., 2019] and the undersampling for imbalanced data classification [Hoyos-Osorio et al., 2021]. The authors feel that PRI provides a solid mathematical foundation for data reduction using information theory.

4 Applications and Emerging Opportunities of Information Theory in DNNs

Information-theoretic measures (such as f-divergence, MMD, mutual information and HSIC) and principles (such as minimizing variational representations of f-divergence and maximizing an "evidence lower bound") have been extensively investigated to the design and the understanding of mainstream generative models [Goodfellow et al., 2014; Kingma and Welling, 2014], with fruitful applications in both computer vision and natural language processing. Here, we additionally introduce two other emerging opportunities.

4.1 Understanding Deep Neural Networks

Information Theory is always a promising way to analyze and understand DNN behaviors in either training phase or decision process. Apart from the aforementioned IB theory that is still under debate (see Section 3.3), information-theoretic methods can also be used to understand the generalization capacity and construct generalization bound for DNNs. Examples in this category include [Xu and Raginsky, 2017; Bu et al., 2020; Steinke and Zakynthinou, 2020]. On the other hand, information-theoretic measures can also quantify the importance (or redundancy) of individual neurons [Liu et al., 2018] or filters [Yu et al., 2020c], which in turn is beneficial to problems like pruning and generalization.

4.2 Learning withMultiple Tasks

A closely related topic to generalization is the joint learning of multiple tasks. Relevant problems include the multi-task learning (MTL) that aims to learn all tasks simultaneously and more proficiently than learning them independently; the meta-learning that aims to learn efficient learning model that can learn new tasks quickly; and the continual learning (CL), where the problem requires learning a sequence of tasks, avoiding negative transfer or catastrophic forgetting.

For all these three problems, information-theoretic tools have demonstrated great potential to strengthen theoretical discoveries and practical performance. For example, in CL, the elastic weight consolidation (EWC) [Kirkpatrick et al., 2017] introduces penalty constructed by Fisher information to force important parameters of the network to remain close to the parameters of the network trained for the previous tasks. In MTL, [Yu et al., 2020b] uses the Bregman-Correntropy (conditional) divergence to quantify the closeness of two tasks and penalize large model discrepancy for related tasks. In meta-learning, [Yin et al., 2019] proposes an informationtheoretic meta-regularizer to mitigate the memorization problem. In general, information theory is beneficial to construct generalization bound for these problems (e.g., the H∆H divergence for both classification [Ben-David et al., 2010] and regression [Cortes and Mohri, 2014]). These bounds can then be used to design different training or optimization algorithms. On the other hand, information theory also offers a new insight to analyze the fundamental trade-offs of existing algorithms [Vera et al., 2018].

5 Topics of Future Interests &Conclusions

The intersection between information theory and deep neural networks (DNNs) a challenging and promising area due to its rapidly increasing prevalence in real-world applications. It is challenging, because most of expressions are defined on probability density functions (PDFs) that are hard to measure for real data. It is also promising, due to the remarkable performance gain and rigorous mathematical foundation. In this paper, we provided a survey of common informationtheoretic estimators, learning principles and regularizations in DNNs, and recent developments of these estimators or principles in practical deep learning applications.

While there is now a significant body of work, there are still several open problems in applying information theory in DNNs. Some of these open problems - certainly not an exhaustive list - include the following.

• Information-theoreticmeasures beyond global PDF: The RKHS is a functional space that is very appropriate for statistical inference [Parzen, 1962], statistical embedding [Sriperumbudur et al., 2010], and statistical modeling. Our current work is extending these methodologies to model local regions of the space of samples, i.e. working beyond the PDF for higher specificity and sensitivity. We have recently applied ideas from quantum mechanics, the famous Schroedinger equation based on the Laplacian of the wave function (here the estimator of the PDF in RKHS) to estimate model and data uncertaintity. Preliminary results show that estimators of uncertainty derived from this approach supplant conventional techniques [Singh and Principe, 2020]. This line of research can lead to better methods for transfer learning, and even causality.
• Information theory beyond i.i.d. data: Most of existing information-theoretic estimators are limited to i.i.d. data, a property that many problems do not meet (e.g., blind source separation on audio data and change detection on stream data). Therefore, it is crucial to extend information-theoretic measures to deal with structured and interdependent observations, including graphs [Han et al., 2015; Wu et al., 2020].
• Fruitful and unorthodox AI applications: It is not hard to foresee more successful applications of informationtheoretic concepts on emerging AI topics. Taking the Explainable AI (XAI) as an example, information theory can help identify informative features to explain given example [Chen et al., 2018] or generate interpretable representation to explain a black-box model [O'Shaughnessy et al., 2020; Bang et al., 2019]. Meanwhile, there is no doubt that information theory will contribute causal inference and offering fresh insights to its relevant modern deep learning topics include generalization and robustness [Sch¨olkopf et al., 2021].