---
title: "Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities"
authors:
  - "Shujian Yu"
  - "Luis Sanchez Giraldo"
  - "Jose Principe"
doi: "10.24963/ijcai.2021/633"
url: "https://ijcai.org/proceedings/2021/633"
pdf: "https://ijcai.org/proceedings/2021/0633.pdf"
venue: "Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence"
year: 2021
pages: "4669-4678"
track: "Survey Track"
keywords:
  - "Machine learning: General"
---

# Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities

Shujian Yu, Luis Sanchez Giraldo, Jose Principe

*Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence*, pages 4669-4678. DOI: [10.24963/ijcai.2021/633](https://doi.org/10.24963/ijcai.2021/633)

## Abstract

We present a review on the recent advances and emerging opportunities around the theme of analyzing deep neural networks (DNNs) with information-theoretic methods. We first discuss popular information-theoretic quantities and their estimators. We then introduce recent developments on information-theoretic learning principles (e.g., loss functions, regularizers and objectives) and their parameterization with DNNs. We finally briefly review current usages of information-theoretic concepts in a few modern machine learning problems and list a few emerging opportunities.