Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities

Yu, Shujian; Sanchez Giraldo, Luis; Principe, Jose

doi:10.24963/ijcai.2021/633

Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities

Shujian Yu, Luis Sanchez Giraldo, Jose Principe

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Survey Track. Pages 4669-4678.

https://doi.org/10.24963/ijcai.2021/633

Keywords

Machine learning: General

Abstract

We present a review on the recent advances and emerging opportunities around the theme of analyzing deep neural networks (DNNs) with information-theoretic methods. We first discuss popular information-theoretic quantities and their estimators. We then introduce recent developments on information-theoretic learning principles (e.g., loss functions, regularizers and objectives) and their parameterization with DNNs. We finally briefly review current usages of information-theoretic concepts in a few modern machine learning problems and list a few emerging opportunities.

Shujian Yu, Luis Sanchez Giraldo, and Jose Principe. 2021. Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 4669–4678. https://doi.org/10.24963/ijcai.2021/633

@inproceedings{ijcai2021p633,
  title     = {Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities},
  author    = {Yu, Shujian and Sanchez Giraldo, Luis and Principe, Jose},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {4669--4678},
  year      = {2021},
  month     = {8},
  note      = {Survey Track},
  doi       = {10.24963/ijcai.2021/633},
  url       = {https://doi.org/10.24963/ijcai.2021/633},
}

TY  - CPAPER
TI  - Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities
AU  - Yu, Shujian
AU  - Sanchez Giraldo, Luis
AU  - Principe, Jose
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 4669
EP  - 4678
DO  - 10.24963/ijcai.2021/633
UR  - https://doi.org/10.24963/ijcai.2021/633
ER  -