Generalization Error of Linear Neural Networks in an Empirical Bayes Approach

Nakajima, Shinichi; Watanabe, Sumio

Generalization Error of Linear Neural Networks in an Empirical Bayes Approach

Download BibTeX Cite

Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence. Pages 804-809.

Abstract

It is well known that in unidentifiable models, the Bayes estimation has the advantage of generalization performance to the maximum likelihood estimation. However, accurate approximation of the posterior distribution requires huge computational costs. In this paper, we consider an empirical Bayes approach where a part of the parameters are regarded as hyperparameters, which we call a subspace Bayes approach, and theoretically analyze the generalization error of three-layer linear neural networks. We show that a subspace Bayes approach is asymptotically equivalent to a positivepart James-Stein type shrinkage estimation, and behaves similarly to the Bayes estimation in typical cases.

Shinichi Nakajima, and Sumio Watanabe. 2005. Generalization Error of Linear Neural Networks in an Empirical Bayes Approach. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, IJCAI-05. IJCAI, 804–809. https://ijcai.org/proceedings/2005/79

@inproceedings{ijcai2005p79,
  title     = {Generalization Error of Linear Neural Networks in an Empirical Bayes Approach},
  author    = {Nakajima, Shinichi and Watanabe, Sumio},
  booktitle = {Proceedings of the Nineteenth International Joint Conference on
               Artificial Intelligence, {IJCAI-05}},
  publisher = {International Joint Conferences on Artificial Intelligence},
  pages     = {804--809},
  year      = {2005},
  url       = {https://ijcai.org/proceedings/2005/79},
}

TY  - CPAPER
TI  - Generalization Error of Linear Neural Networks in an Empirical Bayes Approach
AU  - Nakajima, Shinichi
AU  - Watanabe, Sumio
T2  - Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, IJCAI-05
PB  - International Joint Conferences on Artificial Intelligence
PY  - 2005
SP  - 804
EP  - 809
UR  - https://ijcai.org/proceedings/2005/79
ER  -