{"type":"ScholarlyArticle","title":"Towards Exact Computation of Inductive Bias","authors":[{"name":"Akhilan Boopathy","slug":"akhilan-boopathy","given":"Akhilan","family":"Boopathy","affiliation":"Massachusetts Institute of Technology"},{"name":"William Yue","slug":"william-yue","given":"William","family":"Yue","affiliation":"Massachusetts Institute of Technology"},{"name":"Jaedong Hwang","slug":"jaedong-hwang","given":"Jaedong","family":"Hwang","affiliation":"Massachusetts Institute of Technology"},{"name":"Abhiram Iyer","slug":"abhiram-iyer","given":"Abhiram","family":"Iyer","affiliation":"Massachusetts Institute of Technology"},{"name":"Ila Fiete","slug":"ila-fiete","given":"Ila","family":"Fiete","affiliation":"Massachusetts Institute of Technology"}],"abstract":"Much research in machine learning involves finding appropriate inductive biases (e.g. convolutional neural networks, momentum-based optimizers, transformers) to promote generalization on tasks. However, quantification of the amount of inductive bias associated with these architectures and hyperparameters has been limited. We propose a novel method for efficiently computing the inductive bias required for generalization on a task with a fixed training data budget; formally, this corresponds to the amount of information required to specify well-generalizing models within a specific hypothesis space of models. Our approach involves modeling the loss distribution of random hypotheses drawn from a hypothesis space to estimate the required inductive bias for a task relative to these hypotheses. Unlike prior work, our method provides a direct estimate of inductive bias without using bounds and is applicable to diverse hypothesis spaces. Moreover, we derive approximation error bounds for our estimation approach in terms of the number of sampled hypotheses. Consistent with prior results, our empirical results demonstrate that higher dimensional tasks require greater inductive bias. We show that relative to other expressive model classes, neural networks as a model class encode large amounts of inductive bias. Furthermore, our measure quantifies the relative difference in inductive bias between different neural network architectures. Our proposed inductive bias metric provides an information-theoretic interpretation of the benefits of specific model architectures for certain tasks and provides a quantitative guide to developing tasks requiring greater inductive bias, thereby encouraging the development of more powerful inductive biases.","doi":"10.24963/ijcai.2024/413","url":"https://ijcai.org/proceedings/2024/413","pdf":"https://ijcai.org/proceedings/2024/0413.pdf","markdown":"https://ijcai.org/proceedings/2024/413.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/413","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":413,"pages":{"first":3733,"last":3741},"keywords":["Machine Learning: ML: Other","Machine Learning: ML: Explainable/Interpretable machine learning","Machine Learning: ML: Learning theory","Machine Learning: ML: Evaluation"],"videos":[],"references":[{"position":1,"raw":"[Amari, 1993] Shunichi Amari. A universal theorem on learning curves. Neural Networks, 6(2):161-166, 1993.","title":"A universal theorem on learning curves","authors":"Shunichi Amari","year":"1993","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahri et al., 2021] Yasaman Bahri, Ethan Dyer, Jared Ka- plan, Jaehoon Lee, and Utkarsh Sharma. Explaining neu- ral scaling laws. arXiv preprint arXiv:2102.06701, 2021.","title":"Explaining neural scaling laws","authors":"Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, Utkarsh Sharma","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blumer et al., 1989] Anselm Blumer, A. Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Learnabil- ity and the vapnik-chervonenkis dimension. J. ACM, 36(4):929-965, oct 1989.","title":"Learnability and the vapnik-chervonenkis dimension","authors":"Anselm Blumer, A Ehrenfeucht, David Haussler, Manfred K Warmuth","year":"1989","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boopathy et al., 2023] Akhilan Boopathy, Kevin Liu, Jae- dong Hwang, Shu Ge, Asaad Mohammedsaleh, and Ila Fi- ete. Model-agnostic measure of generalization difficulty. ICML, 2023.","title":"Model-agnostic measure of generalization difficulty","authors":"Boopathy","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chollet, 2019] Franc ¸ois Chollet. On the measure of intelli- gence. arXiv preprint, 2019.","title":"Franc ¸ois Chollet. On the measure of intelligence","authors":"Chollet","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ciregan et al., 2012] Dan Ciregan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for im- age classification. In CVPR, 2012.","title":"Multi-column deep neural networks for image classification","authors":"Dan Ciregan, Ueli Meier, Jürgen Schmidhuber","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cortes et al., 1994] Corinna Cortes, L. D. Jackel, Sara Solla, Vladimir Vapnik, and John Denker. Learning curves: Asymptotic values and rate of convergence. In NeurIPS, 1994.","title":"Learning curves: Asymptotic values and rate of convergence","authors":"Corinna Cortes, L D Jackel, Sara Solla, Vladimir Vapnik, John Denker","year":"1994","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.","title":"ImageNet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du et al., 2018] Simon S. Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Aarti Singh. How many samples are needed to estimate a con- volutional or recurrent neural network? NeurIPS, 2018.","title":"How many samples are needed to estimate a convolutional or recurrent neural network? NeurIPS","authors":"Simon S Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Ruslan Salakhutdinov, Aarti Singh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135. PMLR, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Florian, 2007] Razvan V Florian. Correct equations for the dynamics of the cart-pole system. Center for Cognitive and Neural Studies (Coneural), Romania, 2007.","title":"Correct equations for the dynamics of the cart-pole system","authors":"Florian Razvan","year":"2007","venue":"Center for Cognitive and Neural Studies","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goodfellow et al., 2013] Ian J Goodfellow, Yaroslav Bula- tov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi- digit number recognition from street view imagery us- ing deep convolutional neural networks. arXiv preprint arXiv:1312.6082, 2013.","title":"Multidigit number recognition from street view imagery using deep convolutional neural networks","authors":"Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, Vinay Shet","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Haussler, 1988] David Haussler. Quantifying inductive bias: Ai learning algorithms and valiant's learning frame- work. Artificial Intelligence, 36(2):177-221, 1988.","title":"Quantifying inductive bias: Ai learning algorithms and valiant's learning framework","authors":"David Haussler","year":"1988","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hernández-Orallo, 2016] José Hernández-Orallo. Evalua- tion in artificial intelligence: from task-oriented to ability- oriented measurement. Artificial Intelligence Review, Aug 2016.","title":"José Hernández-Orallo. Evaluation in artificial intelligence: from task-oriented to abilityoriented measurement","authors":"Hernández-Orallo","year":"2016","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hestness et al., 2017] Joel Hestness, Sharan Narang, New- sha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kia- ninejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint, 2017.","title":"Deep learning scaling is predictable, empirically","authors":"Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa, Ali Patwary, Yang Yang, Yanqi Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hutter, 2021] Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021.","title":"Learning curve theory","authors":"Marcus Hutter","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jiang et al., 2021] Yiding Jiang, Parth Natekar, Manik Sharma, Sumukh K Aithal, Dhruva Kashyap, Natara- jan Subramanyam, Carlos Lassance, Daniel M Roy, Gintare Karolina Dziugaite, Suriya Gunasekar, et al. Methods and analysis of the first competition in predicting generalization of deep learning. In NeurIPS 2020 Compe- tition and Demonstration Track, 2021.","title":"Methods and analysis of the first competition in predicting generalization of deep learning","authors":"Jiang","year":"2021","venue":"NeurIPS 2020 Competition and Demonstration Track","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kawaguchi et al., 2022] Kenji Kawaguchi, Zhun Deng, Kyle Luh, and Jiaoyang Huang. Robustness implies gen- eralization via data-dependent generalization bounds. In ICML, 2022.","title":"Robustness implies generalization via data-dependent generalization bounds","authors":"Kenji Kawaguchi, Zhun Deng, Kyle Luh, Jiaoyang Huang","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Koltchinskii and Panchenko, 2000] Vladimir Koltchinskii and Dmitriy Panchenko. Rademacher processes and bounding the risk of function learning. High Dimensional Probability II, page 443-457, 2000.","title":"Vladimir Koltchinskii and Dmitriy Panchenko. Rademacher processes and bounding the risk of function learning","authors":"Panchenko Koltchinskii","year":"2000","venue":"High Dimensional Probability II","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets. 2009.","title":"Cifar-10 and cifar-100 datasets","authors":"Alex Krizhevsky, Vinod Nair, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lake et al., 2015] B. Lake, R. Salakhutdinov, and J. Tenen- baum. Human-level concept learning through probabilistic program induction. Science, 350:1332 -1338, 2015.","title":"Human-level concept learning through probabilistic program induction","authors":"B Lake, R Salakhutdinov, J Tenenbaum","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lecun et al., 1998] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Lee and Choi, 2018] Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise met- ric and subspace. In ICML, pages 2927-2936. PMLR, 2018.","title":"Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace","authors":"Choi Lee","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lee et al., 2015] Chen-Yu Lee, Saining Xie, Patrick Gal- lagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. In AISTATS, 2015.","title":"Deeplysupervised nets","authors":"Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu","year":"2015","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lei et al., 2015] Yunwen Lei, Urun Dogan, Alexander Binder, and Marius Kloft. Multi-class svms: From tighter data-dependent generalization bounds to novel algorithms. NeurIPS, 2015.","title":"Multi-class svms: From tighter data-dependent generalization bounds to novel algorithms","authors":"Lei","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Li et al., 2018] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. NeurIPS, 2018.","title":"Visualizing the loss landscape of neural nets","authors":"Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Li et al., 2021] Zhiyuan Li, Yi Zhang, and Sanjeev Arora. Why are convolutional nets more sample-efficient than fully-connected nets? ICLR, 2021.","title":"Why are convolutional nets more sample-efficient than fully-connected nets? ICLR","authors":"Zhiyuan Li, Yi Zhang, Sanjeev Arora","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Lin et al., 2015] Zhouhan Lin, Roland Memisevic, and Kishore Konda. How far can we go without convolu- tion: Improving fully-connected networks. arXiv preprint, 2015.","title":"How far can we go without convolution: Improving fully-connected networks","authors":"Lin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Lin et al., 2023] Jihao Andreas Lin, Javier Antorán, Shreyas Padhy, David Janz, José Miguel Hernández- Lobato, and Alexander Terenin. Sampling from gaussian process posteriors using stochastic gradient descent. arXiv preprint, 2023.","title":"Sampling from gaussian process posteriors using stochastic gradient descent","authors":"Lin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Mauch and Yang, 2017] Lukas Mauch and Bin Yang. A novel layerwise pruning method for model reduction of fully connected deep neural networks. In ICASSP, 2017.","title":"A novel layerwise pruning method for model reduction of fully connected deep neural networks","authors":"Yang ; Mauch, Lukas Mauch, Bin Yang","year":"2017","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Mhaskar et al., 2017] Hrushikesh Mhaskar, Qianli Liao, and Tomaso Poggio. When and why are deep networks better than shallow ones? In AAAI, 2017.","title":"When and why are deep networks better than shallow ones?","authors":"Mhaskar","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Mrgrhn, 2021] Guido Mrgrhn. Alexnet with tensorflow, 2021.","title":"Alexnet with tensorflow","authors":"Guido Mrgrhn","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Murata et al., 1992] Noboru Murata, Shuji Yoshizawa, and Shun-ichi Amari. Learning curves, model selection and complexity of neural networks. In NeurIPS, 1992.","title":"Learning curves, model selection and complexity of neural networks","authors":"Noboru Murata, Shuji Yoshizawa, Shun-Ichi Amari","year":"1992","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Negrea et al., 2019] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. NeurIPS, 2019.","title":"Information-theoretic generalization bounds for sgld via data-dependent estimates","authors":"Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, Daniel M Roy","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Nishimoto, 2018] Bruno Eidi Nishimoto. Linear classifier, 2018.","title":"Linear classifier","authors":"Bruno Eidi, Nishimoto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Raginsky et al., 2016] Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu. Information-theoretic analysis of stability and bias of learning algorithms. In IEEE Information Theory Workshop (ITW), 2016.","title":"Information-theoretic analysis of stability and bias of learning algorithms","authors":"Raginsky","year":"2016","venue":"IEEE Information Theory Workshop","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Rajeswaran et al., 2019] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. NeurIPS, 32, 2019.","title":"Meta-learning with implicit gradients","authors":"Aravind Rajeswaran, Chelsea Finn, Sergey Sham M Kakade, Levine","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Sankaran, 1959] Munuswamy Sankaran. On the non-central chi-square distribution. Biometrika, 46(1/2):235-237, 1959.","title":"Munuswamy Sankaran. On the non-central chi-square distribution","authors":"Sankaran","year":"1959","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Sharma and Kaplan, 2022] Utkarsh Sharma and Jared Ka- plan. Scaling laws from the data manifold dimension. JMLR, 23:9-1, 2022.","title":"Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension","authors":"Kaplan Sharma","year":"2022","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurIPS, 30, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Veeramacheneni et al., 2022] Lokesh Veeramacheneni, Moritz Wolter, Reinhard Klein, and Jochen Garcke. Canonical convolutional neural networks. arXiv preprint arXiv:2206.01509, 2022.","title":"Canonical convolutional neural networks","authors":"Lokesh Veeramacheneni, Moritz Wolter, Reinhard Klein, Jochen Garcke","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurIPS, 29, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wolpert, 1996] David H Wolpert. The lack of a priori dis- tinctions between learning algorithms. Neural computa- tion, 8(7):1341-1390, 1996.","title":"The lack of a priori distinctions between learning algorithms","authors":"H David, Wolpert","year":"1996","venue":"Neural computation","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}