{"type":"ScholarlyArticle","title":"Meta-Learning via PAC-Bayesian with Data-Dependent Prior: Generalization Bounds from Local Entropy","authors":[{"name":"Shiyu Liu","slug":"shiyu-liu","given":"Shiyu","family":"Liu","affiliation":"University of Electronic Science and Technology of China; Peng Cheng Laboratory"},{"name":"Wei Shi","slug":"wei-shi","given":"Wei","family":"Shi","affiliation":"University of Electronic Science and Technology of China"},{"name":"Zenglin Xu","slug":"zenglin-xu","given":"Zenglin","family":"Xu","affiliation":"Artificial Intelligence Innovation and Incubation (AI; Peng Cheng Laboratory; Harbin Institute of Technology , Shenzhen"},{"name":"Shaogao Lv","slug":"shaogao-lv","given":"Shaogao","family":"Lv","affiliation":") Institute , Fudan University; Nanjing audit university"},{"name":"Yehong Zhang","slug":"yehong-zhang","given":"Yehong","family":"Zhang","affiliation":"Peng Cheng Laboratory"},{"name":"Hui Wang","slug":"hui-wang","given":"Hui","family":"Wang","affiliation":"Peng Cheng Laboratory"}],"abstract":"Meta-learning accelerates the learning process on unseen learning tasks by acquiring prior knowledge through previous related tasks. The PAC-Bayesian theory provides a theoretical framework to analyze the generalization of meta-learning to unseen tasks. However, previous works still encounter two notable limitations: (1) they merely focus on the data-free priors, which often result in inappropriate regularization and loose generalization bounds; (2) more importantly, their optimization process usually involves nested optimization problems, incurring significant computational costs. To address these issues, we derive new generalization bounds and introduce a novel PAC-Bayesian framework for meta-learning that integrates data-dependent priors. This framework enables the extraction of optimal posteriors for each task in closed form, thereby allowing us to minimize generalization bounds incorporated data-dependent priors with only a simple local entropy. The resulting algorithm, which employs SGLD for sampling from the optimal posteriors, is stable, efficient, and computationally lightweight, eliminating the need for nested optimization. Extensive experimental results demonstrate that our proposed method outperforms the other baselines.","doi":"10.24963/ijcai.2024/506","url":"https://ijcai.org/proceedings/2024/506","pdf":"https://ijcai.org/proceedings/2024/0506.pdf","markdown":"https://ijcai.org/proceedings/2024/506.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/506","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":506,"pages":{"first":4578,"last":4586},"keywords":["Machine Learning: ML: Meta-learning","Machine Learning: ML: Bayesian learning"],"videos":[],"references":[{"position":1,"raw":"[Amit and Meir, 2018] Ron Amit and Ron Meir. Meta- learning by adjusting priors based on extended pac-bayes theory. In International Conference on Machine Learning, 2018.","title":"Ron Amit and Ron Meir. Metalearning by adjusting priors based on extended pac-bayes theory","authors":"Meir Amit","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baxter, 2000] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 2000.","title":"A model of inductive bias learning","authors":"Jonathan Baxter","year":"2000","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Catoni, 2007] Olivier Catoni. Pac-bayesian supervised clas- sification: the thermodynamics of statistical learning. arXiv preprint arXiv:0712.0248, 2007.","title":"Olivier Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning","authors":"Catoni","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chaudhari et al., 2017] Pratik Chaudhari, Anna Choroman- ska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In International Conference on Learning Repre- sentations, 2017.","title":"Entropy-sgd: Biasing gradient descent into wide valleys","authors":"Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann Lecun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chaudhari et al., 2019] Pratik Chaudhari, Anna Choroman- ska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Ex- periment, 2019.","title":"Entropy-sgd: Biasing gradient descent into wide valleys","authors":"Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann Lecun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina","year":"2019","venue":"Journal of Statistical Mechanics: Theory and Experiment","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Chen, 2022] Lisha Chen and Tianyi Chen. Is bayesian model-agnostic meta learning better than model- agnostic meta learning, provably? In International Confer- ence on Artificial Intelligence and Statistics, 2022.","title":"Is bayesian model-agnostic meta learning better than modelagnostic meta learning, provably","authors":"Lisha Chen, Tianyi Chen","year":"2022","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dwork et al., 2015] Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity in adaptive data anal- ysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, 2015.","title":"Preserving statistical validity in adaptive data analysis","authors":"Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Leon, Roth","year":"2015","venue":"Proceedings of the forty-seventh annual ACM symposium on Theory of computing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dziugaite and Roy, 2018a] Gintare Karolina Dziugaite and Daniel Roy. Entropy-sgd optimizes the prior of a pac-bayes bound: Generalization properties of entropy-sgd and data- dependent priors. In International Conference on Machine Learning, 2018.","title":"Entropy-sgd optimizes the prior of a pac-bayes bound: Generalization properties of entropy-sgd and datadependent priors","authors":"Roy ; Dziugaite, Karolina Gintare, Daniel Dziugaite, Roy","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dziugaite and Roy, 2018b] Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac-bayes priors via differ- ential privacy. In Advances in neural information process- ing systems, 2018.","title":"Data-dependent pac-bayes priors via differential privacy","authors":"Roy ; Dziugaite, Karolina Gintare, Daniel M Dziugaite, Roy","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dziugaite et al., 2021] Gintare Karolina Dziugaite, Kyle Hsu, Waseem Gharbieh, Gabriel Arpino, and Daniel Roy. On the role of data in pac-bayes bounds. In International Conference on Artificial Intelligence and Statistics, 2021.","title":"On the role of data in pac-bayes bounds","authors":"Karolina Gintare, Kyle Dziugaite, Waseem Hsu, Gabriel Gharbieh, Daniel Arpino, Roy","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Finn et al., 2018] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in neural information processing systems, 2018.","title":"Probabilistic model-agnostic meta-learning","authors":"Finn","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guan and Lu, 2022] Jiechao Guan and Zhiwu Lu. Fast-rate pac-bayesian generalization bounds for meta-learning. In International Conference on Machine Learning, 2022.","title":"Fast-rate pac-bayesian generalization bounds for meta-learning","authors":"Lu ; ] Jiechao Guan, Zhiwu Guan, Lu","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Guan et al., 2022] Jiechao Guan, Zhiwu Lu, and Yong Liu. Improved generalization risk bounds for meta-learning with pac-bayes-kl analysis. 2022.","title":"Improved generalization risk bounds for meta-learning with pac-bayes-kl analysis","authors":"Guan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hospedales et al., 2021] Timothy Hospedales, Antreas An- toniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2021.","title":"Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence","authors":"Hospedales","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2022] Hung-yi Lee, Shang-Wen Li, and Ngoc Thang Vu. Meta learning for natural language processing: A survey. arXiv preprint arXiv:2205.01500, 2022.","title":"Meta learning for natural language processing: A survey","authors":"Lee","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lever et al., 2013] Guy Lever, Franc ¸ois Laviolette, and John Shawe-Taylor. Tighter pac-bayes bounds through distribution-dependent priors. Theoretical Computer Sci- ence, 2013.","title":"Tighter pac-bayes bounds through distribution-dependent priors","authors":"Lever","year":"2013","venue":"Theoretical Computer Sci","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu and Wang, 2016] Qiang Liu and Dilin Wang. Stein vari- ational gradient descent: A general purpose bayesian infer- ence algorithm. In Advances in neural information process- ing systems, 2016.","title":"Stein variational gradient descent: A general purpose bayesian inference algorithm","authors":"Qiang Liu, Dilin Wang","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2021] Tianyu Liu, Jie Lu, Zheng Yan, and Guangquan Zhang. Statistical generalization performance guarantee for meta-learning with data dependent prior. Neu- rocomputing, 2021.","title":"Statistical generalization performance guarantee for meta-learning with data dependent prior","authors":"Tianyu Liu, Jie Lu, Zheng Yan, Guangquan Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Madden, 2004] Samuel Madden. Intel lab data. http://db. csail.mit.edu/labdata/labdata.html, 2004. Accessed: Sep 8, 2020.","title":"Samuel Madden. Intel lab data","authors":"Madden","year":"2004","venue":null,"doi":null,"url":"http://db.csail.mit.edu/labdata/labdata.html,2004","internal_url":null},{"position":22,"raw":"[McAllester, 1999] David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual Conference on Computational learning theory, 1999.","title":"Pac-bayesian model averaging","authors":"A David, Mcallester","year":"1999","venue":"Proceedings of the twelfth annual Conference on Computational learning theory","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Milne et al., 2017] Christopher J Milne, Thomas Schietinger, Masamitsu Aiba, Arturo Alarcon, Jürgen Alex, Alexander Anghel, Vladimir Arsov, Carl Beard, Paul Beaud, Simona Bettoni, et al. Swissfel: the swiss x-ray free electron laser. Applied Sciences, 2017.","title":"Swissfel: the swiss x-ray free electron laser","authors":"Thomas Christopher J Milne, Masamitsu Schietinger, Arturo Aiba, Jürgen Alarcon, Alexander Alex, Vladimir Anghel, Carl Arsov, Paul Beard, Simona Beaud, Bettoni","year":"2017","venue":"Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nichol et al., 2018] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.","title":"On first-order meta-learning algorithms","authors":"Alex Nichol, Joshua Achiam, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pentina and Lampert, 2014] Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for life- long learning. In International Conference on Machine Learning, 2014.","title":"Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for lifelong learning","authors":"Lampert Pentina","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pittorino et al., 2021] Fabrizio Pittorino, Carlo Lucibello, Christoph Feinauer, Gabriele Perugini, Carlo Baldassi, Elizaveta Demyanenko, and Riccardo Zecchina. Entropic gradient descent algorithms and wide flat minima. Journal of Statistical Mechanics: Theory and Experiment, 2021.","title":"Entropic gradient descent algorithms and wide flat minima","authors":"Fabrizio Pittorino, Carlo Lucibello, Christoph Feinauer, Gabriele Perugini, Carlo Baldassi, Elizaveta Demyanenko, Riccardo Zecchina","year":"2021","venue":"Journal of Statistical Mechanics: Theory and Experiment","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Raginsky et al., 2017] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory, 2017.","title":"Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis","authors":"Raginsky","year":"2017","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rezazadeh, 2022] Arezou Rezazadeh. A unified view on pac-bayes bounds for meta-learning. In International Con- ference on Machine Learning, 2022.","title":"A unified view on pac-bayes bounds for meta-learning","authors":"Arezou Rezazadeh","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rivasplata et al., 2020] Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesvári, and John Shawe-Taylor. Pac-bayes anal- ysis beyond the usual bounds. In Advances in Neural Infor- mation Processing Systems, 2020.","title":"Pac-bayes analysis beyond the usual bounds","authors":"Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesvári, John Shawe-Taylor","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rothfuss et al., 2019] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Prox- imal meta-policy search. In International Conference on Machine Learning, 2019.","title":"Promp: Proximal meta-policy search","authors":"Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, Pieter Abbeel","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rothfuss et al., 2021] Jonas Rothfuss, Vincent Fortuin, Mar- tin Josifoski, and Andreas Krause. Pacoh: Bayes-optimal meta-learning with pac-guarantees. In International Con- ference on Machine Learning, 2021.","title":"Pacoh: Bayes-optimal meta-learning with pac-guarantees","authors":"Rothfuss","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Rothfuss et al., 2023] Jonas Rothfuss, Martin Josifoski, Vin- cent Fortuin, and Andreas Krause. Scalable pac-bayesian meta-learning via the pac-optimal hyper-posterior: From theory to practice. arXiv preprint arXiv:2211.07206, 2023.","title":"Scalable pac-bayesian meta-learning via the pac-optimal hyper-posterior: From theory to practice","authors":"Rothfuss","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Silva et al., 2012] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in- hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In Computing in Cardiology, 2012.","title":"Predicting inhospital mortality of icu patients: The physionet/computing in cardiology challenge","authors":"Ikaro Silva, George Moody, J Daniel, Leo A Scott, Roger G Celi, Mark","year":"2012","venue":"Computing in Cardiology","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Thiemann et al., 2017] Niklas Thiemann, Christian Igel, Olivier Wintenberger, and Yevgeny Seldin. A strongly quasiconvex pac-bayesian bound. In International Confer- ence on Algorithmic Learning Theory, 2017.","title":"A strongly quasiconvex pac-bayesian bound","authors":"Thiemann","year":"2017","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Thrun and Pratt, 1998] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, 1998.","title":"Learning to learn: Introduction and overview","authors":"Pratt Thrun, Sebastian Thrun, Lorien Pratt","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Welling and Teh, 2011] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynam- ics. In International conference on machine learning, 2011.","title":"Bayesian learning via stochastic gradient langevin dynamics","authors":"Teh ; ] Max Welling, Yee W Welling, Teh","year":"2011","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xu et al., 2020] Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh. Metafun: Meta- learning with iterative functional updates. In International Conference on Machine Learning, 2020.","title":"Metafun: Metalearning with iterative functional updates","authors":"Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam Kosiorek, Yee Whye Teh","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yoon et al., 2018] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in neural information processing systems, 2018.","title":"Bayesian model-agnostic meta-learning","authors":"Yoon","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhao et al., 2022] Mandi Zhao, Pieter Abbeel, and Stephen James. On the effectiveness of fine-tuning versus meta- reinforcement learning. In Advances in Neural Information Processing Systems, 2022.","title":"On the effectiveness of fine-tuning versus metareinforcement learning","authors":"Mandi Zhao, Pieter Abbeel, Stephen James","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhu et al., 2023] Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.","title":"Transfer learning in deep reinforcement learning: A survey","authors":"Zhu","year":"2023","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}