{"type":"ScholarlyArticle","title":"Fixed-Budget Best-Arm Identification in Structured Bandits","authors":[{"name":"MohammadJavad Azizi","slug":"mohammadjavad-azizi","given":"MohammadJavad","family":"Azizi","affiliation":"University of Southern California"},{"name":"Branislav Kveton","slug":"branislav-kveton","given":"Branislav","family":"Kveton","affiliation":""},{"name":"Mohammad Ghavamzadeh","slug":"mohammad-ghavamzadeh","given":"Mohammad","family":"Ghavamzadeh","affiliation":"Google Research"}],"abstract":"Best-arm identification (BAI) in a fixed-budget setting is a bandit problem where the learning agent maximizes the probability of identifying the optimal (best) arm after a fixed number of observations. Most works on this topic study unstructured problems with a small number of arms, which limits their applicability. We propose a general tractable algorithm that incorporates the structure, by successively eliminating suboptimal arms based on their mean reward estimates from a joint generalization model. We analyze our algorithm in linear and generalized linear models (GLMs), and propose a practical implementation based on a G-optimal design. In linear models, our algorithm has competitive error guarantees to prior works and performs at least as well empirically. In GLMs, this is the first practical algorithm with analysis for fixed-budget BAI.","doi":"10.24963/ijcai.2022/388","url":"https://ijcai.org/proceedings/2022/388","pdf":"https://ijcai.org/proceedings/2022/0388.pdf","markdown":"https://ijcai.org/proceedings/2022/388.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/388","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":388,"pages":{"first":2798,"last":2804},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Active Learning","Machine Learning: Online Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986471","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38986977","length_sec":85,"provider":"slideslive"},{"label":"Video #3","url":"https://slideslive.com/38984614","length_sec":281,"provider":"slideslive"},{"label":"Video #4","url":"https://slideslive.com/38985120","length_sec":628,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abbasi-yadkori et al., 2011] Yasin. Abbasi-yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochas- tic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, D Pál, C Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu et al., 2017] Zeyuan Allen-Zhu, Yuanzhi Li, Aarti Singh, and Yining Wang. Near-optimal design of ex- periments via regret minimization. In International Con- ference on Machine Learning, pages 126-135, 2017.","title":"Near-optimal design of experiments via regret minimization","authors":"Allen-Zhu","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Audibert et al., 2010] Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best Arm Identification in Multi-Armed Bandits. In Proceedings of the 23th Conference on Learning Theory, 2010.","title":"Best Arm Identification in Multi-Armed Bandits","authors":"Audibert","year":"2010","venue":"Proceedings of the 23th Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berthet and Perchet, 2017] Quentin Berthet and Vianney Perchet. Fast rates for bandit optimization with upper- confidence frank-wolfe. In I.","title":"Quentin Berthet and Vianney Perchet. Fast rates for bandit optimization with upperconfidence frank-wolfe","authors":"Perchet Berthet","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 30.","title":"Advances in Neural Information Processing Systems","authors":"U V Guyon, S Luxburg, H Bengio, R Wallach","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck et al., 2009] Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In International conference on Algorithmic learning theory, pages 23-37, 2009.","title":"Pure exploration in multi-armed bandits problems","authors":"Sébastien Bubeck, Rémi Munos, Gilles Stoltz","year":"2009","venue":"International conference on Algorithmic learning theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Carpentier and Locatelli, 2016] Alexandra Carpentier and Andrea Locatelli. Tight (lower) bounds for the fixed bud- get best arm identification bandit problem. In Conference on Learning Theory, pages 590-604, 2016.","title":"Tight (lower) bounds for the fixed budget best arm identification bandit problem","authors":"Alexandra Carpentier, Andrea Locatelli","year":"2016","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Damla Ahipasaoglu et al., 2008] Selin Damla Ahipasaoglu, Peng Sun, and Michael J Todd. Linear convergence of a modified Frank-Wolfe algorithm for computing minimum-volume enclosing ellipsoids. Optimization Methods and Software, 23(1):5-19, 2008.","title":"Linear convergence of a modified Frank-Wolfe algorithm for computing minimum-volume enclosing ellipsoids","authors":"Selin Damla, Ahipasaoglu, Peng Sun, Michael J Todd","year":"2008","venue":"Optimization Methods and Software","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Danskin, 1966] John M. Danskin. The theory of max-min, with applications. SIAM Journal on Applied Mathematics, 14(4):641-664, 1966.","title":"The theory of max-min, with applications","authors":"John M Danskin","year":"1966","venue":"SIAM Journal on Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Degenne et al., 2019] Rémy Degenne, Wouter M Koolen, and Pierre Ménard. Non-asymptotic pure exploration by solving games. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, vol- ume 32.","title":"Non-asymptotic pure exploration by solving games","authors":"Rémy Degenne, Pierre Wouter M Koolen, Ménard","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Degenne et al., 2020] Rémy Degenne, Pierre Ménard, Xue- dong Shang, and Michal Valko. Gamification of pure ex- ploration for linear bandits. In International Conference on Machine Learning, pages 2432-2442, 2020.","title":"Gamification of pure exploration for linear bandits","authors":"Rémy Degenne, Pierre Ménard, Xuedong Shang, Michal Valko","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fedorov, 1972] Valerii Vadimovich Fedorov. Theory of Op- timal Experiments. Probability and Mathematical Statis- tics. Academic Press, 1972.","title":"Theory of Optimal Experiments","authors":"Valerii Vadimovich, Fedorov","year":"1972","venue":"Probability and Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Filippi et al., 2010] Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems, 2010.","title":"Parametric bandits: The generalized linear case","authors":"Sarah Filippi, Olivier Cappe, Aurélien Garivier, Csaba Szepesvári","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gabillon et al., 2012] Victor Gabillon, Mohammad Ghavamzadeh, and Alessandro Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. In Advances in Neural Information Processing Systems, pages 3212-3220, 2012.","title":"Best arm identification: A unified approach to fixed budget and fixed confidence","authors":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hoffman et al., 2014] Matthew Hoffman, Bobak Shahriari, and Nando Freitas. On correlation and budget constraints in model-based bandit optimization with application to au- tomatic machine learning. In Artificial Intelligence and Statistics, pages 365-374, 2014.","title":"On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning","authors":"Matthew Hoffman, Bobak Shahriari, Nando Freitas","year":"2014","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2021] Ruiquan Huang, Weiqiang Wu, Jing Yang, and Cong Shen. Federated linear contextual ban- dits. Advances in Neural Information Processing Systems, 34, 2021.","title":"Federated linear contextual bandits","authors":"Huang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jaggi, 2013] Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In Interna- tional Conference on Machine Learning, pages 427-435, 2013.","title":"Revisiting Frank-Wolfe: Projection-free sparse convex optimization","authors":"Martin Jaggi","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jedra and Proutiere, 2020] Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits. In Advances in Neural Information Processing Systems, pages 10007-10017, 2020.","title":"Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits","authors":"Proutiere Jedra","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Karnin et al., 2013] Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed ban- dits. In Proceedings of the 30th International Conference on International Conference on Machine Learning, page 1238-1246, 2013.","title":"Almost optimal exploration in multi-armed bandits","authors":"Karnin","year":"2013","venue":"Proceedings of the 30th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Katz-Samuels et al., 2020] Julian Katz-Samuels, Lalit Jain, Zohar Karnin, and Kevin Jamieson. An empirical pro- cess approach to the union bound: Practical algorithms for combinatorial and linear bandits. In Advances in Neu- ral Information Processing Systems, pages 10371-10382, 2020.","title":"An empirical process approach to the union bound: Practical algorithms for combinatorial and linear bandits","authors":"Julian Katz-Samuels, Lalit Jain, Zohar Karnin, Kevin Jamieson","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kaufmann et al., 2016] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. The Journal of Machine Learning Research, 17(1):1-42, 2016.","title":"On the complexity of best-arm identification in multi-armed bandit models","authors":"Emilie Kaufmann, Olivier Cappé, Aurélien Garivier","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Khachiyan, 1996] Leonid G Khachiyan. Rounding of poly- topes in the real number model of computation. Mathe- matics of Operations Research, 21(2):307-320, 1996.","title":"Rounding of polytopes in the real number model of computation","authors":"Leonid G Khachiyan","year":"1996","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kiefer and Wolfowitz, 1960] Jack Kiefer and Jacob Wol- fowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics, 12:363-366, 1960.","title":"Jack Kiefer and Jacob Wolfowitz. The equivalence of two extremum problems","authors":"Wolfowitz Kiefer","year":"1960","venue":"Canadian Journal of Mathematics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kumar and Yildirim, 2005] Piyush Kumar and E Alper Yildirim. Minimum-volume enclosing ellipsoids and core sets. Journal of Optimization Theory and Applications, 126(1):1-21, 2005. Kveton et al., 2020] Branislav Kveton, Manzil Zaheer, Csaba Szepesvari, Lihong Li, Mohammad Ghavamzadeh, and Craig Boutilier. Randomized exploration in general- ized linear bandits. In International Conference on Artifi- cial Intelligence and Statistics, pages 2066-2076, 2020.","title":"Randomized exploration in generalized linear bandits","authors":"Piyush Kumar, E Alper Yildirim ; Kveton","year":"2005","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lattimore and Szepesvári, 2020] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.","title":"Tor Lattimore and Csaba Szepesvári. Bandit Algorithms","authors":"Szepesvári Lattimore","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. International Conference on World Wide Web, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Li et al., 2017] Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear con- textual bandits. In International Conference on Machine Learning, pages 2071-2080, 2017.","title":"Provably optimal algorithms for generalized linear contextual bandits","authors":"Lihong Li, Yu Lu, Dengyong Zhou","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[McCullagh and Nelder, 1989] Peter McCullagh and John A Nelder. Generalized Linear Models. Chapman & Hall, 1989.","title":"Generalized Linear Models","authors":"Nelder Mccullagh, Peter Mccullagh, John A Nelder","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Riquelme et al., 2018] Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for Thompson sampling. In International Conference on Learning Representations, 2018.","title":"Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for Thompson sampling","authors":"Carlos Riquelme, George Tucker, Jasper Snoek","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Soare et al., 2014] Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. In Advances in Neural Information Processing Systems, pages 828-836, 2014.","title":"Best-arm identification in linear bandits","authors":"Marta Soare, Alessandro Lazaric, Rémi Munos","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tao et al., 2018] Chao Tao, Saúl Blanco, and Yuan Zhou. Best arm identification in linear bandits with linear dimen- sion dependency. In International Conference on Machine Learning, pages 4877-4886, 2018.","title":"Best arm identification in linear bandits with linear dimension dependency","authors":"Chao Tao, Saúl Blanco, Yuan Zhou","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tripuraneni et al., 2021] Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear rep- resentations. In International Conference on Machine Learning, pages 10434-10443, 2021.","title":"Provable meta-learning of linear representations","authors":"Nilesh Tripuraneni, Chi Jin, Michael Jordan","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vershynin, 2019] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilis- tic Mathematics, 2019.","title":"Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science","authors":"Vershynin","year":"2019","venue":"Cambridge Series in Statistical and Probabilistic Mathematics","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wolke and Schwetlick, 1988] Robert. Wolke and Hartmut Schwetlick. Iteratively reweighted least squares: algo- rithms, convergence analysis, and numerical comparisons. SIAM Journal on Scientific and Statistical Computing, 9(5):907-921, 1988.","title":"Iteratively reweighted least squares: algorithms, convergence analysis, and numerical comparisons","authors":"Robert Wolke, Hartmut Schwetlick","year":"1988","venue":"SIAM Journal on Scientific and Statistical Computing","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xu et al., 2018] Liyuan Xu, Junya Honda, and Masashi Sugiyama. A fully adaptive algorithm for pure exploration in linear bandits. In International Conference on Artificial Intelligence and Statistics, pages 843-851, 2018.","title":"A fully adaptive algorithm for pure exploration in linear bandits","authors":"Liyuan Xu, Junya Honda, Masashi Sugiyama","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yang and Tan, 2021] Junwen Yang and Vincent YF Tan. To- wards minimax optimal best arm identification in linear bandits. arXiv preprint arXiv:2105.13017, 2021.","title":"Towards minimax optimal best arm identification in linear bandits","authors":"Yang, Tan ; ] Junwen Yang, Vincent Yf Tan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}