{"type":"ScholarlyArticle","title":"Improved Pure Exploration in Linear Bandits with No-Regret Learning","authors":[{"name":"Mohammadi Zaki","slug":"mohammadi-zaki","given":"Mohammadi","family":"Zaki","affiliation":"Indian Institute of Science , Bangalore"},{"name":"Avi Mohan","slug":"avi-mohan","given":"Avi","family":"Mohan","affiliation":"Boston University , MA"},{"name":"Aditya Gopalan","slug":"aditya-gopalan","given":"Aditya","family":"Gopalan","affiliation":"Indian Institute of Science , Bangalore"}],"abstract":"We study the best arm identification problem in linear multi-armed bandits (LMAB) in the fixed confidence setting ; this is also the problem of optimizing an unknown linear function over a discrete domain with noisy, zeroth-order access. We propose an explicitly implementable and provably order-optimal sample-complexity algorithm for best arm identification. Existing approaches that achieve optimal sample complexity assume either access to a nontrivial minimax optimization oracle (e.g. RAGE and Lazy-TS) or knowledge of an upper-bound on the norm of the unknown parameter vector(e.g. LinGame). Our algorithm, which we call the Phased Elimination Linear Exploration Game (PELEG), maintains a high-probability confidence ellipsoid containing the unknown vector, and uses it to eliminate suboptimal arms in phases, where a minimax problem is essentially solved in each phase using two-player low regret learners. PELEG does not require to know a bound on norm of the unknown vector, and is asymptotically-optimal, settling an open problem. We show that the sample complexity of PELEG matches, up to order and in a non-asymptotic sense, an instance-dependent universal lower bound for linear bandits. PELEG is thus the first algorithm to achieve both order-optimal sample complexity and explicit implementability for this setting. We also provide numerical results for the proposed algorithm consistent with its theoretical guarantees.","doi":"10.24963/ijcai.2022/515","url":"https://ijcai.org/proceedings/2022/515","pdf":"https://ijcai.org/proceedings/2022/0515.pdf","markdown":"https://ijcai.org/proceedings/2022/515.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/515","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":515,"pages":{"first":3709,"last":3715},"keywords":["Machine Learning: Online Learning","Machine Learning: Learning Theory"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986882","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985025","length_sec":349,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Awerbuch and Kleinberg, 2008a] B. Awerbuch and Robert D. Kleinberg. Online linear optimization and adaptive routing. J. Comput. Syst. Sci., 74:97-114, 2008.","title":"Online linear optimization and adaptive routing","authors":"Kleinberg Awerbuch, B Awerbuch, Robert D Kleinberg","year":"2008","venue":"J. Comput. Syst. Sci","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Awerbuch and Kleinberg, 2008b] Baruch Awerbuch and Robert Kleinberg. Online linear optimization and adaptive routing. Journal of Computer and System Sciences, 74(1):97-114, 2008.","title":"Online linear optimization and adaptive routing","authors":"Awerbuch, Kleinberg ;, Robert Kleinberg","year":"2008","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cam- bridge University Press, 2006.","title":"Nicolò Cesa-Bianchi and Gábor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[de Rooij et al., 2014] Steven de Rooij, Tim van Erven, Pe- ter Grunwald, and Wouter Koolen. Follow the leader if you can, hedge if you must. Journal of Machine Learning Research, 15:1281-1316, 2014.","title":"Follow the leader if you can, hedge if you must","authors":"De Rooij","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Degenne et al., 2019] Rémy Degenne, Wouter M. Koolen, and Pierre Ménard. Non-asymptotic pure exploration by solving games. In NeurIPS, 2019.","title":"Non-asymptotic pure exploration by solving games","authors":"Rémy Degenne, M Wouter, Pierre Koolen, Ménard","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Degenne et al., 2020] Rèmy Degenne, Pierre Mènard, Xue- dong Shang, and Michal Valko. Gamification of pure ex- ploration for linear bandits. Proceedings of the 37th Inter- national Conference on Machine Learning. PMLR, 2020.","title":"Gamification of pure exploration for linear bandits","authors":"Degenne","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Even-Dar et al., 2006] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping condi- tions for the multi-armed bandit and reinforcement learn- ing problems. J. Mach. Learn. Res., 7:1079-1105, Decem- ber 2006.","title":"Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems","authors":"Eyal Even-Dar, Shie Mannor, Yishay Mansour","year":"2006","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fiez et al., 2019] Tanner Fiez, Lalit Jain, Kevin G Jamieson, and Lillian Ratliff. Sequential experimental design for transductive linear bandits. In Advances in Neural Infor- mation Processing Systems, pages 10666-10676, 2019.","title":"Sequential experimental design for transductive linear bandits","authors":"Tanner Fiez, Lalit Jain, Kevin G Jamieson, Lillian Ratliff","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garivier and Kaufmann, 2016] Aurélien Garivier and Emi- lie Kaufmann. Optimal best arm identification with fixed confidence. In Conference On Learning Theory, pages 998-1027, 2016.","title":"Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence","authors":"Kaufmann Garivier","year":"2016","venue":"Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jedra and Proutière, 2020] Yassir Jedra and A. Proutière. Optimal best-arm identification in linear bandits. NeurIPS, abs/2006.16073, 2020.","title":"Yassir Jedra and A. Proutière. Optimal best-arm identification in linear bandits","authors":"Proutière Jedra","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kalyanakrishnan et al., 2012] Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. Pac subset se- lection in stochastic multi-armed bandits. In ICML, 2012.","title":"Pac subset selection in stochastic multi-armed bandits","authors":"Ambuj Shivaram Kalyanakrishnan, Peter Tewari, Peter Auer, Stone","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuroki et al., 2019] Yuko Kuroki, Liyuan Xu, Atsushi Miyauchi, Junya Honda, and Masashi Sugiyama. Polynomial-time algorithms for combinatorial pure exploration with full-bandit feedback. arXiv preprint arXiv:1902.10582, 2019.","title":"Polynomial-time algorithms for combinatorial pure exploration with full-bandit feedback","authors":"Yuko Kuroki, Liyuan Xu, Atsushi Miyauchi, Junya Honda, Masashi Sugiyama","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liyuan Xu, 2017] Masashi Sugiyama Liyuan Xu, Junya Honda. Fully adaptive algorithm for pure ex- ploration in linear bandits. 2017.","title":"Fully adaptive algorithm for pure exploration in linear bandits","authors":"Liyuan Xu, ; ] Masashi Sugiyama, Liyuan Xu, Junya Honda","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shalev-Shwartz and others, 2011] Shai Shalev-Shwartz et al. Online learning and online convex optimiza- tion. Foundations and trends in Machine Learning, 4(2):107-194, 2011.","title":"Online learning and online convex optimization","authors":"Shai Shalev-Shwartz","year":"2011","venue":"Foundations and trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Soare et al., 2014] Marta Soare, Alessandro Lazaric, and Remi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems 27, pages 828-836, 2014.","title":"Best-arm identification in linear bandits","authors":"Marta Soare, Alessandro Lazaric, Remi Munos","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Srinivas et al., 2010] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experi- mental design. In In International Conference on Machine Learning, 2010.","title":"Gaussian process optimization in the bandit setting: no regret and experimental design","authors":"Niranjan Srinivas, Andreas Krause, M Sham, Matthias Kakade, Seeger","year":"2010","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tao et al., 2018] Chao Tao, Saúl Blanco, and Yuan Zhou. Best arm identification in linear bandits with linear dimen- sion dependency. volume 80 of Proceedings of Machine Learning Research, pages 4877-4886. PMLR, 2018.","title":"Best arm identification in linear bandits with linear dimension dependency","authors":"Chao Tao, Saúl Blanco, Yuan Zhou","year":"2018","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Valko et al., 2014] Michal Valko, Rémi Munos, Branislav Kveton, and Tomáš Kocák. Spectral bandits for smooth graph functions. In International Conference on Machine Learning, pages 46-54, 2014.","title":"Spectral bandits for smooth graph functions","authors":"Michal Valko, Rémi Munos, Branislav Kveton, Tomáš Kocák","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021] Po-An Wang, Ruo-Chun Tzeng, and Alexandre Proutiere. Fast pure exploration via frank- wolfe. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Infor- mation Processing Systems, 2021.","title":"Fast pure exploration via frankwolfe","authors":"Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zaki et al., 2019] Mohammadi Zaki, Avinash Mohan, and Aditya Gopalan. Towards optimal and efficient best arm identification in linear bandits. arXiv preprint arXiv:1911.01695, 2019.","title":"Towards optimal and efficient best arm identification in linear bandits","authors":"Mohammadi Zaki, Avinash Mohan, Aditya Gopalan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}