{"type":"ScholarlyArticle","title":"Best Arm Identification in Spectral Bandits","authors":[{"name":"Tomáš Kocák","slug":"tomas-kocak","given":"Tomáš","family":"Kocák","affiliation":"École Normale Supérieur de Lyon"},{"name":"Aurélien Garivier","slug":"aurelien-garivier","given":"Aurélien","family":"Garivier","affiliation":"École Normale Supérieur de Lyon"}],"abstract":"We study best-arm identification with fixed confidence in bandit models with graph smoothness constraint. We provide and analyze an efficient gradient ascent algorithm to compute the sample complexity of this problem as a solution of a non-smooth max-min problem (providing in passing a simplified analysis for the unconstrained case). Building on this algorithm, we propose an asymptotically optimal strategy. We furthermore illustrate by numerical experiments both the strategy's efficiency and the impact of the smoothness constraint on the sample complexity. Best Arm Identification (BAI) is an important challenge in many applications ranging from parameter tuning to clinical trials. It is now very well understood in vanilla bandit models, but real-world problems typically involve some dependency between arms that requires more involved models. Assuming a graph structure on the arms is an elegant practical way to encompass this phenomenon, but this had been done so far only for regret minimization. Addressing BAI with graph constraints involves delicate optimization problems for which the present paper offers a solution.","doi":"10.24963/ijcai.2020/307","url":"https://ijcai.org/proceedings/2020/307","pdf":"https://ijcai.org/proceedings/2020/0307.pdf","markdown":"https://ijcai.org/proceedings/2020/307.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/307","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":307,"pages":{"first":2220,"last":2226},"keywords":["Machine Learning: Online Learning","Machine Learning: Learning Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26641/file_26641.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"[Alon et al., 2013] Noga Alon, Nicolò Cesa-Bianchi, Clau- dio Gentile, and Yishay Mansour. From bandits to experts: A tale of domination and independence. In Neural Inform- ation Processing Systems, 2013.","title":"From bandits to experts: A tale of domination and independence","authors":"Nicolò Noga Alon, Claudio Cesa-Bianchi, Yishay Gentile, Mansour","year":"2013","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alon et al., 2017] Noga Alon, Nicolò Cesa-Bianchi, Clau- dio Gentile, Shie Mannor, Yishay Mansour, and Ohad Shamir. Nonstochastic multi-armed bandits with graph- structured feedback. SIAM Journal on Computing, 46(6):1785-1826, 2017.","title":"Nonstochastic multi-armed bandits with graphstructured feedback","authors":"Nicolò Noga Alon, Claudio Cesa-Bianchi, Shie Gentile, Yishay Mannor, Ohad Mansour, Shamir","year":"2017","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bubeck, 2015] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3-4):231-357, 2015.","title":"Convex optimization: Algorithms and complexity","authors":"Sébastien Bubeck","year":"2015","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chernoff, 1959] Herman Chernoff. Sequential design of Experiments. The Annals of Mathematical Statistics, 30(3):755-770, 1959.","title":"Herman Chernoff. Sequential design of Experiments","authors":"Chernoff","year":"1959","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Degenne and Koolen, 2019] Rémy Degenne and Wouter M. Koolen. Pure Exploration with Multiple Correct Answers. technical report, feb 2019.","title":"Pure Exploration with Multiple Correct Answers","authors":"Rémy Degenne, M Wouter, Koolen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Degenne et al., 2019] Rémy Degenne, Wouter M. Koolen, and Pierre Ménard. Non-Asymptotic Pure Exploration by Solving Games. technical report, jun 2019.","title":"Non-Asymptotic Pure Exploration by Solving Games","authors":"Rémy Degenne, M Wouter, Pierre Koolen, Ménard","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Even-Dar et al., 2006] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping condi- tions for the multi-armed bandit and reinforcement learn- ing problems. Journal of Machine Learning Research, 7:1079-1105, 2006.","title":"Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems","authors":"Eyal Even-Dar, Shie Mannor, Yishay Mansour","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gabillon et al., 2012] Victor Gabillon, Mohammad Ghavamzadeh, and Alessandro Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. In Neural Information Processing Systems, 2012.","title":"Best arm identification: A unified approach to fixed budget and fixed confidence","authors":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric","year":"2012","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garivier and Kaufmann, 2016a] Aurélien Garivier and Em- ilie Kaufmann. Optimal best arm identification with fixed confidence. In Proceedings of the 29th Conference On Learning Theory, 2016.","title":"Optimal best arm identification with fixed confidence","authors":"Kaufmann Garivier, Aurélien Garivier, Emilie Kaufmann","year":"2016","venue":"Proceedings of the 29th Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garivier and Kaufmann, 2016b] Aurélien Garivier and Em- ilie Kaufmann. Optimal Best Arm Identification with Fixed Confidence. 29th Annual Conference on Learning Theory, 2016.","title":"Aurélien Garivier and Emilie Kaufmann. Optimal Best Arm Identification with Fixed Confidence","authors":"Kaufmann Garivier","year":"2016","venue":"29th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garivier and Kaufmann, 2019] Aurélien Garivier and Em- ilie Kaufmann. Non-Asymptotic Sequential Tests for Overlapping Hypotheses Applied to Near Optimal Arm Identification in Bandit Models. technical report, jun 2019.","title":"Aurélien Garivier and Emilie Kaufmann. Non-Asymptotic Sequential Tests for Overlapping Hypotheses Applied to Near Optimal Arm Identification in Bandit Models","authors":"Kaufmann Garivier","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kocák et al., 2014] Tomáš Kocák, Gergely Neu, Michal Valko, and Rémi Munos. Efficient learning by implicit exploration in bandit problems with side observations. In Neural Information Processing Systems, 2014.","title":"Efficient learning by implicit exploration in bandit problems with side observations","authors":"Kocák","year":"2014","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kocák et al., 2016a] Tomáš Kocák, Gergely Neu, and Michal Valko. Online learning with Erdos-Rényi side- observation graphs. In Conference on Uncertainty in Arti- ficial Intelligence, 2016.","title":"Online learning with Erdos-Rényi sideobservation graphs","authors":"Kocák","year":"2016","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kocák et al., 2016b] Tomáš Kocák, Gergely Neu, and Michal Valko. Online learning with noisy side observa- tions. In International Conference on Artificial Intelli- gence and Statistics, 2016.","title":"Online learning with noisy side observations","authors":"Kocák","year":"2016","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kocák et al., 2019] Tomáš Kocák, Rémi Munos, Branislav Kveton, Shipra Agrawal, and Michal Valko. Spectral Ban- dits. Journal of Machine Learning Research, 2019.","title":"Spectral Bandits","authors":"Kocák","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kocák et al., 2014] Tomáš Kocák, Michal Valko, Rémi Munos, and Shipra Agrawal. Spectral Thompson sampling. In Proceedings of the National Conference on Artificial Intelligence, 2014.","title":"Spectral Thompson sampling","authors":"Kocák","year":"2014","venue":"Proceedings of the National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lattimore and Szepesvári, 2019] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2019.","title":"Tor Lattimore and Csaba Szepesvári. Bandit Algorithms","authors":"Szepesvári Lattimore","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mannor and Shamir, 2011] Shie Mannor and Ohad Shamir. From bandits to experts: On the value of side- observations. In Neural Information Processing Systems, 2011.","title":"From bandits to experts: On the value of sideobservations","authors":"Shamir Mannor, Shie Mannor, Ohad Shamir","year":"2011","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Russo, 2016] Daniel Russo. Simple bayesian algorithms for best arm identification. In Proceedings of the 29th Confer- ence On Learning Theory, 2016.","title":"Simple bayesian algorithms for best arm identification","authors":"Daniel Russo","year":"2016","venue":"Proceedings of the 29th Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Valko et al., 2014] Michal Valko, Rémi Munos, Branislav Kveton, and Tomáš Kocák. Spectral bandits for smooth graph functions. In International Conference on Machine Learning, 2014.","title":"Spectral bandits for smooth graph functions","authors":"Michal Valko, Rémi Munos, Branislav Kveton, Tomáš Kocák","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}