{"type":"ScholarlyArticle","title":"Thresholding Bandits with Augmented UCB","authors":[{"name":"Subhojyoti Mukherjee","slug":"subhojyoti-mukherjee","given":"Subhojyoti","family":"Mukherjee","affiliation":""},{"name":"Naveen Kolar Purushothama","slug":"naveen-kolar-purushothama","given":"Naveen Kolar","family":"Purushothama","affiliation":"Department of Electrical Engineering , Indian Institute of Technology Madras"},{"name":"Nandan Sudarsanam","slug":"nandan-sudarsanam","given":"Nandan","family":"Sudarsanam","affiliation":"Department of Management Studies , Indian Institute of Technology Madras"},{"name":"Balaraman Ravindran","slug":"balaraman-ravindran","given":"Balaraman","family":"Ravindran","affiliation":"Department of Computer Science & Engineering , Indian Institute of Technology Madras"}],"abstract":"In this paper we propose the Augmented-UCB (AugUCB) algorithm for a fixed-budget version of the thresholding bandit problem (TBP), where the objective is to identify a set of arms whose quality is above a threshold. A key feature of AugUCB is that it uses both mean and variance estimates to eliminate arms that have been sufficiently explored; to the best of our knowledge this is the first algorithm to employ such an approach for the considered TBP.  Theoretically, we obtain an upper bound on the loss (probability of mis-classification) incurred by AugUCB. Although UCBEV in literature provides a better guarantee, it is important to emphasize that UCBEV has access to problem complexity (whose computation requires arms' mean and variances), and hence is not realistic in practice; this is in contrast to AugUCB whose implementation does not require any such complexity inputs. We conduct extensive simulation experiments to validate the performance of AugUCB. Through our simulation work, we establish that AugUCB, owing to its utilization of variance estimates, performs significantly better than the state-of-the-art APT, CSAR and other non variance-based algorithms.","doi":"10.24963/ijcai.2017/350","url":"https://ijcai.org/proceedings/2017/350","pdf":"https://ijcai.org/proceedings/2017/0350.pdf","markdown":"https://ijcai.org/proceedings/2017/350.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/350","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":350,"pages":{"first":2515,"last":2521},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Learning Theory","Uncertainty in AI: Sequential Decision Making","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"Jacob D Abernethy, Kareem Amin, and Ruihao Zhu. Thresh- old bandits, with and without censored feedback. In Ad- vances In Neural Information Processing Systems, pages 4889-4897, 2016.","title":"Threshold bandits, with and without censored feedback","authors":"Kareem Jacob D Abernethy, Ruihao Amin, Zhu","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Jean-Yves Audibert and Sébastien Bubeck. Minimax poli- cies for adversarial and stochastic bandits. In COLT, pages 217-226, 2009.","title":"Minimax policies for adversarial and stochastic bandits","authors":"Jean-Yves Audibert, Sébastien Bubeck","year":"2009","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Jean-Yves Audibert and Sébastien Bubeck. Best arm identi- fication in multi-armed bandits. In COLT-23th Conference on Learning Theory-2010, pages 13-p, 2010.","title":"Best arm identification in multi-armed bandits","authors":"Jean-Yves Audibert, Sébastien Bubeck","year":"2010","venue":"COLT-23th Conference on Learning Theory-2010","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19):1876-1902, 2009.","title":"Exploration-exploitation tradeoff using variance estimates in multi-armed bandits","authors":"Jean-Yves Audibert, Rémi Munos, Csaba Szepesvári","year":"2009","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Peter Auer and Ronald Ortner. Ucb revisited: Improved re- gret bounds for the stochastic multi-armed bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-65, 2010.","title":"Ucb revisited: Improved regret bounds for the stochastic multi-armed bandit problem","authors":"Peter Auer, Ronald Ortner","year":"2010","venue":"Periodica Mathematica Hungarica","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite- time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finitetime analysis of the multiarmed bandit problem","authors":"Peter Auer, Nicolo Cesa-Bianchi, Paul Fischer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit prob- lems. arXiv preprint arXiv:1204.5721, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Sébastien Bubeck, Nicolo Cesa-Bianchi","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure ex- ploration in finitely-armed and continuous-armed bandits. Theoretical Computer Science, 412(19):1832-1852, 2011.","title":"Pure exploration in finitely-armed and continuous-armed bandits","authors":"Sébastien Bubeck, Rémi Munos, Gilles Stoltz","year":"2011","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Sébastien Bubeck, Tengyao Wang, and Nitin Viswanathan. Multiple identifications in multi-armed bandits. In ICML (1), pages 258-265, 2013.","title":"Multiple identifications in multi-armed bandits","authors":"Sébastien Bubeck, Tengyao Wang, Nitin Viswanathan","year":"2013","venue":"ICML (1)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Shouyuan Chen, Tian Lin, Irwin King, Michael R Lyu, and Wei Chen. Combinatorial pure exploration of multi-armed bandits. In Advances in Neural Information Processing Systems, pages 379-387, 2014.","title":"Combinatorial pure exploration of multi-armed bandits","authors":"Shouyuan Chen, Tian Lin, Irwin King, Wei Michael R Lyu, Chen","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. The Journal of Machine Learning Research, 7:1079-1105, 2006.","title":"Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems","authors":"Eyal Even-Dar, Shie Mannor, Yishay Mansour","year":"2006","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric, and Sébastien Bubeck. Multi-bandit best arm identification. In Advances in Neural Information Process- ing Systems, pages 2222-2230, 2011.","title":"Multi-bandit best arm identification","authors":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric, Sébastien Bubeck","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Victor Gabillon, Mohammad Ghavamzadeh, and Alessandro Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. In Advances in Neural Information Processing Systems, pages 3212-3220, 2012.","title":"Best arm identification: A unified approach to fixed budget and fixed confidence","authors":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Kevin Jamieson and Robert Nowak. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In Information Sciences and Systems (CISS), 2014 48th Annual Conference on, pages 1-6. IEEE, 2014.","title":"Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting","authors":"Kevin Jamieson, Robert Nowak","year":"2014","venue":"Information Sciences and Systems (CISS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. Pac subset selection in stochastic multi-armed bandits. In Proceedings of the 29th International Con- ference on Machine Learning (ICML-12), pages 655-662, 2012.","title":"Pac subset selection in stochastic multi-armed bandits","authors":"Ambuj Shivaram Kalyanakrishnan, Peter Tewari, Peter Auer, Stone","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning (ICML-12)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathemat- ics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in applied mathematics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Yun-Ching Liu and Yoshimasa Tsuruoka. Modification of im- proved upper confidence bounds for regulating exploration in monte-carlo tree search. Theoretical Computer Science, 2016.","title":"Modification of improved upper confidence bounds for regulating exploration in monte-carlo tree search","authors":"Yun-Ching Liu, Yoshimasa Tsuruoka","year":"2016","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Andrea Locatelli, Maurilio Gutzeit, and Alexandra Carpen- tier. An optimal algorithm for the thresholding bandit prob- lem. arXiv preprint arXiv:1605.08671, 2016.","title":"An optimal algorithm for the thresholding bandit problem","authors":"Andrea Locatelli, Maurilio Gutzeit, Alexandra Carpentier","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Herbert Robbins. Some aspects of the sequential design of experiments. In Herbert Robbins Selected Papers, pages 169-177. Springer, 1952.","title":"Some aspects of the sequential design of experiments","authors":"Herbert Robbins","year":"1952","venue":"Herbert Robbins Selected Papers","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, pages 285-294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}