{"type":"ScholarlyArticle","title":"UCBoost: A Boosting Approach to Tame Complexity and Optimality for Stochastic Bandits","authors":[{"name":"Fang Liu","slug":"fang-liu","given":"Fang","family":"Liu","affiliation":"The Ohio State University"},{"name":"Sinong Wang","slug":"sinong-wang","given":"Sinong","family":"Wang","affiliation":"The Ohio State University"},{"name":"Swapna Buccapatnam","slug":"swapna-buccapatnam","given":"Swapna","family":"Buccapatnam","affiliation":"AT&T Labs Research"},{"name":"Ness Shroff","slug":"ness-shroff","given":"Ness","family":"Shroff","affiliation":"The Ohio State University"}],"abstract":"In this work, we address the open problem of finding low-complexity near-optimal multi-armed bandit algorithms for sequential decision making problems. Existing bandit algorithms are either sub-optimal and computationally simple (e.g., UCB1) or optimal and computationally complex (e.g., kl-UCB). We propose a boosting approach to Upper Confidence Bound based algorithms for stochastic bandits, that we call UCBoost. Specifically, we propose two types of UCBoost algorithms. We show that UCBoost(D) enjoys O(1) complexity for each arm per round as well as regret guarantee that is 1/e-close to that of the kl-UCB algorithm. We propose an approximation-based UCBoost algorithm, UCBoost(epsilon), that enjoys a regret guarantee epsilon-close to that of kl-UCB as well as O(log(1/epsilon)) complexity for each arm per round. Hence, our algorithms provide practitioners a practical way to trade optimality with computational complexity. Finally, we present numerical results which show that UCBoost(epsilon) can achieve the same regret performance as the standard kl-UCB while incurring only 1% of the computational cost of kl-UCB.","doi":"10.24963/ijcai.2018/338","url":"https://ijcai.org/proceedings/2018/338","pdf":"https://ijcai.org/proceedings/2018/0338.pdf","markdown":"https://ijcai.org/proceedings/2018/338.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/338","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":338,"pages":{"first":2440,"last":2446},"keywords":["Uncertainty in AI: Sequential Decision Making","Uncertainty in AI: Uncertainty in AI","Machine Learning: Reinforcement Learning","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2017] Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory, pages 12-38, 2017.","title":"Corralling a band of bandit algorithms","authors":"Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, Robert E Schapire","year":"2017","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Goyal, 2012] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi- armed bandit problem. In COLT, pages 39-1, 2012.","title":"Analysis of thompson sampling for the multiarmed bandit problem","authors":"Shipra Agrawal, Navin Goyal","year":"2012","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Audibert and Bubeck, 2010] Jean-Yves Audibert and Sébastien Bubeck. Regret bounds and minimax policies under partial monitoring. Journal of Machine Learning Research, 11(Oct):2785-2836, 2010.","title":"Regret bounds and minimax policies under partial monitoring","authors":"Bubeck ; Jean-Yves Audibert, Sébastien Audibert, Bubeck","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Audibert et al., 2009] Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoret- ical Computer Science, 410(19):1876-1902, 2009.","title":"Exploration-exploitation tradeoff using variance estimates in multi-armed bandits","authors":"Audibert","year":"2009","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buccapatnam et al., 2017] Swapna Buccapatnam, Fang Liu, Atilla Eryilmaz, and Ness B Shroff. Reward maxi- mization under uncertainty: Leveraging side-observations on networks. arXiv preprint arXiv:1704.07943, 2017.","title":"Reward maximization under uncertainty: Leveraging side-observations on networks","authors":"Swapna Buccapatnam, Fang Liu, Atilla Eryilmaz, B Ness, Shroff","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Burnetas and Katehakis, 1996] Apostolos N Burnetas and Michael N Katehakis. Optimal adaptive policies for se- quential allocation problems. Advances in Applied Math- ematics, 17(2):122-142, 1996.","title":"Optimal adaptive policies for sequential allocation problems","authors":"Apostolos N Burnetas, Michael N Katehakis","year":"1996","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Busa-Fekete and Kégl, 2010] Róbert Busa-Fekete and Balázs Kégl. Fast boosting using adversarial bandits. In 27th International Conference on Machine Learning (ICML 2010), pages 143-150, 2010.","title":"Fast boosting using adversarial bandits","authors":"Róbert Busa-Fekete, Balázs Kégl","year":"2010","venue":"27th International Conference on Machine Learning (ICML 2010)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cappé et al., 2012] Olivier Cappé, Aurélien Gariv- ier, and Emilie Kaufmann. py/mabandits: Mat- lab and python packages for multi-armed bandits. http://mloss.org/software/view/415/, 2012.","title":"py/mabandits: Matlab and python packages for multi-armed bandits","authors":"Cappé","year":"2012","venue":null,"doi":null,"url":"http://mloss.org/software/view/415/,2012","internal_url":null},{"position":10,"raw":"[Cappé et al., 2013] Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, 41(3):1516-1541, 2013.","title":"Kullback-leibler upper confidence bounds for optimal sequential allocation","authors":"Cappé","year":"2013","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Garivier and Cappé, 2011] Aurélien Garivier and Olivier Cappé. The kl-ucb algorithm for bounded stochastic ban- dits and beyond. In Proceedings of the 24th annual Con- ference On Learning Theory, pages 359-376, 2011.","title":"Aurélien Garivier and Olivier Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond","authors":"Cappé Garivier","year":"2011","venue":"Proceedings of the 24th annual Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Honda and Takemura, 2010] Junya Honda and Akimichi Takemura. An asymptotically optimal bandit algorithm for bounded support models. In COLT, pages 67-79, 2010.","title":"An asymptotically optimal bandit algorithm for bounded support models","authors":"Takemura Honda, Junya Honda, Akimichi Takemura","year":"2010","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Honda and Takemura, 2012] Junya Honda and Akimichi Takemura. Stochastic bandit based on empirical moments. In Artificial Intelligence and Statistics, pages 529-537, 2012.","title":"Stochastic bandit based on empirical moments","authors":"Takemura Honda, Junya Honda, Akimichi Takemura","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaufmann et al., 2012] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial Intelligence and Statistics, pages 592-600, 2012.","title":"On bayesian upper confidence bounds for bandit problems","authors":"Emilie Kaufmann, Olivier Cappé, Aurélien Garivier","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Korda et al., 2013] Nathaniel Korda, Emilie Kaufmann, and Remi Munos. Thompson sampling for 1-dimensional ex- ponential family bandits. In Advances in Neural Informa- tion Processing Systems, pages 1448-1456, 2013.","title":"Thompson sampling for 1-dimensional exponential family bandits","authors":"Korda","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lai and Robbins, 1985] Tze Leung Lai and Herbert Rob- bins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in applied mathematics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laskey et al., 2015] Michael Laskey, Jeff Mahler, Zoe Mc- Carthy, Florian T Pokorny, Sachin Patil, Jur Van Den Berg, Danica Kragic, Pieter Abbeel, and Ken Goldberg. Multi- armed bandit models for 2d grasp planning with uncer- tainty. In Automation Science and Engineering (CASE), 2015 IEEE International Conference on, pages 572-579. IEEE, 2015.","title":"Multiarmed bandit models for 2d grasp planning with uncertainty","authors":"Michael Laskey, Jeff Mahler, Zoe Mc-Carthy, Florian T Pokorny, Sachin Patil, Jur Van Den, Danica Berg, Pieter Kragic, Ken Abbeel, Goldberg","year":"2015","venue":"Automation Science and Engineering (CASE)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670. ACM, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2017] Fang Liu, Swapna Buccapatnam, and Ness Shroff. Information directed sampling for stochas- tic bandits with graph feedback. arXiv preprint arXiv:1711.03198, 2017.","title":"Information directed sampling for stochastic bandits with graph feedback","authors":"Fang Liu, Swapna Buccapatnam, Ness Shroff","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maillard et al., 2011] Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. A finite-time analysis of multi-armed bandits problems with kullback-leibler diver- gences. In Proceedings of the 24th annual Conference On Learning Theory, pages 497-514, 2011.","title":"A finite-time analysis of multi-armed bandits problems with kullback-leibler divergences","authors":"Maillard","year":"2011","venue":"Proceedings of the 24th annual Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Matikainen et al., 2013] Pyry Matikainen, P Michael Fur- long, Rahul Sukthankar, and Martial Hebert. Multi-armed recommendation bandits for selecting state machine poli- cies for robotic systems. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 4545-4551. IEEE, 2013.","title":"Multi-armed recommendation bandits for selecting state machine policies for robotic systems","authors":"Pyry Matikainen, Michael Furlong, Rahul Sukthankar, Martial Hebert","year":"2013","venue":"Robotics and Automation (ICRA), 2013 IEEE International Conference on","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Moeini et al., 2016] Mahdi Moeini, Oliver Wendt, and Li- nus Krumrey. Portfolio optimization by means of a χ- armed bandit algorithm. In Asian Conference on Intelli- gent Information and Database Systems, pages 620-629. Springer, 2016.","title":"Portfolio optimization by means of a χarmed bandit algorithm","authors":"Moeini","year":"2016","venue":"Asian Conference on Intelligent Information and Database Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Russo and Van Roy, 2014] Dan Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. In Advances in Neural Information Processing Systems, pages 1583-1591, 2014.","title":"Learning to optimize via information-directed sampling","authors":"Van Russo, Dan Roy, Benjamin Russo, Roy Van","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}