{"type":"ScholarlyArticle","title":"Perturbed-History Exploration in Stochastic Multi-Armed Bandits","authors":[{"name":"Branislav Kveton","slug":"branislav-kveton","given":"Branislav","family":"Kveton","affiliation":""},{"name":"Csaba Szepesvári","slug":"csaba-szepesvari","given":"Csaba","family":"Szepesvári","affiliation":"University of Alberta"},{"name":"Mohammad Ghavamzadeh","slug":"mohammad-ghavamzadeh","given":"Mohammad","family":"Ghavamzadeh","affiliation":"Facebook AI Research"},{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":""}],"abstract":"We propose an online algorithm for cumulative regret minimization in a stochastic multi-armed bandit. The algorithm adds O(t) i.i.d. pseudo-rewards to its history in round t and then pulls the arm with the highest average reward in its perturbed history. Therefore, we call it perturbed-history exploration (PHE). The pseudo-rewards are carefully designed to offset potentially underestimated mean rewards of arms with a high probability. We derive near-optimal gap-dependent and gap-free bounds on the n-round regret of PHE. The key step in our analysis is a novel argument that shows that randomized Bernoulli rewards lead to optimism. Finally, we empirically evaluate PHE and show that it is competitive with state-of-the-art baselines.","doi":"10.24963/ijcai.2019/386","url":"https://ijcai.org/proceedings/2019/386","pdf":"https://ijcai.org/proceedings/2019/0386.pdf","markdown":"https://ijcai.org/proceedings/2019/386.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/386","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":386,"pages":{"first":2786,"last":2793},"keywords":["Uncertainty in AI: Sequential Decision Making","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvari. Improved algorithms for lin- ear stochastic bandits. In Advances in Neural Information Processing Systems 24, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, David Pal, Csaba Szepesvari","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abeille and Lazaric, 2017] Marc Abeille and Alessandro Lazaric. Linear Thompson sampling revisited. In Pro- ceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.","title":"Linear Thompson sampling revisited","authors":"Abeille, Lazaric, Marc Abeille, Alessandro Lazaric","year":"2017","venue":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agrawal and Goyal, 2013a] Shipra Agrawal and Navin Goyal. Further optimal regret bounds for Thompson sam- pling. In Proceedings of the 16th International Confer- ence on Artificial Intelligence and Statistics, pages 99- 107, 2013.","title":"Further optimal regret bounds for Thompson sampling","authors":"Goyal Agrawal, Shipra Agrawal, Navin Goyal","year":"2013","venue":"Proceedings of the 16th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Agrawal and Goyal, 2013b] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In Proceedings of the 30th International Conference on Machine Learning, pages 127-135, 2013.","title":"Thompson sampling for contextual bandits with linear payoffs","authors":"Goyal Agrawal, Shipra Agrawal, Navin Goyal","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chapelle and Zhang, 2009] Olivier Chapelle and Ya Zhang. A dynamic Bayesian network click model for web search ranking. In Proceedings of the 18th International Confer- ence on World Wide Web, pages 1-10, 2009.","title":"A dynamic Bayesian network click model for web search ranking","authors":"Chapelle, Zhang, Olivier Chapelle, Ya Zhang","year":"2009","venue":"Proceedings of the 18th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dani et al., 2008] Varsha Dani, Thomas Hayes, and Sham Kakade. Stochastic linear optimization under bandit feed- back. In Proceedings of the 21st Annual Conference on Learning Theory, pages 355-366, 2008.","title":"Stochastic linear optimization under bandit feedback","authors":"Varsha Dani, Thomas Hayes, Sham Kakade","year":"2008","venue":"Proceedings of the 21st Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Filippi et al., 2010] Sarah Filippi, Olivier Cappe, Aurelien Garivier, and Csaba Szepesvari. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems 23, pages 586-594, 2010.","title":"Parametric bandits: The generalized linear case","authors":"Sarah Filippi, Olivier Cappe, Aurelien Garivier, Csaba Szepesvari","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garivier and Cappe, 2011] Aurelien Garivier and Olivier Cappe. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proceeding of the 24th Annual Conference on Learning Theory, pages 359-376, 2011.","title":"The KL-UCB algorithm for bounded stochastic bandits and beyond","authors":"Cappe Garivier, Aurelien Garivier, Olivier Cappe","year":"2011","venue":"Proceeding of the 24th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gopalan et al., 2014] Aditya Gopalan, Shie Mannor, and Yishay Mansour. Thompson sampling for complex online problems. In Proceedings of the 31st International Con- ference on Machine Learning, pages 100-108, 2014.","title":"Thompson sampling for complex online problems","authors":"Aditya Gopalan, Shie Mannor, Yishay Mansour","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hannan, 1957] James Hannan. Approximation to Bayes risk in repeated play. In Contributions to the Theory of Games, volume 3, pages 97-140. Princeton University Press, Princeton, NJ, 1957.","title":"Approximation to Bayes risk in repeated play","authors":"James Hannan","year":"1957","venue":"Contributions to the Theory of Games","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jun et al., 2017] Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable general- ized linear bandits: Online computation and hashing. In Advances in Neural Information Processing Systems 30, pages 98-108, 2017.","title":"Scalable generalized linear bandits: Online computation and hashing","authors":"Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, Rebecca Willett","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kalai and Vempala, 2005] Adam Kalai and Santosh Vem- pala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences, 71(3):291-307, 2005.","title":"Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems","authors":"Vempala Kalai","year":"2005","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Katariya et al., 2016] Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. DCM bandits: Learn- ing to rank with multiple clicks. In Proceedings of the 33rd International Conference on Machine Learning, pages 1215-1224, 2016.","title":"DCM bandits: Learning to rank with multiple clicks","authors":"Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, Zheng Wen","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kawale et al., 2015] Jaya Kawale, Hung Bui, Branislav Kveton, Long Tran-Thanh, and Sanjay Chawla. Efficient Thompson sampling for online matrix-factorization rec- ommendation. In Advances in Neural Information Pro- cessing Systems 28, pages 1297-1305, 2015.","title":"Efficient Thompson sampling for online matrix-factorization recommendation","authors":"Kawale","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kveton et al., 2015] Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learn- ing to rank in the cascade model. In Proceedings of the 32nd International Conference on Machine Learning, 2015.","title":"Cascading bandits: Learning to rank in the cascade model","authors":"Csaba Branislav Kveton, Zheng Szepesvari, Azin Wen, Ashkan","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kveton et al., 2019a] Branislav Kveton, Csaba Szepes- vari, Mohammad Ghavamzadeh, and Craig Boutilier. Perturbed-history exploration in stochastic linear bandits. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, 2019.","title":"Perturbed-history exploration in stochastic linear bandits","authors":"Kveton","year":"2019","venue":"Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kveton et al., 2019b] Branislav Kveton, Csaba Szepesvari, Sharan Vaswani, Zheng Wen, Mohammad Ghavamzadeh, and Tor Lattimore. Garbage in, reward out: Bootstrap- ping exploration in multi-armed bandits. In Proceedings of the 36th International Conference on Machine Learn- ing, pages 3601-3610, 2019.","title":"Garbage in, reward out: Bootstrapping exploration in multi-armed bandits","authors":"Kveton","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lai and Robbins, 1985] T. L. Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Ad- vances in Applied Mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"T L Lai, Herbert Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lattimore and Szepesvari, 2019] Tor Lattimore and Csaba Szepesvari. Bandit Algorithms. Cambridge University Press, 2019.","title":"Tor Lattimore and Csaba Szepesvari. Bandit Algorithms","authors":"Szepesvari Lattimore","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2017] Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contex- tual bandits. In Proceedings of the 34th International Con- ference on Machine Learning, pages 2071-2080, 2017.","title":"Provably optimal algorithms for generalized linear contextual bandits","authors":"Lihong Li, Yu Lu, Dengyong Zhou","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lipton et al., 2018] Zachary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng. BBQ- networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems. In Proceed- ings of the 32nd AAAI Conference on Artificial Intelli- gence, pages 5237-5244, 2018.","title":"BBQnetworks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems","authors":"Zachary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, Li Deng","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2018] Bing Liu, Tong Yu, Ian Lane, and Ole Mengshoel. Customized nonlinear bandits for online re- sponse selection in neural conversation models. In Pro- ceedings of the 32nd AAAI Conference on Artificial Intel- ligence, pages 5245-5252, 2018.","title":"Customized nonlinear bandits for online response selection in neural conversation models","authors":"Liu","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lu and Van Roy, 2017] Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In Advances in Neural Infor- mation Processing Systems 30, pages 3258-3266, 2017.","title":"Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling","authors":"Lu, Van Roy","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Neu and Bartok, 2013] Gergely Neu and Gabor Bartok. An efficient algorithm for learning with semi-bandit feedback. In Proceedings of the 24th International Conference on Al- gorithmic Learning Theory, pages 234-248, 2013.","title":"An efficient algorithm for learning with semi-bandit feedback","authors":"Bartok Neu, Gergely Neu, Gabor Bartok","year":"2013","venue":"Proceedings of the 24th International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Popoviciu, 1935] Tiberiu Popoviciu. Popoviciu's inequality on variances. https://en.wikipedia.org/wiki/Popoviciu's inequality on variances, 1935.","title":"Popoviciu's inequality on variances","authors":"Tiberiu Popoviciu","year":"1935","venue":null,"doi":null,"url":"https://en.wikipedia.org/wiki/Popoviciu'sinequalityonvariances","internal_url":null},{"position":27,"raw":"[Radlinski et al., 2008] Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. Learning diverse rankings with multi-armed bandits. In Proceedings of the 25th Interna- tional Conference on Machine Learning, pages 784-791, 2008.","title":"Learning diverse rankings with multi-armed bandits","authors":"Filip Radlinski, Robert Kleinberg, Thorsten Joachims","year":"2008","venue":"Proceedings of the 25th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Riquelme et al., 2018] Carlos Riquelme, George Tucker, and Jasper Snoek. Deep Bayesian bandits showdown: An empirical comparison of Bayesian deep networks for Thompson sampling. In Proceedings of the 6th Interna- tional Conference on Learning Representations, 2018.","title":"Deep Bayesian bandits showdown: An empirical comparison of Bayesian deep networks for Thompson sampling","authors":"Carlos Riquelme, George Tucker, Jasper Snoek","year":"2018","venue":"Proceedings of the 6th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Russo et al., 2018] Daniel Russo, Benjamin Van Roy, Ab- bas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on Thompson sampling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018.","title":"A tutorial on Thompson sampling","authors":"Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen","year":"2018","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Thompson, 1933] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"R William, Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2016] Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-Hua Zhou. Online stochastic linear optimization under one-bit feedback. In Proceedings of the 33rd International Conference on Machine Learning, pages 392-401, 2016.","title":"Online stochastic linear optimization under one-bit feedback","authors":"Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, Zhi-Hua Zhou","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}