{"type":"ScholarlyArticle","title":"Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs","authors":[{"name":"Bo Xue","slug":"bo-xue","given":"Bo","family":"Xue","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Guanghui Wang","slug":"guanghui-wang","given":"Guanghui","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Yimu Wang","slug":"yimu-wang","given":"Yimu","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Lijun Zhang","slug":"lijun-zhang","given":"Lijun","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"In this paper, we study the problem of stochastic linear bandits with finite action sets. Most of existing work assume the payoffs are bounded or sub-Gaussian, which may be violated in some scenarios such as financial markets. To settle this issue, we analyze the linear bandits with heavy-tailed payoffs, where the payoffs admit finite 1+epsilon moments for some epsilon in (0,1]. Through median of means and dynamic truncation, we propose two novel algorithms which enjoy a sublinear regret bound of widetilde{O}(d^(1/2)T^(1/(1+epsilon))), where d is the dimension of contextual information and T is the time horizon. Meanwhile, we provide an Omega(d^(epsilon/(1+epsilon))T^(1/(1+epsilon))) lower bound, which implies our upper bound matches the lower bound up to polylogarithmic factors in the order of d and T when epsilon=1. Finally, we conduct numerical experiments to demonstrate the effectiveness of our algorithms and the empirical results strongly support our theoretical guarantees.","doi":"10.24963/ijcai.2020/406","url":"https://ijcai.org/proceedings/2020/406","pdf":"https://ijcai.org/proceedings/2020/0406.pdf","markdown":"https://ijcai.org/proceedings/2020/406.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/406","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":406,"pages":{"first":2936,"last":2942},"keywords":["Machine Learning: Online Learning","Machine Learning: Probabilistic Machine Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24795/file_24795.m3u8","length_sec":279,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbasi-yadkori et al., 2011] Yasin Abbasi-yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for lin- ear stochastic bandits. In Advances in Neural Information Processing Systems 24, pages 2312-2320. 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abe et al., 2003] Naoki Abe, Alan W Biermann, and Philip M Long. Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica, 37(4):263- 293, 2003.","title":"Reinforcement learning with immediate rewards and linear hypotheses","authors":"Naoki Abe, Alan W Biermann, Philip M Long","year":"2003","venue":"Algorithmica","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agarwal et al., 2013] A. Agarwal, D. Foster, D. Hsu, S. Kakade, and A. Rakhlin. Stochastic convex optimiza- tion with bandit feedback. SIAM Journal on Optimization, 23(1):213-240, 2013.","title":"Stochastic convex optimization with bandit feedback","authors":"A Agarwal, D Foster, D Hsu, S Kakade, A Rakhlin","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Audibert and Catoni, 2011] Jean-Yves Audibert and Olivier Catoni. Robust linear least squares regression. The Annals of Statistics, 39(5):2766-2794, 2011.","title":"Robust linear least squares regression","authors":"Jean-Yves Audibert, Olivier Catoni","year":"2011","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer, 2002] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3:397-422, 2002.","title":"Using confidence bounds for exploitation-exploration trade-offs","authors":"Peter Auer, Auer","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brownlees et al., 2015] Christian Brownlees, Emilien Joly, and Gábor Lugosi. Empirical risk minimization for heavy- tailed losses. The Annals of Statistics, 43(6):2507-2536, 2015.","title":"Empirical risk minimization for heavytailed losses","authors":"Brownlees","year":"2015","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck and Cesa-Bianchi, 2012] Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1):1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Cesa-Bianchi ; Bubeck, Sébastien Bubeck, Nicolò Cesa-Bianchi","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bubeck et al., 2011] Sébastien Bubeck, Gilles Stoltz, and Jia Yuan Yu. Lipschitz bandits without the lipschitz con- stant. In Proceedings of the 22Nd International Confer- ence on Algorithmic Learning Theory, pages 144-158, 2011.","title":"Lipschitz bandits without the lipschitz constant","authors":"Sébastien Bubeck, Gilles Stoltz, Jia Yuan, Yu","year":"2011","venue":"Proceedings of the 22Nd International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bubeck et al., 2013] S. Bubeck, N. Cesa-Bianchi, and G. Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory, 59(11):7711-7717, 2013.","title":"Bandits with heavy tail","authors":"S Bubeck, N Cesa-Bianchi, G Lugosi","year":"2013","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Bubeck et al., 2015] Sébastien Bubeck, Ofer Dekel, Tomer Koren, and Yuval Peres. Bandit convex optimization: √ T regret in one dimension. In Proceedings of The 28th Con- ference on Learning Theory, pages 266-278, 2015.","title":"Bandit convex optimization: √ T regret in one dimension","authors":"Sébastien Bubeck, Ofer Dekel, Tomer Koren, Yuval Peres","year":"2015","venue":"Proceedings of The 28th Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Catoni, 2012] Olivier Catoni. Challenging the empirical mean and empirical variance: A deviation study. Annales de l'I.H.P. Probabilités et statistiques, 48(4):1148-1185, 2012.","title":"Challenging the empirical mean and empirical variance: A deviation study","authors":"Olivier Catoni, Catoni","year":"2012","venue":"Annales de l'I.H.P. Probabilités et statistiques","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chu et al., 2011] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the 14th International Con- ference on Artificial Intelligence and Statistics, pages 208- 214, 2011.","title":"Contextual bandits with linear payoff functions","authors":"Wei Chu, Lihong Li, Lev Reyzin, Robert Schapire","year":"2011","venue":"Proceedings of the 14th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Cont and Bouchaud, 2000] Rama Cont and Jean-Philipe Bouchaud. Herd behavior and aggregate fluctuations in financial markets. Macroeconomic Dynamics, 4(02):170- 196, 2000.","title":"Rama Cont and Jean-Philipe Bouchaud. Herd behavior and aggregate fluctuations in financial markets","authors":"Bouchaud Cont","year":"2000","venue":"Macroeconomic Dynamics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dani et al., 2008] Varsha Dani, Thomas P. Hayes, and Sham M. Kakade. Stochastic linear optimization under bandit feedback. In Proceedings of the 21st Annual Con- ference on Learning, pages 355-366, 2008.","title":"Stochastic linear optimization under bandit feedback","authors":"Varsha Dani, Thomas P Hayes, Sham M Kakade","year":"2008","venue":"Proceedings of the 21st Annual Conference on Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Foss et al., 2013] Sergey Foss, Dmitry Korshunov, and Stan Zachary. An Introduction to Heavy-Tailed and Subexpo- nential Distributions. Springer, New York, 2013.","title":"An Introduction to Heavy-Tailed and Subexponential Distributions","authors":"Sergey Foss, Dmitry Korshunov, Stan Zachary","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Golub and Van Loan, 1996] Gene H. Golub and Charles F. Van Loan. Matrix Computations (3rd Ed.). Johns Hopkins University Press, USA, 1996.","title":"Matrix Computations","authors":"Gene H Golub, Charles F Van Loan","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hsu and Sabato, 2016] Daniel Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. Journal of Machine Learning Research, 17(18):1-40, 2016.","title":"Loss minimization and parameter estimation with heavy tails","authors":"Daniel Hsu, Sivan Sabato","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kleinberg et al., 2008] Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in met- ric spaces. In Proceedings of the 40th Annual ACM Symposium on Theory of Computing, pages 681-690, 2008.","title":"Multi-armed bandits in metric spaces","authors":"Robert Kleinberg, Aleksandrs Slivkins, Eli Upfal","year":"2008","venue":"Proceedings of the 40th Annual ACM Symposium on Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lai and Robbins, 1985] T. L. Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Ad- vances in Applied Mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"T L Lai, Herbert Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2019] Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Optimal algorithms for Lipschitz bandits with heavy-tailed rewards. In Proceedings of the 36th International Conference on Machine Learning, pages 4154-4163, 2019.","title":"Optimal algorithms for Lipschitz bandits with heavy-tailed rewards","authors":"Shiyin Lu, Guanghui Wang, Yao Hu, Lijun Zhang","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Medina and Yang, 2016] Andres Munoz Medina and Scott Yang. No-regret algorithms for heavy-tailed linear ban- dits. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, pages 1642-1650, 2016.","title":"No-regret algorithms for heavy-tailed linear bandits","authors":"Andres Munoz, Medina, Scott Yang","year":"2016","venue":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the se- quential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Roberts et al., 2015] James A Roberts, Tjeerd W Boonstra, and Michael Breakspear. The heavy tail of the human brain. Current Opinion in Neurobiology, 31:164-172, 2015.","title":"The heavy tail of the human brain","authors":"Tjeerd W James A Roberts, Michael Boonstra, Breakspear","year":"2015","venue":"Current Opinion in Neurobiology","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shao et al., 2018] Han Shao, Xiaotian Yu, Irwin King, and Michael R. Lyu. Almost optimal algorithms for lin- ear stochastic bandits with heavy-tailed payoffs. In Ad- vances in Neural Information Processing Systems 32, pages 8430-8439, 2018.","title":"Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs","authors":"Han Shao, Xiaotian Yu, Irwin King, Michael R Lyu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xue et al., 2020] Bo Xue, Guanghui Wang, Yimu Wang, and Lijun Zhang. Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs. arXiv preprint, abs/2004.13465, 2020.","title":"Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs","authors":"Bo Xue, Guanghui Wang, Yimu Wang, Lijun Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Zhou, 2018] Lijun Zhang and Zhi-Hua Zhou. ℓ 1 - regression with heavy-tailed distributions. In Advances in Neural Information Processing Systems 31, pages 1084- 1094, 2018.","title":"ℓ 1regression with heavy-tailed distributions","authors":"Lijun Zhang, Zhi-Hua Zhou","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2016] Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-Hua Zhou. Online stochastic linear optimization under one-bit feedback. In Proceedings of the 33rd International Conference on Machine Learning, pages 392-401, 2016.","title":"Online stochastic linear optimization under one-bit feedback","authors":"Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, Zhi-Hua Zhou","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}