{"type":"ScholarlyArticle","title":"Thompson Sampling on Symmetric Alpha-Stable Bandits","authors":[{"name":"Abhimanyu Dubey","slug":"abhimanyu-dubey","given":"Abhimanyu","family":"Dubey","affiliation":"Massachusetts Institute of Technology"},{"name":"Alex `Sandy' Pentland","slug":"alex-sandy-pentland","given":"Alex `Sandy'","family":"Pentland","affiliation":"Massachusetts Institute of Technology"}],"abstract":"Thompson Sampling provides an efficient technique to introduce prior knowledge in the multi-armed bandit problem, along with providing remarkable empirical performance. In this paper, we revisit the Thompson Sampling algorithm under rewards drawn from symmetric alpha-stable distributions, which are a class of heavy-tailed probability distributions utilized in finance and economics, in problems such as modeling stock prices and human behavior. We present an efficient framework for posterior inference, which leads to two algorithms for Thompson Sampling in this setting. We prove finite-time regret bounds for both algorithms, and demonstrate through a series of experiments the stronger performance of Thompson Sampling in this setting. With our results, we provide an exposition of symmetric alpha-stable distributions in sequential decision-making, and enable sequential Bayesian inference in applications from diverse fields in finance and complex systems that operate on heavy-tailed features.","doi":"10.24963/ijcai.2019/792","url":"https://ijcai.org/proceedings/2019/792","pdf":"https://ijcai.org/proceedings/2019/0792.pdf","markdown":"https://ijcai.org/proceedings/2019/792.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/792","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Uncertainty in AI","track":"Main track","number":792,"pages":{"first":5715,"last":5721},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Active Learning","Machine Learning: Learning Theory","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2012] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi- armed bandit problem. In Conference on Learning Theory, pages 39-1, 2012.","title":"Analysis of thompson sampling for the multiarmed bandit problem","authors":"Shipra Agrawal, Navin Goyal","year":"2012","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Further optimal regret bounds for thompson sampling. In Artificial Intelligence and Statistics, pages 99-107, 2013.","title":"Further optimal regret bounds for thompson sampling","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andrews and Mallows, 1974] David F Andrews and Colin L Mallows. Scale mixtures of normal distribu- tions. Journal of the Royal Statistical Society: Series B (Methodological), 36(1):99-102, 1974.","title":"Scale mixtures of normal distributions","authors":"F David, Colin L Andrews, Mallows","year":"1974","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Borak et al., 2005] Szymon Borak, Wolfgang Härdle, and Rafał Weron. Stable distributions. In Statistical tools for finance and insurance, pages 21-44. Springer, 2005.","title":"Stable distributions","authors":"Szymon Borak, Wolfgang Härdle, Rafał Weron","year":"2005","venue":"Statistical tools for finance and insurance","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bradley and Taqqu, 2003] Brendan O Bradley and Murad S Taqqu. Financial risk and heavy tails. In Handbook of heavy tailed distributions in finance, pages 35-103. Else- vier, 2003.","title":"Financial risk and heavy tails","authors":"O Brendan, Murad S Bradley, Taqqu","year":"2003","venue":"Handbook of heavy tailed distributions in finance","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck et al., 2013] Sébastien Bubeck, Nicolo Cesa- Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Transactions on Information Theory, 59(11):7711-7717, 2013.","title":"Bandits with heavy tail","authors":"Sébastien Bubeck, Nicolo Cesa-Bianchi, Gábor Lugosi","year":"2013","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cappé et al., 2013] Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz, et al. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, 41(3):1516-1541, 2013.","title":"Kullback-leibler upper confidence bounds for optimal sequential allocation","authors":"Cappé","year":"2013","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Carr and Wu, 2003] Peter Carr and Liuren Wu. The finite moment log stable process and option pricing. The journal of finance, 58(2):753-777, 2003.","title":"The finite moment log stable process and option pricing","authors":"Wu Carr, Peter Carr, Liuren Wu","year":"2003","venue":"The journal of finance","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chambers et al., 1976] John M Chambers, Colin L Mal- lows, and BW Stuck. A method for simulating stable ran- dom variables. Journal of the american statistical associ- ation, 71(354):340-344, 1976.","title":"A method for simulating stable random variables","authors":"Colin L John M Chambers, Mallows, Stuck","year":"1976","venue":"Journal of the american statistical association","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pages 2249- 2257, 2011.","title":"An empirical evaluation of thompson sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Frain, 2009] John C Frain. Studies on the Application of the Alpha-stable Distribution in Economics. 2009.","title":"Studies on the Application of the Alpha-stable Distribution in Economics","authors":"John C Frain","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Godsill and Kuruoglu, 1999] Simon Godsill and Ercan E Kuruoglu. Bayesian inference for time series with heavy- tailed symmetric α-stable noise processes. 1999.","title":"Bayesian inference for time series with heavytailed symmetric α-stable noise processes","authors":"Simon Godsill, Ercan E Kuruoglu","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Korda et al., 2013] Nathaniel Korda, Emilie Kaufmann, and Remi Munos. Thompson sampling for 1-dimensional ex- ponential family bandits. In Advances in Neural Informa- tion Processing Systems, pages 1448-1456, 2013.","title":"Thompson sampling for 1-dimensional exponential family bandits","authors":"Korda","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar and Tomkins, 2010] Ravi Kumar and Andrew Tomkins. A characterization of online browsing behavior. In Proceedings of the 19th international conference on World wide web, pages 561-570. ACM, 2010.","title":"A characterization of online browsing behavior","authors":"Tomkins Kumar, Ravi Kumar, Andrew Tomkins","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lévy, 1925] P Lévy. Calcul des probabilités, vol. 9. Gauthier-Villars Paris, 1925.","title":"Calcul des probabilités","authors":"Lévy","year":"1925","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mahanti et al., 2013] Aniket Mahanti, Niklas Carlsson, Anirban Mahanti, Martin Arlitt, and Carey Williamson. A tale of the tails: Power-laws in internet measurements. IEEE Network, 27(1):59-64, 2013.","title":"A tale of the tails: Power-laws in internet measurements","authors":"Aniket Mahanti, Niklas Carlsson, Anirban Mahanti, Martin Arlitt, Carey Williamson","year":"2013","venue":"IEEE Network","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Medina and Yang, 2016] Andres Munoz Medina and Scott Yang. No-regret algorithms for heavy-tailed linear bandits. In International Conference on Machine Learning, pages 1642-1650, 2016.","title":"No-regret algorithms for heavy-tailed linear bandits","authors":"Andres Munoz, Medina, Scott Yang","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Russo and Van Roy, 2014] Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4):1221-1243, 2014.","title":"Learning to optimize via posterior sampling","authors":"Van Russo, Daniel Roy, Benjamin Russo, Roy Van","year":"2014","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shao and Nikias, 1993] Min Shao and Chrysostomos L Nikias. Signal processing with fractional lower order mo- ments: stable processes and their applications. Proceed- ings of the IEEE, 81(7):986-1010, 1993.","title":"Signal processing with fractional lower order moments: stable processes and their applications","authors":"Nikias Shao, Min Shao, Chrysostomos L Nikias","year":"1993","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shao et al., 2018] Han Shao, Xiaotian Yu, Irwin King, and Michael R Lyu. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. In Advances in Neural Information Processing Systems, pages 8430- 8439, 2018.","title":"Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs","authors":"Han Shao, Xiaotian Yu, Irwin King, Michael R Lyu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vakili et al., 2013] Sattar Vakili, Keqin Liu, and Qing Zhao. Deterministic sequencing of exploration and exploitation for multi-armed bandit problems. IEEE Journal of Se- lected Topics in Signal Processing, 7(5):759-767, 2013.","title":"Deterministic sequencing of exploration and exploitation for multi-armed bandit problems","authors":"Sattar Vakili, Keqin Liu, Qing Zhao","year":"2013","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2018] Xiaotian Yu, Han Shao, Michael R Lyu, and Irwin King. Pure exploration of multi-armed bandits with heavy-tailed payoffs. In Conference on Uncertainty in Ar- tificial Intelligence, pages 937-946, 2018.","title":"Pure exploration of multi-armed bandits with heavy-tailed payoffs","authors":"Xiaotian Yu, Han Shao, Irwin Michael R Lyu, King","year":"2018","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}