{"type":"ScholarlyArticle","title":"A Survey of Risk-Aware Multi-Armed Bandits","authors":[{"name":"Vincent Y. F. Tan","slug":"vincent-y-f-tan","given":"Vincent Y. F.","family":"Tan","affiliation":"National University of Singapore"},{"name":"Prashanth L.A.","slug":"prashanth-l-a","given":"Prashanth","family":"L.A.","affiliation":"IIT Madras"},{"name":"Krishna Jagannathan","slug":"krishna-jagannathan","given":"Krishna","family":"Jagannathan","affiliation":"IIT Madras"}],"abstract":"In several applications such as clinical trials and financial portfolio optimization, the expected value (or the average reward) does not satisfactorily capture the merits of a drug or a portfolio. In such applications, risk plays a crucial role, and a risk-aware performance measure is preferable, so as to capture losses in the case of adverse events. This survey aims to consolidate and summarise the existing research on risk measures, specifically in the context of multi-armed bandits. We review various risk measures of interest, and comment on their properties. Next, we review existing concentration inequalities for various risk measures. Then, we proceed to defining risk-aware bandit problems, We consider algorithms for the regret minimization setting, where the exploration-exploitation tradeoff manifests, as well as the best arm identification setting, which is a pure exploration problem—both in the context of risk-sensitive measures. We conclude by commenting on persisting challenges and fertile areas for future research.","doi":"10.24963/ijcai.2022/784","url":"https://ijcai.org/proceedings/2022/784","pdf":"https://ijcai.org/proceedings/2022/0784.pdf","markdown":"https://ijcai.org/proceedings/2022/784.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/784","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":784,"pages":{"first":5623,"last":5629},"keywords":["Survey Track: Constraint Satisfaction and Optimization","Survey Track: Machine Learning","Survey Track: -","Survey Track: Uncertainty in AI"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987304","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985447","length_sec":712,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Acerbi, 2002] C. Acerbi. Spectral measures of risk: A co- herent representation of subjective risk aversion. Journal of Banking & Finance, 26(7):1505-1518, 2002.","title":"Spectral measures of risk: A coherent representation of subjective risk aversion","authors":"C Acerbi","year":"2002","venue":"Journal of Banking & Finance","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Goyal, 2012] S. Agrawal and N. Goyal. Anal- ysis of Thompson sampling for the multi-armed bandit problem. In Conference on Learning Theory, pages 39- 1, 2012.","title":"Analysis of Thompson sampling for the multi-armed bandit problem","authors":"S Agrawal, N Goyal","year":"2012","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Artzner et al., 1999] P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk. Mathematical Fi- nance, 9(3):203-228, 1999.","title":"Coherent measures of risk","authors":"P Artzner, F Delbaen, J.-M Eber, D Heath","year":"1999","venue":"Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Audibert et al., 2010] J.-Y. Audibert, S. Bubeck, and R. Munos. Best arm identification in multi-armed ban- dits. In Proceedings of the Conference on Learning The- ory, pages 41-53, 2010.","title":"Best arm identification in multi-armed bandits","authors":"J.-Y Audibert, S Bubeck, R Munos","year":"2010","venue":"Proceedings of the Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"P Auer, N Cesa-Bianchi, P Fischer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baudry et al., 2021] D. Baudry, R. Gautron, E. Kaufmann, and O. Maillard. Optimal thompson sampling strategies for support-aware cvar bandits. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 716-726. PMLR, Jul 2021.","title":"Optimal thompson sampling strategies for support-aware cvar bandits","authors":"D Baudry, R Gautron, E Kaufmann, O Maillard","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bhat and Prashanth, 2019] S. P. Bhat and L. A. Prashanth. Concentration of risk measures: A Wasserstein distance approach. In Advances in Neural Information Processing Systems, pages 11739-11748, 2019.","title":"Concentration of risk measures: A Wasserstein distance approach","authors":"S P Bhat, L A Prashanth","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brown, 2007] D. B. Brown. Large deviations bounds for estimating conditional value-at-risk. Operations Research Letters, 35(6):722-730, 2007.","title":"Large deviations bounds for estimating conditional value-at-risk","authors":"D B Brown","year":"2007","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bubeck et al., 2013] S. Bubeck, T. Wang, and N. Viswanathan. Multiple identifications in multi- armed bandits. In International Conference on Machine Learning, pages 258-265. PMLR, 2013.","title":"Multiple identifications in multiarmed bandits","authors":"S Bubeck, T Wang, N Viswanathan","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cassel et al., 2018] A. Cassel, S. Mannor, and A. Zeevi. A general approach to multi-armed bandits under risk crite- ria. In Proceedings of the 31st Conference On Learning Theory, pages 1295-1306, 2018.","title":"A general approach to multi-armed bandits under risk criteria","authors":"A Cassel, S Mannor, A Zeevi","year":"2018","venue":"Proceedings of the 31st Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chang and Tan, 2021] J. Q. L. Chang and V. Y. F. Tan. A unifying theory of thompson sampling for continuous risk- averse bandits. arXiv preprint arXiv:2108.11345, 2021.","title":"A unifying theory of thompson sampling for continuous riskaverse bandits","authors":"J Q L Chang, V Y F Tan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[David and Shimkin, 2016] Y. David and N. Shimkin. Pure exploration for max-quantile bandits. In Joint European Conference on Machine Learning and Knowledge Discov- ery in Databases, pages 556-571. Springer, 2016.","title":"Pure exploration for max-quantile bandits","authors":"Y David, N Shimkin","year":"2016","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[David et al., 2018] Y. David, B. Szörényi, M. Ghavamzadeh, S. Mannor, and N. Shimkin. PAC bandits with risk constraints. In ISAIM, 2018.","title":"PAC bandits with risk constraints","authors":"Y David, B Szörényi, M Ghavamzadeh, S Mannor, N Shimkin","year":"2018","venue":"ISAIM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Even-Dar et al., 2006] E. Even-Dar, S. Mannor, Y. Man- sour, and S. Mahadevan. Action elimination and stop- ping conditions for the multi-armed bandit and reinforce- ment learning problems. Journal of Machine Learning Re- search, 7(39):1079-1105, 2006.","title":"Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems","authors":"E Even-Dar, S Mannor, Y Mansour, S Mahadevan","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gopalan et al., 2017] A. Gopalan, L. A. Prashanth, M. C. Fu, and S. I. Marcus. Weighted bandits or: How bandits learn distorted values that are not expected. In AAAI Con- ference on Artificial Intelligence, pages 1941-1947, 2017.","title":"Weighted bandits or: How bandits learn distorted values that are not expected","authors":"A Gopalan, L A Prashanth, M C Fu, S I Marcus","year":"2017","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hou et al., 2022] Y. Hou, V. Y. F. Tan, and Z. Zhong. Al- most optimal variance-constrained best arm identification, 2022. arXiv 2201.10142.","title":"Almost optimal variance-constrained best arm identification","authors":"Y Hou, V Y F Tan, Z Zhong","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jie et al., 2018] C. Jie, L. A. Prashanth, M. C. Fu, S. I. Mar- cus, and C. Szepesvári. Stochastic optimization in a cu- mulative prospect theory framework. IEEE Transactions on Automatic Control, 63(9):2867-2882, 2018.","title":"Stochastic optimization in a cumulative prospect theory framework","authors":"C Jie, L A Prashanth, M C Fu, S I Marcus, C Szepesvári","year":"2018","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kagrecha et al., 2019] A. Kagrecha, J. Nair, and K. Jagan- nathan. Distribution oblivious, risk-aware algorithms for multi-armed bandits with unbounded rewards. In Ad- vances in Neural Information Processing Systems, pages 11269-11278, 2019.","title":"Distribution oblivious, risk-aware algorithms for multi-armed bandits with unbounded rewards","authors":"A Kagrecha, J Nair, K Jagannathan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kalyanakrishnan et al., 2012] S. Kalyanakrishnan, A. Tewari, P. Auer, and P. Stone. PAC subset selec- tion in stochastic multi-armed bandits. In Proceedings of the 29th International Conference on Machine Learning, pages 227-234.","title":"PAC subset selection in stochastic multi-armed bandits","authors":"S Kalyanakrishnan, A Tewari, P Auer, P Stone","year":null,"venue":"Proceedings of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"PMLR, 2012.","title":null,"authors":"Pmlr","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kaufmann et al., 2012] E. Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically op- timal finite time analysis. In Proceedings of the Interna- tional Conference on Algorithmic Learning Theory, vol- ume 7568, pages 199-213, 2012.","title":"Thompson sampling: An asymptotically optimal finite time analysis","authors":"E Kaufmann, N Korda, R Munos","year":"2012","venue":"Proceedings of the International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Knight, 1921] F. H. Knight. Risk, Uncertainty and Profit. Houghton Mifflin Company, New York, 1921.","title":"Risk, Uncertainty and Profit","authors":"F H Knight","year":"1921","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kolla et al., 2019] R. K. Kolla, L. A. Prashanth, S. P. Bhat, and K. P. Jagannathan. Concentration bounds for empiri- cal conditional value-at-risk: The unbounded case. Oper- ations Research Letters, 47(1):16-20, 2019.","title":"Concentration bounds for empirical conditional value-at-risk: The unbounded case","authors":"R K Kolla, L A Prashanth, S P Bhat, K P Jagannathan","year":"2019","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lattimore and Szepesvári, 2020] T. Lattimore and C. Szepesvári. Bandit Algorithms. Cambridge Uni- versity Press, 2020.","title":"Bandit Algorithms","authors":"T Lattimore, C Szepesvári","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mannor and Tsitsiklis, 2004] S. Mannor and J. N. Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623-648, 2004.","title":"The sample complexity of exploration in the multi-armed bandit problem","authors":"S Mannor, J N Tsitsiklis","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Markowitz, 1952] H. Markowitz. Portfolio selection. The Journal of Finance, 7(1):77-91, 1952.","title":"Portfolio selection","authors":"H Markowitz","year":"1952","venue":"The Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Massart, 1990] P. Massart. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. The Annals of Probability, 18(3):1269-1283, 1990.","title":"The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality","authors":"P Massart","year":"1990","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pandey et al., 2021] A. K. Pandey, L. A. Prashanth, and S. P. Bhat. Estimation of spectral risk measures. Proceed- ings of the AAAI Conference on Artificial Intelligence, 35(13):12166-12173, May 2021.","title":"Estimation of spectral risk measures","authors":"A K Pandey, L A Prashanth, S P Bhat","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Prashanth and Bhat, 2020] L. A. Prashanth and S. P. Bhat. A Wasserstein distance approach for concentration of em- pirical risk estimates, 2020. arXiv 1902.10709v3.","title":"A Wasserstein distance approach for concentration of empirical risk estimates","authors":"Prashanth, ; ] L A Bhat, S P Prashanth, Bhat","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Prashanth et al., 2016] L. A. Prashanth, J. Cheng, M. C. Fu, S. I. Marcus, and C. Szepesvári. Cumulative prospect the- ory meets reinforcement learning: prediction and control. In International Conference on Machine Learning, pages 1406-1415. PMLR, 2016.","title":"Cumulative prospect theory meets reinforcement learning: prediction and control","authors":"L A Prashanth, J Cheng, M C Fu, S I Marcus, C Szepesvári","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Prashanth et al., 2020] L. A. Prashanth, K. Jagannathan, and R. K. Kolla. Concentration bounds for CVaR estima- tion: The cases of light-tailed and heavy-tailed distribu- tions. In International Conference on Machine Learning, volume 119, pages 5577-5586. PMLR, 2020.","title":"Concentration bounds for CVaR estimation: The cases of light-tailed and heavy-tailed distributions","authors":"L A Prashanth, K Jagannathan, R K Kolla","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Riou and Honda, 2020] C. Riou and J. Honda. Bandit algo- rithms based on thompson sampling for bounded reward distributions. In Proceedings of the 31st International Conference on Algorithmic Learning Theory, volume 117, pages 777-826. PMLR, Feb 2020.","title":"Bandit algorithms based on thompson sampling for bounded reward distributions","authors":"C Riou, J Honda","year":"2020","venue":"Proceedings of the 31st International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Rockafellar and Uryasev, 2000] R. T. Rockafellar and S. Uryasev. Optimization of conditional value-at-risk. Journal of Risk, 2(3):21-41, 2000.","title":"Optimization of conditional value-at-risk","authors":"R T Rockafellar, S Uryasev","year":"2000","venue":"Journal of Risk","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Russo et al., 2018] D. J. Russo, B. Van Roy, A. Kazerouni, I. Osband, and Z. Wen. A Tutorial on Thompson Sam- pling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018.","title":"A Tutorial on Thompson Sampling","authors":"D J Russo, B Van Roy, A Kazerouni, I Osband, Z Wen","year":"2018","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sani et al., 2012] A. Sani, A. Lazaric, and R. Munos. Risk- aversion in multi-armed bandits. In Advances in Neural Information Processing Systems, pages 3275-3283, 2012.","title":"Riskaversion in multi-armed bandits","authors":"A Sani, A Lazaric, R Munos","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Sharpe, 1966] W. F. Sharpe. Mutual fund performance. The Journal of Business, 39(1):119-138, 1966.","title":"Mutual fund performance","authors":"W F Sharpe","year":"1966","venue":"The Journal of Business","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Szorenyi et al., 2015] B. Szorenyi, R. Busa-Fekete, P. Weng, and E. Hüllermeier. Qualitative multi-armed bandits: A quantile-based approach. In International Conference on Machine Learning, pages 1660-1668.","title":"Qualitative multi-armed bandits: A quantile-based approach","authors":"B Szorenyi, R Busa-Fekete, P Weng, E Hüllermeier","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"PMLR, 2015.","title":null,"authors":"Pmlr","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Thomas and Learned-Miller, 2019] P. Thomas and E. Learned-Miller. Concentration inequalities for conditional value at risk. In International Conference on Machine Learning, pages 6225-6233. PMLR, 2019.","title":"Concentration inequalities for conditional value at risk","authors":"P Thomas, E Learned-Miller","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Thompson, 1933] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"W R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tversky and Kahneman, 1992] Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative rep- resentation of uncertainty. Journal of Risk and Uncer- tainty, 5(4):297-323, 1992.","title":"Advances in prospect theory: Cumulative representation of uncertainty","authors":"Kahneman Tversky, Amos Tversky, Daniel Kahneman","year":"1992","venue":"Journal of Risk and Uncertainty","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Vakili and Zhao, 2015] S Vakili and Q. Zhao. Mean- variance and value at risk in multi-armed bandit problems. In Annual Allerton Conference on Communication, Con- trol, and Computing (Allerton), pages 1330-1335, 2015.","title":"Meanvariance and value at risk in multi-armed bandit problems","authors":"Zhao ; ] S Vakili, Q Vakili, Zhao","year":"2015","venue":"Annual Allerton Conference on Communication, Control, and Computing","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wainwright, 2019] M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint, volume 48. Cam- bridge University Press, 2019.","title":"High-Dimensional Statistics: A Non-Asymptotic Viewpoint","authors":"M J Wainwright","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang and Gao, 2010] Y. Wang and F. Gao. Deviation in- equalities for an estimator of the conditional value-at-risk. Operations Research Letters, 38(3):236-239, 2010.","title":"Deviation inequalities for an estimator of the conditional value-at-risk","authors":"Y Wang, F Gao","year":"2010","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhang and Ong, 2021] M. Zhang and C. S. Ong. Quan- tile bandits for best arms identification. In International Conference on Machine Learning, pages 12513-12523.","title":"Quantile bandits for best arms identification","authors":"M Zhang, C S Ong","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhu and Tan, 2020] Q. Zhu and V. Y. F. Tan. Thompson sampling algorithms for mean-variance bandits. In Inter- national Conference on Machine Learning, pages 2645- 2654. PMLR, 2020.","title":"Thompson sampling algorithms for mean-variance bandits","authors":"Q Zhu, V Y F Tan","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}