{"type":"ScholarlyArticle","title":"Bandits with Concave Aggregated Reward","authors":[{"name":"Yingqi Yu","slug":"yingqi-yu","given":"Yingqi","family":"Yu","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Sijia Zhang","slug":"sijia-zhang","given":"Sijia","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Shaoang Li","slug":"shaoang-li","given":"Shaoang","family":"Li","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Lan Zhang","slug":"lan-zhang","given":"Lan","family":"Zhang","affiliation":"University of Science and Technology of China , Hefei , China; Institute of Artificial Intelligence , Hefei Comprehensive National Science Center"},{"name":"Wei Xie","slug":"wei-xie","given":"Wei","family":"Xie","affiliation":"University of Science and Technology of China , Hefei , China"},{"name":"Xiang-Yang Li","slug":"xiang-yang-li","given":"Xiang-Yang","family":"Li","affiliation":"University of Science and Technology of China , Hefei , China"}],"abstract":"Multi-armed bandit is a simple but powerful algorithmic framework, and many effective algorithms have been proposed for various online models.\r\nIn numerous applications, the decision-maker faces diminishing marginal utility. With non-linear aggregations, those algorithms often have poor regret bounds. Motivated by this, we study a bandit problem with diminishing marginal utility, which we termed the bandits with concave aggregated reward(BCAR). \r\nTo tackle this problem, we propose two algorithms SW-BCAR and SWUCB-BCAR. Through theoretical analysis, we establish the effectiveness of these algorithms in addressing the BCAR issue. Extensive simulations demonstrate that our algorithms achieve better results than the most advanced bandit algorithms.","doi":"10.24963/ijcai.2024/597","url":"https://ijcai.org/proceedings/2024/597","pdf":"https://ijcai.org/proceedings/2024/0597.pdf","markdown":"https://ijcai.org/proceedings/2024/597.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/597","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":597,"pages":{"first":5398,"last":5406},"keywords":["Machine Learning: ML: Multi-armed bandits"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2009] Deepak Agarwal, Bee-Chung Chen, and Pradheep Elango. Spatio-temporal models for esti- mating click-through rate. In Proceedings of the 18th in- ternational conference on World wide web, pages 21-30, 2009.","title":"Spatio-temporal models for estimating click-through rate","authors":"Deepak Agarwal, Bee-Chung Chen, Pradheep Elango","year":"2009","venue":"Proceedings of the 18th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Devanur, 2014] Shipra Agrawal and Nikhil R Devanur. Bandits with concave rewards and convex knap- sacks. In Proceedings of the fifteenth ACM conference on Economics and computation, pages 989-1006, 2014.","title":"Bandits with concave rewards and convex knapsacks","authors":"Shipra Agrawal, Nikhil R Devanur","year":"2014","venue":"Proceedings of the fifteenth ACM conference on Economics and computation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agrawal et al., 1988] Rajeev Agrawal, MV Hedge, and De- mosthenis Teneketzis. Asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching cost. IEEE Transactions on Automatic Control, 33(10):899-906, 1988.","title":"Asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching cost","authors":"Rajeev Agrawal, Demosthenis Hedge, Teneketzis","year":"1988","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002a] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002b] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Auer","year":"2002","venue":"SIAM journal on computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Badanidiyuru et al., 2012] Ashwinkumar Badanidiyuru, Robert Kleinberg, and Yaron Singer. Learning on a budget: posted price mechanisms for online procurement. In Proceedings of the 13th ACM conference on electronic commerce, pages 128-145, 2012.","title":"Learning on a budget: posted price mechanisms for online procurement","authors":"Badanidiyuru","year":"2012","venue":"Proceedings of the 13th ACM conference on electronic commerce","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Badanidiyuru et al., 2013] Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 207-216. IEEE, 2013.","title":"Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks","authors":"Badanidiyuru","year":"2013","venue":"IEEE 54th Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Besbes et al., 2014] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non- stationary rewards. Advances in neural information pro- cessing systems, 27, 2014.","title":"Stochastic multi-armed-bandit problem with nonstationary rewards","authors":"Omar Besbes, Yonatan Gur, Assaf Zeevi","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bubeck and Cesa-Bianchi, 2012] Sébastien Bubeck and Ni- colo Cesa-Bianchi. Regret analysis of stochastic and non- stochastic multi-armed bandit problems. arXiv preprint arXiv:1204.5721, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Cesa-Bianchi ; Nicolo Bubeck, Cesa-Bianchi","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garivier and Moulines, 2008] Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems. arXiv preprint arXiv:0805.3415, 2008.","title":"Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems","authors":"Moulines Garivier","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gittins, 1979] John C Gittins. Bandit processes and dy- namic allocation indices. Journal of the Royal Statistical Society: Series B (Methodological), 41(2):148-164, 1979.","title":"Bandit processes and dynamic allocation indices","authors":"C John, Gittins","year":"1979","venue":"Journal of the Royal Statistical Society: Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hazan and Kale, 2011] Elad Hazan and Satyen Kale. Better algorithms for benign bandits. Journal of Machine Learn- ing Research, 12(4), 2011.","title":"Better algorithms for benign bandits","authors":"Kale ; ] Elad Hazan, Satyen Hazan, Kale","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lai et al., 1985] Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules. Ad- vances in applied mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in applied mathematics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levine et al., 2017] Nir Levine, Koby Crammer, and Shie Mannor. Rotting bandits. arXiv preprint arXiv:1702.07274, 2017.","title":"Rotting bandits","authors":"Nir Levine, Koby Crammer, Shie Mannor","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2011] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 297-306, 2011.","title":"Unbiased offline evaluation of contextualbandit-based news article recommendation algorithms","authors":"Lihong Li, Wei Chu, John Langford, Xuanhui Wang","year":"2011","venue":"Proceedings of the fourth ACM international conference on Web search and data mining","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2023] Shaoang Li, Lan Zhang, Yingqi Yu, and Xi- angyang Li. Optimal arms identification with knapsacks. In International Conference on Machine Learning, pages 20529-20555. PMLR, 2023.","title":"Optimal arms identification with knapsacks","authors":"Shaoang Li, Lan Zhang, Yingqi Yu, Xiangyang Li","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lykouris et al., 2020] Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Bandits with adversarial scaling. In International Conference on Machine Learning, pages 6511-6521. PMLR, 2020.","title":"Bandits with adversarial scaling","authors":"Thodoris Lykouris, Vahab Mirrokni, Renato Paes, Leme","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maillard et al., 2011] Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. A finite-time analysis of multi-armed bandits problems with kullback-leibler diver- gences. In Proceedings of the 24th annual Conference On Learning Theory, pages 497-514. JMLR Workshop and Conference Proceedings, 2011.","title":"A finite-time analysis of multi-armed bandits problems with kullback-leibler divergences","authors":"Maillard","year":"2011","venue":"Proceedings of the 24th annual Conference On Learning Theory","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Metelli et al., 2022] Alberto Maria Metelli, Francesco Trovo, Matteo Pirola, and Marcello Restelli. Stochastic rising bandits. In International Conference on Machine Learning, pages 15421-15457. PMLR, 2022.","title":"Stochastic rising bandits","authors":"Alberto Maria Metelli, Francesco Trovo, Matteo Pirola, Marcello Restelli","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mohri and Yang, 2016] Mehryar Mohri and Scott Yang. Optimistic bandit convex optimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2297-2305, 2016.","title":"Optimistic bandit convex optimization","authors":"Mehryar Mohri, Scott Yang","year":"2016","venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russac et al., 2019] Yoan Russac, Claire Vernade, and Olivier Cappé. Weighted linear bandits for non-stationary environments. arXiv preprint arXiv:1909.09146, 2019.","title":"Weighted linear bandits for non-stationary environments","authors":"Russac","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saha and Tewari, 2011] Ankan Saha and Ambuj Tewari. Improved regret guarantees for online smooth convex op- timization with bandit feedback. In Proceedings of the Fourteenth International Conference on Artificial Intelli- gence and Statistics, pages 636-642. JMLR Workshop and Conference Proceedings, 2011.","title":"Improved regret guarantees for online smooth convex optimization with bandit feedback","authors":"Tewari ; ] Ankan Saha, Ambuj Saha, Tewari","year":"2011","venue":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Scaman et al., 2017] Kevin Scaman, Francis Bach, Sébastien Bubeck, Yin Tat Lee, and Laurent Mas- soulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In interna- tional conference on machine learning, pages 3027-3036.","title":"Optimal algorithms for smooth and strongly convex distributed optimization in networks","authors":"Kevin Scaman, Francis Bach, Sébastien Bubeck, Yin Tat Lee, Laurent Massoulié","year":null,"venue":"international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"PMLR, 2017.","title":null,"authors":"Pmlr","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Slivkins and Vaughan, 2014] Aleksandrs Slivkins and Jen- nifer Wortman Vaughan. Online decision making in crowdsourcing markets: Theoretical challenges. ACM SIGecom Exchanges, 12(2):4-23, 2014.","title":"Online decision making in crowdsourcing markets: Theoretical challenges","authors":"Vaughan ; Slivkins, Aleksandrs Slivkins, Jennifer Wortman Vaughan","year":"2014","venue":"ACM SIGecom Exchanges","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tekin and Liu, 2012] Cem Tekin and Mingyan Liu. Online learning of rested and restless bandits. IEEE Transactions on Information Theory, 58(8):5588-5611, 2012.","title":"Online learning of rested and restless bandits","authors":"Liu Tekin, Cem Tekin, Mingyan Liu","year":"2012","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Trovò et al., 2020] Francesco Trovò, Stefano Paladino, Marcello Restelli, and Nicola Gatti. Sliding-window thompson sampling for non-stationary settings. J. Artif. Intell. Res., 68:311-364, 2020.","title":"Sliding-window thompson sampling for non-stationary settings","authors":"Francesco Trovò, Stefano Paladino, Marcello Restelli, Nicola Gatti","year":"2020","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[van der Hoeven et al., 2020] Dirk van der Hoeven, Ashok Cutkosky, and Haipeng Luo. Comparator-adaptive convex bandits. arXiv preprint arXiv:2007.08448, 2020.","title":"Comparator-adaptive convex bandits","authors":"Van Der Hoeven","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Whittle, 1988] Peter Whittle. Restless bandits: Activity al- location in a changing world. Journal of applied probabil- ity, 25(A):287-298, 1988.","title":"Restless bandits: Activity allocation in a changing world","authors":"Peter Whittle, Whittle","year":"1988","venue":"Journal of applied probability","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2020] Huanle Xu, Yang Liu, Wing Cheong Lau, and Rui Li. Combinatorial multi-armed bandits with con- cave rewards and fairness constraints. In IJCAI, pages 2554-2560, 2020.","title":"Combinatorial multi-armed bandits with concave rewards and fairness constraints","authors":"Huanle Xu, Yang Liu, Cheong Wing, Rui Lau, Li","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/354"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}