{"type":"ScholarlyArticle","title":"Multi-Player Multi-Armed Bandits with Finite Shareable Resources Arms: Learning Algorithms & Applications","authors":[{"name":"Xuchuang Wang","slug":"xuchuang-wang","given":"Xuchuang","family":"Wang","affiliation":"Department of Computer Science & Engineering , The Chinese University of Hong Kong"},{"name":"Hong Xie","slug":"hong-xie","given":"Hong","family":"Xie","affiliation":"College of Computer Science , Chongqing University , China"},{"name":"John C. S. Lui","slug":"john-c-s-lui","given":"John C.S.","family":"Lui","affiliation":"Department of Computer Science & Engineering , The Chinese University of Hong Kong"}],"abstract":"Multi-player multi-armed bandits (MMAB) study how decentralized players cooperatively play the same multi-armed bandit so as to maximize their total cumulative rewards. Existing MMAB models mostly assume when more than one player pulls the same arm, they either have a collision and obtain zero rewards or have no collision and gain independent rewards, both of which are usually too restrictive in practical scenarios. In this paper, we propose an MMAB with shareable resources as an extension of the collision and non-collision settings. Each shareable arm has finite shareable resources and a “per-load” reward random variable, both of which are unknown to players. The reward from a shareable arm is equal to the “per-load” reward multiplied by the minimum between the number of players pulling the arm and the arm’s maximal shareable resources. We consider two types of feedback: sharing demand information (SDI) and sharing demand awareness (SDA), each of which provides different signals of resource sharing. We design the DPE-SDI and SIC-SDA algorithms to address the shareable arm problem under these two cases of feedback respectively and prove that both algorithms have logarithmic regrets that are tight in the number of rounds. We conduct simulations to validate both algorithms’ performance and show their utilities in wireless networking and edge computing.","doi":"10.24963/ijcai.2022/491","url":"https://ijcai.org/proceedings/2022/491","pdf":"https://ijcai.org/proceedings/2022/0491.pdf","markdown":"https://ijcai.org/proceedings/2022/491.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/491","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":491,"pages":{"first":3537,"last":3543},"keywords":["Machine Learning: Online Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986568","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984711","length_sec":338,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anandkumar et al., 2011] Animashree Anandkumar, Nithin Michael, Ao Kevin Tang, and Ananthram Swami. Dis- tributed algorithms for learning and cognitive medium ac- cess with logarithmic regret. IEEE Journal on Selected Areas in Communications, 29(4):731-745, 2011.","title":"Distributed algorithms for learning and cognitive medium access with logarithmic regret","authors":"Anandkumar","year":"2011","venue":"IEEE Journal on Selected Areas in Communications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anantharam et al., 1987] Venkatachalam Anantharam, Pravin Varaiya, and Jean Walrand. Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards. IEEE Transactions on Automatic Control, 32(11):968-976, 1987.","title":"Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards","authors":"Pravin Venkatachalam Anantharam, Jean Varaiya, Walrand","year":"1987","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Besson and Kaufmann, 2018] Lilian Besson and Emilie Kaufmann. Multi-player bandits revisited. In Algorith- mic Learning Theory, pages 56-92. PMLR, 2018.","title":"Multi-player bandits revisited","authors":"Lilian Besson, Emilie Kaufmann","year":"2018","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bistritz and Leshem, 2018] Ilai Bistritz and Amir Leshem. Distributed multi-player bandits-a game of thrones ap- proach. In Advances in Neural Information Processing Systems, pages 7222-7232, 2018.","title":"Distributed multi-player bandits-a game of thrones approach","authors":"Ilai Bistritz, Amir Leshem","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boursier and Perchet, 2019] Etienne Boursier and Vianney Perchet. Sic-mmab: Synchronisation involves communi- cation in multiplayer multi-armed bandits. In Advances in Neural Information Processing Systems, volume 32, pages 12071-12080, 2019.","title":"Sic-mmab: Synchronisation involves communication in multiplayer multi-armed bandits","authors":"Perchet Boursier, Etienne Boursier, Vianney Perchet","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boursier and Perchet, 2020] Etienne Boursier and Vianney Perchet. Selfish robustness and equilibria in multi-player bandits. In Conference on Learning Theory, pages 530-581.","title":"Etienne Boursier and Vianney Perchet. Selfish robustness and equilibria in multi-player bandits","authors":"Perchet Boursier","year":"2020","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cappé et al., 2013] Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, pages 1516-1541, 2013.","title":"Kullback-leibler upper confidence bounds for optimal sequential allocation","authors":"Cappé","year":"2013","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Combes et al., 2015] Richard Combes, Stefan Magureanu, Alexandre Proutiere, and Cyrille Laroche. Learning to rank: Regret lower bounds and efficient algorithms. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pages 231-244, 2015.","title":"Learning to rank: Regret lower bounds and efficient algorithms","authors":"Richard Combes, Stefan Magureanu, Alexandre Proutiere, Cyrille Laroche","year":"2015","venue":"Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Drolia et al., 2017] Utsav Drolia, Katherine Guo, and Priya Narasimhan. Precog: Prefetching for image recognition applications at the edge. In Proceedings of the Second ACM/IEEE Symposium on Edge Computing, pages 1-13, 2017.","title":"Precog: Prefetching for image recognition applications at the edge","authors":"Utsav Drolia, Katherine Guo, Priya Narasimhan","year":"2017","venue":"Proceedings of the Second ACM/IEEE Symposium on Edge Computing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jouini et al., 2009] Wassim Jouini, Damien Ernst, Christophe Moy, and Jacques Palicot. Multi-armed bandit based policies for cognitive radio's decision making issues. In 2009 3rd International Conference on Signals, Circuits and Systems (SCS), pages 1-6. IEEE, 2009.","title":"Multi-armed bandit based policies for cognitive radio's decision making issues","authors":"Wassim Jouini, Damien Ernst, Christophe Moy, Jacques Palicot","year":"2009","venue":"3rd International Conference on Signals, Circuits and Systems (SCS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kalathil et al., 2014] Dileep Kalathil, Naumaan Nayyar, and Rahul Jain. Decentralized learning for multiplayer multi- armed bandits. IEEE Transactions on Information Theory, 60(4):2331-2345, 2014.","title":"Decentralized learning for multiplayer multiarmed bandits","authors":"Dileep Kalathil, Naumaan Nayyar, Rahul Jain","year":"2014","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lai and Robbins, 1985] Tze Leung Lai and Herbert Rob- bins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in applied mathematics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Landgren et al., 2016] Peter Landgren, Vaibhav Srivastava, and Naomi Ehrich Leonard. Distributed cooperative decision-making in multiarmed bandits: Frequentist and bayesian algorithms. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 167-172. IEEE, 2016.","title":"Distributed cooperative decision-making in multiarmed bandits: Frequentist and bayesian algorithms","authors":"Landgren","year":"2016","venue":"IEEE 55th Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu and Zhao, 2010] Keqin Liu and Qing Zhao. Decentral- ized multi-armed bandit with multiple distributed players. In 2010 Information Theory and Applications Workshop (ITA), pages 1-10. IEEE, 2010.","title":"Decentralized multi-armed bandit with multiple distributed players","authors":"Zhao ; ] Keqin Liu, Qing Liu, Zhao","year":"2010","venue":"2010 Information Theory and Applications Workshop (ITA)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2020] Lydia T Liu, Horia Mania, and Michael Jor- dan. Competing bandits in matching markets. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 1618-1628. PMLR, 2020.","title":"Competing bandits in matching markets","authors":"Lydia T Liu, Horia Mania, Michael Jordan","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Magesh and Veeravalli, 2021] Akshayaa Magesh and Venu- gopal V Veeravalli. Decentralized heterogeneous multi- player multi-armed bandits with non-zero rewards on colli- sions. IEEE Transactions on Information Theory, 2021.","title":"Decentralized heterogeneous multiplayer multi-armed bandits with non-zero rewards on collisions","authors":"Veeravalli ; V Magesh, Veeravalli","year":"2021","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Martínez-Rubio et al., 2019] David Martínez-Rubio, Varun Kanade, and Patrick Rebeschini. Decentralized coopera- tive stochastic bandits. Advances in Neural Information Processing Systems, 32:4529-4540, 2019.","title":"Decentralized cooperative stochastic bandits","authors":"David Martínez-Rubio, Varun Kanade, Patrick Rebeschini","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mehrabian et al., 2020] Abbas Mehrabian, Etienne Boursier, Emilie Kaufmann, and Vianney Perchet. A practical algo- rithm for multiplayer bandits when arm means vary among players. In International Conference on Artificial Intelli- gence and Statistics, pages 1211-1221. PMLR, 2020.","title":"A practical algorithm for multiplayer bandits when arm means vary among players","authors":"Mehrabian","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Narayanan et al., 2020] Arvind Narayanan, Eman Ramadan, Jason Carpenter, Qingxu Liu, Yu Liu, Feng Qian, and Zhi- Li Zhang. A first look at commercial 5g performance on smartphones. In Proceedings of The Web Conference 2020, pages 894-905, 2020.","title":"A first look at commercial 5g performance on smartphones","authors":"Arvind Narayanan, Eman Ramadan, Jason Carpenter, Qingxu Liu, Yu Liu, Feng Qian, Zhi-Li Zhang","year":"2020","venue":"Proceedings of The Web Conference 2020","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rosenski et al., 2016] Jonathan Rosenski, Ohad Shamir, and Liran Szlak. Multi-player bandits-a musical chairs ap- proach. In International Conference on Machine Learning, pages 155-163. PMLR, 2016.","title":"Multi-player bandits-a musical chairs approach","authors":"Jonathan Rosenski, Ohad Shamir, Liran Szlak","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sankararaman et al., 2021] Abishek Sankararaman, Soumya Basu, and Karthik Abinav Sankararaman. Dominate or delete: Decentralized competing bandits in serial dictator- ship. In International Conference on Artificial Intelligence and Statistics, pages 1252-1260. PMLR, 2021.","title":"Dominate or delete: Decentralized competing bandits in serial dictatorship","authors":"Abishek Sankararaman, Soumya Basu, Karthik Abinav Sankararaman","year":"2021","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2021] Chengshuai Shi, Wei Xiong, Cong Shen, and Jing Yang. Heterogeneous multi-player multi-armed bandits: Closing the gap and generalization. Advances in Neural Information Processing Systems, 34, 2021.","title":"Heterogeneous multi-player multi-armed bandits: Closing the gap and generalization","authors":"Chengshuai Shi, Wei Xiong, Cong Shen, Jing Yang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020] Po-An Wang, Alexandre Proutiere, Kaito Ariu, Yassir Jedra, and Alessio Russo. Optimal algorithms for multiplayer multi-armed bandits. In International Con- ference on Artificial Intelligence and Statistics, pages 4120- 4129. PMLR, 2020.","title":"Optimal algorithms for multiplayer multi-armed bandits","authors":"Wang","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}