{"type":"ScholarlyArticle","title":"Unifying the Stochastic and the Adversarial Bandits with Knapsack","authors":[{"name":"Anshuka Rangi","slug":"anshuka-rangi","given":"Anshuka","family":"Rangi","affiliation":"University of California , San Diego ,"},{"name":"Massimo Franceschetti","slug":"massimo-franceschetti","given":"Massimo","family":"Franceschetti","affiliation":"University of California , San Diego ,"},{"name":"Long Tran-Thanh","slug":"long-tran-thanh","given":"Long","family":"Tran-Thanh","affiliation":"University of Southampton"}],"abstract":"This work investigates the adversarial Bandits with Knapsack (BwK) learning problem, where a player repeatedly chooses to perform an action, pays the corresponding cost of the action, and receives a reward associated with the action. The player is constrained by the maximum budget that can be spent to perform the actions, and the rewards and the costs of these actions are assigned by an adversary. This setting is studied in terms of expected regret, defined as the difference between the total expected rewards per unit cost corresponding the best fixed action and the total expected rewards per unit cost of the learning algorithm. We  propose a novel algorithm EXP3.BwK   and show that the expected regret of the algorithm is order optimal in the budget. We then propose another algorithm EXP3++.BwK, \r\nwhich is order optimal in the adversarial BwK setting, and incurs an almost optimal expected regret in the  stochastic BwK setting  where the rewards and the costs are drawn from unknown underlying distributions. These results are then extended to a more general online learning setting, by designing another algorithm EXP3++.LwK and providing its performance guarantees. Finally, we investigate the scenario where the costs of the actions are large and comparable to the budget. \r\nWe show that for the adversarial setting, the achievable regret bounds scale at least linearly with the maximum cost for any learning algorithm, and are significantly worse in comparison to the case of having costs bounded  by a constant, which is a common assumption in the  BwK literature.","doi":"10.24963/ijcai.2019/459","url":"https://ijcai.org/proceedings/2019/459","pdf":"https://ijcai.org/proceedings/2019/0459.pdf","markdown":"https://ijcai.org/proceedings/2019/459.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/459","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":459,"pages":{"first":3311,"last":3317},"keywords":["Machine Learning: Learning Theory","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Devanur, 2014] Shipra Agrawal and Nikhil R Devanur. Bandits with concave rewards and convex knap- sacks. In Proceedings of the fifteenth ACM conference on Economics and computation, pages 989-1006, 2014.","title":"Bandits with concave rewards and convex knapsacks","authors":"Shipra Agrawal, Nikhil R Devanur","year":"2014","venue":"Proceedings of the fifteenth ACM conference on Economics and computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Devanur, 2016] Shipra Agrawal and Nikhil Devanur. Linear contextual bandits with knapsacks. In Ad- vances in Neural Information Processing Systems, pages 3450-3458, 2016.","title":"Linear contextual bandits with knapsacks","authors":"Shipra Agrawal, Nikhil Devanur","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alon et al., 2015] Noga Alon, Nicolo Cesa-Bianchi, Ofer Dekel, and Tomer Koren. Online learning with feedback graphs: Beyond bandits. In JMLR WORKSHOP AND CONFERENCE PROCEEDINGS, volume 40. Microtome Publishing, 2015.","title":"Online learning with feedback graphs: Beyond bandits","authors":"Nicolo Noga Alon, Ofer Cesa-Bianchi, Tomer Dekel, Koren","year":"2015","venue":"JMLR WORKSHOP AND CONFERENCE PROCEEDINGS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer and Chiang, 2016] Peter Auer and Chao-Kai Chiang. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Conference on Learning Theory, pages 116-120, 2016.","title":"An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits","authors":"Chiang Auer, Peter Auer, Chao-Kai Chiang","year":"2016","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.","title":"The nonstochastic mul-tiarmed bandit problem","authors":"Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, Robert E Schapire","year":"2002","venue":"SIAM journal on computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Badanidiyuru et al., 2013] Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In Foundations of Computer Science (FOCS), 2013 IEEE 54th Annual Symposium on, pages 207-216. IEEE, 2013.","title":"Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks","authors":"Badanidiyuru","year":"2013","venue":"Foundations of Computer Science (FOCS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck and Slivkins, 2012] Sébastien Bubeck and Alek- sandrs Slivkins. The best of both worlds: stochastic and adversarial bandits. In Conference on Learning Theory, pages 42-1, 2012.","title":"The best of both worlds: stochastic and adversarial bandits","authors":"Slivkins Bubeck, Sébastien Bubeck, Aleksandrs Slivkins","year":"2012","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2013] Wenkui Ding, Tao Qin, Xu-Dong Zhang, and Tie-Yan Liu. Multi-armed bandit with budget con- straint and variable costs. In Twenty-Seventh AAAI Con- ference on Artificial Intelligence, 2013.","title":"Multi-armed bandit with budget constraint and variable costs","authors":"Wenkui Ding, Tao Qin, Xu-Dong Zhang, Tie-Yan Liu","year":"2013","venue":"Twenty-Seventh AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Immorlica et al., 2018] Nicole Immorlica, Karthik Abinav Sankararaman, Robert Schapire, and Aleksandrs Slivkins. Adversarial bandits with knapsacks. arXiv preprint arXiv:1811.11881, 2018.","title":"Adversarial bandits with knapsacks","authors":"Nicole Immorlica, Karthik Abinav Sankararaman, Robert Schapire, Aleksandrs Slivkins","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kellerer et al., 2004] H. Kellerer, U. Pferschy, and D. Pisinger. Knapsack Problems. Springer, Berlin, Germany, 2004.","title":"Knapsack Problems","authors":"H Kellerer, U Pferschy, D Pisinger","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lykouris et al., 2018] Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to ad- versarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 114-122. ACM, 2018.","title":"Stochastic bandits robust to adversarial corruptions","authors":"Thodoris Lykouris, Vahab Mirrokni, Renato Paes, Leme","year":"2018","venue":"Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rangi and Franceschetti, 2018a] Anshuka Rangi and Mas- simo Franceschetti. Multi-armed bandit algorithms for crowdsourcing systems with online estimation of work- ers' ability. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Sys- tems, pages 1345-1352. International Foundation for Au- tonomous Agents and Multiagent Systems, 2018.","title":"Anshuka Rangi and Massimo Franceschetti. Multi-armed bandit algorithms for crowdsourcing systems with online estimation of workers' ability","authors":"Franceschetti Rangi","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rangi and Franceschetti, 2018b] Anshuka Rangi and Mas- simo Franceschetti. Online learning with feedback graphs and switching costs. arXiv preprint arXiv:1810.09666, 2018.","title":"Anshuka Rangi and Massimo Franceschetti. Online learning with feedback graphs and switching costs","authors":"Franceschetti Rangi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rangi et al., 2018a] Anshuka Rangi, Massimo Franceschetti, and Stefano Marano. Consensus-based chernoff test in sensor networks. In 2018 IEEE Confer- ence on Decision and Control (CDC), pages 6773-6778. IEEE, 2018.","title":"Consensus-based chernoff test in sensor networks","authors":"Rangi","year":"2018","venue":"IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rangi et al., 2018b] Anshuka Rangi, Massimo Franceschetti, and Stefano Marano. Decentralized chernoff test in sensor networks. In 2018 IEEE Interna- tional Symposium on Information Theory (ISIT), pages 501-505. IEEE, 2018.","title":"Decentralized chernoff test in sensor networks","authors":"Rangi","year":"2018","venue":"IEEE International Symposium on Information Theory (ISIT)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rangi et al., 2018c] Anshuka Rangi, Massimo Franceschetti, and Stefano Marano. Distributed chernoff test: Optimal decision systems over networks. arXiv preprint arXiv:1809.04587, 2018.","title":"Distributed chernoff test: Optimal decision systems over networks","authors":"Rangi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rangi et al., 2018d] Anshuka Rangi, Massimo Franceschetti, and Long Tran-Thanh. Unifying the stochastic and the adversarial bandits with knapsack. arXiv preprint arXiv:1811.12253, 2018.","title":"Unifying the stochastic and the adversarial bandits with knapsack","authors":"Rangi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sankararaman and Slivkins, 2018] Karthik Abinav Sankararaman and Aleksandrs Slivkins. Combina- torial semi-bandits with knapsacks. In International Conference on Artificial Intelligence and Statistics, pages 1760-1770, 2018.","title":"Combinatorial semi-bandits with knapsacks","authors":"Abinav Karthik, Aleksandrs Sankararaman, Slivkins","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seldin and Lugosi, 2017] Yevgeny Seldin and Gábor Lu- gosi. An improved parametrization and analysis of the exp3++ algorithm for stochastic and adversarial bandits. In Conference on Learning Theory, pages 1743-1759, 2017.","title":"An improved parametrization and analysis of the exp3++ algorithm for stochastic and adversarial bandits","authors":"Yevgeny Seldin, Gábor Lugosi","year":"2017","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Seldin and Slivkins, 2014] Yevgeny Seldin and Aleksandrs Slivkins. One practical algorithm for both stochastic and adversarial bandits. In ICML, pages 1287-1295, 2014.","title":"One practical algorithm for both stochastic and adversarial bandits","authors":"Yevgeny Seldin, Aleksandrs Slivkins","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tran-Thanh et al., 2010] Long Tran-Thanh, Archie Chap- man, Enrique Munoz de Cote, Alex Rogers, and Nicholas R Jennings. Epsilon-first policies for budget- limited multi-armed bandits. In Twenty-Fourth AAAI Con- ference on Artificial Intelligence, 2010.","title":"Epsilon-first policies for budgetlimited multi-armed bandits","authors":"Tran-Thanh","year":"2010","venue":"Twenty-Fourth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran-Thanh et al., 2012a] Long Tran-Thanh, Archie C Chapman, Alex Rogers, and Nicholas R Jennings. Knap- sack based optimal policies for budget-limited multi- armed bandits. In AAAI, pages 1134-1140, 2012.","title":"Knapsack based optimal policies for budget-limited multiarmed bandits","authors":"Tran-Thanh","year":"2012","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tran-Thanh et al., 2012b] Long Tran-Thanh, Sebastian Stein, Alex Rogers, and Nicholas R Jennings. Efficient crowdsourcing of unknown experts using multi-armed bandits. In Proceedings of the 20th European Conference on Artificial Intelligence, pages 768-773, 2012.","title":"Efficient crowdsourcing of unknown experts using multi-armed bandits","authors":"Tran-Thanh","year":"2012","venue":"Proceedings of the 20th European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xia et al., 2016] Yingce Xia, Tao Qin, Weidong Ma, Neng- hai Yu, and Tie-Yan Liu. Budgeted multi-armed bandits with multiple plays. In IJCAI, pages 2210-2216, 2016.","title":"Budgeted multi-armed bandits with multiple plays","authors":"Yingce Xia, Tao Qin, Weidong Ma, Nenghai Yu, Tie-Yan Liu","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou and Tomlin, 2018] Datong P Zhou and Claire J Tom- lin. Budget-constrained multi-armed bandits with multiple plays. In Thirty-Second AAAI Conference on Artificial In- telligence, 2018.","title":"Budget-constrained multi-armed bandits with multiple plays","authors":"P Datong, Claire J Zhou, Tomlin","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}