{"type":"ScholarlyArticle","title":"Bounded Memory Adversarial Bandits with Composite Anonymous Delayed Feedback","authors":[{"name":"Zongqi Wan","slug":"zongqi-wan","given":"Zongqi","family":"Wan","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Xiaoming Sun","slug":"xiaoming-sun","given":"Xiaoming","family":"Sun","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Jialin Zhang","slug":"jialin-zhang","given":"Jialin","family":"Zhang","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"}],"abstract":"We study the adversarial bandit problem with composite anonymous delayed feedback. In this setting, losses of an action are split into d components, spreading over consecutive rounds after the action is chosen. And in each round, the algorithm observes the aggregation of losses that come from the latest d rounds. Previous works focus on oblivious adversarial setting, while we investigate the harder nonoblivious setting. We show nonoblivious setting incurs Omega(T) pseudo regret even when the loss sequence is bounded memory. However, we propose a wrapper algorithm which enjoys o(T) policy regret on many adversarial bandit problems with the assumption that the loss sequence is bounded memory. Especially, for K armed bandit and bandit convex optimization, our policy regret bound is in the order of T to the two third. We also prove a matching lower bound for K armed bandit. Our lower bound works even when the loss sequence is oblivious but the delay is nonoblivious. It answers the open problem proposed in [Wang, Wang, Huang 2021], showing that nonoblivious delay is enough to incur the regret in the order of T to the two third.","doi":"10.24963/ijcai.2022/486","url":"https://ijcai.org/proceedings/2022/486","pdf":"https://ijcai.org/proceedings/2022/0486.pdf","markdown":"https://ijcai.org/proceedings/2022/486.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/486","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":486,"pages":{"first":3501,"last":3507},"keywords":["Machine Learning: Online Learning","Machine Learning: Learning Theory"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986847","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984990","length_sec":343,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anava et al., 2015] Oren Anava, Elad Hazan, and Shie Mannor. Online learning for adversaries with memory: price of past mistakes. In NeurIPS, pages 784-792, 2015.","title":"Online learning for adversaries with memory: price of past mistakes","authors":"Oren Anava, Elad Hazan, Shie Mannor","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2012] Raman Arora, Ofer Dekel, and Ambuj Tewari. Online bandit learning against an adaptive adver- sary: from regret to policy regret. In ICML, pages 1747- 1754, 2012.","title":"Online bandit learning against an adaptive adversary: from regret to policy regret","authors":"Raman Arora, Ofer Dekel, Ambuj Tewari","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora et al., 2018] Raman Arora, Michael Dinitz, Teodor V Marinov, and Mehryar Mohri. Policy regret in repeat- ed games. In NeurIPS, volume 2018, pages 6732-6741, 2018.","title":"Policy regret in repeated games","authors":"Raman Arora, Michael Dinitz, Teodor V Marinov, Mehryar Mohri","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, Robert E Schapire","year":"2002","venue":"SIAM journal on computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bistritz et al., 2019] Ilai Bistritz, Zhengyuan Zhou, X- i Chen, Nicholas Bambos, and Jose Blanchet. Exp3 learning in adversarial bandits with delayed feedback. In NeurIPS, 2019.","title":"Exp3 learning in adversarial bandits with delayed feedback","authors":"Ilai Bistritz, Zhengyuan Zhou, Xi Chen, Nicholas Bambos, Jose Blanchet","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bubeck et al., 2017] Sébastien Bubeck, Yin Tat Lee, and Ronen Eldan. Kernel-based methods for bandit convex op- timization. In STOC, pages 72-85, 2017.","title":"Kernel-based methods for bandit convex optimization","authors":"Sébastien Bubeck, Yin Tat Lee, Ronen Eldan","year":"2017","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cella and Cesa-Bianchi, 2020] Leonardo Cella and Nicolò Cesa-Bianchi. Stochastic bandits with delay-dependent payoffs. In AISTATS, pages 1168-1177. PMLR, 2020.","title":"Stochastic bandits with delay-dependent payoffs","authors":"Cesa-Bianchi ; Cella, Leonardo Cella, Nicolò Cesa-Bianchi","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cesa-Bianchi et al., 2018] Nicolo Cesa-Bianchi, Claudio Gentile, and Yishay Mansour. Nonstochastic bandits with composite anonymous feedback. In COLT, pages 750- 773, 2018.","title":"Nonstochastic bandits with composite anonymous feedback","authors":"Cesa-Bianchi","year":"2018","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chapelle et al., 2014] Olivier Chapelle, Eren Manavoglu, and Romer Rosales. Simple and scalable response pre- diction for display advertising. ACM Transactions on In- telligent Systems and Technology (TIST), 5(4):1-34, 2014.","title":"Simple and scalable response prediction for display advertising","authors":"Chapelle","year":"2014","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dekel et al., 2014] Ofer Dekel, Jian Ding, Tomer Koren, and Yuval Peres. Bandits with switching costs: T 2/3 re- gret. In STOC, pages 459-467, 2014.","title":"Bandits with switching costs: T 2/3 regret","authors":"Ofer Dekel, Jian Ding, Tomer Koren, Yuval Peres","year":"2014","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Flaxman et al., 2005] Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimiza- tion in the bandit setting: gradient descent without a gra- dient. In SODA, pages 385-394, 2005.","title":"Online convex optimization in the bandit setting: gradient descent without a gradient","authors":"Adam Abraham D Flaxman, Brendan Tauman Kalai, Mcmahan","year":"2005","venue":"SODA","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gergely Neu et al., 2010] András György Gergely Neu, C- saba Szepesvári, and András Antos. Online markov deci- sion processes under bandit feedback. In NeurIPS, 2010.","title":"Online markov decision processes under bandit feedback","authors":"Gergely Neu","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gyorgy and Joulani, 2021] Andras Gyorgy and Pooria Joulani. Adapting to delays and data in adversarial multi-armed bandits. In ICML, pages 3988-3997, 2021.","title":"Adapting to delays and data in adversarial multi-armed bandits","authors":"Joulani Gyorgy, Andras Gyorgy, Pooria Joulani","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Heidari et al., 2016] Hoda Heidari, Michael Kearns, and Aaron Roth. Tight policy regret bounds for improving and decaying bandits. In IJCAI, pages 1562-1570, 2016.","title":"Tight policy regret bounds for improving and decaying bandits","authors":"Hoda Heidari, Michael Kearns, Aaron Roth","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/224"},{"position":15,"raw":"[Jaghargh et al., 2019] Mohammad Reza Karimi Jaghargh, Andreas Krause, Silvio Lattanzi, and Sergei Vassilvtiskii. Consistent online optimization: Convex and submodular. In AISTATS, pages 2241-2250, 2019.","title":"Consistent online optimization: Convex and submodular","authors":"Mohammad Reza, Karimi Jaghargh, Andreas Krause, Silvio Lattanzi, Sergei Vassilvtiskii","year":"2019","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Joulani et al., 2013] Pooria Joulani, Andras Gyorgy, and C- saba Szepesvári. Online learning under delayed feedback. In ICML, pages 1453-1461, 2013.","title":"Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback","authors":"Joulani","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Discounted ucb. In 2nd PASCAL Challenges Workshop, volume 2, 2006.","title":"Discounted ucb","authors":"Szepesvári ; Kocsis, Csaba Kocsis, Szepesvári","year":"2006","venue":"2nd PASCAL Challenges Workshop","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lei et al., 2017] Huitian Lei, Ambuj Tewari, and Susan A Murphy. An actor-critic contextual bandit algorith- m for personalized mobile health interventions. arX- iv:1706.09090, 2017.","title":"An actor-critic contextual bandit algorithm for personalized mobile health interventions","authors":"Lei","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2019] Bingcong Li, Tianyi Chen, and Georgios B Giannakis. Bandit online learning with unknown delays. In AISTATS, pages 993-1002, 2019.","title":"Bandit online learning with unknown delays","authors":"Li","year":"2019","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pike-Burke et al., 2018] Ciara Pike-Burke, Shipra Agrawal, Csaba Szepesvari, and Steffen Grunewalder. Bandits with delayed, aggregated anonymous feedback. In ICML, pages 4105-4113, 2018.","title":"Bandits with delayed, aggregated anonymous feedback","authors":"Pike-Burke","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Quanrud and Khashabi, 2015] Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays. In NeurIPS, pages 1270-1278, 2015.","title":"Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays","authors":"Khashabi Quanrud","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saha and Tewari, 2011] Ankan Saha and Ambuj Tewari. Improved regret guarantees for online smooth convex op- timization with bandit feedback. In AISTATS, pages 636- 642. JMLR Workshop and Conference Proceedings, 2011.","title":"Improved regret guarantees for online smooth convex optimization with bandit feedback","authors":"Tewari ; ] Ankan Saha, Ambuj Saha, Tewari","year":"2011","venue":"JMLR Workshop and Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thune et al., 2019] Tobias Sommer Thune, Nicolò Cesa- Bianchi, and Yevgeny Seldin. Nonstochastic multiarmed bandits with unrestricted delays. In NeurIPS, pages 6541- 6550, 2019.","title":"Nonstochastic multiarmed bandits with unrestricted delays","authors":"Thune","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Villar et al., 2015] Sofía S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal de- sign of clinical trials: benefits and challenges. Statistical science: a review journal of the Institute of Mathematical Statistics, 30(2):199, 2015.","title":"Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges","authors":"S Sofía, Jack Villar, James Bowden, Wason","year":"2015","venue":"Statistical science: a review journal of the Institute of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Siwei Wang, Haoyun Wang, and Longbo Huang. Adaptive algorithms for multi-armed bandit with composite and anonymous feedback. In AAAI, volume 35, pages 10210-10217, 2021.","title":"Adaptive algorithms for multi-armed bandit with composite and anonymous feedback","authors":"Siwei Wang, Haoyun Wang, Longbo Huang","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zimmert and Seldin, 2020] Julian Zimmert and Yevgeny Seldin. An optimal algorithm for adversarial bandits with arbitrary delays. In AISTATS, pages 3285-3294, 2020.","title":"Julian Zimmert and Yevgeny Seldin. An optimal algorithm for adversarial bandits with arbitrary delays","authors":"Seldin Zimmert","year":"2020","venue":"AISTATS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}