{"type":"ScholarlyArticle","title":"Dynamic Bandits with Temporal Structure","authors":[{"name":"Qinyi Chen","slug":"qinyi-chen","given":"Qinyi","family":"Chen","affiliation":"Massachusetts Institute of Technology; Djallel Bouneffouf ( IBM Research) ."}],"abstract":"In this work, we study a dynamic multi-armed bandit (MAB) problem, where the expected reward of each arm evolves over time following an auto-regressive model. We present an algorithm whose per-round regret upper bound almost matches the regret lower bound, and numerically demonstrate its efficacy in adapting to the changing environment.","doi":"10.24963/ijcai.2022/823","url":"https://ijcai.org/proceedings/2022/823","pdf":"https://ijcai.org/proceedings/2022/0823.pdf","markdown":"https://ijcai.org/proceedings/2022/823.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/823","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":823,"pages":{"first":5841,"last":5842},"keywords":["Machine Learning (ML): General"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2002a] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The non-stochastic multi-armed bandit problem. SIAM journal of computing, 32:48-77, 2002.","title":"The non-stochastic multi-armed bandit problem","authors":"Auer","year":"2002","venue":"SIAM journal of computing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002b] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Besbes et al., 2014] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non- stationary rewards. In Advances in Neural Information Processing Systems, volume 27, 2014.","title":"Stochastic multi-armed-bandit problem with nonstationary rewards","authors":"Omar Besbes, Yonatan Gur, Assaf Zeevi","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Garivier et al., 2016] Aurelien Garivier, Tor Lattimore, and Emilie Kaufmann. On explore-then-commit strategies. In Advances in Neural Information Processing Systems, vol- ume 29, 2016.","title":"On explore-then-commit strategies","authors":"Aurelien Garivier, Tor Lattimore, Emilie Kaufmann","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ortner et al., 2012] Ronald Ortner, Daniil Ryabko, Peter Auer, and Rémi Munos. Regret bounds for restless markov bandits. In Proceedings of the 23rd International Con- ference on Algorithmic Learning Theory, ALT'12, page 214-228, Berlin, Heidelberg, 2012. Springer-Verlag.","title":"Regret bounds for restless markov bandits","authors":"Ronald Ortner, Daniil Ryabko, Peter Auer, Rémi Munos","year":"2012","venue":"Proceedings of the 23rd International Conference on Algorithmic Learning Theory, ALT'12","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the se- quential design of experiments. Bulletin of the American Mathematical Society, 58(5):527 -535, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Slivkins and Upfal, 2008] Alex Slivkins and Eli Upfal. Adapting to a changing environment: the brownian restless bandits. In 21st Conference on Learning Theory (COLT), pages 343-354, 2008.","title":"Adapting to a changing environment: the brownian restless bandits","authors":"Alex Slivkins, Eli Upfal","year":"2008","venue":"21st Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Thompson, 1933] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25:285 -294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"W R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}