{"type":"ScholarlyArticle","title":"Online Learning with Off-Policy Feedback in Adversarial MDPs","authors":[{"name":"Francesco Bacchiocchi","slug":"francesco-bacchiocchi","given":"Francesco","family":"Bacchiocchi","affiliation":"Politecnico di Milano"},{"name":"Francesco Emanuele Stradi","slug":"francesco-emanuele-stradi","given":"Francesco Emanuele","family":"Stradi","affiliation":"Politecnico di Milano"},{"name":"Matteo Papini","slug":"matteo-papini","given":"Matteo","family":"Papini","affiliation":"Politecnico di Milano"},{"name":"Alberto Maria Metelli","slug":"alberto-maria-metelli","given":"Alberto Maria","family":"Metelli","affiliation":"Politecnico di Milano"},{"name":"Nicola Gatti","slug":"nicola-gatti","given":"Nicola","family":"Gatti","affiliation":"Politecnico di Milano"}],"abstract":"In this paper, we face the challenge of online learning in adversarial Markov decision processes with off-policy feedback. In this setting, the learner chooses a policy, but, differently from the traditional on-policy setting, the environment is explored by means of a different, fixed, and possibly unknown policy (named colleague's policy). \r\nThe off-policy feedback presents an additional issue that is not present in traditional settings: the learner is charged with the regret of its chosen policy but it observes only the rewards gained by the colleague's policy.\r\nFirst, we present a lower-bound for the setting we propose, which shows that the optimal dependency of the sublinear regret is w.r.t. the dissimilarity between the optimal policy in hindsight and the colleague's policy.\r\nThen, we propose novel algorithms that, by employing pessimistic estimators---commonly adopted in the off-line reinforcement learning literature---ensure sublinear regret bounds depending on the desired dissimilarity, even when the colleague's policy is unknown.","doi":"10.24963/ijcai.2024/409","url":"https://ijcai.org/proceedings/2024/409","pdf":"https://ijcai.org/proceedings/2024/0409.pdf","markdown":"https://ijcai.org/proceedings/2024/409.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/409","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":409,"pages":{"first":3697,"last":3705},"keywords":["Machine Learning: ML: Online learning","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2008] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. In Advances in Neural Information Processing Systems, volume 21.","title":"Near-optimal regret bounds for reinforcement learning","authors":"Auer","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Curran Associates, Inc., 2008.","title":null,"authors":"Curran Associates, Inc","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Azar et al., 2017] Mohammad Gheshlaghi Azar, Ian Os- band, and Rémi Munos. Minimax regret bounds for re- inforcement learning. In Proceedings of the 34th Inter- national Conference on Machine Learning, (ICML), vol- ume 70, pages 263-272. PMLR, 2017.","title":"Minimax regret bounds for reinforcement learning","authors":"Mohammad Gheshlaghi Azar, Ian Osband, Rémi Munos","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cam- bridge university press, 2006.","title":"Nicolo Cesa-Bianchi and Gábor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2022] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning (ICML), volume 162, pages 3852-3878. PMLR, 2022.","title":"Adversarially trained actor critic for offline reinforcement learning","authors":"Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal","year":"2022","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ernst et al., 2005] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. J. Mach. Learn. Res., 6:503-556, 2005.","title":"Tree-based batch mode reinforcement learning","authors":"Damien Ernst, Pierre Geurts, Louis Wehenkel","year":"2005","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Even-Dar et al., 2009] Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Online markov decision pro- cesses. Mathematics of Operations Research, 34(3):726- 736, 2009.","title":"Online markov decision processes","authors":"Eyal Even-Dar, Yishay Sham M Kakade, Mansour","year":"2009","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gabbianelli et al., 2023] Germano Gabbianelli, Gergely Neu, and Matteo Papini. Online learning with off-policy feedback. In International Conference on Algorithmic Learning Theory (ALT), volume 201, pages 620-641.","title":"Online learning with off-policy feedback","authors":"Germano Gabbianelli, Gergely Neu, Matteo Papini","year":null,"venue":"International Conference on Algorithmic Learning Theory (ALT)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"PMLR, 2023.","title":null,"authors":"Pmlr","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hazan, 2016] Elad Hazan. Introduction to online convex optimization. Found. Trends Optim., 2(3-4):157-325, 2016.","title":"Introduction to online convex optimization","authors":"Elad Hazan","year":"2016","venue":"Found. Trends Optim","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaksch et al., 2010] Thomas Jaksch, Ronald Ortner, and Pe- ter Auer. Near-optimal regret bounds for reinforcement learning. J. Mach. Learn. Res., 11:1563-1600, 2010.","title":"Near-optimal regret bounds for reinforcement learning","authors":"Thomas Jaksch, Ronald Ortner, Peter Auer","year":"2010","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jin et al., 2020] Chi Jin, Tiancheng Jin, Haipeng Luo, Su- vrit Sra, and Tiancheng Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the 37th International Con- ference on Machine Learning (ICML), volume 119, pages 4860-4869. PMLR, 2020.","title":"Learning adversarial markov decision processes with bandit feedback and unknown transition","authors":"Jin","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2021] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In Proceedings of the 38th International Conference on Ma- chine Learning (ICML), volume 139, pages 5084-5096. PMLR, 18-24 Jul 2021.","title":"Is pessimism provably efficient for offline rl?","authors":"Ying Jin, Zhuoran Yang, Zhaoran Wang","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lagoudakis and Parr, 2003] Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. J. Mach. Learn. Res., 4:1107-1149, 2003.","title":"Least-squares policy iteration","authors":"Michail G Lagoudakis, Ronald Parr","year":"2003","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lattimore and Szepesvári, 2020] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.","title":"Tor Lattimore and Csaba Szepesvári. Bandit algorithms","authors":"Szepesvári Lattimore","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Munos and Szepesvári, 2008] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. J. Mach. Learn. Res., 9:815-857, 2008.","title":"Finite-time bounds for fitted value iteration","authors":"Munos, Szepesvári, Rémi Munos, Csaba Szepesvári","year":"2008","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Neu et al., 2014] Gergely Neu, András György, Csaba Szepesvári, and András Antos. Online markov decision processes under bandit feedback. IEEE Trans. Autom. Control., 59(3):676-691, 2014.","title":"Online markov decision processes under bandit feedback","authors":"Gergely Neu, András György, Csaba Szepesvári, András Antos","year":"2014","venue":"IEEE Trans. Autom. Control","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Neu, 2015] Gergely Neu. Explore no more: Improved high- probability regret bounds for non-stochastic bandits. In Advances in Neural Information Processing Systems 28 (NIPS), pages 3168-3176, 2015.","title":"Explore no more: Improved highprobability regret bounds for non-stochastic bandits","authors":"Gergely Neu, Neu","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rashidinejad et al., 2022] Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. IEEE Trans. Inf. Theory, 68(12):8156- 8196, 2022.","title":"Bridging offline reinforcement learning and imitation learning: A tale of pessimism","authors":"Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, Stuart Russell","year":"2022","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rosenberg and Mansour, 2019a] Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversar- ial Markov decision processes. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pages 5478-5486. PMLR, 09-15 Jun 2019.","title":"Online convex optimization in adversarial Markov decision processes","authors":"Mansour Rosenberg, Aviv Rosenberg, Yishay Mansour","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rosenberg and Mansour, 2019b] Aviv Rosenberg and Yishay Mansour. Online stochastic shortest path with bandit feedback and unknown transition function. In Advances in Neural Information Processing Systems, volume 32.","title":"Online stochastic shortest path with bandit feedback and unknown transition function","authors":"Mansour Rosenberg, Aviv Rosenberg, Yishay Mansour","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Uehara and Sun, 2022] Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In The Tenth International Con- ference on Learning Representations (ICLR). OpenRe- view.net, 2022.","title":"Pessimistic model-based offline reinforcement learning under partial coverage","authors":"Masatoshi Uehara, Wen Sun","year":"2022","venue":"The Tenth International Conference on Learning Representations (ICLR). OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiao et al., 2021] Chenjun Xiao, Yifan Wu, Jincheng Mei, Bo Dai, Tor Lattimore, Lihong Li, Csaba Szepesvari, and Dale Schuurmans. On the optimality of batch policy opti- mization algorithms. In Proceedings of the 38th Interna- tional Conference on Machine Learning (ICML), volume 139, pages 11362-11371. PMLR, 18-24 Jul 2021.","title":"On the optimality of batch policy optimization algorithms","authors":"Xiao","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zanette et al., 2021] Andrea Zanette, Martin J. Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), pages 13626-13640, 2021.","title":"Provable benefits of actor-critic methods for offline reinforcement learning","authors":"Andrea Zanette, Martin J Wainwright, Emma Brunskill","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zimin and Neu, 2013] Alexander Zimin and Gergely Neu. Online learning in episodic markovian decision processes by relative entropy policy search. In Advances in Neural Information Processing Systems, volume 26.","title":"Online learning in episodic markovian decision processes by relative entropy policy search","authors":"Neu Zimin, Alexander Zimin, Gergely Neu","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}