{"type":"ScholarlyArticle","title":"Causal Deep Reinforcement Learning Using Observational Data","authors":[{"name":"Wenxuan Zhu","slug":"wenxuan-zhu","given":"Wenxuan","family":"Zhu","affiliation":"Dalian University of Technology"},{"name":"Chao Yu","slug":"chao-yu","given":"Chao","family":"Yu","affiliation":"Sun Yat-sen University"},{"name":"Qiang Zhang","slug":"qiang-zhang","given":"Qiang","family":"Zhang","affiliation":"Dalian University of Technology"}],"abstract":"Deep reinforcement learning (DRL) requires the collection of interventional data, which is sometimes expensive and even unethical in the real world, such as in the autonomous driving and the medical field. Offline reinforcement learning promises to alleviate this issue by exploiting the vast amount of observational data available in the real world. However, observational data may mislead the learning agent to undesirable outcomes if the behavior policy that generates the data depends on unobserved random variables (i.e., confounders). In this paper, we propose two deconfounding methods in DRL to address this problem. The methods first calculate the importance degree of different samples based on the causal inference technique, and then adjust the impact of different samples on the loss function by reweighting or resampling the offline dataset to ensure its unbiasedness. These deconfounding methods can be flexibly combined with existing model-free DRL algorithms such as soft actor-critic and deep Q-learning, provided that a weak condition can be satisfied by the loss functions of these algorithms. We prove the effectiveness of our deconfounding methods and validate them experimentally.","doi":"10.24963/ijcai.2023/524","url":"https://ijcai.org/proceedings/2023/524","pdf":"https://ijcai.org/proceedings/2023/0524.pdf","markdown":"https://ijcai.org/proceedings/2023/524.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/524","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":524,"pages":{"first":4711,"last":4719},"keywords":["Uncertainty in AI: UAI: Causality, structural causal models and causal inference","Machine Learning: ML: Deep reinforcement learning","Machine Learning: ML: Causality"],"videos":[],"references":[{"position":1,"raw":"[Agarwal et al., 2020] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on of- fline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.","title":"An optimistic perspective on offline reinforcement learning","authors":"Rishabh Agarwal, Dale Schuurmans, Mohammad Norouzi","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ernst et al., 2005] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.","title":"Tree-based batch mode reinforcement learning","authors":"Damien Ernst, Pierre Geurts, Louis Wehenkel","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fujimoto et al., 2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Ma- chine Learning, pages 2052-2062. PMLR, 2019.","title":"Off-policy deep reinforcement learning without exploration","authors":"Scott Fujimoto, David Meger, Doina Precup","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gasse et al., 2021] Maxime Gasse, Damien Grasset, Guil- laume Gaudron, and Pierre-Yves Oudeyer. Causal rein- forcement learning using observational and interventional data. arXiv preprint arXiv:2106.14421, 2021.","title":"Causal reinforcement learning using observational and interventional data","authors":"Maxime Gasse, Damien Grasset, Guillaume Gaudron, Pierre-Yves Oudeyer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gelada and Bellemare, 2019] Carles Gelada and Marc G Bellemare. Off-policy deep reinforcement learning by bootstrapping the covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3647-3655, 2019.","title":"Off-policy deep reinforcement learning by bootstrapping the covariate shift","authors":"Bellemare Gelada, Carles Gelada, Marc G Bellemare","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learn- ing with deep energy-based policies. In International Con- ference on Machine Learning, pages 1352-1361. PMLR, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hallak and Mannor, 2017] Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation. In Interna- tional Conference on Machine Learning, pages 1372- 1383. PMLR, 2017.","title":"Consistent on-line off-policy evaluation","authors":"Mannor ; ] Assaf Hallak, Shie Hallak, Mannor","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaques et al., 2019] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of im- plicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.","title":"Way off-policy batch deep reinforcement learning of implicit human preferences in dialog","authors":"Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, Rosalind Picard","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2020] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learn- ing with linear function approximation. In Conference on Learning Theory, pages 2137-2143. PMLR, 2020.","title":"Provably efficient reinforcement learning with linear function approximation","authors":"Jin","year":"2020","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kallus and Uehara, 2019] Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. arXiv preprint arXiv:1909.05850, 2019.","title":"Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning","authors":"Uehara Kallus, Nathan Kallus, Masatoshi Uehara","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kallus and Zhou, 2018] Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treat- ments. In International conference on artificial intelli- gence and statistics, pages 1243-1251. PMLR, 2018.","title":"Policy evaluation and optimization with continuous treatments","authors":"Zhou Kallus, Nathan Kallus, Angela Zhou","year":"2018","venue":"International conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kallus and Zhou, 2020] Nathan Kallus and Angela Zhou. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. arXiv preprint arXiv:2002.04518, 2020.","title":"Confounding-robust policy evaluation in infinite-horizon reinforcement learning","authors":"Zhou Kallus, Nathan Kallus, Angela Zhou","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2019] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q- learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949, 2019.","title":"Stabilizing off-policy qlearning via bootstrapping error reduction","authors":"Aviral Kumar, Justin Fu, George Tucker, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar et al., 2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020.","title":"Conservative q-learning for offline reinforcement learning","authors":"Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Langford and Zhang, 2007] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang ; ] Langford, John Langford, Tong Zhang","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Curran Associates, Inc., 2007.","title":null,"authors":"Curran Associates, Inc","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lattimore et al., 2016] Finnian Lattimore, Tor Lattimore, and Mark D Reid. Causal bandits: Learning good interventions via causal inference. arXiv preprint arXiv:1606.03203, 2016.","title":"Causal bandits: Learning good interventions via causal inference","authors":"Finnian Lattimore, Tor Lattimore, Mark D Reid","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Levine et al., 2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tu- torial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.","title":"Offline reinforcement learning: Tutorial, review, and perspectives on open problems","authors":"Sergey Levine, Aviral Kumar, George Tucker, Justin Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2018] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. arXiv preprint arXiv:1810.12429, 2018.","title":"Breaking the curse of horizon: Infinite-horizon off-policy estimation","authors":"Qiang Liu, Lihong Li, Ziyang Tang, Dengyong Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Louizos et al., 2017] Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, vol- ume 30.","title":"Causal effect inference with deep latent-variable models","authors":"Christos Louizos, Uri Shalit, M Joris, David Mooij, Richard Sontag, Max Zemel, Welling","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lu et al., 2018] Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández-Lobato. Deconfounding re- inforcement learning in observational settings. arXiv preprint arXiv:1812.10576, 2018.","title":"Deconfounding reinforcement learning in observational settings","authors":"Lu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nagler, 2018] Thomas Nagler. A generic approach to non- parametric function estimation with mixed data. Statistics & Probability Letters, 137:326-330, 2018.","title":"A generic approach to nonparametric function estimation with mixed data","authors":"Thomas Nagler","year":"2018","venue":"Statistics & Probability Letters","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nair and Jiang, 2021] Yash Nair and Nan Jiang. A spec- tral approach to off-policy evaluation for pomdps. arXiv preprint arXiv:2109.10502, 2021.","title":"A spectral approach to off-policy evaluation for pomdps","authors":"Jiang Nair, Yash Nair, Nan Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pearl, 2009] Judea Pearl. Causality. Cambridge university press, 2009.","title":"Judea Pearl. Causality","authors":"Pearl","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pearl, 2012] Judea Pearl. The do-calculus revisited. arXiv preprint arXiv:1210.4852, 2012.","title":"Judea Pearl. The do-calculus revisited","authors":"Pearl","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Peters et al., 2017] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foun- dations and learning algorithms. The MIT Press, 2017.","title":"Elements of causal inference: foundations and learning algorithms","authors":"Peters","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rissanen and Marttinen, 2021] Severi Rissanen and Pekka Marttinen. A critical look at the consistency of causal es- timation with deep latent variable models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Pro- cessing Systems, volume 34, pages 4207-4217.","title":"A critical look at the consistency of causal estimation with deep latent variable models","authors":"Severi Rissanen, Pekka Marttinen","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rothfuss et al., 2019] Jonas Rothfuss, Fabio Ferreira, Si- mon Walther, and Maxim Ulrich. Conditional density es- timation with neural networks: Best practices and bench- marks. arXiv:1903.00954, 2019.","title":"Conditional density estimation with neural networks: Best practices and benchmarks","authors":"Jonas Rothfuss, Fabio Ferreira, Simon Walther, Maxim Ulrich","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sen et al., 2017] Rajat Sen, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Iden- tifying best interventions through online importance sampling. In International Conference on Machine Learning, pages 3057-3066. PMLR, 2017.","title":"Identifying best interventions through online importance sampling","authors":"Rajat Sen, Karthikeyan Shanmugam, Alexandros G Dimakis, Sanjay Shakkottai","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Siegel et al., 2020] Noah Y Siegel, Jost Tobias Springen- berg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behav- ioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.","title":"Keep doing what worked: Behavioral modelling priors for offline reinforcement learning","authors":"Siegel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sugiyama et al., 2010] Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Daisuke Okanohara. Conditional density estimation via least-squares density ratio estimation. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thir- teenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learn- ing Research, pages 781-788, Chia Laguna Resort, Sar- dinia, Italy, 13-15 May 2010. PMLR.","title":"Conditional density estimation via least-squares density ratio estimation","authors":"Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Daisuke Okanohara","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Takuma Seno, 2021] Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurIPS 2021 Offline Reinforcement Learning Workshop, Decem- ber 2021.","title":"d3rlpy: An offline deep reinforcement library","authors":"Michita Imai, Takuma Seno","year":"2021","venue":"NeurIPS 2021 Offline Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tennenholtz et al., 2020] Guy Tennenholtz, Uri Shalit, and Shie Mannor. Off-policy evaluation in partially observ- able environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10276-10283, 2020.","title":"Off-policy evaluation in partially observable environments","authors":"Tennenholtz","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[van Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Dale Schuurmans and Michael P. Well- man, editors, Proceedings of the Thirtieth AAAI Con- ference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094-2100. AAAI Press, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2021] Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Provably efficient causal reinforcement learning with confounded observational data. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21164-21175.","title":"Provably efficient causal reinforcement learning with confounded observational data","authors":"Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yang and Wang, 2019] Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995-7004. PMLR, 2019.","title":"Sample-optimal parametric q-learning using linearly additive features","authors":"Yang, Wang ;, Lin Yang, Mengdi Wang","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yang and Wang, 2020] Lin Yang and Mengdi Wang. Rein- forcement learning in feature space: Matrix bandit, ker- nels, and regret bound. In International Conference on Machine Learning, pages 10746-10756. PMLR, 2020.","title":"Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound","authors":"Yang, Wang ;, Lin Yang, Mengdi Wang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}