{"type":"ScholarlyArticle","title":"Transfer Learning in Multi-Armed Bandits: A Causal Approach","authors":[{"name":"Junzhe Zhang","slug":"junzhe-zhang","given":"Junzhe","family":"Zhang","affiliation":"Purdue University , USA"},{"name":"Elias Bareinboim","slug":"elias-bareinboim","given":"Elias","family":"Bareinboim","affiliation":"Purdue University , USA"}],"abstract":"Reinforcement learning (RL) agents have been deployed in complex environments where interactions are costly, and learning is usually slow. One prominent task in these settings is to reuse interactions performed by other agents to accelerate the learning process.  Causal inference provides a family of methods to infer the effects of actions from a combination of data and qualitative assumptions about the underlying environment. Despite its success of transferring invariant knowledge across domains in the empirical sciences, causal inference has not been fully realized in the context of transfer learning in interactive domains. In this paper, we use causal inference as a basis to support a principled and more robust transfer of knowledge in RL settings. In particular, we tackle the problem of transferring knowledge  across bandit agents in settings where causal effects cannot be identified by do-calculus [Pearl, 2000] and standard learning techniques.  Our new identification strategy combines two steps -- first, deriving bounds over the arm’s distribution based on structural knowledge; second, incorporating these bounds in a dynamic allocation procedure so as to guide the search towards more promising actions. We formally prove that our strategy dominates previously known algorithms and achieves orders of magnitude faster convergence rates than these algorithms. Finally, we perform simulations and empirically demonstrate that our strategy is consistently more efficient than the current (non-causal) state-of-the-art methods","doi":"10.24963/ijcai.2017/186","url":"https://ijcai.org/proceedings/2017/186","pdf":"https://ijcai.org/proceedings/2017/0186.pdf","markdown":"https://ijcai.org/proceedings/2017/186.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/186","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Knowledge Representation, Reasoning, and Logic","track":"Main track","number":186,"pages":{"first":1340,"last":1346},"keywords":["Knowledge Representation, Reasoning, and Logic: Action, Change and Causality","Uncertainty in AI: Graphical Models","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2011] S. Agrawal and N. Goyal. Anal- ysis of thompson sampling for the multi-armed bandit problem. CoRR, abs/1111.1797, 2011.","title":"Analysis of thompson sampling for the multi-armed bandit problem","authors":"S Agrawal, N Goyal","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bareinboim and Pearl, 2012] E. Bareinboim and J. Pearl. Causal inference by surrogate experiments: z- identifiability. In Nando de Freitas and Kevin Murphy, editors, Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, pages 113-120, Corvallis, OR, 2012. AUAI Press.","title":"Causal inference by surrogate experiments: zidentifiability","authors":"E Bareinboim, J Pearl","year":"2012","venue":"Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bareinboim and Pearl, 2016] E. Bareinboim and J. Pearl. Causal inference and the data-fusion problem. Proceed- ings of the National Academy of Sciences, 113:7345-7352, 2016.","title":"Causal inference and the data-fusion problem","authors":"E Bareinboim, J Pearl","year":"2016","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bareinboim et al., 2015] Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In Advances in Neural Information Processing Systems, pages 1342-1350, 2015.","title":"Bandits with unobserved confounders: A causal approach","authors":"Elias Bareinboim, Andrew Forney, Judea Pearl","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cappé et al., 2013] Olivier Cappé, Aurélien Garivier, Odalric-Ambrym Maillard, Rémi Munos, Gilles Stoltz, et al. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, 41(3):1516-1541, 2013.","title":"Kullback-leibler upper confidence bounds for optimal sequential allocation","authors":"Cappé","year":"2013","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pages 2249- 2257, 2011.","title":"An empirical evaluation of thompson sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heckerman and Shachter, 1995] D. Heckerman and R. Shachter. A definition and graphical representation for causality. In P. Besnard and S. Hanks, editors, Proceedings of the 11th Conference on Uncertainty in Artificial Intelligence, pages 262-273, San Francisco, 1995.","title":"A definition and graphical representation for causality","authors":"D Heckerman, R Shachter","year":"1995","venue":"Proceedings of the 11th Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Morgan Kaufmann. Huang and Valtorta, 2006] Y. Huang and M. Valtorta. Pearl's calculus of intervention is complete. In R. Dechter and T.S. Richardson, editors, Proceedings of the Twenty- Second Conference on Uncertainty in Artificial Intelli- gence, pages 217-224. AUAI Press, Corvallis, OR, 2006.","title":"Pearl's calculus of intervention is complete","authors":"Y Huang, M Valtorta","year":"2006","venue":"Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konidaris and Barto, 2007] George Konidaris and An- drew G Barto. Building portable options: Skill transfer in reinforcement learning. In IJCAI, volume 7, pages 895-900, 2007.","title":"George Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement learning","authors":"Barto Konidaris","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neural information processing systems, pages 817-824, 2008.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lazaric, 2012] Alessandro Lazaric. Transfer in reinforce- ment learning: a framework and a survey. In Reinforce- ment Learning, pages 143-173. Springer, 2012.","title":"Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey","authors":"Lazaric","year":"2012","venue":"Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Stone, 2006] Yaxin Liu and Peter Stone. Value- function-based transfer for reinforcement learning using structure mapping. 2006.","title":"Valuefunction-based transfer for reinforcement learning using structure mapping","authors":"Stone Liu, Yaxin Liu, Peter Stone","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mehta et al., 2008] Neville Mehta, Soumya Ray, Prasad Tadepalli, and Thomas Dietterich. Automatic discovery and transfer of maxq hierarchies. In Proceedings of the 25th international conference on Machine learning, pages 648-655. ACM, 2008.","title":"Automatic discovery and transfer of maxq hierarchies","authors":"Neville Mehta, Soumya Ray, Prasad Tadepalli, Thomas Dietterich","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mehta et al., 2011] Neville Mehta, Soumya Ray, Prasad Tadepalli, and Thomas Dietterich. Automatic discovery and transfer of task hierarchies in reinforcement learning. AI Magazine, 32(1):35-51, 2011.","title":"Automatic discovery and transfer of task hierarchies in reinforcement learning","authors":"Neville Mehta, Soumya Ray, Prasad Tadepalli, Thomas Dietterich","year":"2011","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pearl, 1995] J. Pearl. Causal diagrams for empirical re- search. Biometrika, 82(4):669-710, 1995.","title":"Causal diagrams for empirical research","authors":"J Pearl","year":"1995","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pearl, 2000] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, 2000. 2nd edition, 2009.","title":"Causality: Models, Reasoning, and Inference","authors":"J Pearl","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pearl, 2014] Judea Pearl. Is scientific knowledge useful for policy analysis? a peculiar theorem says: No. Journal of Causal Inference J. Causal Infer., 2(1):109-112, 2014.","title":"Judea Pearl. Is scientific knowledge useful for policy analysis? a peculiar theorem says: No","authors":"Pearl","year":"2014","venue":"Journal of Causal Inference J. Causal Infer","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shpitser and Pearl, 2006] I. Shpitser and J Pearl. Identi- fication of conditional interventional distributions. In R. Dechter and T.S. Richardson, editors, Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, pages 437-444. AUAI Press, Corvallis, OR, 2006.","title":"Identification of conditional interventional distributions","authors":"Pearl ; Shpitser, I Shpitser, Pearl","year":"2006","venue":"Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Strehl et al., 2010] Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit ex- ploration data. In Advances in Neural Information Pro- cessing Systems, pages 2217-2225, 2010.","title":"Learning from logged implicit exploration data","authors":"Alex Strehl, John Langford, Lihong Li, M Sham, Kakade","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Swaminathan and Joachims, 2015] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In ICML, pages 814-823, 2015.","title":"Counterfactual risk minimization: Learning from logged bandit feedback","authors":"Joachims Swaminathan, Adith Swaminathan, Thorsten Joachims","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Taylor and Stone, 2009] Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"E Matthew, Peter Taylor, Stone","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian and Pearl, 2002] J. Tian and J. Pearl. A general identi- fication condition for causal effects. In Proceedings of the Eighteenth National Conference on Artificial Intelligence, pages 567-573, Menlo Park, CA, 2002. AAAI Press/The MIT Press.","title":"A general identification condition for causal effects","authors":"J Tian, J Pearl","year":"2002","venue":"Proceedings of the Eighteenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Bareinboim, 2017] Junzhe Zhang and Elias Bareinboim. Transfer learning in multi-armed bandit: a causal approach. Technical Report R-25, Purdue AI Lab., 2017.","title":"Transfer learning in multi-armed bandit: a causal approach","authors":"Bareinboim Zhang, Junzhe Zhang, Elias Bareinboim","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}