{"type":"ScholarlyArticle","title":"Self-Attentional Credit Assignment for Transfer in Reinforcement Learning","authors":[{"name":"Johan Ferret","slug":"johan-ferret","given":"Johan","family":"Ferret","affiliation":"Google Research , Brain Team"},{"name":"Raphael Marinier","slug":"raphael-marinier","given":"Raphael","family":"Marinier","affiliation":"Google Research , Brain Team"},{"name":"Matthieu Geist","slug":"matthieu-geist","given":"Matthieu","family":"Geist","affiliation":"Google Research , Brain Team"},{"name":"Olivier Pietquin","slug":"olivier-pietquin","given":"Olivier","family":"Pietquin","affiliation":"Google Research , Brain Team"}],"abstract":"The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample-efficient. Our main contribution is SECRET, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.","doi":"10.24963/ijcai.2020/368","url":"https://ijcai.org/proceedings/2020/368","pdf":"https://ijcai.org/proceedings/2020/0368.pdf","markdown":"https://ijcai.org/proceedings/2020/368.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/368","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":368,"pages":{"first":2655,"last":2661},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24483/file_24483.m3u8","length_sec":268,"provider":"hls"}],"references":[{"position":1,"raw":"[Arjona-Medina et al., 2019] Jose A Arjona-Medina, Michael Gill- hofer, Michael Widrich, Thomas Unterthiner, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. NeurIPS, 2019.","title":"Rudder: Return decomposition for delayed rewards","authors":"Michael Jose A Arjona-Medina, Michael Gillhofer, Thomas Widrich, Sepp Unterthiner, Hochreiter","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beattie et al., 2016] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.","title":"Deepmind lab","authors":"Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[de Bruin et al., 2018] Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters, 2018.","title":"Integrating state representation learning into deep reinforcement learning","authors":"De Bruin","year":"2018","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duan et al., 2017] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl$ˆ2$: Fast reinforce- ment learning via slow reinforcement learning. ICLR, 2017.","title":"Rl$ˆ2$: Fast reinforcement learning via slow reinforcement learning","authors":"Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ecoffet et al., 2019] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019.","title":"Go-explore: a new approach for hard-exploration problems","authors":"Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, Jeff Clune","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goertzel et al., 2012] Ben Goertzel, Matt Iklé, and Jared Wigmore. The architecture of human-like general intelligence. In Theoretical Foundations of Artificial General Intelligence. Springer, 2012.","title":"The architecture of human-like general intelligence","authors":"Ben Goertzel, Matt Iklé, Jared Wigmore","year":"2012","venue":"Theoretical Foundations of Artificial General Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goyal et al., 2019] Anirudh Goyal, Philemon Brakel, William Fe- dus, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, and Yoshua Bengio. Recall traces: Backtracking models for efficient reinforcement learning. ICLR, 2019.","title":"Recall traces: Backtracking models for efficient reinforcement learning","authors":"Anirudh Goyal, Philemon Brakel, William Fedus, Timothy Lillicrap, Sergey Levine, Hugo Larochelle, Yoshua Bengio","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Houthooft et al., 2019] Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In NeurIPS, 2019.","title":"Evolved policy gradients","authors":"Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, Openai, Jonathan Ho, Pieter Abbeel","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hung et al., 2018] Chia-Chun Hung, Timothy P. Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. CoRR, abs/1810.06721, 2018.","title":"Optimizing agent behavior over long time scales by transporting value","authors":"Chia-Chun Hung, Timothy P Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, Greg Wayne","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ke et al., 2018] Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Ben- gio. Sparse attentive backtracking: Temporal credit assignment through reminding. In NeurIPS, 2018.","title":"Sparse attentive backtracking: Temporal credit assignment through reminding","authors":"Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, C Michael, Chris Mozer, Yoshua Pal, Bengio","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In ICLR, 2017.","title":"A structured self-attentive sentence embedding","authors":"Lin","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin, 1992] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 1992.","title":"Self-improving reactive agents based on reinforcement learning, planning and teaching","authors":"Long-Ji Lin","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mishra et al., 2018] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018.","title":"A simple neural attentive meta-learner","authors":"Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and ap- plication to reward shaping. In ICML, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Parisotto et al., 2016] Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer re- inforcement learning. In ICLR, 2016.","title":"Actor-mimic: Deep multitask and transfer reinforcement learning","authors":"Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes. Wiley, 1994.","title":"Markov Decision Processes","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rusu et al., 2016a] Andrei A. Rusu, Sergio Gomez Colmenarejo, C ¸aglar Gülc ¸ehre, Guillaume Desjardins, James Kirkpatrick, Raz- van Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2016.","title":"Policy distillation","authors":"Rusu","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rusu et al., 2016b] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1606.04671, 2016.","title":"Progressive neural networks","authors":"Rusu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Savinov et al., 2019] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. ICLR, 2019.","title":"Episodic curiosity through reachability","authors":"Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, Sylvain Gelly","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NeurIPS, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taylor and Stone, 2009] Matthew E Taylor and Peter Stone. Trans- fer learning for reinforcement learning domains: A survey. JMLR, 2009.","title":"Transfer learning for reinforcement learning domains: A survey","authors":"E Matthew, Peter Taylor, Stone","year":"2009","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Teh et al., 2017] Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In NeurIPS, 2017.","title":"Distral: Robust multitask reinforcement learning","authors":"Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, Razvan Pascanu","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2016] Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tiru- mala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to rein- forcement learn. CoRR, abs/1611.05763, 2016.","title":"Learning to reinforcement learn","authors":"Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, Matthew Botvinick","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zoph et al., 2016] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. ACL 2016, 2016.","title":"Transfer learning for low-resource neural machine translation","authors":"Barret Zoph, Deniz Yuret, Jonathan May, Kevin Knight","year":"2016","venue":"ACL 2016","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}