{"type":"ScholarlyArticle","title":"Towards Long-delayed Sparsity: Learning a Better Transformer through Reward Redistribution","authors":[{"name":"Tianchen Zhu","slug":"tianchen-zhu","given":"Tianchen","family":"Zhu","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University; School of Computer Science and Engineering , Beihang University; Shenyuan Honors College , Beihang University"},{"name":"Yue Qiu","slug":"yue-qiu","given":"Yue","family":"Qiu","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University; School of Computer Science and Engineering , Beihang University"},{"name":"Haoyi Zhou","slug":"haoyi-zhou","given":"Haoyi","family":"Zhou","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University; School of Software , Beihang University"},{"name":"Jianxin Li","slug":"jianxin-li","given":"Jianxin","family":"Li","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University; School of Computer Science and Engineering , Beihang University"}],"abstract":"Recently, Decision Transformer (DT) pioneered the offline RL into a contextual conditional sequence modeling paradigm, which leverages self-attended autoregression to learn from global target rewards, states, and actions. However, many applications have a severe delay of the above signals, such as the agent can only obtain a reward signal at the end of each trajectory. This delay causes an unwanted bias cumulating in autoregressive learning global signals. In this paper, we focused its virtual example on episodic reinforcement learning with trajectory feedback. We propose a new reward redistribution algorithm for learning parameterized reward functions, and it decomposes the long-delayed reward onto each timestep. To improve the redistributing's adaptation ability, we formulate the previous decomposition as a bi-level optimization problem for global optimal. We extensively evaluate the proposed method on various benchmarks and demonstrate an overwhelming performance improvement under long-delayed settings.","doi":"10.24963/ijcai.2023/522","url":"https://ijcai.org/proceedings/2023/522","pdf":"https://ijcai.org/proceedings/2023/0522.pdf","markdown":"https://ijcai.org/proceedings/2023/522.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/522","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":522,"pages":{"first":4693,"last":4701},"keywords":["Planning and Scheduling: PS: POMDPs","Machine Learning: ML: Deep reinforcement learning","Uncertainty in AI: UAI: Sequential decision making"],"videos":[],"references":[{"position":1,"raw":"[Arjona-Medina et al., 2019] Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. RUDDER: return decomposition for delayed rewards. In NeurIPS, pages 13544-13555, 2019.","title":"RUDDER: return decomposition for delayed rewards","authors":"Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, Sepp Hochreiter","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2015] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). In IJCAI, pages 4148-4152, 2015.","title":"The arcade learning environment: An evaluation platform for general agents (extended abstract)","authors":"G Marc, Yavar Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/585"},{"position":3,"raw":"[Bellman, 1966] Richard E. Bellman. Dynamic program- ming. Science, 153(3731):34-37, 1966.","title":"Dynamic programming","authors":"Richard E Bellman","year":"1966","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Lili Chen, Kevin Lu, Aravind Ra- jeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. De- cision transformer: Reinforcement learning via sequence modeling. In NeurIPS, pages 15084-15097, 2021.","title":"Decision transformer: Reinforcement learning via sequence modeling","authors":"Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chevalier-Boisvert et al., 2018] Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Min- imalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018. Accessed: 2023-06-25.","title":"Minimalistic gridworld environment for openai gym","authors":"Maxime Chevalier-Boisvert, Lucas Willems, Suman Pal","year":"2023","venue":null,"doi":null,"url":"https://github.com/maximecb/gym-minigrid,2018","internal_url":null},{"position":7,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. In NeurIPS, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2011] Sam Devlin, Daniel Kudenko, and Marek Grzes. An empirical study of potential-based re- ward shaping and advice in complex, multi-agent systems. Adv. Complex Syst., 14(2):251-278, 2011.","title":"An empirical study of potential-based reward shaping and advice in complex, multi-agent systems","authors":"Devlin","year":"2011","venue":"Adv. Complex Syst","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. [Efroni et al., 2021] Yonathan Efroni, Nadav Merlis, and Shie Mannor. Reinforcement learning with trajectory feed- back. In AAAI, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Emmons et al., 2022] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In ICLR, 2022.","title":"Rvs: What is essential for offline RL via supervised learning?","authors":"Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, volume 70, pages 1126-1135, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fu et al., 2018] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In ICLR, 2018.","title":"Learning robust rewards with adverserial inverse reinforcement learning","authors":"Justin Fu, Katie Luo, Sergey Levine","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fu et al., 2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.","title":"D4rl: Datasets for deep data-driven reinforcement learning","authors":"Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Fu et al., 2021] Xingcheng Fu, Jianxin Li, Jia Wu, Qingyun Sun, Cheng Ji, Senzhang Wang, Jiajun Tan, Hao Peng, and S Yu Philip. Ace-hgnn: Adaptive curvature exploration hyperbolic graph neural network. In ICDM, pages 111- 120, 2021.","title":"Ace-hgnn: Adaptive curvature exploration hyperbolic graph neural network","authors":"Xingcheng Fu, Jianxin Li, Jia Wu, Qingyun Sun, Cheng Ji, Senzhang Wang, Jiajun Tan, Hao Peng, S Yu, Philip","year":"2021","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gangwani et al., 2019] Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. In ICLR, 2019.","title":"Learning self-imitating diverse policies","authors":"Tanmay Gangwani, Qiang Liu, Jian Peng","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Gangwani et al., 2020] Tanmay Gangwani, Yuan Zhou, and Jian Peng. Learning guidance rewards with trajectory- space smoothing. In NeurIPS, 2020.","title":"Learning guidance rewards with trajectoryspace smoothing","authors":"Tanmay Gangwani, Yuan Zhou, Jian Peng","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Guo et al., 2016] Xiaoxiao Guo, Satinder Singh, Richard L. Lewis, and Honglak Lee. Deep learning for reward design to improve monte carlo tree search in ATARI games. In IJCAI, pages 1519-1525, 2016.","title":"Deep learning for reward design to improve monte carlo tree search in ATARI games","authors":"Xiaoxiao Guo, Satinder Singh, Richard L Lewis, Honglak Lee","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/218"},{"position":19,"raw":"[Guo et al., 2018] Yijie Guo, Junhyuk Oh, Satinder Singh, and Honglak Lee. Generative adversarial self-imitation learning. arXiv preprint arXiv:1812.00950, 2018.","title":"Generative adversarial self-imitation learning","authors":"Yijie Guo, Junhyuk Oh, Satinder Singh, Honglak Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Harutyunyan et al., 2019] Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Heess, Hado van Hasselt, Gregory Wayne, Satinder Singh, Doina Precup, and Rémi Munos. Hind- sight credit assignment. In NeurIPS, pages 12467-12476, 2019.","title":"Hindsight credit assignment","authors":"Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Gheshlaghi Azar, Bilal Piot, Nicolas Heess, Gregory Hado Van Hasselt, Satinder Wayne, Doina Singh, Rémi Precup, Munos","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NeurIPS, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hu et al., 2020] Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. In NeurIPS, 2020.","title":"Learning to utilize shaping rewards: A new approach of reward shaping","authors":"Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, Changjie Fan","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Janner et al., 2021] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big se- quence modeling problem. In NeurIPS, pages 1273-1286, 2021.","title":"Offline reinforcement learning as one big sequence modeling problem","authors":"Michael Janner, Qiyang Li, Sergey Levine","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Jiang et al., 2016] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard L. Lewis. The dependence of effective plan- ning horizon on model accuracy. In IJCAI, pages 4180- 4189, 2016.","title":"The dependence of effective planning horizon on model accuracy","authors":"Nan Jiang, Alex Kulesza, Satinder Singh, Richard L Lewis","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/626"},{"position":25,"raw":"[Li et al., 2023] Jianxin Li, Qingyun Sun, Hao Peng, Bein- ing Yang, Jia Wu, and S Yu Phillp. Adaptive subgraph neural network with reinforced critical structure mining. TPAMI, 2023.","title":"Adaptive subgraph neural network with reinforced critical structure mining","authors":"Jianxin Li, Qingyun Sun, Hao Peng, Beining Yang, Jia Wu, S Yu, Phillp","year":"2023","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2018] Hanxiao Liu, Karen Simonyan, and Yim- ing Yang. Darts: Differentiable architecture search. In ICLR, 2018.","title":"Darts: Differentiable architecture search","authors":"Liu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2019] Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, and Jian Peng. Sequence modeling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420, 2019.","title":"Sequence modeling of temporal credit assignment for episodic reinforcement learning","authors":"Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, Jian Peng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mataric, 1994] Maja J. Mataric. Reward functions for ac- celerated learning. In ICML, pages 181-189, 1994.","title":"Reward functions for accelerated learning","authors":"J Maja, Mataric","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ng and Russell, 2000] Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In ICML, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Andrew Y Ng, Stuart Russell","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ng et al., 1999] Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, pages 278-287, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Daishi Harada, Stuart Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Petrik and Scherrer, 2008] Marek Petrik and Bruno Scher- rer. Biasing approximate dynamic programming with a lower discount factor. In NeurIPS, pages 1265-1272, 2008.","title":"Biasing approximate dynamic programming with a lower discount factor","authors":"Scherrer Petrik, Marek Petrik, Bruno Scherrer","year":"2008","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Radford and Narasimhan, 2018] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. https://openai.com/research/ language-unsupervised, 2018. Accessed: 2023-06-25.","title":"Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training","authors":"Narasimhan Radford","year":"2018","venue":null,"doi":null,"url":"https://openai.com/research/language-unsupervised,2018","internal_url":null},{"position":34,"raw":"[Rajeswaran et al., 2020] Aravind Rajeswaran, Igor Mor- datch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. In ICML, volume 119, pages 7953-7963, 2020.","title":"A game theoretic framework for model based reinforcement learning","authors":"Aravind Rajeswaran, Igor Mordatch, Vikash Kumar","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ren et al., 2022] Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition. ICLR, 2022.","title":"Learning long-term reward redistribution via randomized return decomposition","authors":"Ren","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Schmidhuber, 2019] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards -just map them to actions. arXiv preprint arXiv:1912.02875, 2019.","title":"Reinforcement learning upside down: Don't predict rewards -just map them to actions","authors":"Juergen Schmidhuber","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Song et al., 2019] Shihong Song, Jiayi Weng, Hang Su, Dong Yan, Haosheng Zou, and Jun Zhu. Playing FPS games with environment-aware hierarchical reinforcement learning. In IJCAI, pages 3475-3482, 2019.","title":"Playing FPS games with environment-aware hierarchical reinforcement learning","authors":"Shihong Song, Jiayi Weng, Hang Su, Dong Yan, Haosheng Zou, Jun Zhu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/482"},{"position":38,"raw":"[Sorg et al., 2010] Jonathan Sorg, Satinder Singh, and Richard L. Lewis. Reward design via online gradient as- cent. In NeurIPS, pages 2190-2198, 2010.","title":"Reward design via online gradient ascent","authors":"Jonathan Sorg, Satinder Singh, Richard L Lewis","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Srivastava et al., 2019] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Wall Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber. Training agents using upside-down rein- forcement learning. arXiv preprint arXiv:1912.02877, 2019.","title":"Training agents using upside-down reinforcement learning","authors":"Rupesh Kumar Srivastava, Pranav Shyam, Filipe Wall Mutz, Wojciech Jaśkowski, Jürgen Schmidhuber","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 9(5):1054-1054, 1998.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":"IEEE Trans. Neural Networks","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Sutton, 1988] Richard S. Sutton. Learning to predict by the methods of temporal differences. Mach. Learn., 3:9-44, 1988.","title":"Learning to predict by the methods of temporal differences","authors":"Richard S Sutton","year":"1988","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Torabi et al., 2018] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In IJCAI, page 4950-4957, 2018.","title":"Behavioral cloning from observation","authors":"Faraz Torabi, Garrett Warnell, Peter Stone","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/687"},{"position":43,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wu and Tian, 2017] Yuxin Wu and Yuandong Tian. Train- ing agent for first-person shooter game with actor-critic curriculum learning. In ICLR, 2017.","title":"Training agent for first-person shooter game with actor-critic curriculum learning","authors":"Tian ; ] Yuxin Wu, Yuandong Wu, Tian","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Xu et al., 2018] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In NeurIPS, pages 2402-2413, 2018.","title":"Meta-gradient reinforcement learning","authors":"Xu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Yang et al., 2023] Yang Yang, Yurui Huang, Weili Guo, Baohua Xu, and Dingyin Xia. Towards global video scene segmentation with context-aware transformer. In AAAI, 2023.","title":"Towards global video scene segmentation with context-aware transformer","authors":"Yang Yang, Yurui Huang, Weili Guo, Baohua Xu, Dingyin Xia","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zheng et al., 2018] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In NeurIPS, pages 4649-4659, 2018.","title":"On learning intrinsic rewards for policy gradient methods","authors":"Zheng","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zheng et al., 2020] Zeyu Zheng, Junhyuk Oh, Matteo Hes- sel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, and Satinder Singh. What can learned intrin- sic rewards capture? In ICML, volume 119, pages 11436- 11446, 2020.","title":"What can learned intrinsic rewards capture?","authors":"Zheng","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zheng et al., 2022] Zeyu Zheng, Risto Vuorio, Richard L. Lewis, and Satinder Singh. Adaptive pairwise weights for temporal credit assignment. In AAAI, pages 9225-9232, 2022.","title":"Adaptive pairwise weights for temporal credit assignment","authors":"Zheng","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, pages 11106- 11115, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}