{"type":"ScholarlyArticle","title":"Experience Replay Optimization","authors":[{"name":"Daochen Zha","slug":"daochen-zha","given":"Daochen","family":"Zha","affiliation":"Department of Computer Science and Engineering , Texas A&M University"},{"name":"Kwei-Herng Lai","slug":"kwei-herng-lai","given":"Kwei-Herng","family":"Lai","affiliation":"Department of Computer Science and Engineering , Texas A&M University"},{"name":"Kaixiong Zhou","slug":"kaixiong-zhou","given":"Kaixiong","family":"Zhou","affiliation":"Department of Computer Science and Engineering , Texas A&M University"},{"name":"Xia Hu","slug":"xia-hu","given":"Xia","family":"Hu","affiliation":"Department of Computer Science and Engineering , Texas A&M University"}],"abstract":"Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consider learning a replay policy to optimize the cumulative reward. Replay learning is challenging because the replay memory is noisy and large, and the cumulative reward is unstable. To address these issues, we propose a novel experience replay optimization (ERO) framework which alternately updates two policies: the agent policy, and the replay policy. The agent is updated to maximize the cumulative reward based on the replayed data, while the replay policy is updated to provide the agent with the most useful experiences. The conducted experiments on various continuous control tasks demonstrate the effectiveness of ERO, empirically showing promise in experience replay learning to improve the performance of off-policy reinforcement learning algorithms.","doi":"10.24963/ijcai.2019/589","url":"https://ijcai.org/proceedings/2019/589","pdf":"https://ijcai.org/proceedings/2019/0589.pdf","markdown":"https://ijcai.org/proceedings/2019/589.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/589","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":589,"pages":{"first":4243,"last":4249},"keywords":["Machine Learning: Deep Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Andrychowicz et al., 2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Pe- ter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In NeurIPS, 2017.","title":"Hindsight experience replay","authors":"Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mcgrew, Josh Tobin, Openai, Pieter Abbeel, Wojciech Zaremba","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fan et al., 2018] Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu. Learning to teach. In ICLR, 2018.","title":"Learning to teach","authors":"Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, Tie-Yan Liu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hessel et al., 2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Sil- ver. Rainbow: Combining improvements in deep rein- forcement learning. In AAAI, 2018.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Isele and Cosgun, 2018] David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In AAAI, 2018.","title":"Selective experience replay for lifelong learning","authors":"David Isele, Akansel Cosgun","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lin, 1992] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.","title":"Self-improving reactive agents based on reinforcement learning, planning and teaching","authors":"Long-Ji Lin","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin, 1993] Long-Ji Lin. Reinforcement learning for robots using neural networks. Technical report, Carnegie-Mellon Univ Pittsburgh PA School of Computer Science, 1993.","title":"Reinforcement learning for robots using neural networks","authors":"Long-Ji Lin","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu and Zou, 2017] Ruishan Liu and James Zou. The ef- fects of memory replay in reinforcement learning. arXiv preprint arXiv:1710.06574, 2017.","title":"The effects of memory replay in reinforcement learning","authors":"Zou Liu, Ruishan Liu, James Zou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mattar and Daw, 2018] Marcelo Gomes Mattar and Nathaniel D Daw. Prioritized memory access explains planning and hippocampal replay. bioRxiv, page 225664, 2018.","title":"Prioritized memory access explains planning and hippocampal replay","authors":"Marcelo Gomes, Mattar, Nathaniel D Daw","year":"2018","venue":"bioRxiv","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Work- shop, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"NIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Novati and Koumoutsakos, 2018] Guido Novati and Petros Koumoutsakos. Remember and forget for experience re- play. arXiv preprint arXiv:1807.05827, 2018.","title":"Remember and forget for experience replay","authors":"Guido Novati, Petros Koumoutsakos","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan et al., 2018] Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Organiz- ing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains. In IJ- CAI, 2018.","title":"Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains","authors":"Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, Martha White","year":"2018","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2018/666"},{"position":15,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. In ICML, 2016. [Shohamy and Daw, 2015] Daphna Shohamy and Nathaniel D Daw. Integrating memories to guide decisions. Current Opinion in Behavioral Sciences, 5:85-90, 2015.","title":"Integrating memories to guide decisions","authors":"Schaul","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In IROS, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In AAAI, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with expe- rience replay. In ICLR, 2017.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2018] Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie-Yan Liu. Learning to teach with dynamic loss functions. In NeurIPS, 2018.","title":"Learning to teach with dynamic loss functions","authors":"Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, Tie-Yan Liu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2018a] Tianbing Xu, Qiang Liu, Liang Zhao, Wei Xu, and Jian Peng. Learning to explore with meta-policy gradient. In ICML, 2018.","title":"Learning to explore with meta-policy gradient","authors":"Xu","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2018b] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In NeurIPS, 2018.","title":"Meta-gradient reinforcement learning","authors":"Xu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin and Pan, 2017] Haiyan Yin and Sinno Jialin Pan. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In AAAI, 2017.","title":"Knowledge transfer for deep reinforcement learning with hierarchical experience replay","authors":"Haiyan Yin, Sinno Jialin Pan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Sutton, 2017] Shangtong Zhang and Richard S Sutton. A deeper look at experience replay. NIPS Deep Reinforcement Learning Symposium, 2017.","title":"A deeper look at experience replay","authors":"Sutton ; Zhang, Shangtong Zhang, Richard S Sutton","year":"2017","venue":"NIPS Deep Reinforcement Learning Symposium","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zheng et al., 2018] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In NeurIPS, 2018.","title":"On learning intrinsic rewards for policy gradient methods","authors":"Zheng","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}