{"type":"ScholarlyArticle","title":"Episodic Memory Deep Q-Networks","authors":[{"name":"Zichuan Lin","slug":"zichuan-lin","given":"Zichuan","family":"Lin","affiliation":""},{"name":"Tianqi Zhao","slug":"tianqi-zhao","given":"Tianqi","family":"Zhao","affiliation":""},{"name":"Guangwen Yang","slug":"guangwen-yang","given":"Guangwen","family":"Yang","affiliation":"Tsinghua University"},{"name":"Lintao Zhang","slug":"lintao-zhang","given":"Lintao","family":"Zhang","affiliation":"Microsoft Research"}],"abstract":"Reinforcement learning (RL) algorithms have made huge progress in recent years by leveraging the power of deep neural networks (DNN). Despite the success, deep RL algorithms are known to be sample inefficient, often requiring many rounds of interactions with the environments to obtain satisfactory performances. Recently, episodic memory based RL has attracted attention due to its ability to latch on good actions quickly. In this paper, we present a simple yet effective biologically inspired RL algorithm called Episodic Memory Deep Q-Networks (EMDQN), which leverages episodic memory to supervise an agent during training. Experiments show that our proposed method leads to better sample efficiency and is more likely to find good policy. It only requires 1/5 of the interactions of DQN to achieve many state-of-the-art performances on Atari games, significantly outperforming regular DQN and other episodic memory based RL algorithms.","doi":"10.24963/ijcai.2018/337","url":"https://ijcai.org/proceedings/2018/337","pdf":"https://ijcai.org/proceedings/2018/0337.pdf","markdown":"https://ijcai.org/proceedings/2018/337.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/337","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":337,"pages":{"first":2433,"last":2439},"keywords":["Machine Learning: Reinforcement Learning","Heuristic Search and Game Playing: General Game Playing and General Video Game Playing"],"videos":[],"references":[{"position":1,"raw":"[Amarjyoti, 2017] Smruti Amarjyoti. Deep reinforcement learning for robotic manipulation-the state of the art. arXiv preprint arXiv:1701.08878, 2017.","title":"Deep reinforcement learning for robotic manipulation-the state of the art","authors":"Smruti Amarjyoti","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] M. G. Bellemare, Y. Naddaf, J. Ve- ness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Ar- tificial Intelligence Research, 47:253-279, jun 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"M G Bellemare, Y Naddaf, J Veness, M Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bentley, 1975] Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communica- tions of the ACM, 18(9):509-517, 1975.","title":"Multidimensional binary search trees used for associative searching","authors":"Jon Louis Bentley, Bentley","year":"1975","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blundell et al., 2016] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hass- abis. Model-free episodic control. arXiv preprint arXiv:1606.04460, 2016.","title":"Model-free episodic control","authors":"Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, Demis Hassabis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gershman and Daw, 2017] Samuel J Gershman and Nathaniel D Daw. Reinforcement learning and episodic memory in humans and animals: An integrative frame- work. Annual Review of Psychology, 68:101-128, 2017.","title":"Reinforcement learning and episodic memory in humans and animals: An integrative framework","authors":"Daw ; Gershman, J Samuel, Nathaniel D Gershman, Daw","year":"2017","venue":"Annual Review of Psychology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2016] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q- learning with model-based acceleration. In Proceedings of The 33rd International Conference on Machine Learning, pages 2829-2838, 2016.","title":"Continuous deep qlearning with model-based acceleration","authors":"Gu","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Harutyunyan et al., 2016] Anna Harutyunyan, Marc G Bellemare, Tom Stepleton, and Rémi Munos. Q (\\lambda) with off-policy corrections. In International Confer- ence on Algorithmic Learning Theory, pages 305-320. Springer, 2016.","title":"Q (\\lambda) with off-policy corrections","authors":"Anna Harutyunyan, Tom Marc G Bellemare, Rémi Stepleton, Munos","year":"2016","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Hado V Hasselt. Double q-learning. In Ad- vances in Neural Information Processing Systems, pages 2613-2621, 2010.","title":"Double q-learning","authors":"V Hado, Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Frank S He, Yang Liu, Alexander G Schwing, and Jian Peng. Learning to play in a day: Faster deep reinforcement learning by optimality tighten- ing. arXiv preprint arXiv:1611.01606, 2016.","title":"Learning to play in a day: Faster deep reinforcement learning by optimality tightening","authors":"Yang Frank S He, Alexander G Liu, Jian Schwing, Peng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heess et al., 2015] Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradi- ents. In Advances in Neural Information Processing Sys- tems, pages 2944-2952, 2015.","title":"Learning continuous control policies by stochastic value gradients","authors":"Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, Yuval Tassa","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson and Lindenstrauss, 1984] William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189- 206):1, 1984.","title":"Extensions of lipschitz mappings into a hilbert space","authors":"B William, Joram Johnson, Lindenstrauss","year":"1984","venue":"Contemporary mathematics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 1995] Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time se- ries. The handbook of brain theory and neural networks, 3361(10):1995, 1995.","title":"Convolutional networks for images, speech, and time series","authors":"Yann Lecun, Yoshua Bengio","year":"1995","venue":"The handbook of brain theory and neural networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lengyel and Dayan, 2007] Máté Lengyel and Peter Dayan. Hippocampal contributions to control: The third way. In NIPS, volume 20, pages 889-896, 2007.","title":"Hippocampal contributions to control: The third way","authors":"Dayan ; ] Máté Lengyel, Peter Lengyel, Dayan","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin, 1992] Long-H Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3/4):69-97, 1992.","title":"Self-improving reactive agents based on reinforcement learning, planning and teaching","authors":"Long-H Lin","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Munos et al., 2016] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off- policy reinforcement learning. In Advances in Neural In- formation Processing Systems, pages 1054-1062, 2016.","title":"Safe and efficient offpolicy reinforcement learning","authors":"Rémi Munos, Tom Stepleton, Anna Harutyunyan, Marc Bellemare","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennartz et al., 2011] CMA Pennartz, R Ito, PFMJ Ver- schure, FP Battaglia, and TW Robbins. The hippocampal- striatal axis in learning, prediction and goal-directed be- havior. Trends in neurosciences, 34(10):548-559, 2011.","title":"The hippocampalstriatal axis in learning, prediction and goal-directed behavior","authors":"Pennartz","year":"2011","venue":"Trends in neurosciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pritzel et al., 2017] Alexander Pritzel, Benigno Uria, Sri- ram Srinivasan, Adrià Puigdomènech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. arXiv preprint arXiv:1703.01988, 2017.","title":"Neural episodic control","authors":"Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, Charles Blundell","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schaul et al., 2016] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.","title":"Prioritized experience replay","authors":"Schaul","year":"2016","venue":"Proceedings of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Singh and Sutton, 1996] Satinder P Singh and Richard S Sutton. Reinforcement learning with replacing eligibility traces. Recent Advances in Reinforcement Learning, pages 123-158, 1996.","title":"Reinforcement learning with replacing eligibility traces","authors":"P Satinder, Richard S Singh, Sutton","year":"1996","venue":"Recent Advances in Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton, 1991] Richard S Sutton. Dyna, an integrated archi- tecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163, 1991.","title":"Dyna, an integrated architecture for learning, planning, and reacting","authors":"Richard S Sutton","year":"1991","venue":"ACM SIGART Bulletin","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thrun and Schwartz, 1993] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for re- inforcement learning. In Proceedings of the 1993 Con- nectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.","title":"Issues in using function approximation for reinforcement learning","authors":"Schwartz Thrun, Sebastian Thrun, Anton Schwartz","year":"1993","venue":"Proceedings of the 1993 Connectionist Models Summer School","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In AAAI, pages 2094-2100, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hes- sel, Hado van Hasselt, Marc Lanctot, and Nando de Fre- itas. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Con- ference on Machine Learning, pages 1995-2003, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Marc Hado Van Hasselt, Nando Lanctot, De Freitas","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}