{"type":"ScholarlyArticle","title":"DeepMellow: Removing the Need for a Target Network in Deep Q-Learning","authors":[{"name":"Seungchan Kim","slug":"seungchan-kim","given":"Seungchan","family":"Kim","affiliation":"Brown University Department of Computer Science"},{"name":"Kavosh Asadi","slug":"kavosh-asadi","given":"Kavosh","family":"Asadi","affiliation":"Brown University Department of Computer Science"},{"name":"Michael Littman","slug":"michael-littman","given":"Michael","family":"Littman","affiliation":"Brown University Department of Computer Science"},{"name":"George Konidaris","slug":"george-konidaris","given":"George","family":"Konidaris","affiliation":"Brown University Department of Computer Science"}],"abstract":"Deep Q-Network (DQN) is an algorithm that achieves human-level performance in complex domains like Atari games. One of the important elements of DQN is its use of a target network, which is necessary to stabilize learning. We argue that using a target network is incompatible with online reinforcement learning, and it is possible to achieve faster and more stable learning without a target network when we use Mellowmax, an alternative softmax operator. We derive novel properties of Mellowmax, and empirically show that the combination of DQN and Mellowmax, but without a target network, outperforms DQN with a target network.","doi":"10.24963/ijcai.2019/379","url":"https://ijcai.org/proceedings/2019/379","pdf":"https://ijcai.org/proceedings/2019/0379.pdf","markdown":"https://ijcai.org/proceedings/2019/379.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/379","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":379,"pages":{"first":2733,"last":2739},"keywords":["Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Asadi and Littman, 2017] Kavosh Asadi and Michael L. Littman. An Alternative Softmax Operator for Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning, pages 243-252, 2017.","title":"An Alternative Softmax Operator for Reinforcement Learning","authors":"Littman Asadi, Kavosh Asadi, Michael L Littman","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The Arcade Learning Environment: An Evaluation Platform for General Agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellman, 1957] Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, 6(5), 1957.","title":"A Markovian Decision Process","authors":"Richard Bellman","year":"1957","venue":"Journal of Mathematics and Mechanics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boyd and Vandenberghe, 2004] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge university press, 2004.","title":"Convex Optimization","authors":"Vandenberghe Boyd","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brafman and Tennenholtz, 2002] Ronen I. Brafman and Moshe Tennenholtz. R-MAX-A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3:213-231, 2002.","title":"R-MAX-A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dai et al., 2018] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation. In Proceedings of the International Conference on Machine Learning, pages 1133-1142, 2018.","title":"SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation","authors":"Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, Le Song","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dearden et al., 1998] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian Q-learning. In AAAI/IAAI, pages 761-768, 1998.","title":"Bayesian Q-learning","authors":"Richard Dearden, Nir Friedman, Stuart Russell","year":"1998","venue":"AAAI/IAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fox et al., 2016] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the Noise in Reinforcement Learning via Soft Updates. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, 2016.","title":"Taming the Noise in Reinforcement Learning via Soft Updates","authors":"Roy Fox, Ari Pakman, Naftali Tishby","year":"2016","venue":"Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao and Pavel, 2017] Bolin Gao and Lacra Pavel. On the Properties of the Softmax function with Application in Game Theory and Reinforcement Learning. arXiv preprint arXiv:1704.00805, 2017.","title":"On the Properties of the Softmax function with Application in Game Theory and Reinforcement Learning","authors":"Pavel ; Gao, Bolin Gao, Lacra Pavel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep Learning. nature, 521(7553):436, 2015.","title":"Deep Learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Littman and Szepesvári, 1996] Michael L Littman and Csaba Szepesvári. A Generalized Reinforcement-Learning Model: Convergence and Applications. In ICML, volume 96, pages 310-318, 1996.","title":"A Generalized Reinforcement-Learning Model: Convergence and Applications","authors":"L Michael, Csaba Littman, Szepesvári","year":"1996","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level Control through Deep Reinforcement Learning. Nature, 518(7540):529-533, 2015.","title":"Human-level Control through Deep Reinforcement Learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nachum et al., 2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the Gap Between Value and Policy Based Reinforcement Learning. In Advances in Neural Information Processing Systems, pages 2775-2785, 2017.","title":"Bridging the Gap Between Value and Policy Based Reinforcement Learning","authors":"Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Neu et al., 2017] Gergely Neu, Anders Jonsson, and Vicenc ¸ Gómez. A Unified View of Entropy-Regularized Markov Decision Processes. arXiv preprint arXiv:1705.07798, 2017.","title":"A Unified View of Entropy-Regularized Markov Decision Processes","authors":"Gergely Neu, Anders Jonsson, Vicenc, Gómez","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rummery and Niranjan, 1994] G. A. Rummery and M. Niranjan. On-line Q-learning Using Connectionist Systems. Technical Report CUED/F-INFENG/TR 166, Cambridge University Engineering Department, 1994.","title":"On-line Q-learning Using Connectionist Systems","authors":"G A Rummery, M Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Safak, 1993] Aysel Safak. Statistical Analysis of the Power Sum of Multiple Correlated Log-Normal Components. IEEE Transactions on Vehicular Technology, 42(1):58-61, 1993.","title":"Statistical Analysis of the Power Sum of Multiple Correlated Log-Normal Components","authors":"Aysel Safak, Safak","year":"1993","venue":"IEEE Transactions on Vehicular Technology","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stahl II and Wilson, 1994] Dale O Stahl II and Paul W Wilson. Experimental Evidence on Players' Models of Other Players. Journal of Economic Behavior & Organization, 25(3):309-327, 1994.","title":"Experimental Evidence on Players' Models of Other Players","authors":"I I Stahl, ; ] Wilson, O Dale, I I Stahl, Paul W Wilson","year":"1994","venue":"Journal of Economic Behavior & Organization","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Strehl et al., 2006] Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L Littman. PAC Model-Free Reinforcement Learning. In Proceedings of the 23rd International Conference on Machine Learning, pages 881-888, 2006.","title":"PAC Model-Free Reinforcement Learning","authors":"Lihong Alexander L Strehl, Eric Li, John Wiewiora, Langford, Michael L Littman","year":"2006","venue":"Proceedings of the 23rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning -An Introduction. MIT Press, 1998.","title":"Reinforcement Learning -An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[van Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2094-2100, 2016.","title":"Deep Reinforcement Learning with Double Q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[van Hasselt, 2010] Hado van Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems 23, pages 2613-2621, 2010.","title":"Hado van Hasselt. Double Q-learning","authors":"Van Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van Seijen and Sutton, 2014] Harm van Seijen and Rich Sutton. True Online TD (lambda). In Proceedings of the International Conference on Machine Learning, pages 692-700, 2014.","title":"Harm van Seijen and Rich Sutton. True Online TD (lambda)","authors":"Sutton Van Seijen","year":"2014","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of the 33nd International Conference on Machine Learning, pages 1995-2003, 2016.","title":"Dueling Network Architectures for Deep Reinforcement Learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Marc Hado Van Hasselt, Nando Lanctot, De Freitas","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}