{"type":"ScholarlyArticle","title":"Reinforcement Learning with Dynamic Boltzmann Softmax Updates","authors":[{"name":"Ling Pan","slug":"ling-pan","given":"Ling","family":"Pan","affiliation":"IIIS , Tsinghua University"},{"name":"Qingpeng Cai","slug":"qingpeng-cai","given":"Qingpeng","family":"Cai","affiliation":"Alibaba Group"},{"name":"Qi Meng","slug":"qi-meng","given":"Qi","family":"Meng","affiliation":"Microsoft Research"},{"name":"Wei Chen","slug":"wei-chen","given":"Wei","family":"Chen","affiliation":"Microsoft Research"},{"name":"Longbo Huang","slug":"longbo-huang","given":"Longbo","family":"Huang","affiliation":"IIIS , Tsinghua University"}],"abstract":"Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the non-expansion property, and its direct use may fail to converge even in value iteration. In this paper, we propose to update the value function with dynamic Boltzmann softmax (DBS) operator, which has good convergence property in the setting of planning and learning. Experimental results on GridWorld show that the DBS operator enables better estimation of the value function, which rectifies the convergence issue of the softmax operator. Finally, we propose the DBS-DQN algorithm by applying the DBS operator, which outperforms DQN substantially in 40 out of 49 Atari games.","doi":"10.24963/ijcai.2020/276","url":"https://ijcai.org/proceedings/2020/276","pdf":"https://ijcai.org/proceedings/2020/0276.pdf","markdown":"https://ijcai.org/proceedings/2020/276.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/276","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":276,"pages":{"first":1992,"last":1998},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24718/file_24718.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Asadi and Littman, 2017] Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforce- ment learning. In International Conference on Machine Learning, pages 243-252, 2017.","title":"An alternative softmax operator for reinforcement learning","authors":"Littman Asadi, Kavosh Asadi, Michael L Littman","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Azar et al., 2012] Mohammad Gheshlaghi Azar, Vicenc ¸ Gómez, and Hilbert J Kappen. Dynamic policy pro- gramming. Journal of Machine Learning Research, 13(Nov):3207-3245, 2012.","title":"Dynamic policy programming","authors":"Mohammad Gheshlaghi Azar, ¸ Vicenc, Hilbert J Gómez, Kappen","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banach, 1922] Stefan Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fund. math, 3(1):133-181, 1922.","title":"Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales","authors":"Stefan Banach","year":"1922","venue":"Fund. math","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellemare et al., 2016] Marc G Bellemare, Georg Ostro- vski, Arthur Guez, Philip S Thomas, and Rémi Munos. In- creasing the action gap: New operators for reinforcement learning. In AAAI, pages 1476-1483, 2016.","title":"Increasing the action gap: New operators for reinforcement learning","authors":"Georg Marc G Bellemare, Arthur Ostrovski, Philip S Guez, Rémi Thomas, Munos","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bellman, 1957] Richard Ernest Bellman. Dynamic pro- gramming. 1957.","title":"Dynamic programming","authors":"Richard Ernest, Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cesa-Bianchi et al., 2017] Nicolò Cesa-Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. Boltzmann ex- ploration done right. In Advances in Neural Information Processing Systems, pages 6284-6293, 2017.","title":"Boltzmann exploration done right","authors":"Nicolò Cesa-Bianchi, Claudio Gentile, Gábor Lugosi, Gergely Neu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2018] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear func- tion approximation. In International Conference on Ma- chine Learning, pages 1133-1142, 2018.","title":"Sbeed: Convergent reinforcement learning with nonlinear function approximation","authors":"Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, Le Song","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fox et al., 2015] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft up- dates. arXiv preprint arXiv:1512.08562, 2015.","title":"Taming the noise in reinforcement learning via soft updates","authors":"Roy Fox, Ari Pakman, Naftali Tishby","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learn- ing with deep energy-based policies. arXiv preprint arXiv:1702.08165, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Se- hoon Ha, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. arXiv preprint arXiv:1812.11103, 2018.","title":"Learning to walk via deep reinforcement learning","authors":"Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, George Tucker, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Hado V Hasselt. Double q-learning. In Ad- vances in Neural Information Processing Systems, pages 2613-2621, 2010.","title":"Double q-learning","authors":"V Hado, Hasselt","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hessel et al., 2017] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep re- inforcement learning. arXiv preprint arXiv:1710.02298, 2017.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kober et al., 2013] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238- 1274, 2013.","title":"Reinforcement learning in robotics: A survey","authors":"Kober","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Littman and Szepesvári, 1996] Michael L Littman and Csaba Szepesvári. A generalized reinforcement-learning model: Convergence and applications. In International Conference on Machine Learning, pages 310-318, 1996.","title":"A generalized reinforcement-learning model: Convergence and applications","authors":"L Michael, Csaba Littman, Szepesvári","year":"1996","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Littman et al., 1996] Michael L Littman, Andrew W Moore, et al. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4(11, 28):237-285, 1996.","title":"Reinforcement learning: A survey","authors":"Andrew W Michael L Littman, Moore","year":"1996","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[O'Donoghue et al., 2016] Brendan O'Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626, 2016.","title":"Combining policy gradient and q-learning","authors":"O'donoghue","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pan et al., 2019] Ling Pan, Qingpeng Cai, and Longbo Huang. Multi-path policy optimization. arXiv preprint arXiv:1911.04207, 2019.","title":"Multi-path policy optimization","authors":"Ling Pan, Qingpeng Cai, Longbo Huang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Singh et al., 2000] Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning, 38(3):287-308, 2000.","title":"Convergence results for single-step on-policy reinforcement-learning algorithms","authors":"Satinder Singh, Tommi Jaakkola, Csaba Michael L Littman, Szepesvári","year":"2000","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2019] Zhao Song, Ronald Parr, and Lawrence Carin. Revisiting the softmax bellman operator: New ben- efits and new perspective. 2019.","title":"Revisiting the softmax bellman operator: New benefits and new perspective","authors":"Zhao Song, Ronald Parr, Lawrence Carin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton, 1992] Richard S Sutton. Adapting bias by gradi- ent descent: An incremental version of delta-bar-delta. In AAAI, pages 171-176, 1992.","title":"Adapting bias by gradient descent: An incremental version of delta-bar-delta","authors":"Richard S Sutton","year":"1992","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Thirtieth AAAI conference on artificial intelligence, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"Thirtieth AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[van Hasselt, 2013] Hado van Hasselt. Estimating the max- imum expected value: an analysis of (nested) cross vali- dation and the maximum sample average. arXiv preprint arXiv:1302.7175, 2013.","title":"Hado van Hasselt. Estimating the maximum expected value: an analysis of (nested) cross validation and the maximum sample average","authors":"Van Hasselt","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2015] Ziyu Wang, Tom Schaul, Matteo Hes- sel, Hado Van Hasselt, Marc Lanctot, and Nando De Fre- itas. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581, 2015.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, Nando De Freitas","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's Col- lege, Cambridge, 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2018] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. arXiv preprint arXiv:1805.09801, 2018.","title":"Meta-gradient reinforcement learning","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}