{"type":"ScholarlyArticle","title":"I4R: Promoting Deep Reinforcement Learning by the Indicator for Expressive Representations","authors":[{"name":"Xufang Luo","slug":"xufang-luo","given":"Xufang","family":"Luo","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , Beijing"},{"name":"Qi Meng","slug":"qi-meng","given":"Qi","family":"Meng","affiliation":"Microsoft Research , Beijing , China"},{"name":"Di He","slug":"di-he","given":"Di","family":"He","affiliation":"Key Laboratory of Machine Perception (MOE) , School of EECS , Peking University"},{"name":"Wei Chen","slug":"wei-chen","given":"Wei","family":"Chen","affiliation":"Microsoft Research , Beijing , China"},{"name":"Yunhong Wang","slug":"yunhong-wang","given":"Yunhong","family":"Wang","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , Beijing"}],"abstract":"Learning expressive representations is always crucial for well-performed policies in deep reinforcement learning (DRL). Different from supervised learning, in DRL, accurate targets are not always available, and some inputs with different actions only have tiny differences, which stimulates the demand for learning expressive representations. In this paper, firstly, we empirically compare the representations of DRL models with different performances. We observe that the representations of a better state extractor (SE) are more scattered than a worse one when they are visualized. Thus, we investigate the singular values of representation matrix, and find that, better SEs always correspond to smaller differences among these singular values. Next, based on such observations, we define an indicator of the representations for DRL model, which is the Number of Significant Singular Values (NSSV) of a representation matrix. Then, we propose I4R algorithm, to improve DRL algorithms by adding the corresponding regularization term to enhance the NSSV. Finally, we apply I4R to both policy gradient and value based algorithms on Atari games, and the results show the superiority of our proposed method.","doi":"10.24963/ijcai.2020/370","url":"https://ijcai.org/proceedings/2020/370","pdf":"https://ijcai.org/proceedings/2020/0370.pdf","markdown":"https://ijcai.org/proceedings/2020/370.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/370","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":370,"pages":{"first":2669,"last":2675},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning Applications: Game Playing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25018/file_25018.m3u8","length_sec":262,"provider":"hls"}],"references":[{"position":1,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. JAIR, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Donahue et al., 2017] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[François-Lavet et al., 2019] Vincent François-Lavet, Yoshua Bengio, Doina Precup, and Joelle Pineau. Com- bined reinforcement learning via abstract representations. In AAAI, 2019.","title":"Combined reinforcement learning via abstract representations","authors":"Vincent François-Lavet, Yoshua Bengio, Doina Precup, Joelle Pineau","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goel et al., 2018] Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. In NeurIPS, pages 5683-5694, 2018.","title":"Unsupervised video object segmentation for deep reinforcement learning","authors":"Vikash Goel, Jameson Weng, Pascal Poupart","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Higgins et al., 2017] Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexan- der Lerchner. Darla: Improving zero-shot transfer in rein- forcement learning. arXiv, 2017.","title":"Darla: Improving zero-shot transfer in reinforcement learning","authors":"Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, P Christopher, Alexander Burgess, Matthew Pritzel, Charles Botvinick, Alexander Blundell, Lerchner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hjelm et al., 2019] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. In ICLR, 2019.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Houle, 2017] Michael E Houle. Local intrinsic dimensional- ity i: An extreme-value-theoretic foundation for similarity applications. In SISAP, pages 64-79, 2017.","title":"Local intrinsic dimensionality i: An extreme-value-theoretic foundation for similarity applications","authors":"E Michael, Houle","year":"2017","venue":"SISAP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jonschkowski and Brock, 2015] Rico Jonschkowski and Oliver Brock. Learning state representations with robotic priors. Autonomous Robots, 39(3):407-428, 2015.","title":"Rico Jonschkowski and Oliver Brock. Learning state representations with robotic priors","authors":"Brock Jonschkowski","year":"2015","venue":"Autonomous Robots","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kostrikov, 2018] Ilya Kostrikov. PyTorch implementations of A3C. https://github.com/ikostrikov/pytorch-a3c, 2018. Accessed on Feb 28, 2018.","title":"PyTorch implementations of A3C","authors":"Kostrikov","year":"2018","venue":null,"doi":null,"url":"https://github.com/ikostrikov/pytorch-a3c","internal_url":null},{"position":11,"raw":"[Lerer and Peysakhovich, 2018] Adam Lerer and Alexander Peysakhovich. Learning social conventions in markov games. arXiv, 2018.","title":"Learning social conventions in markov games","authors":"Peysakhovich Lerer, Adam Lerer, Alexander Peysakhovich","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mattner et al., 2012] Jan Mattner, Sascha Lange, and Martin Riedmiller. Learn to swing up and balance a real pole based on raw visual input data. In ICONIP, pages 126-133, 2012.","title":"Learn to swing up and balance a real pole based on raw visual input data","authors":"Mattner","year":"2012","venue":"ICONIP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Munk et al., 2016] Jelle Munk, Jens Kober, and Robert Babuška. Learning state representation for deep actor-critic control. In CDC, pages 4667-4673, 2016.","title":"Learning state representation for deep actor-critic control","authors":"Jelle Munk, Jens Kober, Robert Babuška","year":"2016","venue":"CDC","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oh et al., 2017] Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In NeurIPS, pages 6118- 6128, 2017.","title":"Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network","authors":"Oh","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peysakhovich and Lerer, 2018] Alexander Peysakhovich and Adam Lerer. Consequentialist conditional cooperation in social dilemmas with imperfect information. In ICLR, 2018.","title":"Consequentialist conditional cooperation in social dilemmas with imperfect information","authors":"Lerer Peysakhovich, Alexander Peysakhovich, Adam Lerer","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Racanière et al., 2017] Sébastien Racanière, Théophane We- ber, David P Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adria Puigdomènech Badia, Oriol Vinyals, Nico- las Heess, Yujia Li, et al. Imagination-augmented agents for deep reinforcement learning. In NeurIPS, pages 5690- 5701, 2017.","title":"Imagination-augmented agents for deep reinforcement learning","authors":"Sébastien Racanière, Théophane Weber, David P Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adria Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shelhamer et al., 2016] Evan Shelhamer, Parsa Mah- moudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. arXiv, 2016.","title":"Loss is its own reward: Self-supervision for reinforcement learning","authors":"Evan Shelhamer, Parsa Mahmoudieh, Max Argus, Trevor Darrell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tran et al., 2017] Luan Tran, Xi Yin, and Xiaoming Liu. Dis- entangled representation learning GAN for pose-invariant face recognition. In CVPR, pages 1415-1424, 2017.","title":"Disentangled representation learning GAN for pose-invariant face recognition","authors":"Luan Tran, Xi Yin, Xiaoming Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. JMLR, 11(Dec.):3371-3408, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In ICML, pages 1995-2003, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, Nando Freitas","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Watter et al., 2015] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to con- trol: A locally linear latent dynamics model for control from raw images. In NeurIPS, pages 2746-2754, 2015.","title":"Embed to control: A locally linear latent dynamics model for control from raw images","authors":"Manuel Watter, Jost Springenberg, Joschka Boedecker, Martin Riedmiller","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learn- ing. arXiv, 2018.","title":"Decoupling dynamics and reward for transfer learning","authors":"Amy Zhang, Harsh Satija, Joelle Pineau","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao and Eskenazi, 2016] Tiancheng Zhao and Maxine Es- kenazi. Towards end-to-end learning for dialog state track- ing and management using deep reinforcement learning. arXiv, 2016.","title":"Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning","authors":"Eskenazi Zhao, Tiancheng Zhao, Maxine Eskenazi","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}