{"type":"ScholarlyArticle","title":"Only Relevant Information Matters: Filtering Out Noisy Samples To Boost RL","authors":[{"name":"Yannis Flet-Berliac","slug":"yannis-flet-berliac","given":"Yannis","family":"Flet-Berliac","affiliation":"Inria CRIStAL Univ. Lille CNRS"},{"name":"Philippe Preux","slug":"philippe-preux","given":"Philippe","family":"Preux","affiliation":"Inria CRIStAL Univ. Lille CNRS"}],"abstract":"In reinforcement learning, policy gradient algorithms optimize the policy directly and rely on sampling efficiently an environment. Nevertheless, while most sampling procedures are based on direct policy sampling, self-performance measures could be used to improve such sampling prior to each policy update. Following this line of thought, we introduce SAUNA, a method where non-informative transitions are rejected from the gradient update. The level of information is estimated according to the fraction of variance explained by the value function: a measure of the discrepancy between V and the empirical returns. In this work, we use this criterion to select samples that are useful to learn from, and we demonstrate that this selection can significantly improve the performance of policy gradient methods. In this paper: (a) We introduce the SAUNA method to filter transitions. (b) We conduct experiments on a set of benchmark continuous control problems. SAUNA significantly improves performance. (c) We investigate how SAUNA reliably selects samples with the most positive impact on learning and study its improvement on both performance and sample efficiency.","doi":"10.24963/ijcai.2020/376","url":"https://ijcai.org/proceedings/2020/376","pdf":"https://ijcai.org/proceedings/2020/0376.pdf","markdown":"https://ijcai.org/proceedings/2020/376.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/376","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":376,"pages":{"first":2711,"last":2717},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24831/file_24831.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Amari, 1998] S. Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251-276, 1998.","title":"Natural gradient works efficiently in learning","authors":"S Amari","year":"1998","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andrew et al., 1983] A. Andrew, R. Sutton, and C. Ander- son. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13(5):834-846, 1983.","title":"Neuronlike adaptive elements that can solve difficult learning control problems","authors":"A Andrew, R Sutton, C Anderson","year":"1983","venue":"IEEE Transactions on Systems, Man, and Cybernetics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2013] M. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. JAIR, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"M Bellemare, Y Naddaf, J Veness, M Bowling","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] G. Brockman, V. Cheung, L. Pet- tersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI Gym. arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Espeholt et al., 2018] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architec- tures. In Proc. ICML, 2018.","title":"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures","authors":"L Espeholt, H Soyer, R Munos, K Simonyan, V Mnih, T Ward, Y Doron, V Firoiu, T Harley, I Dunning","year":"2018","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Flet-Berliac and Preux, 2019] Y. Flet-Berliac and P. Preux. MERL: Multi-Head Reinforcement Learning. In Deep Re- inforcement Learning Workshop, NeurIPS, 2019.","title":"MERL: Multi-Head Reinforcement Learning","authors":"Y Flet-Berliac, P Preux","year":"2019","venue":"Deep Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Freeman et al., 2019] D. Freeman, L. Metz, and D. Ha. Learning to predict without looking ahead: World models without forward prediction. In Proc. NeurIPS, 2019.","title":"Learning to predict without looking ahead: World models without forward prediction","authors":"D Freeman, L Metz, D Ha","year":"2019","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ha and Schmidhuber, 2018] D. Ha and J. Schmidhuber. Re- current world models facilitate policy evolution. In Proc. NIPS, 2018.","title":"Recurrent world models facilitate policy evolution","authors":"D Ha, J Schmidhuber","year":"2018","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haarnoja et al., 2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. ICML, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"T Haarnoja, A Zhou, P Abbeel, S Levine","year":"2018","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heess et al., 2015] N. Heess, G. Wayne, D. Silver, T. Lil- licrap, T. Erez, and Y. Tassa. Learning continuous con- trol policies by stochastic value gradients. In Proc. NIPS, 2015.","title":"Learning continuous control policies by stochastic value gradients","authors":"N Heess, G Wayne, D Silver, T Lillicrap, T Erez, Y Tassa","year":"2015","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hill et al., 2018] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schul- man, S. Sidor, and Y. Wu. Stable baselines, 2018.","title":"Stable baselines","authors":"A Hill, A Raffin, M Ernestus, A Gleave, A Kanervisto, R Traore, P Dhariwal, C Hesse, O Klimov, A Nichol, M Plappert, A Radford, J Schulman, S Sidor, Y Wu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kakade, 2002] S. Kakade. A natural policy gradient. In Proc. NIPS, 2002.","title":"A natural policy gradient","authors":"S Kakade","year":"2002","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kakade, 2003] S. Kakade. On the sample complexity of re- inforcement learning. PhD thesis, University of London, 2003.","title":"On the sample complexity of reinforcement learning","authors":"S Kakade","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klimov and Schulman, 2017] O. Klimov and J. Schulman. Roboschool, 2017. [Kvålseth, 1985]","title":"Roboschool","authors":"O Klimov, J Schulman","year":"1985","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"T. Kvålseth. Cautionary Note about R 2 . The American Statistician, 39(4):279-285, 1985.","title":"Cautionary Note about R 2","authors":"T Kvålseth","year":"1985","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lapan, 2018] M. Lapan. Deep Reinforcement Learning Hands-On. Packt Publishing, 2018.","title":"Deep Reinforcement Learning Hands-On","authors":"M Lapan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Levine and Abbeel, 2014] S. Levine and P. Abbeel. Learn- ing neural network policies with guided policy search un- der unknown dynamics. In Proc. NIPS, 2014.","title":"Learning neural network policies with guided policy search under unknown dynamics","authors":"S Levine, P Abbeel","year":"2014","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lillicrap et al., 2016] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In Proc. ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"T Lillicrap, J Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2016] V. Mnih, A. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proc. ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"V Mnih, A Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu","year":"2016","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nachum et al., 2017] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Bridging the gap between value and pol- icy based reinforcement learning. In Proc. NIPS, 2017.","title":"Bridging the gap between value and policy based reinforcement learning","authors":"O Nachum, M Norouzi, K Xu, D Schuurmans","year":"2017","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters and Schaal, 2008] J. Peters and S. Schaal. Reinforce- ment learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008.","title":"Reinforcement learning of motor skills with policy gradients","authors":"] J Peters, S Peters, Schaal","year":"2008","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Precup, 2000] D. Precup. Temporal Abstraction in Re- inforcement Learning. PhD thesis, University of Mas- sachusetts, 2000.","title":"Temporal Abstraction in Reinforcement Learning","authors":"D Precup","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Puterman, 1994] M. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"M Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Robinson and Fallside, 1989] A. Robinson and F. Fallside. Dynamic reinforcement driven error propagation networks with application to game playing. In Conference of the Cognitive Science Society, 1989.","title":"Dynamic reinforcement driven error propagation networks with application to game playing","authors":"A Robinson, F Fallside","year":"1989","venue":"Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schaul et al., 2016] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In Proc. ICLR, 2016.","title":"Prioritized experience replay","authors":"T Schaul, J Quan, I Antonoglou, D Silver","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schmidhuber and Huber, 1991] J. Schmidhuber and R. Hu- ber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2(1/2):135-141, 1991.","title":"Learning to generate artificial fovea trajectories for target detection","authors":"Schmidhuber, ] J Huber, R Schmidhuber, Huber","year":"1991","venue":"International Journal of Neural Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schmidhuber, 1991] J. Schmidhuber. Curious model- building control systems. In Proc. IEEE IJCNN, 1991.","title":"Curious modelbuilding control systems","authors":"J Schmidhuber","year":"1991","venue":"Proc. IEEE IJCNN","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schulman et al., 2016] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proc. ICLR, 2016.","title":"High-dimensional continuous control using generalized advantage estimation","authors":"J Schulman, P Moritz, S Levine, M Jordan, P Abbeel","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Silver et al., 2014] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gra- dient algorithms. In Proc. ICML, 2014.","title":"Deterministic policy gradient algorithms","authors":"D Silver, G Lever, N Heess, T Degris, D Wierstra, M Riedmiller","year":"2014","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Silver et al., 2016] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529:484, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"D Silver, A Huang, C Maddison, A Guez, L Sifre, G Van Den Driessche, J Schrittwieser, I Antonoglou, V Panneershelvam, M Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Srivastava et al., 2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"N Srivastava, G Hinton, A Krizhevsky, I Sutskever, R Salakhutdinov","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sutton and Barto, 2018] R. Sutton and A. Barto. Reinforce- ment learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"R Sutton, A Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sutton et al., 1999] R. Sutton, D. Precup, and S. Singh. Be- tween mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelli- gence, 112(1-2):181-211, 1999.","title":"Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning","authors":"R Sutton, D Precup, S Singh","year":"1999","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sutton et al., 2000] R. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proc. NIPS, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"R Sutton, D Mcallester, S Singh, Y Mansour","year":"2000","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Todorov et al., 2012] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ ICIRS, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"E Todorov, T Erez, Y Tassa","year":"2012","venue":"IEEE/RSJ ICIRS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2017] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. Sample efficient actor-critic with experience replay. In Proc. ICLR, 2017.","title":"Sample efficient actor-critic with experience replay","authors":"Z Wang, V Bapst, N Heess, V Mnih, R Munos, K Kavukcuoglu, N De Freitas","year":"2017","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Werbos, 1989] P. Werbos. Neural networks for control and system identification. In IEEE CDC, 1989.","title":"Neural networks for control and system identification","authors":"P Werbos","year":"1989","venue":"IEEE CDC","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Williams, 1992] R. Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradientfollowing algorithms for connectionist reinforcement learning","authors":"R Williams","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wu et al., 2017] Y. Wu, E. Mansimov, R. Grosse, S. Liao, and J. Ba. Scalable trust-region method for deep reinforce- ment learning using kronecker-factored approximation. In Proc. NIPS, 2017.","title":"Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation","authors":"Y Wu, E Mansimov, R Grosse, S Liao, J Ba","year":"2017","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}