{"type":"ScholarlyArticle","title":"Learning Sparse Representations in Reinforcement Learning with Sparse Coding","authors":[{"name":"Lei Le","slug":"lei-le","given":"Lei","family":"Le","affiliation":"Dept. of Computer Science Indiana University Bloomington , IN , USA"},{"name":"Raksha Kumaraswamy","slug":"raksha-kumaraswamy","given":"Raksha","family":"Kumaraswamy","affiliation":"Dept. of Computer Science Indiana University Bloomington , IN , USA"},{"name":"Martha White","slug":"martha-white","given":"Martha","family":"White","affiliation":"Dept. of Computer Science Indiana University Bloomington , IN , USA"}],"abstract":"A variety of representation learning approaches have been investigated for reinforcement learning; much less attention, however, has been given to investigating the utility of sparse coding. Outside of reinforcement learning, sparse coding representations have been widely used, with non-convex objectives that result in discriminative representations. In this work, we develop a supervised sparse coding objective for policy evaluation. Despite the non-convexity of this objective, we prove that all local minima are global minima, making the approach amenable to simple optimization strategies. We empirically show that it is key to use a supervised objective, rather than the more straightforward unsupervised sparse coding approach. We then compare the learned representations to a canonical fixed sparse representation, called tile-coding, demonstrating that the sparse coding representation outperforms a wide variety of tile-coding representations.","doi":"10.24963/ijcai.2017/287","url":"https://ijcai.org/proceedings/2017/287","pdf":"https://ijcai.org/proceedings/2017/0287.pdf","markdown":"https://ijcai.org/proceedings/2017/287.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/287","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":287,"pages":{"first":2067,"last":2073},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Feature Selection/Construction"],"videos":[],"references":[{"position":1,"raw":"Alekh Agarwal, Animashree Anandkumar, Prateek Jain, and Praneeth Netrapalli. Learning sparsely used overcomplete dictionaries via alternating minimization. In Ann. Conf. on Learning Theory, 2014.","title":"Learning sparsely used overcomplete dictionaries via alternating minimization","authors":"Alekh Agarwal, Animashree Anandkumar, Prateek Jain, Praneeth Netrapalli","year":"2014","venue":"Ann. Conf. on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on Signal Processing, 2006.","title":"K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation","authors":"Michal Aharon, Michael Elad, Alfred Bruckstein","year":"2006","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Haitham B Ammar, Karl Tuyls, Matthew E Taylor, Kurt Driessens, and Gerhard Weiss. Reinforcement learning transfer via sparse coding. In Inter. Conf. on Autonomous Agents and Multiagent Systems, 2012.","title":"Reinforcement learning transfer via sparse coding","authors":"Karl Haitham B Ammar, Matthew E Tuyls, Kurt Taylor, Gerhard Driessens, Weiss","year":"2012","venue":"Inter. Conf. on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. arXiv:1503.00778v1 [cs.LG], 2015.","title":"Simple, efficient, and neural algorithms for sparse coding","authors":"Sanjeev Arora, Rong Ge, Tengyu Ma, Ankur Moitra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Christopher G Atkeson and Jun Morimoto. Nonparametric represen- tation of policies and value functions: a trajectory-based approach. In Advances in Neural Information Processing Systems, 2003.","title":"Nonparametric representation of policies and value functions: a trajectory-based approach","authors":"Jun Christopher G Atkeson, Morimoto","year":"2003","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Inter. Conf. on Mach. Learning, 1995.","title":"Residual algorithms: Reinforcement learning with function approximation","authors":"Leemon Baird","year":"1995","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Shalabh Bhatnagar, Doina Precup, David Silver, Richard S Sutton, Hamid R Maei, and Csaba Szepesvári. Convergent temporal- difference learning with arbitrary smooth function approximation. In Advances in Neural Information Processing Systems, 2009.","title":"Convergent temporaldifference learning with arbitrary smooth function approximation","authors":"Shalabh Bhatnagar, Doina Precup, David Silver, Richard S Sutton, Hamid R Maei, Csaba Szepesvári","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Andrea Braides. Local minimization, variational evolution and Γ- convergence. Lecture Notes in Mathematics, 2013.","title":"Local minimization, variational evolution and Γconvergence","authors":"Andrea Braides","year":"2013","venue":"Lecture Notes in Mathematics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Rémi Coulom. Reinforcement learning using neural networks, with applications to motor control. PhD thesis, INPG, 2002.","title":"Reinforcement learning using neural networks, with applications to motor control","authors":"Rémi Coulom","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Gianni Dal Maso. An Introduction to Γ-Convergence. Springer, 2012.","title":"An Introduction to Γ-Convergence","authors":"Gianni Dal, Maso","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Dotan Di Castro and Shie Mannor. Adaptive bases for Q-learning. In Conference on Decision and Control, 2010.","title":"Adaptive bases for Q-learning","authors":"Dotan Di, Castro, Shie Mannor","year":"2010","venue":"Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Kimon Fountoulakis and Jacek Gondzio. A second-order method for strongly convex ℓ1-regularization problems. Math. Prog., 2013.","title":"A second-order method for strongly convex ℓ1-regularization problems","authors":"Kimon Fountoulakis, Jacek Gondzio","year":"2013","venue":"Math. Prog","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Benjamin D Haeffele and Rene Vidal. Global Optimality in Tensor Factorization, Deep Learning, and Beyond. arXiv.org, 2015.","title":"Global Optimality in Tensor Factorization","authors":"D Benjamin, Rene Haeffele, Vidal","year":"2015","venue":"Deep Learning, and Beyond","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"J Zico Kolter and Andrew Y Ng. Regularization and feature selection in least-squares temporal difference learning. In Inter. Conf. on Mach. Learning, 2009.","title":"Regularization and feature selection in least-squares temporal difference learning","authors":"Zico Kolter, Andrew Y Ng","year":"2009","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"George Konidaris, Sarah Osentoski, and Philip Thomas. Value func- tion approximation in reinforcement learning using the Fourier basis. In Inter. Conf. on Mach. Learning, 2011.","title":"Value function approximation in reinforcement learning using the Fourier basis","authors":"George Konidaris, Sarah Osentoski, Philip Thomas","year":"2011","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Lei Le and Martha White. Global optimization of factor models using alternating minimization. arXiv.org:1604.04942v3, 2017.","title":"Global optimization of factor models using alternating minimization","authors":"Lei Le, Martha White","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Manuel Loth, Manuel Davy, and Philippe Preux. Sparse temporal difference learning using LASSO. In Symposium on Approximate Dynamic Programming and Reinforcement Learning, 2007.","title":"Sparse temporal difference learning using LASSO","authors":"Manuel Loth, Manuel Davy, Philippe Preux","year":"2007","venue":"Symposium on Approximate Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: a Laplacian framework for learning representation and control in Markov decision processes. J. Machine Learning Research, 2007.","title":"Proto-value functions: a Laplacian framework for learning representation and control in Markov decision processes","authors":"Sridhar Mahadevan, Mauro Maggioni","year":"2007","venue":"J. Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Sridhar Mahadevan, Stephen Giguere, and Nicholas Jacek. Basis Adaptation for Sparse Nonlinear Reinforcement Learning. In AAAI Conference on Artificial Intelligence, 2013.","title":"Basis Adaptation for Sparse Nonlinear Reinforcement Learning","authors":"Stephen Sridhar Mahadevan, Nicholas Giguere, Jacek","year":"2013","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Sridhar Mahadevan. Learning Representation and Control in Markov Decision Processes: New Frontiers. Foundations and Trends R in Machine Learning, 2009.","title":"Learning Representation and Control in Markov Decision Processes: New Frontiers","authors":"Sridhar Mahadevan","year":"2009","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Supervised dictionary learning. In Advances in Neural Information Processing Systems, 2009.","title":"Supervised dictionary learning","authors":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, Andrew Zisserman","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. On- line learning for matrix factorization and sparse coding. J. Machine Learning Res., 2010.","title":"Online learning for matrix factorization and sparse coding","authors":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro","year":"2010","venue":"J. Machine Learning Res","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Ishai Menache, Shie Mannor, and Nahum Shimkin. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research, 2005.","title":"Basis function adaptation in temporal difference reinforcement learning","authors":"Ishai Menache, Shie Mannor, Nahum Shimkin","year":"2005","venue":"Annals of Operations Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, An- dreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Trung Nguyen, Zhuoru Li, Tomi Silander, and Tze Yun Leong. On- line feature selection for model-based reinforcement learning. J. Machine Learning Research, 2013.","title":"Online feature selection for model-based reinforcement learning","authors":"Trung Nguyen, Zhuoru Li, Tomi Silander, Tze Yun Leong","year":"2013","venue":"J. Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"Bruno Olshausen and David Field. Sparse coding with an overcom- plete basis set: a strategy employed by V1? Vision Research, 1997.","title":"Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Research","authors":"Bruno Olshausen, David Field","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"Christopher Painter-Wakefield and Ronald Parr. Greedy algorithms for sparse reinforcement learning. In Inter. Conf. on Mach. Learn- ing, 2012.","title":"Greedy algorithms for sparse reinforcement learning","authors":"Christopher Painter, - Wakefield, Ronald Parr","year":"2012","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter-Wakefield, and Michael L Littman. An analysis of linear models linear value function approximation and feature selection for reinforcement learning. In Inter. Conf. on Mach. Learning, 2008.","title":"An analysis of linear models linear value function approximation and feature selection for reinforcement learning","authors":"Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter-Wakefield, Michael L Littman","year":"2008","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"Bohdana Ratitch and Doina Precup. Sparse distributed memories for on-line value-based reinforcement learning. In ECML PKDD, 2004.","title":"Sparse distributed memories for on-line value-based reinforcement learning","authors":"Bohdana Ratitch, Doina Precup","year":"2004","venue":"ECML PKDD","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"Martin Riedmiller. Neural fitted Q iteration -first experiences with a data efficient neural reinforcement learning method. In Machine Learning: ECML PKDD, 2005.","title":"Neural fitted Q iteration -first experiences with a data efficient neural reinforcement learning method","authors":"Martin Riedmiller","year":"2005","venue":"Machine Learning: ECML PKDD","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"Bruno Scherrer. Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view. In Inter. Conf. on Mach. Learning, 2010.","title":"Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view","authors":"Bruno Scherrer","year":"2010","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"Ajit Singh and Geoffrey Gordon. A unified view of matrix factoriza- tion models. In ECML PKDD, 2008.","title":"A unified view of matrix factorization models","authors":"Ajit Singh, Geoffrey Gordon","year":"2008","venue":"ECML PKDD","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"Kenneth O Stanley and Risto Miikkulainen. Efficient evolution of neural network topologies. In CEC, 2002.","title":"Efficient evolution of neural network topologies","authors":"O Kenneth, Risto Stanley, Miikkulainen","year":"2002","venue":"CEC","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"Wen Sun and J Andrew Bagnell. Online Bellman Residual Algo- rithms with Predictive Error Guarantees. In Conference on Uncer- tainty in Artificial Intelligence, 2015.","title":"Online Bellman Residual Algorithms with Predictive Error Guarantees","authors":"Wen Sun, Andrew Bagnell","year":"2015","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"Richard S Sutton and Andrew G Barto. Representation search through generate and test. In Proceedings of the AAAI Workshop on Learn- ing Rich Representations from Low-Level Sensors, 2013.","title":"Representation search through generate and test","authors":"S Richard, Andrew G Sutton, Barto","year":"2013","venue":"Proceedings of the AAAI Workshop on Learning Rich Representations from Low-Level Sensors","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"Richard S Sutton and Steven D Whitehead. Online learning with random representations. In Inter. Conf. on Mach. Learning, 1993.","title":"Online learning with random representations","authors":"S Richard, Steven D Sutton, Whitehead","year":"1993","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"Richard S Sutton, Hamid Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora. Fast gradient- descent methods for temporal-difference learning with linear func- tion approximation. In Inter. Conf. on Mach. Learning, 2009.","title":"Fast gradientdescent methods for temporal-difference learning with linear function approximation","authors":"Hamid Richard S Sutton, Doina Maei, Shalabh Precup, David Bhatnagar, Csaba Silver, Eric Szepesvári, Wiewiora","year":"2009","venue":"Inter. Conf. on Mach. Learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"Richard S Sutton. Generalization in reinforcement learning: Success- ful examples using sparse coarse coding. In Advances in Neural Information Processing Systems, 1996.","title":"Generalization in reinforcement learning: Successful examples using sparse coarse coding","authors":"Richard S Sutton","year":"1996","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"Shimon Whiteson, Matthew E Taylor, and Peter Stone. Adaptive tile coding for value function approximation. Technical report, 2007.","title":"Adaptive tile coding for value function approximation","authors":"Shimon Whiteson, Matthew E Taylor, Peter Stone","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"Yangyang Xu and Wotao Yin. A block coordinate descent method for regularized multiconvex optimization with applications to non- negative tensor factorization and completion. SIAM J. Imaging Sciences, 2013.","title":"A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion","authors":"Yangyang Xu, Wotao Yin","year":"2013","venue":"SIAM J. Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"Huizhen Yu and Dimitri P Bertsekas. Basis function adaptation meth- ods for cost approximation in MDP. In Symposium on Adaptive Dynamic Programming and Reinforcement Learning, 2009.","title":"Basis function adaptation methods for cost approximation in MDP","authors":"Huizhen Yu, Dimitri P Bertsekas","year":"2009","venue":"Symposium on Adaptive Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}