{"type":"ScholarlyArticle","title":"Automatic Successive Reinforcement Learning with Multiple Auxiliary Rewards","authors":[{"name":"Zhao-Yang Fu","slug":"zhao-yang-fu","given":"Zhao-Yang","family":"Fu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"De-Chuan Zhan","slug":"de-chuan-zhan","given":"De-Chuan","family":"Zhan","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Xin-Chun Li","slug":"xin-chun-li","given":"Xin-Chun","family":"Li","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"},{"name":"Yi-Xing Lu","slug":"yi-xing-lu","given":"Yi-Xing","family":"Lu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing , China"}],"abstract":"Reinforcement learning has played an important role in decision making related applications, e.g., robotics motion, self-driving, recommendation, etc. The reward function, as a crucial component, affects the efficiency and effectiveness of reinforcement learning to a large extent. In this paper, we focus on the investigation of reinforcement learning with more than one auxiliary reward. It is found that different auxiliary rewards can boost up the learning rate and effectiveness in different stages, and consequently we propose the Automatic Successive Reinforcement Learning (ASR) for auxiliary rewards grading selection for efficient reinforcement learning by stages. Experiments and simulations have shown the superiority of our proposed ASR on a range of environments, including OpenAI classical control domains and video games; Freeway and Catcher.","doi":"10.24963/ijcai.2019/324","url":"https://ijcai.org/proceedings/2019/324","pdf":"https://ijcai.org/proceedings/2019/0324.pdf","markdown":"https://ijcai.org/proceedings/2019/324.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/324","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":324,"pages":{"first":2336,"last":2342},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Multi-instance;Multi-label;Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Amari, 1998] Shun-ichi Amari. Natural gradient works ef- ficiently in learning. Neural Computation, 10(2):251-276, 1998.","title":"Shun-ichi Amari. Natural gradient works efficiently in learning","authors":"Amari","year":"1998","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learn- ing. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41-48, Montréal, Canada, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brys et al., 2014a] Tim Brys, Anna Harutyunyan, Peter Vrancx, Matthew E. Taylor, Daniel Kudenko, and Ann Nowé. Multi-objectivization of reinforcement learning problems by reward shaping. In Proceedings of the 2014 International Joint Conference on Neural Networks, pages 2315-2322, Beijing, China, 2014.","title":"Multi-objectivization of reinforcement learning problems by reward shaping","authors":"Brys","year":"2014","venue":"Proceedings of the 2014 International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brys et al., 2014b] Tim Brys, Ann Nowé, Daniel Kudenko, and Matthew E. Taylor. Combining multiple correlated reward and shaping signals by measuring confidence. In Proceedings of the 28th AAAI Conference on Artificial In- telligence, pages 1687-1693, Québec, Canada, 2014.","title":"Combining multiple correlated reward and shaping signals by measuring confidence","authors":"Brys","year":"2014","venue":"Proceedings of the 28th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brys et al., 2017] Tim Brys, Anna Harutyunyan, Peter Vrancx, Ann Nowé, and Matthew E. Taylor. Multi- objectivization and ensembles of shapings in reinforce- ment learning. Neurocomputing, 263:48-59, 2017.","title":"Multiobjectivization and ensembles of shapings in reinforcement learning","authors":"Tim Brys, Anna Harutyunyan, Peter Vrancx, Ann Nowé, Matthew E Taylor","year":"2017","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2011] Sam Devlin, Daniel Kudenko, and Marek Grzes. An empirical study of potential-based re- ward shaping and advice in complex, multi-agent systems. Advances in Complex Systems, 14(2):251-278, 2011.","title":"An empirical study of potential-based reward shaping and advice in complex, multi-agent systems","authors":"Devlin","year":"2011","venue":"Advances in Complex Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Efthymiadis and Kudenko, 2013] Kyriakos Efthymiadis and Daniel Kudenko. Using plan-based reward shaping to learn strategies in StarCraft: Broodwar. In Proceedings of the 2013 IEEE Conference on Computational Inteligence in Games, pages 1-8, Niagara Falls, Canada, 2013.","title":"Using plan-based reward shaping to learn strategies in StarCraft: Broodwar","authors":"Kyriakos Efthymiadis, Daniel Kudenko","year":"2013","venue":"Proceedings of the 2013 IEEE Conference on Computational Inteligence in Games","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karpathy and van de Panne, 2012] Andrej Karpathy and Michiel van de Panne. Curriculum learning for motor skills. In Advances in Artificial Intelligence, pages 325-330, Heidelberg, Germany, 2012.","title":"Curriculum learning for motor skills","authors":"Van Karpathy, Andrej De Panne, Michiel Karpathy, Van De Panne","year":"2012","venue":"Advances in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim and De Weck, 2006] I. Y. Kim and O. L. De Weck. Adaptive weighted sum method for multiobjective op- timization: A new method for Pareto front generation. Structural and Multidisciplinary Optimization, 31(2):105- 116, 2006.","title":"Adaptive weighted sum method for multiobjective optimization: A new method for Pareto front generation","authors":"De Kim, ; Weck, I Y Kim, O L De Weck","year":"2006","venue":"Structural and Multidisciplinary Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin, 2005] JiGuan G. Lin. On min-norm and min-max methods of multi-objective optimization. Mathematical Programming, 103(1):1-33, 2005.","title":"On min-norm and min-max methods of multi-objective optimization","authors":"G Jiguan, Lin","year":"2005","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Ma- chine Learning, pages 1928-1937, New York, NY., 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moffaert et al., 2013] Kristof Van Moffaert, Madalina M. Drugan, and Ann Nowé. Scalarized multi-objective re- inforcement learning: Novel design techniques. In Pro- ceedings of the 2013 IEEE Symposium on Adaptive Dy- namic Programming and Reinforcement Learning, pages 191-199, Singapore, 2013.","title":"Scalarized multi-objective reinforcement learning: Novel design techniques","authors":"Moffaert","year":"2013","venue":"Proceedings of the 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ng et al., 1999] Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learn- ing, pages 278-287, Bled, Slovenia, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Daishi Harada, Stuart J Russell","year":"1999","venue":"Proceedings of the 16th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Parisi et al., 2014] Simone Parisi, Matteo Pirotta, Nicola Smacchia, Luca Bascetta, and Marcello Restelli. Policy gradient approaches for multi-objective sequential deci- sion making. In Proceedings of the 2014 International Joint Conference on Neural Networks, pages 2323-2330, Beijing, China, 2014.","title":"Policy gradient approaches for multi-objective sequential decision making","authors":"Simone Parisi, Matteo Pirotta, Nicola Smacchia, Luca Bascetta, Marcello Restelli","year":"2014","venue":"Proceedings of the 2014 International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pirotta et al., 2015] Matteo Pirotta, Simone Parisi, and Mar- cello Restelli. Multi-objective reinforcement learning with continuous Pareto frontier approximation. In Proceed- ings of the 29th AAAI Conference on Artificial Intelligence, pages 2928-2934, Austin, TX., 2015.","title":"Multi-objective reinforcement learning with continuous Pareto frontier approximation","authors":"Matteo Pirotta, Simone Parisi, Marcello Restelli","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Plappert et al., 2018] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464, 2018.","title":"Multi-goal reinforcement learning: Challenging robotics environments and request for research","authors":"Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob Mcgrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, Wojciech Zaremba","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Platt, 1998] John C. Platt. Sequential minimal optimization: A fast algorithm for training support vector machines. Advances in Kernel Methods -Support Vector Learning, 208:185-208, 1998.","title":"Sequential minimal optimization: A fast algorithm for training support vector machines","authors":"John C Platt","year":"1998","venue":"Advances in Kernel Methods -Support Vector Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schmidhuber, 2013] Jürgen Schmidhuber. PowerPlay: Training an increasingly general problem solver by contin- ually searching for the simplest still unsolvable problem. Frontiers in Psychology, 4:313, 2013.","title":"PowerPlay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem","authors":"Jürgen Schmidhuber","year":"2013","venue":"Frontiers in Psychology","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pages 1889-1897, Lille, France, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA., 1998.","title":"Introduction to Reinforcement Learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with expe- rience replay. In Proceedings of 5th International Confer- ence on Learning Representations, Toulon, France, 2017.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2017","venue":"Proceedings of 5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}