{"type":"ScholarlyArticle","title":"Learning Environmental Calibration Actions for Policy Self-Evolution","authors":[{"name":"Chao Zhang","slug":"chao-zhang","given":"Chao","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Yang Yu","slug":"yang-yu","given":"Yang","family":"Yu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"Reinforcement learning in physical world is often expensive. Simulators are commonly employed to train policies. Due to the simulation error, trained-in-simulator policies are hard to be directly deployed in physical world. Therefore, how to efficiently reuse these policies to the real environment is a key issue. To address this issue, this paper presents a policy self-evolution process: in the target environment, the agent firstly executes a few calibration actions to perceive the environment, and then reuses the previous policies according to the observation of the environment. In this way, the mission of policy learning in the target environment is reduced to the task of environment identification through executing the calibration actions, which needs much less samples than learning a policy from scratch. We propose the POSEC (POlicy Self-Evolution by Calibration) approach, which learns the most informative calibration actions for policy self-evolution. Taking three robotic arm controlling tasks as the test beds, we show that the proposed method can learn a fine policy for a new arm with only a few (e.g. five) samples of the target environment.","doi":"10.24963/ijcai.2018/425","url":"https://ijcai.org/proceedings/2018/425","pdf":"https://ijcai.org/proceedings/2018/0425.pdf","markdown":"https://ijcai.org/proceedings/2018/425.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/425","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":425,"pages":{"first":3061,"last":3067},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Brafman and Tennenholtz, 2002] Ronen I. Brafman and Moshe Tennenholtz. R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning. Jour- nal of Machine Learning Research, 3:213-231, 2002.","title":"R-MAX -A general polynomial time algorithm for near-optimal reinforcement learning","authors":"Ronen I Brafman, Moshe Tennenholtz","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":"/proceedings/2001/130"},{"position":2,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv:1606.01540, 2016.","title":"OpenAI Gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deisenroth and Rasmussen, 2011] Marc Deisenroth and Carl E. Rasmussen. PILCO: A model-based and data- efficient approach to policy search. In Proceedings of the 28th International Conference on Machine Learning, pages 465-472, Bellevue, WA, 2011.","title":"PILCO: A model-based and dataefficient approach to policy search","authors":"Rasmussen Deisenroth, Marc Deisenroth, Carl E Rasmussen","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Pro- ceedings of the 33rd International Conference on Machine Learning, pages 1329-1338, New York, NY, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1126-1135, Syd- ney, Australia, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hansen and Ostermeier, 2001] Nikolaus Hansen and An- dreas Ostermeier. Completely derandomized self- adaptation in evolution strategies. Evolutionary compu- tation, 9(2):159-195, 2001.","title":"Completely derandomized selfadaptation in evolution strategies","authors":"Ostermeier Hansen","year":"2001","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2017] Yi-Qi Hu, Hong Qian, and Yang Yu. Se- quential classification-based optimization for direct pol- icy search. In Proceedings of the 31st Association for the Advancement of Artificial Intelligence, pages 2029-2035, San Francisco, CA, 2017.","title":"Sequential classification-based optimization for direct policy search","authors":"Yi-Qi Hu, Hong Qian, Yang Yu","year":"2017","venue":"Proceedings of the 31st Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jong and Stone, 2007] Nicholas K. Jong and Peter Stone. Model-based exploration in continuous state spaces. In Proceedings of the 7th International Symposium on Ab- straction, Reformulation, and Approximation, pages 258- 272, Berlin,Germany, 2007.","title":"Model-based exploration in continuous state spaces","authors":"Nicholas K Jong, Peter Stone","year":"2007","venue":"Proceedings of the 7th International Symposium on Abstraction, Reformulation, and Approximation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kirkpatrick et al., 1983] Scott Kirkpatrick, Daniel Gelatt, and Mario P. Vecchi. Optimization by simulated anneal- ing. Science, 220(4598):671-680, 1983.","title":"Optimization by simulated annealing","authors":"Scott Kirkpatrick, Daniel Gelatt, Mario P Vecchi","year":"1983","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lazaric et al., 2008] Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch rein- forcement learning. In Proceedings of the 25th Interna- tional Conference on Machine Learning, pages 544-551, Helsinki, Finland, 2008.","title":"Transfer of samples in batch reinforcement learning","authors":"Alessandro Lazaric, Marcello Restelli, Andrea Bonarini","year":"2008","venue":"Proceedings of the 25th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mehta et al., 2008] Neville Mehta, Sriraam Natarajan, Prasad Tadepalli, and Alan Fern. Transfer in variable- reward hierarchical reinforcement learning. Machine Learning, 73(3):289-312, 2008.","title":"Transfer in variablereward hierarchical reinforcement learning","authors":"Neville Mehta, Sriraam Natarajan, Prasad Tadepalli, Alan Fern","year":"2008","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidje- land, and Georg Ostrovski. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Munos, 2014] Rémi Munos. From bandits to Monte-Carlo tree search: The optimistic principle applied to optimiza- tion and planning. Foundations and Trends in Machine Learning, 7(1):1-130, 2014.","title":"From bandits to Monte-Carlo tree search: The optimistic principle applied to optimization and planning","authors":"Rémi Munos, Munos","year":"2014","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan et al., 2011] Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Net- works, 22(2):199-210, 2011.","title":"Domain adaptation via transfer component analysis","authors":"Ivor W Sinno Jialin Pan, James T Tsang, Qiang Kwok, Yang","year":"2011","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Peng et al., 2017] Xue-Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Real transfer of robotic control with dynamics randomization. arXiv:1710.06537, 2017.","title":"Sim-to-Real transfer of robotic control with dynamics randomization","authors":"Xue-Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ratitch and Precup, 2004] Bohdana Ratitch and Doina Pre- cup. Sparse distributed memories for on-line value-based reinforcement learning. In Proceedings of the 15th Euro- pean Conference on Machine Learning, pages 347-358, Pisa, Italy, 2004.","title":"Bohdana Ratitch and Doina Precup. Sparse distributed memories for on-line value-based reinforcement learning","authors":"Precup Ratitch","year":"2004","venue":"Proceedings of the 15th European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 33nd In- ternational conference on Machine Learning, pages 1889- 1897, Lille, France, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 33nd International conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shahriari et al., 2016] Bobak Shahriari, Kevin Swersky, Zi- Yu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of Bayesian optimiza- tion. Proceedings of the IEEE, 104(1):148-175, 2016.","title":"Taking the human out of the loop: A review of Bayesian optimization","authors":"Bobak Shahriari, Kevin Swersky, Zi-Yu Wang, Ryan P Adams, Nando De Freitas","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, George A. Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George A Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Strehl et al., 2006] Alexander L. Strehl, Li-Hong Li, Eric Wiewiora, John Langford, and Michael L. Littman. PAC model-free reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning, pages 881-888, Pittsburgh, PA, 2006.","title":"PAC model-free reinforcement learning","authors":"Alexander L Strehl, Li-Hong Li, Eric Wiewiora, John Langford, Michael L Littman","year":"2006","venue":"Proceedings of the 23rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Introduction to reinforcement learning. MIT Press Cambridge, 1998.","title":"Introduction to reinforcement learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton et al., 2000] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 14th Advances in Neural Information Processing Systems, pages 1057-1063, Denver, CO, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Proceedings of the 14th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Watkins and Dayan, 1992] Chris Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.","title":"Q-learning","authors":"Chris Watkins, Peter Dayan","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2016] Yang Yu, Hong Qian, and Yi-Qi Hu. Derivative-free optimization via classification. In Proceed- ings of the 13th Association for the Advancement of Artifi- cial Intelligence, pages 2286-2292, Phoenix, AZ, 2016.","title":"Derivative-free optimization via classification","authors":"Yang Yu, Hong Qian, Yi-Qi Hu","year":"2016","venue":"Proceedings of the 13th Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou, 2016] Zhi-Hua Zhou. Learnware: On the future of machine learning. Frontiers of Computer Science, 10(4):589-590, 2016.","title":"Learnware: On the future of machine learning","authors":"Zhi-Hua Zhou","year":"2016","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}