{"type":"ScholarlyArticle","title":"Contextual Covariance Matrix Adaptation Evolutionary Strategies","authors":[{"name":"Abbas Abdolmaleki","slug":"abbas-abdolmaleki","given":"Abbas","family":"Abdolmaleki","affiliation":": DSI , University of Minho; : LIACC , University of Porto , Portugal"},{"name":"Bob Price","slug":"bob-price","given":"Bob","family":"Price","affiliation":": PARC , A Xerox Company , USA"},{"name":"Nuno Lau","slug":"nuno-lau","given":"Nuno","family":"Lau","affiliation":"University of Aveiro"},{"name":"Luis Paulo Reis","slug":"luis-paulo-reis","given":"Luis Paulo","family":"Reis","affiliation":": DSI , University of Minho; : LIACC , University of Porto , Portugal"},{"name":"Gerhard Neumann","slug":"gerhard-neumann","given":"Gerhard","family":"Neumann","affiliation":": CLAS , TU Darmstadt , Germany"}],"abstract":"Many stochastic search algorithms are designed to optimize a fixed objective function to learn a task, \r\ni.e., if the objective function changes slightly, for example, due to a change in the situation or context of the task, \r\nrelearning is required to adapt to the new context. For instance, if we want to learn a kicking movement for a soccer robot, \r\nwe have to relearn the movement for different ball locations. Such relearning is undesired as it is highly inefficient \r\nand many applications require a fast adaptation to a new context/situation. Therefore, we investigate contextual stochastic search \r\nalgorithms that can learn multiple, similar tasks simultaneously. Current contextual stochastic search methods are based \r\non policy search algorithms and suffer from premature convergence and the need for parameter tuning. \r\nIn this paper, we extend the well known CMA-ES algorithm to the contextual setting and illustrate its performance on several contextual tasks. \r\nOur new algorithm, called contextual CMA-ES, leverages from contextual learning while it preserves all the features of standard CMA-ES such \r\nas stability, avoidance of premature convergence, step size control and a minimal amount of parameter tuning.","doi":"10.24963/ijcai.2017/191","url":"https://ijcai.org/proceedings/2017/191","pdf":"https://ijcai.org/proceedings/2017/0191.pdf","markdown":"https://ijcai.org/proceedings/2017/191.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/191","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":191,"pages":{"first":1378,"last":1385},"keywords":["Machine Learning: Reinforcement Learning","Combinatorial & Heuristic Search: Combinatorial search/optimisation"],"videos":[],"references":[{"position":1,"raw":"[Abdolmaleki et al., 2015a] A. Abdolmaleki, N. Lua, L.P. Reis, and G. Neumann. Regularized covariance estima- tion for weighted maximum likelihood policy search meth- ods. In Proceedings of the International Conference on Humanoid Robots (HUMANOIDS), 2015.","title":"Regularized covariance estimation for weighted maximum likelihood policy search methods","authors":"Abdolmaleki","year":"2015","venue":"Proceedings of the International Conference on Humanoid Robots (HUMANOIDS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abdolmaleki et al., 2015b] A. Abdolmaleki, N. Lua, L.P. Reis, J. Peters, and G. Neumann. Contextual Policy Search for Generalizing a Parameterized Biped Walking Con- troller. In IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), 2015.","title":"Contextual Policy Search for Generalizing a Parameterized Biped Walking Controller","authors":"Abdolmaleki","year":"2015","venue":"IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Abdolmaleki et al., 2016] A. Abdolmaleki, D. Simoes, N. Lua, L.P. Reis, and G. Neumann. Learning a Humanoid Kick With Controlled Distance. In Robocup Symposium, 2016.","title":"Learning a Humanoid Kick With Controlled Distance","authors":"A Abdolmaleki, D Simoes, N Lua, L P Reis, G Neumann","year":"2016","venue":"Robocup Symposium","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Akimoto et al., 2012] Y. Akimoto, Y. Nagata, I. Ono, and S. Kobayashi. Theoretical foundation for cma-es from in- formation geometry perspective. Algorithmica, 2012.","title":"Theoretical foundation for cma-es from information geometry perspective","authors":"Y Akimoto, Y Nagata, I Ono, S Kobayashi","year":"2012","venue":"Algorithmica","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caruana, 1997] Rich Caruana. Multitask Learning. PhD thesis, CMU, 1997.","title":"Multitask Learning","authors":"Rich Caruana","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Da Silva et al., 2012] Bruno Da Silva, George Konidaris, and Andrew Barto. Learning parameterized skills. Inter- national Conference on Machine Learning (ICML), 2012.","title":"Learning parameterized skills","authors":"Da Silva","year":"2012","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deisenroth et al., 2014] Marc Peter Deisenroth, Peter En- glert, Jan Peters, and Dieter Fox. Multi-task policy search for robotics. In IEEE International Conference on Robotics and Automation (ICRA), 2014.","title":"Multi-task policy search for robotics","authors":"Marc Peter Deisenroth, Peter Englert, Jan Peters, Dieter Fox","year":"2014","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ha and Liu, 2016] S. Ha and C.K Liu. Evolutionary op- timization for parameterized whole-body dynamic motor skills. In Proceedings of IEEE International Conference on Robotics and Automation (ICRA), 2016.","title":"Evolutionary optimization for parameterized whole-body dynamic motor skills","authors":"S Ha, C K Liu","year":"2016","venue":"Proceedings of IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hansen et al., 2003] N. Hansen, S.D. Muller, and P. Koumoutsakos. Reducing the Time Complexity of the Derandomized Evolution Strategy with Covari- ance Matrix Adaptation (CMA-ES). Evolutionary Computation, 2003.","title":"Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation","authors":"N Hansen, S D Muller, P Koumoutsakos","year":"2003","venue":"CMA-ES)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hansen, 2016] N. Hansen. The cma evolution strategy: A tutorial. Tutorial, 2016.","title":"The cma evolution strategy: A tutorial","authors":"N Hansen","year":"2016","venue":"Tutorial","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Igel et al., 2006] C. Igel, T. Suttorp, and N. Hansen. A com- putational efficient covariance matrix update and a (1+ 1)- CMA for evolution strategies. In Proceedings of the 8th annual conference on Genetic and evolutionary computa- tion, 2006.","title":"A computational efficient covariance matrix update and a (1+ 1)-CMA for evolution strategies","authors":"C Igel, T Suttorp, N Hansen","year":"2006","venue":"Proceedings of the 8th annual conference on Genetic and evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kober et al., 2010] J. Kober, E. Oztop, and J. Peters. Re- inforcement Learning to adjust Robot Movements to New Situations. In Proceedings of the Robotics: Science and Systems Conference (RSS), 2010.","title":"Reinforcement Learning to adjust Robot Movements to New Situations","authors":"J Kober, E Oztop, J Peters","year":"2010","venue":"Proceedings of the Robotics: Science and Systems Conference (RSS)","doi":null,"url":null,"internal_url":"/proceedings/2011/441"},{"position":13,"raw":"[Kupcsik et al., 2013] A. Kupcsik, M. P. Deisenroth, J. Pe- ters, and G. Neumann. Data-Efficient Contextual Policy Search for Robot Movement Skills. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2013.","title":"Data-Efficient Contextual Policy Search for Robot Movement Skills","authors":"A Kupcsik, M P Deisenroth, J Peters, G Neumann","year":"2013","venue":"Proceedings of the National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Molga and Smutnicki, 2005] M. Molga and C. Smut- nicki. Test Functions for Optimization Needs. In http://www.zsd.ict.pwr.wroc.pl/files/docs/functions.pdf, 2005.","title":"Smutnicki","authors":"M Molga, C","year":"2005","venue":"Test Functions for Optimization Needs","doi":null,"url":"http://www.zsd.ict.pwr.wroc.pl/files/docs/functions.pdf","internal_url":null},{"position":15,"raw":"[Peters et al., 2010] J. Peters, K. Mülling, and Y. Altun. Rel- ative Entropy Policy Search. In Proceedings of the 24th National Conference on Artificial Intelligence (AAAI). AAAI Press, 2010.","title":"Relative Entropy Policy Search","authors":"J Peters, K Mülling, Y Altun","year":"2010","venue":"Proceedings of the 24th National Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rückstieß et al., 2008] T. Rückstieß, M. Felder, and J. Schmidhuber. State-dependent Exploration for Policy Gradient Methods. In Proceedings of the European Conference on Machine Learning (ECML), 2008.","title":"State-dependent Exploration for Policy Gradient Methods","authors":"T Rückstieß, M Felder, J Schmidhuber","year":"2008","venue":"Proceedings of the European Conference on Machine Learning (ECML)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stulp and Sigaud, 2012] F. Stulp and O. Sigaud. Path Inte- gral Policy Improvement with Covariance Matrix Adapta- tion. In International Conference on Machine Learning (ICML), 2012.","title":"Path Integral Policy Improvement with Covariance Matrix Adaptation","authors":"F Stulp, O Sigaud","year":"2012","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Stulp et al., 2013] Freek Stulp, Gennaro Raiola, Antoine Hoarau, Serena Ivaldi, and Olivier Sigaud. Learning com- pact parameterized skills with a single regression. In IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2013.","title":"Learning compact parameterized skills with a single regression","authors":"Freek Stulp, Gennaro Raiola, Antoine Hoarau, Serena Ivaldi, Olivier Sigaud","year":"2013","venue":"IEEE-RAS International Conference on Humanoid Robots","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2009] Y. Sun, D. Wierstra, T. Schaul, and J. Schmidhuber. Efficient Natural Evolution Strategies. In Proceedings of the 11th Annual conference on Genetic and evolutionary computation(GECCO), 2009.","title":"Efficient Natural Evolution Strategies","authors":"Y Sun, D Wierstra, T Schaul, J Schmidhuber","year":"2009","venue":"Proceedings of the 11th Annual conference on Genetic and evolutionary computation(GECCO)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}