{"type":"ScholarlyArticle","title":"Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies (Extended Abstract)","authors":[{"name":"Paul Vicol","slug":"paul-vicol","given":"Paul","family":"Vicol","affiliation":"University of Toronto"},{"name":"Luke Metz","slug":"luke-metz","given":"Luke","family":"Metz","affiliation":"Google Brain"},{"name":"Jascha Sohl-Dickstein","slug":"jascha-sohl-dickstein","given":"Jascha","family":"Sohl-Dickstein","affiliation":"Google Brain"}],"abstract":"Current approaches for optimizing parameters in unrolled computation graphs suffer from high variance gradients, bias, slow updates, or large memory usage. We introduce a method called Persistent Evolution Strategies (PES), which divides the computation graph into a series of truncated unrolls, and performs an evolution strategies-based update step after each unroll. PES eliminates bias from these truncations by accumulating correction terms over the entire sequence of unrolls. PES allows for rapid parameter updates, has low memory usage, is unbiased, and has reasonable variance.","doi":"10.24963/ijcai.2022/750","url":"https://ijcai.org/proceedings/2022/750","pdf":"https://ijcai.org/proceedings/2022/0750.pdf","markdown":"https://ijcai.org/proceedings/2022/750.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/750","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":750,"pages":{"first":5354,"last":5358},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987266","length_sec":64,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985409","length_sec":601,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Beatson and Adams, 2019] Alex Beatson and Ryan P Adams. Efficient optimization of loops and limits with randomized telescoping sums. arXiv preprint arXiv:1905.07006, 2019.","title":"Efficient optimization of loops and limits with randomized telescoping sums","authors":"Alex Beatson, Ryan P Adams","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Benzing et al., 2019] Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal Kronecker-sum approximation of real time recurrent learning. arXiv preprint arXiv:1902.03993, 2019.","title":"Optimal Kronecker-sum approximation of real time recurrent learning","authors":"Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, Angelika Steger","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui et al., 2018] Xiaodong Cui, Wei Zhang, Zoltán Tüske, and Michael Picheny. Evolutionary stochastic gradient descent for optimization of deep neural networks. In Ad- vances in Neural Information Processing Systems, pages 6048-6058, 2018.","title":"Evolutionary stochastic gradient descent for optimization of deep neural networks","authors":"Xiaodong Cui, Wei Zhang, Zoltán Tüske, Michael Picheny","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Franceschi et al., 2018] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. arXiv preprint arXiv:1806.04910, 2018.","title":"Bilevel programming for hyperparameter optimization and meta-learning","authors":"Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, Massimilano Pontil","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ha and Schmidhuber, 2018] David Ha and Jürgen Schmid- huber. World models. arXiv preprint arXiv:1803.10122, 2018.","title":"World models","authors":"David Ha, Jürgen Schmidhuber","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ha, 2020] David Ha. Neuroevolution for deep reinforcement learning problems. In Genetic and Evolutionary Computa- tion Conference Companion, pages 404-427, 2020.","title":"Neuroevolution for deep reinforcement learning problems","authors":"David Ha","year":"2020","venue":"Genetic and Evolutionary Computation Conference Companion","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mania et al., 2018] Horia Mania, Aurelia Guy, and Ben- jamin Recht. Simple random search provides a compet- itive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018.","title":"Simple random search provides a competitive approach to reinforcement learning","authors":"Horia Mania, Aurelia Guy, Benjamin Recht","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Metz et al., 2018] Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning up- date rules for unsupervised representation learning. arXiv preprint arXiv:1804.00222, 2018.","title":"Meta-learning update rules for unsupervised representation learning","authors":"Metz","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Metz et al., 2019] Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Ma- chine Learning, pages 4556-4565, 2019.","title":"Understanding and correcting pathologies in the training of learned optimizers","authors":"Metz","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mujika et al., 2018] Asier Mujika, Florian Meier, and An- gelika Steger. Approximating real-time recurrent learning with random Kronecker factors. In Advances in Neural Information Processing Systems, pages 6594-6603, 2018.","title":"Approximating real-time recurrent learning with random Kronecker factors","authors":"Asier Mujika, Florian Meier, Angelika Steger","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Parmas et al., 2018] Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4062-4071, 2018.","title":"PIPPS: Flexible model-based policy search robust to the curse of chaos","authors":"Paavo Parmas, Carl Edward Rasmussen, Jan Peters, Kenji Doya","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pascanu et al., 2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310-1318, 2013.","title":"On the difficulty of training recurrent neural networks","authors":"Razvan Pascanu, Tomas Mikolov, Yoshua Bengio","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Staines and Barber, 2012] Joe Staines and David Barber. Variational optimization. arXiv preprint arXiv:1212.4507, 2012.","title":"Variational optimization","authors":"Joe Staines, David Barber","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tallec and Ollivier, 2017] Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043, 2017.","title":"Unbiased online recurrent optimization","authors":"Corentin Tallec, Yann Ollivier","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tieleman and Hinton, 2012] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-RMSprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.","title":"Lecture 6.5-RMSprop: Divide the gradient by a running average of its recent magnitude","authors":"Hinton Tieleman, Tijmen Tieleman, Geoffrey Hinton","year":"2012","venue":"COURSERA: Neural Networks for Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vicol et al., 2021] Paul Vicol, Luke Metz, and Jascha Sohl- Dickstein. Unbiased gradient estimation in unrolled compu- tation graphs with persistent evolution strategies. In Inter- national Conference on Machine Learning, pages 10553- 10563, 2021.","title":"Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies","authors":"Paul Vicol, Luke Metz, Jascha Sohl-Dickstein","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Werbos, 1990] Paul J Werbos. Backpropagation through time: What it does and how to do it. Proceedings of the IEEE, 78(10):1550-1560, 1990.","title":"Backpropagation through time: What it does and how to do it","authors":"Werbos Paul","year":"1990","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wichrowska et al., 2017] Olga Wichrowska, Niru Mah- eswaranathan, Matthew W Hoffman, Sergio Gomez Col- menarejo, Misha Denil, Nando de Freitas, and Jascha Sohl- Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv:1703.04813, 2017.","title":"Learned optimizers that scale and generalize","authors":"Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando De Freitas, Jascha Sohl-Dickstein","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Williams and Peng, 1990] Ronald J Williams and Jing Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural Computation, 2(4):490-501, 1990.","title":"An efficient gradient-based algorithm for on-line training of recurrent network trajectories","authors":"J Ronald, Jing Williams, Peng","year":"1990","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams and Zipser, 1989] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270- 280, 1989.","title":"A learning algorithm for continually running fully recurrent neural networks","authors":"J Ronald, David Williams, Zipser","year":"1989","venue":"Neural Computation","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}