{"type":"ScholarlyArticle","title":"Learning in the Machine: Random Backpropagation and the Deep Learning Channel (Extended Abstract)","authors":[{"name":"Pierre Baldi","slug":"pierre-baldi","given":"Pierre","family":"Baldi","affiliation":"Department of Computer Science , University of California , Irvine"},{"name":"Peter Sadowski","slug":"peter-sadowski","given":"Peter","family":"Sadowski","affiliation":"Department of Computer Science , University of California , Irvine"},{"name":"Zhiqin Lu","slug":"zhiqin-lu","given":"Zhiqin","family":"Lu","affiliation":"Department of Mathematics , University of California , Irvine"}],"abstract":"Random backpropagation (RBP) is a variant of the backpropagation algorithm for training neural networks, where the transpose of the forward matrices are replaced by fixed random matrices in the calculation of the weight updates. It is remarkable both because of its effectiveness, in spite of using random matrices to communicate error information, and because it completely removes the requirement of maintaining symmetric weights in a physical neural system. To better understand RBP, we compare different algorithms in terms of the information available locally to each neuron. In the process, we derive several alternatives to RBP, including skipped RBP (SRBP), adaptive RBP (ARBP), sparse RBP, and study their behavior through simulations. These simulations show that many variants are also robust deep learning algorithms, but that the derivative of the transfer function is important in the learning rule. Finally, we prove several mathematical results including the convergence to fixed points of linear chains of arbitrary length, the convergence to fixed points of linear autoencoders with decorrelated data, the long-term existence of solutions for linear systems with a single hidden layer and convergence in special cases, and the convergence to fixed points of non-linear chains, when the derivative of the activation functions is included.","doi":"10.24963/ijcai.2019/885","url":"https://ijcai.org/proceedings/2019/885","pdf":"https://ijcai.org/proceedings/2019/0885.pdf","markdown":"https://ijcai.org/proceedings/2019/885.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/885","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Journal track","track":"Journal track","number":885,"pages":{"first":6348,"last":6352},"keywords":["Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Baldi and Sadowski, 2016] Pierre Baldi and Peter Sad- owski. A theory of local learning, the learning channel, and the optimality of backpropagation. Neural Networks, 83:51-74, 2016.","title":"Pierre Baldi and Peter Sadowski. A theory of local learning, the learning channel, and the optimality of backpropagation","authors":"Sadowski Baldi","year":"2016","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baldi and Sadowski, 2018] Pierre Baldi and Peter Sad- owski. Learning in the machine: Recirculation is random backpropagation. Neural Networks, 108:479-494, 2018.","title":"Learning in the machine: Recirculation is random backpropagation","authors":"Sadowski Baldi","year":"2018","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baldi et al., 2014] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 2014.","title":"Searching for exotic particles in high-energy physics with deep learning","authors":"Baldi","year":"2014","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baldi et al., 2017] Pierre Baldi, Peter Sadowski, and Zhiqin Lu. Learning in the machine: The symmetries of the deep learning channel. Neural Networks, 95:110-133, 2017.","title":"Learning in the machine: The symmetries of the deep learning channel","authors":"Baldi","year":"2017","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baldi et al., 2018] Pierre Baldi, Peter Sadowski, and Zhiqin Lu. Learning in the machine: Random backpropaga- tion and the deep learning channel. Artificial intelligence, 260:1-35, 2018.","title":"Learning in the machine: Random backpropagation and the deep learning channel","authors":"Baldi","year":"2018","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bengio et al., 2007] Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle, Université De Montréal, and Montréal Québec. Greedy layer-wise training of deep net- works. In Advances in Neural Information Processing Sys- tems 19. MIT Press, 2007.","title":"Greedy layer-wise training of deep networks","authors":"Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle, Université De Montréal, Montréal Québec","year":"2007","venue":"Advances in Neural Information Processing Systems 19","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Crick, 1989] Francis Crick. The recent excitement about neural networks. Nature, 337(6203):129-132, 1989.","title":"The recent excitement about neural networks","authors":"Francis Crick","year":"1989","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gilmer et al., 2017] Justin Gilmer, Colin Raffel, Samuel S. Schoenholz, Maithra Raghu, and Jascha Sohl-Dickstein. Explaining the learning dynamics of direct feedback align- ment. 2017.","title":"Explaining the learning dynamics of direct feedback alignment","authors":"Justin Gilmer, Colin Raffel, Samuel S Schoenholz, Maithra Raghu, Jascha Sohl-Dickstein","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton and McClelland, 1988] Geoffrey E Hinton and James L McClelland. Learning representations by recir- culation. In Neural information processing systems, pages 358-366. New York: American Institute of Physics, 1988.","title":"Learning representations by recirculation","authors":"Geoffrey E Hinton, James L Mcclelland","year":"1988","venue":"Neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton et al., 2006] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7):1527-1554, 2006.","title":"A Fast Learning Algorithm for Deep Belief Nets","authors":"Geoffrey E Hinton, Simon Osindero, Yee-Whye Teh","year":"2006","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ilyashenko, 2002] Yu Ilyashenko. Centennial history of hilbert's 16th problem. Bulletin of the American Mathe- matical Society, 39(3):301-354, 2002.","title":"Centennial history of hilbert's 16th problem","authors":"Yu Ilyashenko","year":"2002","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jaderberg et al., 2017] Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neu- ral interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1627-1635. JMLR. org, 2017.","title":"Decoupled neural interfaces using synthetic gradients","authors":"Jaderberg","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liao et al., 2016] Qianli Liao, Joel Leibo, and Tomaso Pog- gio. How important is weight symmetry in backpropaga- tion? In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 1837-1844, 2016.","title":"How important is weight symmetry in backpropagation","authors":"Liao","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Daniel Cown- den, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7:13276, November 2016.","title":"Random synaptic feedback weights support error backpropagation for deep learning","authors":"Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, Colin J Akerman","year":"2016","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mostafa et al., 2017] Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. arXiv preprint arXiv:1711.06756, 2017.","title":"Deep supervised learning using local errors","authors":"Hesham Mostafa, Ramesh Vishwajith, Gert Cauwenberghs","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nøkland, 2016] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. In Advances in Neural Information Processing Systems, pages 1037- 1045, 2016.","title":"Direct feedback alignment provides learning in deep neural networks","authors":"Arild Nøkland, Nøkland","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Smale, 1998] Steve Smale. Mathematical problems for the next century. The Mathematical Intelligencer, 20(2):7-15, 1998.","title":"Mathematical problems for the next century","authors":"Steve Smale, Smale","year":"1998","venue":"The Mathematical Intelligencer","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}