{"type":"ScholarlyArticle","title":"Sequence Learning Using Equilibrium Propagation","authors":[{"name":"Malyaban Bal","slug":"malyaban-bal","given":"Malyaban","family":"Bal","affiliation":"School of Electrical Engineering and Computer Science The Pennsylvania State University"},{"name":"Abhronil Sengupta","slug":"abhronil-sengupta","given":"Abhronil","family":"Sengupta","affiliation":"School of Electrical Engineering and Computer Science The Pennsylvania State University"}],"abstract":"Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset). Our implementation source code is available at https://github.com/NeuroCompLab-psu/EqProp-SeqLearning.","doi":"10.24963/ijcai.2023/329","url":"https://ijcai.org/proceedings/2023/329","pdf":"https://ijcai.org/proceedings/2023/0329.pdf","markdown":"https://ijcai.org/proceedings/2023/329.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/329","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":329,"pages":{"first":2949,"last":2957},"keywords":["Machine Learning: ML: Attention models","Humans and AI: HAI: Cognitive systems"],"videos":[],"references":[{"position":1,"raw":"[Almeida, 1990] Luis B Almeida. A learning rule for asyn- chronous perceptrons with feedback in a combinatorial en- vironment. In Artificial neural networks: concept learn- ing, pages 102-111. 1990.","title":"A learning rule for asynchronous perceptrons with feedback in a combinatorial environment","authors":"B Luis, Almeida","year":"1990","venue":"Artificial neural networks: concept learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berkes et al., 2011] Pietro Berkes, Gergő Orbán, Máté Lengyel, and József Fiser. Spontaneous cortical activity reveals hallmarks of an optimal internal model of the en- vironment. Science, 331(6013):83-87, 2011.","title":"Spontaneous cortical activity reveals hallmarks of an optimal internal model of the environment","authors":"Pietro Berkes, Gergő Orbán, Máté Lengyel, József Fiser","year":"2011","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bi and Poo, 1998] Guo-qiang Bi and Mu-ming Poo. Synap- tic modifications in cultured hippocampal neurons: depen- dence on spike timing, synaptic strength, and postsynaptic cell type. Journal of neuroscience, 18(24):10464-10472, 1998.","title":"Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type","authors":"Guo-Qiang Bi, Mu-Ming Poo","year":"1998","venue":"Journal of neuroscience","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Gabor Samuel R Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bowman et al., 2016] Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. arXiv preprint arXiv:1603.06021, 2016.","title":"A fast unified model for parsing and sentence understanding","authors":"Jon Samuel R Bowman, Abhinav Gauthier, Raghav Rastogi, Christopher D Gupta, Christopher Manning, Potts","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Campos et al., 2017] Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural net- works. arXiv preprint arXiv:1708.06834, 2017.","title":"Skip rnn: Learning to skip state updates in recurrent neural networks","authors":"Víctor Campos, Brendan Jou, Xavier Giró-I Nieto, Jordi Torres, Shih-Fu Chang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chilkuri and Eliasmith, 2021] Narsimha Reddy Chilkuri and Chris Eliasmith. Parallelizing legendre memory unit training. In International Conference on Machine Learn- ing, pages 1898-1907. PMLR, 2021.","title":"Parallelizing legendre memory unit training","authors":"Narsimha Reddy, Chilkuri, Chris Eliasmith","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Crick, 1989] Francis Crick. The recent excitement about neural networks. Nature, 337(6203):129-132, 1989.","title":"The recent excitement about neural networks","authors":"Francis Crick","year":"1989","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Demircigil et al., 2017] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2):288-299, 2017.","title":"On a model of associative memory with huge storage capacity","authors":"Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, Franck Vermet","year":"2017","venue":"Journal of Statistical Physics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dey and Salem, 2017] Rahul Dey and Fathi M Salem. Gate- variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on cir- cuits and systems (MWSCAS), pages 1597-1600. IEEE, 2017.","title":"Gatevariants of gated recurrent unit (gru) neural networks","authors":"Rahul Dey, M Fathi, Salem","year":"2017","venue":"IEEE 60th international midwest symposium on circuits and systems (MWSCAS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ernoult et al., 2019] Maxence Ernoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, and Benjamin Scellier. Updates of equilibrium prop match gradients of backprop through time in an rnn with static input. Advances in neural information processing systems, 32, 2019.","title":"Updates of equilibrium prop match gradients of backprop through time in an rnn with static input","authors":"Maxence Ernoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, Benjamin Scellier","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gammell et al., 2021] Jimmy Gammell, Sonia Buckley, Sae Woo Nam, and Adam N McCaughan. Layer- skipping connections improve the effectiveness of equilib- rium propagation on layered networks. Frontiers in com- putational neuroscience, 15:627357, 2021.","title":"Layerskipping connections improve the effectiveness of equilibrium propagation on layered networks","authors":"Gammell","year":"2021","venue":"Frontiers in computational neuroscience","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2019] Kun Han, Junwen Chen, Hui Zhang, Haiyang Xu, Yiping Peng, Yun Wang, Ning Ding, Hui Deng, Yonghu Gao, Tingwei Guo, et al. Delta: A deep learning based language technology platform. arXiv preprint arXiv:1908.01853, 2019.","title":"Delta: A deep learning based language technology platform","authors":"Kun Han, Junwen Chen, Hui Zhang, Haiyang Xu, Yiping Peng, Yun Wang, Ning Ding, Hui Deng, Yonghu Gao, Tingwei Guo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hinton, 2002] Geoffrey E Hinton. Training products of ex- perts by minimizing contrastive divergence. Neural com- putation, 14(8):1771-1800, 2002.","title":"Training products of experts by minimizing contrastive divergence","authors":"Geoffrey E Hinton","year":"2002","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hopfield, 1982] John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558, 1982.","title":"Neural networks and physical systems with emergent collective computational abilities","authors":"J John, Hopfield","year":"1982","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krotov and Hopfield, 2016] Dmitry Krotov and John J Hop- field. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.","title":"Dense associative memory for pattern recognition","authors":"Dmitry Krotov, John J Hopfield","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krotov and Hopfield, 2018] Dmitry Krotov and John Hop- field. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12):3151-3167, 2018.","title":"Dense associative memory is robust to adversarial inputs","authors":"Dmitry Krotov, John Hopfield","year":"2018","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Laborieux et al., 2021] Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, and Damien Querlioz. Scaling equilibrium propagation to deep convnets by drastically reducing its gradient estima- tor bias. Frontiers in neuroscience, 15:129, 2021.","title":"Scaling equilibrium propagation to deep convnets by drastically reducing its gradient estimator bias","authors":"Axel Laborieux, Maxence Ernoult, Benjamin Scellier, Yoshua Bengio, Julie Grollier, Damien Querlioz","year":"2021","venue":"Frontiers in neuroscience","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lillicrap et al., 2020] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuro- science, 21(6):335-346, 2020.","title":"Backpropagation and the brain","authors":"Adam Timothy P Lillicrap, Luke Santoro, Colin J Marris, Geoffrey Akerman, Hinton","year":"2020","venue":"Nature Reviews Neuroscience","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maas et al., 2011] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Pro- ceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Martin et al., 2021] Erwann Martin, Maxence Ernoult, Jérémie Laydevant, Shuai Li, Damien Querlioz, Teodora Petrisor, and Julie Grollier. Eqspike: spike-driven equi- librium propagation for neuromorphic implementations. Iscience, 24(3):102222, 2021.","title":"Eqspike: spike-driven equilibrium propagation for neuromorphic implementations","authors":"Erwann Martin, Maxence Ernoult, Jérémie Laydevant, Shuai Li, Damien Querlioz, Teodora Petrisor, Julie Grollier","year":"2021","venue":"Iscience","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Parikh et al., 2016] Ankur P Parikh, Oscar Täckström, Di- panjan Das, and Jakob Uszkoreit. A decomposable atten- tion model for natural language inference. arXiv preprint arXiv:1606.01933, 2016.","title":"A decomposable attention model for natural language inference","authors":"P Ankur, Oscar Parikh, Dipanjan Täckström, Jakob Das, Uszkoreit","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pineda, 1987] Fernando Pineda. Generalization of back propagation to recurrent and higher order neural networks. In Neural information processing systems, 1987.","title":"Generalization of back propagation to recurrent and higher order neural networks","authors":"Fernando Pineda","year":"1987","venue":"Neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ramsauer et al., 2020] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks is all you need. arXiv preprint arXiv:2008.02217, 2020.","title":"Hopfield networks is all you need","authors":"Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rusch and Mishra, 2020] T Konstantin Rusch and Sid- dhartha Mishra. Coupled oscillatory recurrent neural net- work (cornn): An accurate and (gradient) stable architec- ture for learning long time dependencies. arXiv preprint arXiv:2010.00951, 2020.","title":"Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies","authors":"Mishra Rusch, Konstantin Rusch, Siddhartha Mishra","year":"2020","venue":"T","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rusch and Mishra, 2021] T Konstantin Rusch and Sid- dhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. In International Conference on Machine Learning, pages 9168-9178. PMLR, 2021.","title":"Unicornn: A recurrent model for learning very long time dependencies","authors":"Mishra ; Rusch, Konstantin Rusch, Siddhartha Mishra","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Scellier and Bengio, 2017] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap be- tween energy-based models and backpropagation. Fron- tiers in computational neuroscience, 11:24, 2017.","title":"Equilibrium propagation: Bridging the gap between energy-based models and backpropagation","authors":"Bengio Scellier, Benjamin Scellier, Yoshua Bengio","year":"2017","venue":"Frontiers in computational neuroscience","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Scellier and Bengio, 2019] Benjamin Scellier and Yoshua Bengio. Equivalence of equilibrium propagation and re- current backpropagation. Neural computation, 31(2):312- 329, 2019.","title":"Equivalence of equilibrium propagation and recurrent backpropagation","authors":"Bengio Scellier, Benjamin Scellier, Yoshua Bengio","year":"2019","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Scellier et al., 2018] Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, and Yoshua Bengio. Generalization of equilibrium propagation to vector field dynamics. arXiv preprint arXiv:1808.04873, 2018.","title":"Generalization of equilibrium propagation to vector field dynamics","authors":"Benjamin Scellier, Anirudh Goyal, Jonathan Binas, Thomas Mesnard, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bow- man. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.","title":"Glue: A multi-task benchmark and analysis platform for natural language understanding","authors":"Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R Bowman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Widrich et al., 2020] Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al. Modern hop- field networks and attention for immune repertoire classi- fication. Advances in Neural Information Processing Sys- tems, 33:18832-18845, 2020.","title":"Modern hopfield networks and attention for immune repertoire classification","authors":"Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yuille and Rangarajan, 2001] Alan L Yuille and Anand Rangarajan. The concave-convex procedure (cccp). Ad- vances in neural information processing systems, 14, 2001.","title":"The concave-convex procedure (cccp). Advances in neural information processing systems","authors":"Alan L Yuille, Anand Rangarajan","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yuille and Rangarajan, 2003] Alan L Yuille and Anand Rangarajan. The concave-convex procedure. Neural com- putation, 15(4):915-936, 2003.","title":"The concave-convex procedure","authors":"Alan L Yuille, Anand Rangarajan","year":"2003","venue":"Neural computation","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}