{"type":"ScholarlyArticle","title":"Modeling Hebb Learning Rule for Unsupervised Learning","authors":[{"name":"Jia Liu","slug":"jia-liu","given":"Jia","family":"Liu","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding , Xidian University , Xi'an , China"},{"name":"Maoguo Gong","slug":"maoguo-gong","given":"Maoguo","family":"Gong","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding , Xidian University , Xi'an , China"},{"name":"Qiguang Miao","slug":"qiguang-miao","given":"Qiguang","family":"Miao","affiliation":"School of Computer Science and Technology , Xidian University , Xi'an , China"}],"abstract":"This paper presents to model the Hebb learning rule and proposes a neuron learning machine (NLM). Hebb learning rule describes the plasticity of the connection between presynaptic and postsynaptic neurons and it is unsupervised itself. It formulates the updating gradient of the connecting weight in artificial neural networks. In this paper, we construct an objective function via modeling the Hebb rule. We make a hypothesis to simplify the model and introduce a correlation based constraint according to the hypothesis and stability of solutions. By analysis from the perspectives of maintaining abstract information and increasing the energy based probability of observed data, we find that this biologically inspired model has the capability of learning useful features. NLM can also be stacked to learn hierarchical features and reformulated into convolutional version to extract features from 2-dimensional data. Experiments on single-layer and deep networks demonstrate the effectiveness of NLM in unsupervised feature learning.","doi":"10.24963/ijcai.2017/322","url":"https://ijcai.org/proceedings/2017/322","pdf":"https://ijcai.org/proceedings/2017/0322.pdf","markdown":"https://ijcai.org/proceedings/2017/322.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/322","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":322,"pages":{"first":2315,"last":2321},"keywords":["Machine Learning: Machine Learning","Machine Learning: Unsupervised Learning","Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Arel et al., 2010] Itamar Arel, Derek Rose, and Thomas Karnowski. Deep machine learning-a new frontier in arti- ficial intelligence research [research frontier]. IEEE Com- putational Intelligence Magazine, 5(11):13-18, 2010.","title":"Deep machine learning-a new frontier in artificial intelligence research [research frontier","authors":"Arel","year":"2010","venue":"IEEE Computational Intelligence Magazine","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2006] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise train- ing of deep networks. In Proc. NIPS, 2006.","title":"Greedy layer-wise training of deep networks","authors":"Yoshua Bengio, Pascal Lamblin, Dan Popovici, Hugo Larochelle","year":"2006","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. PAMI, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gong et al., 2015] Mao Guo Gong, Jia Liu, Hao Li, Qing Cai, and Lin Zhi Su. A multiobjective sparse feature learn- ing model for deep neural networks. IEEE Trans. NNLS, 26(12):3263-3277, 2015.","title":"A multiobjective sparse feature learning model for deep neural networks","authors":"Mao Guo, Gong, Jia Liu, Hao Li, Qing Cai, Lin Zhi, Su","year":"2015","venue":"IEEE Trans. NNLS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2013] Yun Long He, Koray Kavukcuoglu, Yun Wang, Arthur Szlam, and Yan Jun Qi. Unsupervised fea- ture learning by deep sparse coding. arXiv:1312.5783, 2013.","title":"Unsupervised feature learning by deep sparse coding","authors":"Yun Long He, Koray Kavukcuoglu, Yun Wang, Arthur Szlam, Yan Jun Qi","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hebb, 1949] Donald Olding Hebb. In The Organization of Behavior: A Neuropsychological Theory. Wiley, New Y- ork, 1949.","title":"The Organization of Behavior: A Neuropsychological Theory","authors":"Donald Olding, Hebb","year":"1949","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton and Salakhutdinov, 2006] Geoffrey Hinton and Rus- lan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-507, 2006.","title":"Geoffrey Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks","authors":"Salakhutdinov Hinton","year":"2006","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2006] Geoffrey Hinton, Simon Osindero, and YeeWhye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527-1554, 2006.","title":"A fast learning algorithm for deep belief nets","authors":"Geoffrey Hinton, Simon Osindero, Yeewhye Teh","year":"2006","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hinton, 2010] Geoffrey Hinton. A practical guide to train- ing restricted boltzmann machines. Momentum, 9(1):599- 619, 2010.","title":"A practical guide to training restricted boltzmann machines","authors":"Geoffrey Hinton","year":"2010","venue":"Momentum","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ji et al., 2014] Nan Nan Ji, Jiang She Zhang, and Chun Xia Zhang. A sparse-response deep belief network based on rate distortion theory. Pattern Recognition, 47(9):3179- 3191, 2014.","title":"A sparse-response deep belief network based on rate distortion theory","authors":"Nan Nan, Ji, Jiang She Zhang, Chun Xia Zhang","year":"2014","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kruger et al., 2013] Norbert Kruger, Peter Janssen, Sinan Kalkan, Markus Lappe, Ales Leonardis, Justus Piater, Antonio Jose Rodriguez-Sanchez, and Laurenz Wiskot- t. Deep hierarchies in the primate visual cortex: What can we learn for computer vision? IEEE Trans. PAMI, 35(8):1847-1871, 2013.","title":"Deep hierarchies in the primate visual cortex: What can we learn for computer vision?","authors":"Norbert Kruger, Peter Janssen, Sinan Kalkan, Markus Lappe, Ales Leonardis, Justus Piater, Antonio Jose Rodriguez-Sanchez, Laurenz Wiskott","year":"2013","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuriscak et al., 2015] Eduard Kuriscak, Petr Marsalek, Julius Stroffek, and Peter Toth. Biological context of Hebb learning in artificial neural networks, a review. Neurocom- puting, 152:27-35, 2015.","title":"Biological context of Hebb learning in artificial neural networks, a review","authors":"Kuriscak","year":"2015","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offery Hinton. Deep learning. Nature, 521(7553):436-44, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffery Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2007] Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area V2. In Proc. NIPS, 2007.","title":"Sparse deep belief net model for visual area V2","authors":"Lee","year":"2007","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Masci et al., 2011] Jonathan Masci, Ueli Meier, Ciresan Dan, and Jrgen Schmidhuber. Stacked convolutional auto- encoders for hierarchical feature extraction. In Proc. I- CANN, pages 52-59, 2011.","title":"Stacked convolutional autoencoders for hierarchical feature extraction","authors":"Masci","year":"2011","venue":"Proc. I-CANN","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Norouzi et al., 2009] Mohammad Norouzi, Mani Ranjbar, and Greg Mori. Stacks of convolutional restricted boltz- mann machines for shift-invariant feature learning. In Proc. CVPR, pages 2735-2742, 2009.","title":"Stacks of convolutional restricted boltzmann machines for shift-invariant feature learning","authors":"Mohammad Norouzi, Mani Ranjbar, Greg Mori","year":"2009","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ranzato et al., 2007] Marc'Aurelio Ranzato, Y-Lan Boureau, and Yann LeCun. Sparse feature learning for deep belief networks. In Proc. NIPS, 2007.","title":"Sparse feature learning for deep belief networks","authors":"Ranzato","year":"2007","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rifai et al., 2011] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extrac- tion. In Proc. ICML, 2011.","title":"Contractive auto-encoders: Explicit invariance during feature extraction","authors":"Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, Yoshua Bengio","year":"2011","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Scellier and Bengio, 2016] Benjamin Scellier and Yoshua Bengio. Towards a biologically plausible backprop. arX- iv:1602.05179, 2016.","title":"Towards a biologically plausible backprop","authors":"Benjamin Scellier, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Valpola, 2014] Harri Valpola. From neural PCA to deep un- supervised learning. arXiv:1411.7783, 2014.","title":"From neural PCA to deep unsupervised learning","authors":"Harri Valpola","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning use- ful representations in a deep network with a local de- noising criterion. Journal of Machine Learning Research, 11:3371-3408, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}