{"type":"ScholarlyArticle","title":"Neurogenesis-Inspired Dictionary Learning: Online Model Adaption in a Changing World","authors":[{"name":"Sahil Garg","slug":"sahil-garg","given":"Sahil","family":"Garg","affiliation":"Research Center"},{"name":"Irina Rish","slug":"irina-rish","given":"Irina","family":"Rish","affiliation":"Research Center"},{"name":"Guillermo Cecchi","slug":"guillermo-cecchi","given":"Guillermo","family":"Cecchi","affiliation":"Research Center"},{"name":"Aurelie Lozano","slug":"aurelie-lozano","given":"Aurelie","family":"Lozano","affiliation":"Research Center"}],"abstract":"We address the problem of online model adaptation when learning representations from non-stationary data streams. Specifically, we focus here on online dictionary learning (i.e. sparse linear autoencoder), and propose a simple but effective online model selection approach involving “birth” (addition) and “death” (removal) of hidden units representing dictionary\r\nelements, in response to changing inputs; we draw inspiration from the adult neurogenesis phenomenon in the dentate gyrus of the hippocampus, known to be associated with better adaptation to new environments. Empirical evaluation on real-life datasets (images and text), as well as on synthetic data, demonstrates that the proposed approach can considerably outperform the state-of-art non-adaptive online sparse coding of [Mairal et al., 2009] in the presence of non-stationary data. Moreover, we identify certain data- and model properties\r\nassociated with such improvements.","doi":"10.24963/ijcai.2017/235","url":"https://ijcai.org/proceedings/2017/235","pdf":"https://ijcai.org/proceedings/2017/0235.pdf","markdown":"https://ijcai.org/proceedings/2017/235.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/235","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":235,"pages":{"first":1696,"last":1702},"keywords":["Machine Learning: Machine Learning","Multidisciplinary Topics and Applications: Cognitive Modeling","Machine Learning: Online Learning","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Aharon et al., 2006] Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. Signal Processing, IEEE Transactions on, 2006.","title":"K-svd: An algorithm for designing overcomplete dictionaries for sparse representation","authors":"Michal Aharon, Michael Elad, Alfred Bruckstein","year":"2006","venue":"Signal Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information pro- cessing systems, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Samy Bengio, Fernando Pereira, Yoram Singer, and Dennis Strelow. Group sparse coding. In Advances in Neural Information Processing Systems 22. 2009.","title":"Group sparse coding","authors":"Samy Bengio, Fernando Pereira, Yoram Singer, Dennis Strelow","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bucilu et al., 2006] Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006.","title":"Model compression","authors":"Cristian Bucilu, Rich Caruana, Alexandru Niculescu-Mizil","year":"2006","venue":"Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Draelos et al., 2016] Timothy J Draelos, Nadine E Miner, Christo- pher C Lamb, Craig M Vineyard, Kristofor D Carlson, Conrad D James, and James B Aimone. Neurogenesis deep learning. arXiv preprint arXiv:1612.03770, 2016.","title":"Neurogenesis deep learning","authors":"Timothy J Draelos, Nadine E Miner, Christopher C Lamb, Craig M Vineyard, Conrad D Kristofor D Carlson, James B James, Aimone","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fahlman and Lebiere, 1989] Scott E Fahlman and Christian Lebiere. The cascade-correlation learning architecture. 1989.","title":"The cascade-correlation learning architecture","authors":"E Scott, Christian Fahlman, Lebiere","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jenatton et al., 2011] Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, and Francis Bach. Proximal methods for hierarchical sparse coding. Journal of Machine Learning Research, 2011.","title":"Proximal methods for hierarchical sparse coding","authors":"Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, Francis Bach","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kempermann, 2006] Gerd Kempermann. Adult neurogenesis: stem cells and neuronal development in the adult brain. 2006.","title":"Adult neurogenesis: stem cells and neuronal development in the adult brain","authors":"Gerd Kempermann, Kempermann","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kreutz-Delgado et al., 2003] Kenneth Kreutz-Delgado, Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te-Won Lee, and Ter- rence J Sejnowski. Dictionary learning algorithms for sparse rep- resentation. Neural computation, 2003.","title":"Dictionary learning algorithms for sparse representation","authors":"Kenneth Kreutz-Delgado, Joseph F Murray, Kjersti Bhaskar D Rao, Te-Won Engan, Terrence J Lee, Sejnowski","year":"2003","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2006] Honglak Lee, Alexis Battle, Rajat Raina, and An- drew Y Ng. Efficient sparse coding algorithms. In Advances in neural information processing systems, 2006.","title":"Efficient sparse coding algorithms","authors":"Lee","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mairal et al., 2009] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online dictionary learning for sparse coding. In Proceedings of the 26th annual international conference on machine learning, 2009.","title":"Online dictionary learning for sparse coding","authors":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro","year":"2009","venue":"Proceedings of the 26th annual international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mariet and Sra, 2015] Zelda Mariet and Suvrit Sra. Diversity net- works. arXiv preprint arXiv:1511.05077, 2015.","title":"Zelda Mariet and Suvrit Sra. Diversity networks","authors":"Sra Mariet","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Olshausen and Field, 1997] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy em- ployed by v1? Vision research, 1997.","title":"Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research","authors":"A Bruno, David J Olshausen, Field","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ramirez and Sapiro, 2012] Ignacio Ramirez and Guillermo Sapiro. An mdl framework for sparse coding and dictionary learning. IEEE Transactions on Signal Processing, 60(6):2913- 2927, 2012.","title":"Ignacio Ramirez and Guillermo Sapiro. An mdl framework for sparse coding and dictionary learning","authors":"Sapiro Ramirez","year":"2012","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rusu et al., 2016] Andrei A Rusu, Neil C Rabinowitz, Guil- laume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.","title":"Progressive neural networks","authors":"Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, Raia Hadsell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Stuchlik, 2014] Ales Stuchlik. Dynamic learning and memory, synaptic plasticity and neurogenesis: an update. Frontiers in be- havioral neuroscience, 2014.","title":"Dynamic learning and memory, synaptic plasticity and neurogenesis: an update","authors":"Ales Stuchlik, Stuchlik","year":"2014","venue":"Frontiers in behavioral neuroscience","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and se- lection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yogatama et al., 2015] Dani Yogatama, Manaal Faruqui, Chris Dyer, and Noah A Smith. Learning word representations with hierarchical sparse coding. In Proc. of ICML, 2015.","title":"Learning word representations with hierarchical sparse coding","authors":"Dani Yogatama, Manaal Faruqui, Chris Dyer, Noah A Smith","year":"2015","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yuan and Lin, 2006] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2006.","title":"Model selection and estimation in regression with grouped variables","authors":"Ming Yuan, Yi Lin","year":"2006","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}