{"type":"ScholarlyArticle","title":"Towards Understanding the Invertibility of Convolutional Neural Networks","authors":[{"name":"Anna Gilbert","slug":"anna-gilbert","given":"Anna","family":"Gilbert","affiliation":"University of Michigan , Ann Arbor , MI 48109"},{"name":"Yi Zhang","slug":"yi-zhang","given":"Yi","family":"Zhang","affiliation":"University of Michigan , Ann Arbor , MI 48109"},{"name":"Kibok Lee","slug":"kibok-lee","given":"Kibok","family":"Lee","affiliation":"University of Michigan , Ann Arbor , MI 48109"},{"name":"Yuting Zhang","slug":"yuting-zhang","given":"Yuting","family":"Zhang","affiliation":"University of Michigan , Ann Arbor , MI 48109"},{"name":"Honglak Lee","slug":"honglak-lee","given":"Honglak","family":"Lee","affiliation":"University of Michigan , Ann Arbor , MI 48109; Google Brain , Mountain View , CA 94043"}],"abstract":"Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is consistent with CNNs with random weights. We give an exact connection to a particular model of model-based compressive sensing (and its recovery algorithms) and random-weight CNNs. We show empirically that several learned networks are consistent with our mathematical analysis and then demonstrate that with such a simple theoretical framework, we can obtain reasonable reconstruction results on real images. We also discuss gaps between our model assumptions and the CNN trained for classification in practical scenarios.","doi":"10.24963/ijcai.2017/236","url":"https://ijcai.org/proceedings/2017/236","pdf":"https://ijcai.org/proceedings/2017/0236.pdf","markdown":"https://ijcai.org/proceedings/2017/236.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/236","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":236,"pages":{"first":1703,"last":1710},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2014] Sanjeev Arora, Aditya Bhaskara, Rong Ge, and Tengyu Ma. Provable Bounds for Learning Some Deep Represen- tations. In ICML, 2014.","title":"Provable Bounds for Learning Some Deep Representations","authors":"Sanjeev Arora, Aditya Bhaskara, Rong Ge, Tengyu Ma","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2015] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. Why are deep nets reversible: A simple theory, with implications for training. arXiv:1511.05653, 2015.","title":"Why are deep nets reversible: A simple theory, with implications for training","authors":"Sanjeev Arora, Yingyu Liang, Tengyu Ma","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baraniuk et al., 2010] R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde. Model-Based Compressive Sensing. IEEE Transactions on Information Theory, 56(4):1982-2001, 2010.","title":"Model-Based Compressive Sensing","authors":"R G Baraniuk, V Cevher, M F Duarte, C Hegde","year":"2010","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse prob- lems. SIAM Journal of Imaging Science, 2:183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM Journal of Imaging Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blumensath and Davies, 2009] Thomas Blumensath and Mike E Davies. Iterative hard thresholding for compressed sensing. Ap- plied and Computational Harmonic Analysis, 27(3):265-274, 2009.","title":"Iterative hard thresholding for compressed sensing","authors":"Thomas Blumensath, Mike E Davies","year":"2009","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boureau et al., 2008] Y-lan Boureau, Yann L Cun, et al. Sparse feature learning for deep belief networks. In NIPS, 2008.","title":"Sparse feature learning for deep belief networks","authors":"Y-Lan Boureau, Yann L Cun","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Boyd, 2015] Stephen Boyd. l1-norm methods for convex- cardinality problems, ee364b: Convex optimization II lecture notes, 2014-2015 spring. 2015.","title":"Stephen Boyd. l1-norm methods for convexcardinality problems","authors":"Boyd","year":"2015","venue":"Convex optimization II lecture notes","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bruna et al., 2014] Joan Bruna, Arthur Szlam, and Yann LeCun. Signal recovery from pooling representations. In ICML, 2014.","title":"Signal recovery from pooling representations","authors":"Bruna","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Candés, 2008] Emmanuel J. Candés. The restricted isometry prop- erty and its implications for compressed sensing. Comptes Rendus Mathematique, 346(9):589-592, 2008.","title":"The restricted isometry property and its implications for compressed sensing","authors":"Emmanuel J Candés","year":"2008","venue":"Comptes Rendus Mathematique","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations us- ing rnn encoder-decoder for statistical machine translation. arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bot- tou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natu- ral language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, R Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dosovitskiy and Brox, 2016] Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional net- works. In CVPR, 2016.","title":"Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks","authors":"Brox Dosovitskiy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Giryes et al., 2016] Raja Giryes, Guillermo Sapiro, and Alex M Bronstein. Deep neural networks with random gaussian weights: A universal classification strategy? IEEE Transactions on Signal Processing, 64(13):3444-3457, 2016.","title":"Deep neural networks with random gaussian weights: A universal classification strategy","authors":"Raja Giryes, Guillermo Sapiro, Alex M Bronstein","year":"2016","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hannun et al., 2014] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv:1412.5567, 2014.","title":"Deep speech: Scaling up end-to-end speech recognition","authors":"Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kun He, Yan Wang, and John Hopcroft. A power- ful generative model using random weights for the deep image representation. arXiv:1606.04801, 2016.","title":"A powerful generative model using random weights for the deep image representation","authors":"Kun He, Yan Wang, John Hopcroft","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hinton et al., 2012] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82-97, 2012.","title":"Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups","authors":"Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath","year":"2012","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jarrett et al., 2009] Kevin Jarrett, Koray Kavukcuoglu, Marc'Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In ICCV, 2009.","title":"What is the best multi-stage architecture for object recognition?","authors":"Kevin Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadar- rama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G. E. Hin- ton. Imagenet classification with deep convolutional neural net- works. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Le et al., 2013] Q. V. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng. Building high- level features using large scale unsupervised learning. In ICML, 2013.","title":"Building highlevel features using large scale unsupervised learning","authors":"Q V Le, M Ranzato, R Monga, M Devin, K Chen, G S Corrado, J Dean, A Y Ng","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[LeCun et al., 1989] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.","title":"Backpropagation applied to handwritten zip code recognition","authors":"Yann Lecun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, Lawrence D Jackel","year":"1989","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lee et al., 2008] Honglak Lee, Chaitanya Ekanadham, and An- drew Y Ng. Sparse deep belief net model for visual area v2. In NIPS, 2008.","title":"Honglak Lee, Chaitanya Ekanadham, and Andrew Y Ng. Sparse deep belief net model for visual area v2","authors":"Lee","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mallat and Zhang, 1993] Stephane Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Trans- actions on Signal Processing, 41:3397 -3415, 1993.","title":"Matching pursuits with time-frequency dictionaries","authors":"Stephane Mallat, Zhifeng Zhang","year":"1993","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Olshausen and others, 1996] Bruno A Olshausen et al. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607-609, 1996.","title":"Emergence of simple-cell receptive field properties by learning a sparse code for natural images","authors":"Bruno A Olshausen","year":"1996","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Park et al., 2011] Jae Young Park, Han Lun Yap, C.J. Rozell, and M. B. Wakin. Concentration of Measure for Block Diagonal Ma- trices With Applications to Compressive Signal Processing. IEEE Transactions on Signal Processing, 59(12):5859-5875, 2011.","title":"Concentration of Measure for Block Diagonal Matrices With Applications to Compressive Signal Processing","authors":"Jae Young Park, Han Lun Yap, C J Rozell, M B Wakin","year":"2011","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Paul and Venkatasubramanian, 2014] Arnab Paul and Suresh Venkatasubramanian. Why does Deep Learning work? -A perspective from Group Theory. arXiv.org, December 2014.","title":"Why does Deep Learning work? -A perspective from Group Theory","authors":"Arnab Paul, Suresh Venkatasubramanian","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ranzato et al., 2007] Marc Aurelio Ranzato, Fu Jie Huang, Y-Lan Boureau, and Yann LeCun. Unsupervised learning of invariant feature hierarchies with applications to object recognition. In CVPR, 2007.","title":"Unsupervised learning of invariant feature hierarchies with applications to object recognition","authors":"Aurelio Marc, Ranzato, Jie Fu, Y-Lan Huang, Yann Boureau, Lecun","year":"2007","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Saxe et al., 2011] Andrew Saxe, Pang W Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. On random weights and unsupervised feature learning. In ICML, 2011.","title":"On random weights and unsupervised feature learning","authors":"Andrew Saxe, W Pang, Zhenghao Koh, Maneesh Chen, Bipin Bhand, Andrew Y Suresh, Ng","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Shang et al., 2016] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In ICML, 2016.","title":"Understanding and improving convolutional neural networks via concatenated rectified linear units","authors":"Wenling Shang, Kihyuk Sohn, Diogo Almeida, Honglak Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vedaldi and Lenc, 2015] A. Vedaldi and K. Lenc. Matconvnet - convolutional neural networks for matlab. In Proceeding of the ACM International Conference on Multimedia, 2015.","title":"Matconvnetconvolutional neural networks for matlab","authors":"A Vedaldi, K Lenc","year":"2015","venue":"Proceeding of the ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vershynin, 2010] Roman Vershynin. Introduction to the non- asymptotic analysis of random matrices. arXiv:1011.3027, November 2010.","title":"Roman Vershynin. Introduction to the nonasymptotic analysis of random matrices","authors":"Vershynin","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2010] Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11):2861-2873, 2010.","title":"Image super-resolution via sparse representation","authors":"Jianchao Yang, John Wright, Thomas S Huang, Yi Ma","year":"2010","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2016] Yuting Zhang, Kibok Lee, and Honglak Lee. Augmenting neural networks with reconstructive decoding path- ways for large-scale image classification. In ICML, 2016.","title":"Augmenting neural networks with reconstructive decoding pathways for large-scale image classification","authors":"Yuting Zhang, Kibok Lee, Honglak Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhao et al., 2016] Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann Lecun. Stacked what-where auto-encoders. arXiv:1506.02351, 2016.","title":"Stacked what-where auto-encoders","authors":"Junbo Zhao, Michael Mathieu, Ross Goroshin, Yann Lecun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}