{"type":"ScholarlyArticle","title":"Exchangeability and Kernel Invariance in Trained MLPs","authors":[{"name":"Russell Tsuchida","slug":"russell-tsuchida","given":"Russell","family":"Tsuchida","affiliation":"The University of Queensland"},{"name":"Fred Roosta","slug":"fred-roosta","given":"Fred","family":"Roosta","affiliation":"The University of Queensland; International Computer Science Institute"},{"name":"Marcus Gallagher","slug":"marcus-gallagher","given":"Marcus","family":"Gallagher","affiliation":"The University of Queensland"}],"abstract":"In the analysis of machine learning models, it is often convenient to assume that the parameters are IID. This assumption is not satisfied when the parameters are updated through training processes such as Stochastic Gradient Descent. A relaxation of the IID condition is a probabilistic symmetry known as exchangeability. We show the sense in which the weights in MLPs are exchangeable. This yields the result that in certain instances, the layer-wise kernel of fully-connected layers remains approximately constant during training. Our results shed light on such kernel properties throughout training while limiting the use of unrealistic assumptions.","doi":"10.24963/ijcai.2019/498","url":"https://ijcai.org/proceedings/2019/498","pdf":"https://ijcai.org/proceedings/2019/0498.pdf","markdown":"https://ijcai.org/proceedings/2019/498.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/498","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":498,"pages":{"first":3592,"last":3598},"keywords":["Machine Learning: Deep Learning","Machine Learning: Kernel Methods"],"videos":[],"references":[{"position":1,"raw":"[Adamczak et al., 2016] R. Adamczak, D. Chafaï, and P. Wolff. Circular law for random matrices with exchange- able entries. Random Structures & Algorithms, 48(3):454- 479, 2016.","title":"Circular law for random matrices with exchangeable entries","authors":"R Adamczak, D Chafaï, P Wolff","year":"2016","venue":"Random Structures & Algorithms","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aldous, 1981] D.J. Aldous. Representations for partially exchangeable arrays of random variables. Journal of Multi- variate Analysis, 11(4):581-598, 1981.","title":"Representations for partially exchangeable arrays of random variables","authors":"D J Aldous","year":"1981","venue":"Journal of Multivariate Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bach, 2017] F. Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(19):1-53, 2017.","title":"Breaking the curse of dimensionality with convex neural networks","authors":"F Bach","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bietti and Mairal, 2017] A. Bietti and J. Mairal. Invariance and stability of deep convolutional representations. In Ad- vances in Neural Information Processing Systems, pages 6210-6220, 2017.","title":"Invariance and stability of deep convolutional representations","authors":"A Bietti, J","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chatterjee, 2006] S. Chatterjee. A generalization of the Lin- deberg principle. The Annals of Probability, 34(6):2061- 2076, 2006.","title":"A generalization of the Lindeberg principle","authors":"S Chatterjee","year":"2006","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chaudhari and Soatto, 2018] P. Chaudhari and S. Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In Interna- tional Conference on Learning Representations, 2018.","title":"Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks","authors":"P Chaudhari, S Soatto","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chizat and Bach, 2018] L. Chizat and F. Bach. A note on lazy training in supervised differentiable programming. Technical report, 2018.","title":"A note on lazy training in supervised differentiable programming","authors":"L Chizat, F Bach","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cho and Saul, 2009] Y. Cho and L.K. Saul. Kernel meth- ods for deep learning. In Advances in Neural Information Processing Systems, pages 342-350, 2009.","title":"Kernel methods for deep learning","authors":"Y Cho, L K Saul","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daniely et al., 2016] A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances In Neural Information Processing Systems, pages 2253-2261, 2016.","title":"Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity","authors":"A Daniely, R Frostig, Y Singer","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dozat, 2016] T. Dozat. Incorporating nesterov momentum into adam. Technical report, 2016.","title":"Incorporating nesterov momentum into adam","authors":"T Dozat","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Du et al., 2019] S.S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes over-parameterized neural networks. International Conference on Learning Representations, 2019.","title":"Gradient descent provably optimizes over-parameterized neural networks","authors":"S S Du, X Zhai, B Poczos, A Singh","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2015] K. He, X. Zhang, S. Ren, and J. Sun. Delv- ing deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In International Confer- ence on Computer Vision, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"K He, X Zhang, S Ren, J Sun","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jacot et al., 2018] A. Jacot, F. Gabriel, and C. Hongler. Neu- ral tangent kernel: Convergence and generalization in neu- ral networks. In Advances in neural information processing systems, pages 8571-8580, 2018.","title":"Neural tangent kernel: Convergence and generalization in neural networks","authors":"A Jacot, F Gabriel, C Hongler","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kallenberg, 2006] O. Kallenberg. Probabilistic symmetries and invariance principles. Springer Science & Business Media, 2006.","title":"Probabilistic symmetries and invariance principles","authors":"O Kallenberg","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Khinchin, 1949] A.I. Khinchin. Mathematical foundations of statistical mechanics. Courier Corporation, 1949.","title":"Mathematical foundations of statistical mechanics","authors":"A I Khinchin","year":"1949","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2015] D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Con- ference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"D P Kingma, J Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Tech- nical report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kurth, 2014] R. Kurth. Axiomatics of classical statistical mechanics. Elsevier, 2014.","title":"Axiomatics of classical statistical mechanics","authors":"R Kurth","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[MacKay, 1992] D.J.C. MacKay. A practical Bayesian frame- work for backpropagation networks. Neural Computation, 4(3):448-472, 1992.","title":"A practical Bayesian framework for backpropagation networks","authors":"D J C Mackay","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Martin and Mahoney, 2017] C.H. Martin and M.W. Ma- honey. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learn- ing behavior. arXiv preprint arXiv:1710.09553, 2017.","title":"Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior","authors":"C H Martin, M W Mahoney","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Moore, 2015] C.C. Moore. Ergodic theorem, ergodic theory, and statistical mechanics. Proceedings of the National Academy of Sciences, 112(7):1907-1911, 2015.","title":"Ergodic theorem, ergodic theory, and statistical mechanics","authors":"C C Moore","year":"2015","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Neal, 1994] R.M. Neal. Bayesian Learning for Neural Net- works. PhD thesis, University of Toronto, 1994.","title":"Bayesian Learning for Neural Networks","authors":"R M Neal","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pennington and Bahri, 2017] J. Pennington and Y. Bahri. Geometry of neural network loss surfaces via random ma- trix theory. In International Conference on Machine Learn- ing, pages 2798-2806, 2017.","title":"Geometry of neural network loss surfaces via random matrix theory","authors":"J Pennington, Y Bahri","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pennington and Worah, 2017] J. Pennington and P. Worah. Nonlinear random matrix theory for deep learning. In Advances in Neural Information Processing Systems, pages 2634-2643, 2017.","title":"Nonlinear random matrix theory for deep learning","authors":"J Pennington, P Worah","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Poole et al., 2016] B. Poole, S. Lahiri, M. Raghu, J. Sohl- Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems, pages 3360-3368, 2016.","title":"Exponential expressivity in deep neural networks through transient chaos","authors":"B Poole, S Lahiri, M Raghu, J Sohl-Dickstein, S Ganguli","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raghu et al., 2017] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein. On the expressive power of deep neural networks. In International Conference on Machine Learning, pages 2847-2854, 2017.","title":"On the expressive power of deep neural networks","authors":"M Raghu, B Poole, J Kleinberg, S Ganguli, J Sohl-Dickstein","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Royden and Fitzpatrick, 2010] H.L. Royden and P. Fitz- patrick. Real Analysis. Prentice Hall, 2010.","title":"Real Analysis","authors":"H L Royden, P","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Seung et al., 1992] H.S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Physical Review A, 45(8):6056, 1992.","title":"Statistical mechanics of learning from examples","authors":"H S Seung, H Sompolinsky, N Tishby","year":"1992","venue":"Physical Review A","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tsuchida et al., 2018] R. Tsuchida, F. Roosta-Khorasani, and M. Gallagher. Invariance of weight distributions in rectified MLPs. International Conference on Machine Learning, 2018.","title":"Invariance of weight distributions in rectified MLPs","authors":"R Tsuchida, F Roosta-Khorasani, M Gallagher","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Watkin et al., 1993] T.L.H. Watkin, A. Rau, and M. Biehl. The statistical mechanics of learning a rule. Reviews of Modern Physics, 65(2):499, 1993.","title":"The statistical mechanics of learning a rule","authors":"T L H Watkin, A Rau, M Biehl","year":"1993","venue":"Reviews of Modern Physics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}