{"type":"ScholarlyArticle","title":"The Orthogonality of Weight Vectors: The Key Characteristics of Normalization and Residual Connections","authors":[{"name":"Zhixing Lu","slug":"zhixing-lu","given":"Zhixing","family":"Lu","affiliation":"College of Computer Science and Technology , Dalian University of Technology , Dalian , 116081 , Liaoning , China"},{"name":"Yuanyuan Sun","slug":"yuanyuan-sun","given":"Yuanyuan","family":"Sun","affiliation":"College of Computer Science and Technology , Dalian University of Technology , Dalian , 116081 , Liaoning , China"},{"name":"Zhihao Yang","slug":"zhihao-yang","given":"Zhihao","family":"Yang","affiliation":"College of Computer Science and Technology , Dalian University of Technology , Dalian , 116081 , Liaoning , China"},{"name":"Qin Zhou","slug":"qin-zhou","given":"Qin","family":"Zhou","affiliation":"College of Computer Science and Technology , Dalian University of Technology , Dalian , 116081 , Liaoning , China"},{"name":"Hongfei Lin","slug":"hongfei-lin","given":"Hongfei","family":"Lin","affiliation":"College of Computer Science and Technology , Dalian University of Technology , Dalian , 116081 , Liaoning , China"}],"abstract":"Normalization and residual connections find extensive application within the intricate architecture of deep neural networks, contributing significantly to their heightened performance. Nevertheless, the precise factors responsible for this elevated performance have remained elusive. Our theoretical investigations have unveiled a noteworthy revelation: the utilization of normalization and residual connections results in an enhancement of the orthogonality within the weight vectors of deep neural networks. This, in turn, induces the Gram matrix of neural network weights to exhibit a pronounced tendency towards strict diagonal dominance, thereby amplifying the neural network's capacity for feature learning. Meanwhile, we have designed the parameters independence index (PII) to precisely characterize the orthogonality of parameter vectors. In tandem with our theoretical findings, we undertook empirical validations through experiments conducted on prevalent network models, including fully connected networks (FNNs), convolutional neural networks (CNNs), Transformers,  pre-trained language models(PLMs) and large language models (LLMs) composed of Transformers. Finally, we have found that a fine-tuning technique (LoRA) preserves the orthogonality of parameter vectors, a revelation that carries importance within the framework of fine-tuning techniques for LLMs.","doi":"10.24963/ijcai.2024/518","url":"https://ijcai.org/proceedings/2024/518","pdf":"https://ijcai.org/proceedings/2024/0518.pdf","markdown":"https://ijcai.org/proceedings/2024/518.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/518","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":518,"pages":{"first":4687,"last":4695},"keywords":["Machine Learning: ML: Explainable/Interpretable machine learning","Machine Learning: ML: Deep learning architectures"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.","title":"Gpt-4 technical report","authors":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2018] Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch nor- malization. arXiv preprint arXiv:1812.03981, 2018.","title":"Theoretical analysis of auto rate-tuning by batch normalization","authors":"Sanjeev Arora, Zhiyuan Li, Kaifeng Lyu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.","title":"Qwen technical report","authors":"Bai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Balduzzi et al., 2017] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning, pages 342-350. PMLR, 2017.","title":"The shattered gradients problem: If resnets are the answer, then what is the question?","authors":"David Balduzzi, Marcus Frean, Lennox Leary, Kurt Lewis, Wan-Duo Ma, Brian Mcwilliams","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Beaglehole et al., 2023] Daniel Beaglehole, Aditya- narayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Mechanism of feature learning in convolutional neural networks. arXiv preprint arXiv:2309.00570, 2023.","title":"Mechanism of feature learning in convolutional neural networks","authors":"Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, Mikhail Belkin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chowdhery et al., 2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling lan- guage modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.","title":"Palm: Scaling language modeling with pathways","authors":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Daneshmand et al., 2020] Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Advances in Neural Information Processing Systems, 33:18387-18398, 2020.","title":"Batch normalization provably avoids ranks collapse for randomly initialised deep networks","authors":"Jonas Hadi Daneshmand, Francis Kohler, Thomas Bach, Aurelien Hofmann, Lucchi","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daneshmand et al., 2021] Hadi Daneshmand, Amir Joudaki, and Francis Bach. Batch normalization or- thogonalizes representations in deep random networks. Advances in Neural Information Processing Systems, 34:4896-4906, 2021.","title":"Batch normalization orthogonalizes representations in deep random networks","authors":"Amir Hadi Daneshmand, Francis Joudaki, Bach","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[De and Smith, 2020] Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Informa- tion Processing Systems, 33:19964-19975, 2020.","title":"Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks","authors":"Smith De","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[DeMaris, 1995] Alfred DeMaris. A tutorial in logistic re- gression. Journal of Marriage and the Family, pages 956- 968, 1995.","title":"A tutorial in logistic regression","authors":"Alfred Demaris, Demaris","year":"1995","venue":"Journal of Marriage and the Family","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fan, 2000] Engui Fan. Extended tanh-function method and its applications to nonlinear equations. Physics Letters A, 277(4-5):212-218, 2000.","title":"Engui Fan. Extended tanh-function method and its applications to nonlinear equations","authors":"Fan","year":"2000","venue":"Physics Letters A","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Furusho and Ikeda, 2019] Yasutaka Furusho and Kazushi Ikeda. Resnet and batch-normalization improve data sepa- rability. In Asian Conference on Machine Learning, pages 94-108. PMLR, 2019.","title":"Resnet and batch-normalization improve data separability","authors":"Ikeda Furusho, Yasutaka Furusho, Kazushi Ikeda","year":"2019","venue":"Asian Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Furusho and Ikeda, 2020] Yasutaka Furusho and Kazushi Ikeda. Theoretical analysis of skip connections and batch normalization from generalization and optimization per- spectives. APSIPA Transactions on Signal and Informa- tion Processing, 9:e9, 2020.","title":"Theoretical analysis of skip connections and batch normalization from generalization and optimization perspectives","authors":"Ikeda Furusho, Yasutaka Furusho, Kazushi Ikeda","year":"2020","venue":"APSIPA Transactions on Signal and Information Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth in- ternational conference on artificial intelligence and statis- tics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proceedings of the thirteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.","title":"Gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Huang et al., 2018] Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over mul- tiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 32, 2018.","title":"Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks","authors":"Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, Bo Li","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Karakida et al., 2019] Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. The normalization method for alle- viating pathological sharpness in wide neural networks. Advances in neural information processing systems, 32, 2019.","title":"The normalization method for alleviating pathological sharpness in wide neural networks","authors":"Karakida","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Katsman et al., 2023] Isay Katsman, Eric Ming Chen, Sid- hanth Holalkere, Anna Asch, Aaron Lou, Ser-Nam Lim, and Christopher De Sa. Riemannian residual neural net- works. arXiv preprint arXiv:2310.10013, 2023.","title":"Riemannian residual neural networks","authors":"Isay Katsman, Eric, Ming Chen, Sidhanth Holalkere, Anna Asch, Aaron Lou, Ser-Nam Lim, Christopher De Sa","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kenton and Toutanova, 2019] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton, Lee Kristina, Toutanova","year":"2019","venue":"Proceedings of naacL-HLT","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kohler et al., 2019] Jonas Kohler, Hadi Daneshmand, Au- relien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch nor- malization: The power of length-direction decoupling in non-convex optimization. In The 22nd International Con- ference on Artificial Intelligence and Statistics, pages 806- 815. PMLR, 2019.","title":"Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization","authors":"Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, Klaus Neymeyr","year":"2019","venue":"The 22nd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[LeCun et al., 2002] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Effi- cient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.","title":"Efficient backprop","authors":"Yann Lecun, Léon Bottou, Genevieve B Orr, Klaus-Robert Müller","year":"2002","venue":"Neural networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2019] Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intel- ligence, 43(4):1352-1368, 2019.","title":"Orthogonal deep neural networks","authors":"Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, Dacheng Tao","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2020] Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, and Yuexian Zou. Rethinking skip con- nection with layer normalization. In Proceedings of the 28th international conference on computational linguis- tics, pages 3586-3598, 2020.","title":"Rethinking skip connection with layer normalization","authors":"Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, Yuexian Zou","year":"2020","venue":"Proceedings of the 28th international conference on computational linguistics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In Proceedings of the 27th international con- ference on machine learning (ICML-10), pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"Proceedings of the 27th international conference on machine learning (ICML-10)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Orhan and Pitkow, 2017] A Emin Orhan and Xaq Pitkow. Skip connections eliminate singularities. arXiv preprint arXiv:1701.09175, 2017.","title":"Skip connections eliminate singularities","authors":"Pitkow ; A Emin Orhan, Xaq Orhan, Pitkow","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Oyedotun et al., 2022] Oyebade K Oyedotun, Kassem Al Ismaeil, and Djamila Aouada. Why is everyone training very deep neural network with skip connections? IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Why is everyone training very deep neural network with skip connections","authors":"Oyedotun","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Radhakrishnan et al., 2022] Adityanarayanan Radhakrish- nan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Feature learning in neural networks and kernel machines that recursively learn features. arXiv preprint arXiv:2212.13881, 2022.","title":"Adityanarayanan Radhakrishnan","authors":"Radhakrishnan","year":"2022","venue":"Feature learning in neural networks and kernel machines that recursively learn features","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485- 5551, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234- 241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shao et al., 2020] Jie Shao, Kai Hu, Changhu Wang, Xi- angyang Xue, and Bhiksha Raj. Is normalization indis- pensable for training deep neural network? Advances in Neural Information Processing Systems, 33:13434-13444, 2020.","title":"Is normalization indispensable for training deep neural network","authors":"Jie Shao, Kai Hu, Changhu Wang, Xiangyang Xue, Bhiksha Raj","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tanton, 2005] James S Tanton. Encyclopedia of mathemat- ics. Infobase Publishing, 2005.","title":"Encyclopedia of mathematics","authors":"James S Tanton","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard","authors":"Touvron","year":"2023","venue":"Llama: Open and efficient foundation language models","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wang et al., 2020] Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 11505-11515, 2020.","title":"Orthogonal convolutional neural networks","authors":"Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, Stella X Yu","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Xiao et al., 2018] Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural net- works. In International Conference on Machine Learning, pages 5393-5402. PMLR, 2018.","title":"Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks","authors":"Xiao","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yang et al., 2019] Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08129, 2019.","title":"A mean field theory of batch normalization","authors":"Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S Schoenholz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yong et al., 2020] Hongwei Yong, Jianqiang Huang, Xian- sheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 635-652. Springer, 2020.","title":"Gradient centralization: A new optimization technique for deep neural networks","authors":"Hongwei Yong, Jianqiang Huang, Xiansheng Hua, Lei Zhang","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}