{"type":"ScholarlyArticle","title":"Efficiency Calibration of Implicit Regularization in Deep Networks via Self-paced Curriculum-Driven Singular Value Selection","authors":[{"name":"Zhe Li","slug":"zhe-li","given":"Zhe","family":"Li","affiliation":"PCA Lab , Nanjing University of Science and Technology"},{"name":"Shuo Chen","slug":"shuo-chen","given":"Shuo","family":"Chen","affiliation":"RIKEN"},{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"PCA Lab , Nanjing University of Science and Technology"},{"name":"Lei Luo","slug":"lei-luo","given":"Lei","family":"Luo","affiliation":"PCA Lab , Nanjing University of Science and Technology"}],"abstract":"The generalization of neural networks has been a major focus of research in deep learning. It is often interpreted as an implicit bias towards solutions with specific properties. Especially, in practical applications, it has been observed that linear neural networks (LNN) tend to favor low-rank solutions for matrix completion tasks. However, most existing methods rely on increasing the depth of the neural network to enhance the low rank of solutions, resulting in higher complexity. In this paper, we propose a new explicit regularization method that calibrates the implicit bias towards low-rank trends in matrix completion tasks. Our approach automatically incorporates smaller singular values into the training process using a self-paced learning strategy, gradually restoring matrix information. By jointly using both implicit and explicit regularization, we effectively capture the low-rank structure of LNN and accelerate its convergence. We also analyze how our proposed penalty term interacts with implicit regularization and provide theoretical guarantees for our new model. To evaluate the effectiveness of our method, we conduct a series of experiments on both simulated and real-world data. Our experimental results clearly demonstrate that our method has better robustness and generalization ability compared with other methods.","doi":"10.24963/ijcai.2024/499","url":"https://ijcai.org/proceedings/2024/499","pdf":"https://ijcai.org/proceedings/2024/0499.pdf","markdown":"https://ijcai.org/proceedings/2024/499.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/499","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":499,"pages":{"first":4515,"last":4523},"keywords":["Machine Learning: ML: Representation learning","Data Mining: DM: Recommender systems","Machine Learning: ML: Theory of deep learning"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2018a] Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neu- ral networks with rectified linear units. In ICLR. OpenRe- view.net, 2018.","title":"Understanding deep neural networks with rectified linear units","authors":"Arora","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2018b] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit ac- celeration by overparameterization. In ICML, volume 80, pages 244-253. PMLR, 2018.","title":"On the optimization of deep networks: Implicit acceleration by overparameterization","authors":"Arora","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora et al., 2019] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. NIPS, 32:7411-7422, 2019.","title":"Implicit regularization in deep matrix factorization","authors":"Sanjeev Arora, Nadav Cohen, Wei Hu, Yuping Luo","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Belkin et al., 2019] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade- off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.","title":"Reconciling modern machinelearning practice and the classical bias-variance tradeoff","authors":"Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal","year":"2019","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cai et al., 2010] Jian-Feng Cai, Emmanuel J Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956-1982, 2010.","title":"A singular value thresholding algorithm for matrix completion","authors":"Jian-Feng Cai, Emmanuel J Candès, Zuowei Shen","year":"2010","venue":"SIAM Journal on optimization","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Candes and Recht, 2012] Emmanuel Candes and Benjamin Recht. Exact matrix completion via convex optimization. Communications of the ACM, 55(6):111-119, 2012.","title":"Exact matrix completion via convex optimization","authors":"Recht Candes, Emmanuel Candes, Benjamin Recht","year":"2012","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong et al., 2016] Tieliang Gong, Qian Zhao, Deyu Meng, and Zongben Xu. Why curriculum learning & self-paced learning work in big/noisy data: A theoretical perspective. Big Data and Information Analytics, 1(1):111-127, 2016.","title":"Why curriculum learning & self-paced learning work in big/noisy data: A theoretical perspective","authors":"Gong","year":"2016","venue":"Big Data and Information Analytics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2014] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2862-2869, 2014.","title":"Weighted nuclear norm minimization with application to image denoising","authors":"Gu","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gunasekar et al., 2017] Suriya Gunasekar, Blake E Wood- worth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factoriza- tion. NIPS, 30:6151-6159, 2017.","title":"Implicit regularization in matrix factorization","authors":"Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, Nati Srebro","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2013] Guibing Guo, Jie Zhang, and Neil Yorke- Smith. A novel bayesian similarity measure for recom- mender systems. In IJCAI, pages 2619-2625, 2013.","title":"A novel bayesian similarity measure for recommender systems","authors":"Guo","year":"2013","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2013/386"},{"position":11,"raw":"[Hardt et al., 2016] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochas- tic gradient descent. In ICML, volume 48, pages 1225- 1234. PMLR, 2016.","title":"Train faster, generalize better: Stability of stochastic gradient descent","authors":"Moritz Hardt, Ben Recht, Yoram Singer","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Harper and Konstan, 2016] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: His- tory and context. ACM Trans. Interact. Intell. Syst., 5(4):19:1-19:19, 2016.","title":"The movielens datasets: History and context","authors":"F, Maxwell Harper, Joseph A Konstan","year":"2016","venue":"ACM Trans. Interact. Intell. Syst","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2012] Yao Hu, Debing Zhang, Jieping Ye, Xue- long Li, and Xiaofei He. Fast and accurate matrix com- pletion via truncated nuclear norm regularization. IEEE transactions on pattern analysis and machine intelligence, 35(9):2117-2130, 2012.","title":"Fast and accurate matrix completion via truncated nuclear norm regularization","authors":"Yao Hu, Debing Zhang, Jieping Ye, Xuelong Li, Xiaofei He","year":"2012","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang et al., 2014] Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G Hauptmann. Easy samples first: Self- paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia, pages 547-556, 2014.","title":"Easy samples first: Selfpaced reranking for zero-example multimedia search","authors":"Jiang","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2015] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning. In AAAI, volume 29, pages 2694- 2700, 2015.","title":"Self-paced curriculum learning","authors":"Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander Hauptmann","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2015] Eunwoo Kim, Minsik Lee, and Songhwai Oh. Elastic-net regularization of singular values for robust subspace learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 915- 923, 2015.","title":"Elastic-net regularization of singular values for robust subspace learning","authors":"Kim","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krizhevsky et al., 2017] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2017","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kumar et al., 2010] M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. NIPS, 23:1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"M Pawan Kumar, Benjamin Packer, Daphne Koller","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In ICLR. OpenReview.net, 2021.","title":"Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning","authors":"Zhiyuan Li, Yuping Luo, Kaifeng Lyu","year":"2021","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2023] Jiangyuan Li, Thanh V. Nguyen, Chinmay Hegde, and Raymond K. W. Wong. Implicit regularization for group sparsity. In ICLR, 2023.","title":"Implicit regularization for group sparsity","authors":"Jiangyuan Li, Thanh V Nguyen, Chinmay Hegde, Raymond K W Wong","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Meng et al., 2017] Deyu Meng, Qian Zhao, and Lu Jiang. A theoretical understanding of self-paced learning. Informa- tion Sciences, 414:319-328, 2017.","title":"A theoretical understanding of self-paced learning","authors":"Deyu Meng, Qian Zhao, Lu Jiang","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Neyshabur et al., 2015] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In ICLR, 2015.","title":"In search of the real inductive bias: On the role of implicit regularization in deep learning","authors":"Ryota Behnam Neyshabur, Nathan Tomioka, Srebro","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Neyshabur et al., 2017] Behnam Neyshabur, Srinadh Bho- janapalli, David McAllester, and Nati Srebro. Explor- ing generalization in deep learning. NIPS, 30:5947-5956, 2017.","title":"Exploring generalization in deep learning","authors":"Srinadh Behnam Neyshabur, David Bhojanapalli, Nati Mcallester, Srebro","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nie et al., 2012] Feiping Nie, Heng Huang, and Chris H. Q. Ding. Low-rank matrix recovery via efficient schatten p- norm minimization. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada, pages 655-661. AAAI Press, 2012.","title":"Low-rank matrix recovery via efficient schatten pnorm minimization","authors":"Feiping Nie, Heng Huang, Chris H Q Ding","year":"2012","venue":"Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Orvieto et al., 2022] Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, and Aurelien Lucchi. Anti- correlated noise injection for improved generalization. In ICML, volume 162, pages 17094-17116. PMLR, 2022.","title":"Anticorrelated noise injection for improved generalization","authors":"Orvieto","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Peng et al., 2015] Chong Peng, Zhao Kang, Huiqing Li, and Qiang Cheng. Subspace clustering using log-determinant rank approximation. In Proceedings of the 21th ACM SIGKDD international conference on Knowledge Discov- ery and Data Mining, pages 925-934, 2015.","title":"Subspace clustering using log-determinant rank approximation","authors":"Chong Peng, Zhao Kang, Huiqing Li, Qiang Cheng","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD international conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Raghu et al., 2017] Maithra Raghu, Ben Poole, Jon Klein- berg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In ICLR, pages 2847-2854. PMLR, 2017.","title":"On the expressive power of deep neural networks","authors":"Raghu","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Razin and Cohen, 2020] Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be ex- plainable by norms. NIPS, 33:21174-21187, 2020.","title":"Implicit regularization in deep learning may not be explainable by norms","authors":"Razin, Cohen, Noam Razin, Nadav Cohen","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Smith et al., 2021] Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De. On the origin of implicit reg- ularization in stochastic gradient descent. In ICLR. Open- Review.net, 2021.","title":"On the origin of implicit regularization in stochastic gradient descent","authors":"L Samuel, Smith, Benoit Dherin, G T David, Soham Barrett, De","year":"2021","venue":"ICLR. Open-Review.net","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun et al., 2017] Ying Sun, Prabhu Babu, and Daniel P. Palomar. Majorization-minimization algorithms in signal processing, communications, and machine learning. IEEE Trans. Signal Process., 65(3):794-816, 2017.","title":"Majorization-minimization algorithms in signal processing, communications, and machine learning","authors":"Ying Sun, Prabhu Babu, Daniel P Palomar","year":"2017","venue":"IEEE Trans. Signal Process","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vardi, 2023] Gal Vardi. On the implicit bias in deep- learning algorithms. Communications of the ACM, 66(6):86-93, 2023.","title":"On the implicit bias in deeplearning algorithms","authors":"Gal Vardi, Vardi","year":"2023","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2015] Zheng Wang, Ming-Jun Lai, Zhaosong Lu, Wei Fan, Hasan Davulcu, and Jieping Ye. Orthogo- nal rank-one matrix pursuit for low rank matrix comple- tion. SIAM Journal on Scientific Computing, 37(1):A488- A514, 2015.","title":"Orthogonal rank-one matrix pursuit for low rank matrix completion","authors":"Zheng Wang, Ming-Jun Lai, Zhaosong Lu, Wei Fan, Hasan Davulcu, Jieping Ye","year":"2015","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2022] Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, Zhi-Ming Ma, and Tie-Yan Liu. Does momentum change the implicit regularization on separable data? NIPS, 35:26764-26776, 2022.","title":"Does momentum change the implicit regularization on separable data?","authors":"Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, Zhi-Ming Ma, Tie-Yan Liu","year":"2022","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Williams et al., 2019] Francis Williams, Matthew Trager, Daniele Panozzo, Claudio Silva, Denis Zorin, and Joan Bruna. Gradient dynamics of shallow univariate relu net- works. NIPS, 32:8376-8385, 2019.","title":"Gradient dynamics of shallow univariate relu networks","authors":"Francis Williams, Matthew Trager, Daniele Panozzo, Claudio Silva, Denis Zorin, Joan Bruna","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu and Su, 2023] Lei Wu and Weijie J. Su. The implicit regularization of dynamical stability in stochastic gradient descent. In ICML, volume 202, pages 37656-37684, 2023.","title":"The implicit regularization of dynamical stability in stochastic gradient descent","authors":"Lei Wu, J Weijie, Su","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2022] Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis. NIPS, 35:4680-4693, 2022.","title":"The alignment property of sgd noise and how it helps select flat minima: A stability analysis","authors":"Lei Wu, Mingze Wang, Weijie Su","year":"2022","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yan et al., 2013] Ming Yan, Yi Yang, and Stanley Osher. Exact low-rank matrix completion from sparsely corrupted entries via adaptive outlier pursuit. Journal of Scientific Computing, 56:433-449, 2013.","title":"Exact low-rank matrix completion from sparsely corrupted entries via adaptive outlier pursuit","authors":"Ming Yan, Yi Yang, Stanley Osher","year":"2013","venue":"Journal of Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yang et al., 2018] Shuyuan Yang, Zhixi Feng, Min Wang, and Kai Zhang. Self-paced learning-based probability subspace projection for hyperspectral image classification. IEEE transactions on neural networks and learning sys- tems, 30(2):630-635, 2018.","title":"Self-paced learning-based probability subspace projection for hyperspectral image classification","authors":"Shuyuan Yang, Zhixi Feng, Min Wang, Kai Zhang","year":"2018","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zaremba et al., 2014] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regulariza- tion. arXiv preprint arXiv:1409.2329, 2014.","title":"Recurrent neural network regularization","authors":"Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang et al., 2017] Dingwen Zhang, Le Yang, Deyu Meng, Dong Xu, and Junwei Han. Spftn: A self-paced fine-tuning network for segmenting objects in weakly labelled videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4429-4437, 2017.","title":"Spftn: A self-paced fine-tuning network for segmenting objects in weakly labelled videos","authors":"Zhang","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhang et al., 2019] Dingwen Zhang, Junwei Han, Long Zhao, and Deyu Meng. Leveraging prior-knowledge for weakly supervised object detection under a collabora- tive self-paced curriculum learning framework. IJCV, 127:363-380, 2019.","title":"Leveraging prior-knowledge for weakly supervised object detection under a collaborative self-paced curriculum learning framework","authors":"Zhang","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhang et al., 2021] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.","title":"Understanding deep learning (still) requires rethinking generalization","authors":"Zhang","year":"2021","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhao et al., 2015] Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, and Alexander Hauptmann. Self- paced learning for matrix factorization. In AAAI, vol- ume 29, pages 3196-3202, 2015.","title":"Selfpaced learning for matrix factorization","authors":"Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, Alexander Hauptmann","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhao, 2022] Dan Zhao. Combining explicit and implicit regularization for efficient learning in deep networks. NIPS, 35:3024-3038, 2022.","title":"Combining explicit and implicit regularization for efficient learning in deep networks","authors":"Dan Zhao","year":"2022","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhou et al., 2020] Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. Curriculum learning by dynamic instance hard- ness. NIPS, 33:8602-8613, 2020.","title":"Curriculum learning by dynamic instance hardness","authors":"Tianyi Zhou, Shengjie Wang, Jeffrey Bilmes","year":"2020","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}