{"type":"ScholarlyArticle","title":"Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models","authors":[{"name":"Shi-Yu Xia","slug":"shi-yu-xia","given":"Shi-Yu","family":"Xia","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing 210096 , China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast"},{"name":"Wenxuan Zhu","slug":"wenxuan-zhu","given":"Wenxuan","family":"Zhu","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing 210096 , China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast"},{"name":"Xu Yang","slug":"xu-yang","given":"Xu","family":"Yang","affiliation":"University) , Ministry of Education , China"},{"name":"Xin Geng","slug":"xin-geng","given":"Xin","family":"Geng","affiliation":"University) , Ministry of Education , China"}],"abstract":"In practice, we usually need to build variable-sized models adapting for diverse resource constraints in different application scenarios, where weight initialization is an important step prior to training. The Learngene framework, introduced recently, firstly learns one compact part termed as learngene from a large well-trained model, after which learngene is expanded to initialize variable-sized models. In this paper, we start from analysing the importance of guidance for the expansion of well-trained learngene layers, inspiring the design of a simple but highly effective Learngene approach termed SWS (Stage-wise Weight Sharing), where both learngene layers and their learning process critically contribute to providing knowledge and guidance for initializing models at varying scales. Specifically, to learn learngene layers, we build an auxiliary model comprising multiple stages where the layer weights in each stage are shared, after which we train it through distillation. Subsequently, we expand these learngene layers containing stage information at their corresponding stage to initialize models of variable depths. Extensive experiments on ImageNet-1K demonstrate that SWS achieves consistent better performance compared to many models trained from scratch, while reducing around 6.6× total training costs. In some cases, SWS performs better only after 1 epoch tuning. When initializing variable-sized models adapting for different resource constraints, SWS achieves better results while reducing around 20× parameters stored to initialize these models and around 10× pre-training costs, in contrast to the pre-training and fine-tuning approach.","doi":"10.24963/ijcai.2024/581","url":"https://ijcai.org/proceedings/2024/581","pdf":"https://ijcai.org/proceedings/2024/0581.pdf","markdown":"https://ijcai.org/proceedings/2024/581.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/581","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":581,"pages":{"first":5254,"last":5262},"keywords":["Machine Learning: ML: Classification","Machine Learning: ML: Deep learning architectures"],"videos":[],"references":[{"position":1,"raw":"[Arpit et al., 2019] Devansh Arpit, Víctor Campos, and Yoshua Bengio. How to initialize your network? robust ini- tialization for weightnorm & resnets. Advances in Neural Information Processing Systems, 32, 2019.","title":"How to initialize your network? robust initialization for weightnorm & resnets","authors":"Devansh Arpit, Víctor Campos, Yoshua Bengio","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2019] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.","title":"Deep equilibrium models","authors":"Bai","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bao et al., 2022] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022.","title":"Beit: Bert pre-training of image transformers","authors":"Hangbo Bao, Li Dong, Songhao Piao, Furu Wei","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bossard et al., 2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative com- ponents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.","title":"Food-101-mining discriminative components with random forests","authors":"Lukas Bossard, Matthieu Guillaumin, Luc Van Gool","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dabre and Fujita, 2019] Raj Dabre and Atsushi Fujita. Re- current stacking of layers for compact neural machine trans- lation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6292-6299, 2019.","title":"Recurrent stacking of layers for compact neural machine translation models","authors":"Fujita Dabre, Raj Dabre, Atsushi Fujita","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[DING et al., 2023] Yahao DING, Mohammad SHIKH- BAHAEI, Zhaohui YANG, Chongwen HUANG, and Weijie YUAN. Secure federated learning over wireless communi- cation networks with model compression. ZTE Communi- cations, 21(1):46-54, 2023.","title":"Secure federated learning over wireless communication networks with model compression","authors":"Ding","year":"2023","venue":"ZTE Communications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feng et al., 2024] Fu Feng, Jing Wang, and Xin Geng. Trans- ferring core knowledge via learngenes. arXiv preprint arXiv:2401.08139, 2024.","title":"Transferring core knowledge via learngenes","authors":"Jing Fu Feng, Xin Wang, Geng","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth interna- tional conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceed- ings, 2010.","title":"Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot, Xavier Glorot, Yoshua Bengio","year":"2010","venue":"Proceedings of the thirteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Graham et al., 2021] Benjamin Graham, Alaaeldin El- Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12259-12269, 2021.","title":"Levit: a vision transformer in convnet's clothing for faster inference","authors":"Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, Matthijs Douze","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2020] Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer opti- mization through better initialization. In International Con- ference on Machine Learning, pages 4475-4483. PMLR, 2020.","title":"Improving transformer optimization through better initialization","authors":"Xiao Shi Huang, Felipe Perez, Jimmy Ba, Maksims Volkovs","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ji et al., 2023] Zhong Ji, Jingwei Ni, Xiyao Liu, and Yanwei Pang. Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning. Frontiers of Computer Science, 17(2):172312, 2023.","title":"Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning","authors":"Zhong Ji, Jingwei Ni, Xiyao Liu, Yanwei Pang","year":"2023","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kovaleva et al., 2019] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. EMNLP, 2019.","title":"Revealing the dark secrets of bert","authors":"Olga Kovaleva, Alexey Romanov, Anna Rogers, Anna Rumshisky","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krause et al., 2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE inter- national conference on computer vision workshops, pages 554-561, 2013.","title":"3d object representations for finegrained categorization","authors":"Jonathan Krause, Michael Stark, Jia Deng, Li Fei-Fei","year":"2013","venue":"Proceedings of the IEEE international conference on computer vision workshops","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technique Report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lan et al., 2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. ICLR, 2020.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[LeCun et al., 2002] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.","title":"Efficient backprop","authors":"Yann Lecun, Léon Bottou, Genevieve B Orr, Klaus-Robert Müller","year":"2002","venue":"Neural networks: Tricks of the trade","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2024] Lei Li, Chengyu Wang, Minghui Qiu, Cen Chen, Ming Gao, and Aoying Zhou. Accelerating bert inference with gpu-efficient exit prediction. Frontiers of Computer Science, 18(3):183308, 2024.","title":"Accelerating bert inference with gpu-efficient exit prediction","authors":"Lei Li, Chengyu Wang, Minghui Qiu, Cen Chen, Ming Gao, Aoying Zhou","year":"2024","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2025] Shaoyuan Li, Yuxiang Zheng, Ye Shi, Shengjun Huang, and Songcan Chen. Kd-crowd: A knowl- edge distillation framework for learning from crowds. Fron- tiers of Computer Science, 19(1):191302, 2025.","title":"Kd-crowd: A knowledge distillation framework for learning from crowds","authors":"Li","year":"2025","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yix- uan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Mishkin and Matas, 2015] Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422, 2015.","title":"All you need is a good init","authors":"Dmytro Mishkin, Jiri Matas","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Oquab et al., 2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.","title":"Dinov2: Learning robust visual features without supervision","authors":"Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Proceedings of the 33th Annual Conference on Neural Information Processing Systems, pages 8024-8035, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Proceedings of the 33th Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ren et al., 2023] Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3687-3697, 2023.","title":"Tinymim: An empirical study of distilling mim pre-trained models","authors":"Ren","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Samragh et al., 2023] Mohammad Samragh, Mehrdad Fara- jtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Raste- gari. Weight subcloning: direct initialization of trans- formers using larger pretrained ones. arXiv preprint arXiv:2312.09299, 2023.","title":"Weight subcloning: direct initialization of transformers using larger pretrained ones","authors":"Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, Mohammad Rastegari","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Shi et al., 2024] Boyu Shi, Shiyu Xia, Xu Yang, Haokun Chen, Zhiqiang Kou, and Xin Geng. Building variable- sized models via learngene pool. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14946-14954, 2024.","title":"Building variablesized models via learngene pool","authors":"Boyu Shi, Shiyu Xia, Xu Yang, Haokun Chen, Zhiqiang Kou, Xin Geng","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Takase and Kiyono, 2021] Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. arXiv preprint arXiv:2104.06022, 2021.","title":"Lessons on parameter sharing across layers in transformers","authors":"Kiyono ; ] Takase, Takase Sho, Shun Kiyono","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image trans- formers & distillation through attention. In International conference on machine learning, pages 10347-10357.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2022a] Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi-Yu Xia, Lei Qi, and Ning Xu. Learngene: From open-world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8557-8565, 2022.","title":"Learngene: From open-world to your learning task","authors":"Wang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2022b] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggar- wal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022.","title":"Image as a foreign language: Beit pretraining for all vision and vision-language tasks","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2023] Qiufeng Wang, Xu Yang, Shuxia Lin, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint arXiv:2305.02279, 2023.","title":"Learngene: Inheriting condensed knowledge from the ancestry model to descendant models","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Xia et al., 2024] Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, and Xin Geng. Transformer as linear expansion of learngene. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16014-16022, 2024.","title":"Transformer as linear expansion of learngene","authors":"Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, Xin Geng","year":"2024","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Xu et al., 2023] Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. arXiv preprint arXiv:2311.18823, 2023.","title":"Initializing models with larger ones","authors":"Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, Zhuang Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yan et al., 2023] Jintao Yan, Tan Chen, Bowen Xie, Yuxuan Sun, Sheng Zhou, and Zhisheng Niu. Hierarchical federated learning: Architecture, challenges, and its implementation in vehicular networks. ZTE Communications, 21(1):38-45, 2023.","title":"Hierarchical federated learning: Architecture, challenges, and its implementation in vehicular networks","authors":"Jintao Yan, Tan Chen, Bowen Xie, Yuxuan Sun, Sheng Zhou, Zhisheng Niu","year":"2023","venue":"ZTE Communications","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2022] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Minivit: Compressing vision transformers with weight mul- tiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12145- 12154, 2022.","title":"Minivit: Compressing vision transformers with weight multiplexing","authors":"Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, Lu Yuan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}