{"type":"ScholarlyArticle","title":"Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers","authors":[{"name":"Hongyi Pan","slug":"hongyi-pan","given":"Hongyi","family":"Pan","affiliation":"Machine and Hybrid Intelligence Lab , Northwestern University , Chicago , USA"},{"name":"Emadeldeen Hamdan","slug":"emadeldeen-hamdan","given":"Emadeldeen","family":"Hamdan","affiliation":"Department of Electrical and Computer Engineering , University of Illinois Chicago , Chicago , USA"},{"name":"Xin Zhu","slug":"xin-zhu","given":"Xin","family":"Zhu","affiliation":"Department of Electrical and Computer Engineering , University of Illinois Chicago , Chicago , USA"},{"name":"Ahmet Enis Cetin","slug":"ahmet-enis-cetin","given":"Ahmet Enis","family":"Cetin","affiliation":"Department of Electrical and Computer Engineering , University of Illinois Chicago , Chicago , USA"},{"name":"Ulas Bagci","slug":"ulas-bagci","given":"Ulas","family":"Bagci","affiliation":"Machine and Hybrid Intelligence Lab , Northwestern University , Chicago , USA"}],"abstract":"Self-attention is central to the success of Transformer architectures; however, learning the query, key, and value projections from random initialization remains challenging and computationally expensive. In this paper, we propose two complementary methods that leverage the Discrete Cosine Transform (DCT) to enhance the efficiency and performance of Vision Transformers. First, we address the initialization problem by introducing a simple yet effective DCT-based initialization strategy for self-attention, where projection weights are initialized using DCT coefficients. This structure-preserving approach consistently improves classification accuracy on the CIFAR-10 and ImageNet-1K benchmarks. Second, we propose a DCT-based attention compression technique that exploits the decorrelation properties of the frequency domain. By observing that high-frequency DCT coefficients typically correspond to noise, we truncate high-frequency components of the input patches, thereby reducing the dimensionality of the query, key, and value projections without sacrificing accuracy. Experiments on Swin Transformer models demonstrate that the proposed compression method achieves a substantial reduction in computational overhead while maintaining comparable performance. Code: https://github.com/NUBagciLab/DCT-Transformer.","doi":"10.24963/ijcai.2026/522","url":"https://ijcai.org/proceedings/2026/522","pdf":"https://ijcai.org/proceedings/2026/0522.pdf","markdown":"https://ijcai.org/proceedings/2026/522.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/522","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Machine Learning","track":"Main Track","number":522,"pages":{"first":4687,"last":4695},"keywords":["Machine Learning: Deep learning architectures","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"Swin-T 28.29M/4.49G 81.474%/95.776% PoolFormer-S24 21M/3.5G 80.3%/- DCFormer-SW-T(τ =1) 28.29M/4.5G 81.2%/- DCFormer-SW-T(τ =2) 28.29M/1.3G 79.2%/- gSwin-T 22/3.6 81.715%/- Swin-T-DCT-Q 28.29M/4.49G 81.568%/95.862% Swin-T-DCT-K 28.29M/4.49G 81.728%/95.862% Swin-T-DCT-V 28.29M/4.49G 81.528%/95.850% Swin-T-DCT-0.25 22.21M/3.24G 79.580%/94.936% Swin-T-DCT-0.5 23.43M/3.64G 80.762%/95.520% Swin-T-DCT-0.75 25.45M/4.18G 81.474%/95.728% Swin-S 49.61M/8.74G 83.196%/96.360% PoolFormer-M36 56M/8.8G 82.1%/- DCFormer-SW-S(τ =1) 49.61M/8.7G 82.8%/- DCFormer-SW-S(τ =2) 49.61M/2.7G 80.9%/- gSwin-S 40M/7.0G 83.014%/- Swin-S-DCT-Q 49.61M/8.74G 83.298%/96.298% Swin-S-DCT-K 49.61M/8.74G 83.378%/ 96.398% Swin-S-DCT-V 49.61M/8.74G 83.200%/96.364% Swin-S-DCT-0.25 38.54M/6.28G 81.840%/95.704% Swin-S-DCT-0.5 40.76M/7.07G 82.622%/96.036% Swin-S-DCT-0.75 44.45M/8.13G 83.202%/96.388% ViT-B-32 88.22M/4.41G 73.120%/90.306% ViT-B-32-DCT-Q 88.22M/4.41G 73.278%/90.574% ViT-B-32-DCT-K 88.22M/4.41G 73.614%/90.870% ViT-B-32-DCT-V 88.22M/4.41G 73.306%/90.502% ViT-B-32-DCT-0.5 66.97M/3.51G 72.948%/90.390% ViT-B-32-DCT-0.75 75.83M/4.05G 73.226%/90.642% Table 3: ImageNet-1K validation dataset evaluation. as standard attention; however, this overhead can be further butterfly-based DCT implementations.","title":"Swin-T","authors":null,"year":null,"venue":"Swin-T","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahmed et al., 1974] Nasir Ahmed, T Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE trans- actions on Computers, 100(1):90-93, 1974.","title":"Discrete cosine transform","authors":"Nasir Ahmed, T Natarajan, Kamisetty R Rao","year":"1974","venue":"IEEE transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akansu and Torun, 2012] Ali N Akansu and Mustafa U Torun. Toeplitz approximation to empirical correlation matrix of asset returns: A signal processing perspective. IEEE Journal of Selected Topics in Signal Processing, 6(4):319-326, 2012.","title":"Toeplitz approximation to empirical correlation matrix of asset returns: A signal processing perspective","authors":"N Ali, Mustafa U Akansu, Torun","year":"2012","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bolya et al., 2022] Daniel Bolya, Cheng-Yang Fu, Xiao- liang Dai, Peizhao Zhang, and Judy Hoffman. Hydra at- tention: Efficient attention with many heads. In European Conference on Computer Vision, pages 35-49. Springer, 2022.","title":"Hydra attention: Efficient attention with many heads","authors":"Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Judy Hoffman","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bozinovski, 2020] Stevo Bozinovski. Reminder of the first paper on transfer learning in neural networks, 1976. Infor- matica, 44(3), 2020.","title":"Reminder of the first paper on transfer learning in neural networks","authors":"Stevo Bozinovski, Bozinovski","year":"1976","venue":"Informatica","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Buchholz and Jug, 2022] Tim-Oliver Buchholz and Florian Jug. Fourier image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1846-1854, 2022.","title":"Fourier image transformer","authors":"Jug ; Tim-Oliver Buchholz, Florian Buchholz, Jug","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cai and Vasconcelos, 2018] Zhaowei Cai and Nuno Vas- concelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 6154-6162, 2018.","title":"Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection","authors":"Vasconcelos Cai","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 1977] Wen-Hsiung Chen, CH Smith, and Sam Fralick. A fast computational algorithm for the discrete cosine transform. IEEE Transactions on communications, 25(9):1004-1009, 1977.","title":"A fast computational algorithm for the discrete cosine transform","authors":"Wen-Hsiung Chen, C H Smith, Sam Fralick","year":"1977","venue":"IEEE Transactions on communications","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chi et al., 2020] Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. Advances in Neural Information Pro- cessing Systems, 33:4479-4488, 2020.","title":"Fast fourier convolution","authors":"Lu Chi, Borui Jiang, Yadong Mu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cubuk et al., 2018] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018.","title":"Autoaugment: Learning augmentation policies from data","authors":"Barret Ekin D Cubuk, Dandelion Zoph, Vijay Mane, Quoc V Vasudevan, Le","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dony and others, 2001] R Dony et al. Karhunen-loeve transform. The transform and data compression hand- book, 1(1-34):29, 2001.","title":"Karhunen-loeve transform","authors":"Dony","year":"2001","venue":"The transform and data compression handbook","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ebrahimi and Horne, 2000] Touradj Ebrahimi and Caspar Horne. Mpeg-4 natural video coding-an overview. Sig- nal Processing: Image Communication, 15(4-5):365-385, 2000.","title":"Mpeg-4 natural video coding-an overview","authors":"Horne Ebrahimi","year":"2000","venue":"Signal Processing: Image Communication","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth in- ternational conference on artificial intelligence and statis- tics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proceedings of the thirteenth in-ternational conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Go and Tachibana, 2023] Mocho Go and Hideyuki Tachibana. gswin: Gated mlp vision model with hi- erarchical structure of shifted window. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.","title":"Mocho Go and Hideyuki Tachibana. gswin: Gated mlp vision model with hierarchical structure of shifted window","authors":"Tachibana Go","year":"2023","venue":"ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hu et al., 2019a] Shell Xu Hu, Sergey Zagoruyko, and Nikos Komodakis. Exploring weight symmetry in deep neural networks. Computer Vision and Image Understand- ing, 187:102786, 2019.","title":"Exploring weight symmetry in deep neural networks","authors":"Hu","year":"2019","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hu et al., 2019b] Wei Hu, Lechao Xiao, and Jeffrey Pen- nington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In International Confer- ence on Learning Representations, 2019.","title":"Provable benefit of orthogonal initialization in optimizing deep linear networks","authors":"Hu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ji et al., 2022] Xiaozhong Ji, Boyuan Jiang, Donghao Luo, Guangpin Tao, Wenqing Chu, Zhifeng Xie, Chengjie Wang, and Ying Tai. Colorformer: Image colorization via color memory assisted hybrid-attention transformer. In European Conference on Computer Vision, pages 20-36. Springer, 2022.","title":"Colorformer: Image colorization via color memory assisted hybrid-attention transformer","authors":"Ji","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2023] Xinyu Li, Yanyi Zhang, Jianbo Yuan, Han- lin Lu, and Yibo Zhu. Discrete cosin transformer: Im- age modeling from frequency domain. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 5468-5478, 2023.","title":"Discrete cosin transformer: Image modeling from frequency domain","authors":"Xinyu Li, Yanyi Zhang, Jianbo Yuan, Hanlin Lu, Yibo Zhu","year":"2023","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF in- ternational conference on computer vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Martens and others, 2010] James Martens et al. Deep learn- ing via hessian-free optimization. In ICML, volume 27, pages 735-742, 2010.","title":"Deep learning via hessian-free optimization","authors":"James Martens","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Masood and Chandra, 2012] Sarfaraz Masood and Pravin Chandra. Training neural network with zero weight ini- tialization. In Proceedings of the CUBE International In- formation Technology Conference, pages 235-239, 2012.","title":"Training neural network with zero weight ini-tialization","authors":"Sarfaraz Masood, Pravin Chandra","year":"2012","venue":"Proceedings of the CUBE International Information Technology Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mehta and Rastegari, 2021] Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer. In International Con- ference on Learning Representations, 2021.","title":"Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer","authors":"Sachin Mehta, Mohammad Rastegari","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pan et al., 2023] Hongyi Pan, Xin Zhu, Salih Furkan Atici, and Ahmet Cetin. A hybrid quantum-classical approach based on the hadamard transform for the convolutional layer. In International Conference on Machine Learning, pages 26891-26903. PMLR, 2023.","title":"A hybrid quantum-classical approach based on the hadamard transform for the convolutional layer","authors":"Xin Hongyi Pan, Salih Zhu, Ahmet Furkan Atici, Cetin","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural in- formation processing systems, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Patro et al., 2025] Badri N Patro, Vinay P Namboodiri, and Vijay S Agneeswaran. Spectformer: Frequency and atten- tion is what you need in a vision transformer. In 2025 IEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV), pages 9543-9554. IEEE, 2025.","title":"Spectformer: Frequency and attention is what you need in a vision transformer","authors":"Badri N Patro, P Vinay, Vijay S Namboodiri, Agneeswaran","year":"2025","venue":"IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Radünz et al., 2022] Anabeth P Radünz, Fábio M Bayer, and Renato J Cintra. Low-complexity rounded klt approx- imation for image compression. Journal of Real-Time Im- age Processing, pages 1-11, 2022.","title":"Low-complexity rounded klt approximation for image compression","authors":"Radünz","year":"2022","venue":"Journal of Real-Time Image Processing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Saxe et al., 2013] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.","title":"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks","authors":"James L Andrew M Saxe, Surya Mcclelland, Ganguli","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Scribano et al., 2023] Carmelo Scribano, Giorgia Fran- chini, Marco Prato, and Marko Bertogna. Dct-former: Ef- ficient self-attention with discrete cosine transform. Jour- nal of Scientific Computing, 94(3):67, 2023.","title":"Dct-former: Efficient self-attention with discrete cosine transform","authors":"Scribano","year":"2023","venue":"Journal of Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Setyawan et al., 2025] Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, and Jun-Wei Hsieh. Mi- crovit: a vision transformer with low complexity self atten- tion for edge device. In 2025 IEEE International Sympo- sium on Circuits and Systems (ISCAS), pages 1-5. IEEE, 2025.","title":"Microvit: a vision transformer with low complexity self attention for edge device","authors":"Setyawan","year":"2025","venue":"IEEE International Symposium on Circuits and Systems (ISCAS)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shen et al., 2021] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531-3539, 2021.","title":"Efficient attention: Attention with linear complexities","authors":"Shen","year":"2021","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Testa and Magli, 2016] Matteo Testa and Enrico Magli. Compressive estimation and imaging based on autoregres- sive models. IEEE Transactions on Image Processing, 25(11):5077-5087, 2016.","title":"Compressive estimation and imaging based on autoregressive models","authors":"Magli Testa, Matteo Testa, Enrico Magli","year":"2016","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Trockman and Kolter, 2023] Asher Trockman and J Zico Kolter. Mimetic initialization of self-attention layers. arXiv preprint arXiv:2305.09828, 2023.","title":"Mimetic initialization of self-attention layers","authors":"Kolter ; Trockman, J Asher Trockman, Kolter Zico","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wallace, 1991] Gregory K Wallace. The jpeg still pic- ture compression standard. Communications of the ACM, 34(4):30-44, 1991.","title":"The jpeg still picture compression standard","authors":"Wallace Gregory","year":"1991","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Weiss et al., 2016] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.","title":"A survey of transfer learning","authors":"Karl Weiss, Taghi M Khoshgoftaar, Dingding Wang","year":"2016","venue":"Journal of Big data","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Xu et al., 2023] Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. In The Twelfth International Conference on Learning Represen- tations, 2023.","title":"Initializing models with larger ones","authors":"Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, Zhuang Liu","year":"2023","venue":"The Twelfth International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yu et al., 2022] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819- 10829, 2022.","title":"Metaformer is actually what you need for vision","authors":"Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023- 6032, 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zhang et al., 2018] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, David Yann N Dauphin, Lopez-Paz","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhang et al., 2019] Biao Zhang, Ivan Titov, and Rico Sen- nrich. Improving deep transformer with depth-scaled ini- tialization and merged attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pages 898-909, 2019.","title":"Improving deep transformer with depth-scaled initialization and merged attention","authors":"Biao Zhang, Ivan Titov, Rico Sennrich","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhao et al., 2022] Jiawei Zhao, Florian Tobias Schaefer, and Anima Anandkumar. Zero initialization: Initializing neural networks with only zeros and ones. Transactions on Machine Learning Research, 2022.","title":"Zero initialization: Initializing neural networks with only zeros and ones","authors":"Jiawei Zhao, Florian Tobias Schaefer, Anima Anandkumar","year":"2022","venue":"Transactions on Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zheng et al., 2024] Jianqiao Zheng, Xueqian Li, and Simon Lucey. Structured initialization for attention in vision transformers. arXiv preprint arXiv:2404.01139, 2024.","title":"Structured initialization for attention in vision transformers","authors":"Zheng","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhong et al., 2020] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug- mentation. In Proceedings of the AAAI conference on arti- ficial intelligence, volume 34, pages 13001-13008, 2020.","title":"Random erasing data augmentation","authors":"Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, Yi Yang","year":"2020","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}