Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers

1 Introduction

In the last decade, convolutional neural networks (CNNs) have dominated image-processing tasks. However, inspired by the success of transformer architectures in natural language processing, Vision Transformers (ViTs) have redefined the norms of visual data interpretation in recent years. Selfattention mechanism, a key component of transformers, is pivotal in reshaping how models perceive and analyze visual information [Vaswani et al., 2017].

Unlike CNNs, ViTs view an image as a sequence of informative patches, adopting a more holistic approach. By treating image patches like "visual words", ViTs can leverage powerful transformer architectures, originally designed for natural language processing, to learn global context and long-range dependencies. This novel approach has led to remarkable performance gains on a wide range of computer vision tasks, demonstrating the potential of ViTs to revolutionize the field.

The initialization of self-attention weights is a critical step significantly impacting the training dynamics and performance of the Transformers. The core of Transformers is the self-attention mechanism, which includes three weight matrices: queries (WQ), keys (WK), and values (WV ). Proper initialization of these matrices ensures that the model starts from a stable state, facilitating effective learning. The most common approach is to initialize the weights randomly, often using distributions like the normal or uniform distribution. For example, Xavier initialization [Glorot and Bengio, 2010] and He initialization [He et al., 2015] are popular choices. They are designed to keep the scale of the gradients roughly the same in all layers. Although less common, in some cases, biases or certain parts of the weight matrices are initialized to zeros [Masood and Chandra, 2012; Zhao et al., 2022]. This is less common for queries, keys, and values matrices, as it can lead to symmetry-breaking issues during training [Hu et al., 2019a].

Efficiency is an ongoing concern in Transformers. While conventional initialization strategies are often suitable for generic applications, new algorithms are increasingly tailored to the specific characteristics and needs of Transformers. These advancements in initialization techniques contribute to the ongoing improvements in the performance and efficiency of Transformer-based models. Orthogonal initialization [Saxe et al., 2013; Hu et al., 2019b], sparse initialization [Martens and others, 2010], variance scaling [Zhang et al., 2019], pre-trained initialization [Weiss et al., 2016; Bozinovski, 2020; Xu et al., 2023], layer-wise, domainspecific and attention-specific initialization [Trockman and Kolter, 2023] are some of the new trends in initialization approaches. In this study, we propose a new family of algorithms for initialization strategy in the intersection of attention and domain-specific approaches: Discrete Cosine Transform-Based Decorrelated Attention.

Can DCT in Vision Transformer be a game changer? DCT has been widely used in signal processing and data compression applications. It reduces the dimensionality of the input data while preserving its essential features. Transforming the input data into the frequency domain helps capture the most important features of the input data. In this paper, we propose two methods that apply DCT to the attention for vision transformers. First, we initialize the weight matrices for the queries, keys, and values computation as the DCT matrix. Our DCT-based initialization covers the entire spectrum utilizing the DCT basis vectors, and each weight vector has a different initial bandwidth. As a comparison, to cover the entire spectrum in the classical initialization, all the weight matrices are initialized as random numbers due to their independent generation. These random numbers can be treated as white noise. All the weight initialization starts with the same spectra. Second, we introduce a DCT-based compressed attention for vision transformers. With our compression methodology, compared to the vanilla Swin Transformers, the number of parameters and computational overhead of the Swin Transformers are reduced. Meanwhile, the accuracy of the models is comparable.

2 RelatedWorks

Vision Transformer Transformer architectures have merged as a dominant standard for natural language processing tasks [Vaswani et al., 2017]. Vision Transformer (ViT) [Dosovitskiy et al., 2020] built a bridge between language and vision. It shows that pure transformers can also be applied directly to sequences of image patches for classification tasks. In other words, it partitions images into patches and treats them as word sequences. Then, a shifted windowing scheme was proposed in Swin Transformer [Liu et al., 2021] to obtain greater efficiency. This limits self-attention computation to non-overlapping local windows while allowing for cross-window connection. Later, to overcome the limitations of the heavyweights in ViTs, MobileViT [Mehta and Rastegari, 2021] combined the strengths of MobileNets [Sandler et al., 2018] and ViTs. It is a lightweight and low-latency network for mobile vision tasks. Moreover, gSwin [Go and Tachibana, 2023] combined MLP-based architecture with Swin Transformer and attains superior accuracy with a smaller model size.

Frequency-basedNeural Network Orthogonal transforms such as the Fourier transform [Chi et al., 2020; Buchholz and Jug, 2022; Patro et al., 2025] and Hadamard transform [Pan et al., 2023] have been used to improve the performance of the neural networks. In the orthogonal transform family, DCT is a mathematical technique commonly used in signal processing and image and video compression. Compared to the Fourier transform, which is the standard frequency representation, DCT does not generate complex-valued numbers. Therefore, it is computationally more efficient. DCTFormer [Scribano et al., 2023], for instance, applied DCT compression on the input sequence along the length channel before feeding it into the self-attention layers to reduce the computational overhead in the NLP problems. Furthermore, inspired by JPEG compression, DCFormer [Li et al., 2023] highlighted the visually significant signals by leveraging the input information compression on its frequency domain representation. These two DCT-based works and our DCT-based compression all aim to reduce the computational overhead. Therefore, we mainly compare our DCT-based compression method with them. DCT-Former [Scribano et al., 2023] is designed for the NLP transformers and takes DCT along the width and height of the input tensor patches, while our method is designed for the vision transformers and it compresses the tensor along the channel axis. DCFormer [Li et al., 2023] averages DCT coefficients via average pooling, which does not take advantage of the property that DCT components in low-frequency bands contain more important information. On the contrary, our DCT-based compression method retains the low-frequency components very accurately and discards the unnecessary extremely highfrequency components corresponding to noise. Under the same input image size, compared to the vanilla transformer models, DCFormer reduces the FLOPs and retains the number of parameters while sacrificing accuracy. Our DCT-based compression method, on the other hand, saves both trainable parameters and FLOPs and obtains on-par or better performance in addition to other benefits that our DCT-based compression method has. Furthermore, we introduce a DCTbased initialization and we have not found any related initialization in the literature.

Self-Attention Initialization DS-Init [Zhang et al., 2019] improves convergence in deep Transformers for NLP by reducing parameter variance during initialization and minimizing output variance of residual connections, facilitating better gradient flow. It pairs with a merged attention sublayer (MAtt) that combines average-based self-attention with encoder-decoder attention to reduce the computational cost. Additionally, Mimetic initialization [Trockman and Kolter, 2023] is a compute-free method that initializes the weights of self-attention layers by simulating the patterns found in pre-trained weights, resulting in improved Transformer performance. Moreover, weight selection [Xu et al., 2023] enables the initialization of smaller models by selecting a subset of weights from a larger pre-trained model, facilitating efficient knowledge transfer and faster training. This method enhances small model performance, reduces training time, and integrates with knowledge distillation, making it particularly valuable for resource-constrained environments. Furthermore, structured initialization [Zheng et al., 2024] tackles the challenge of training vision transformers on small-scale datasets by introducing a structured initialization strategy inspired by the architectural biases of Convolutional Neural Networks (CNNs). While these methods focus on mimicking pre-trained patterns, selecting weights from larger models, or using CNN-inspired biases, our DCT-based initialization utilizes well-established DCT matrices to initialize attention weights.

Efficient Self-Attention Avariety of approaches have been proposed to reduce the memory and computational costs of dot-product attention, including linear-complexity attention mechanisms [Shen et al., 2021], increasing attention heads as in Hydra Attention [Bolya et al., 2022], and architectural rethinking exemplified by MetaFormer [Yu et al., 2022], which shows that overall architecture often matters more than the specific token mixer. MicroViT [Setyawan et al., 2025] further targets resource-constrained settings by introducing an Efficient Single Head Attention mechanism that reduces selfattention cost through group convolution and partial channel processing. Despite these advances, existing methods mainly pursue efficiency through linearization, head scaling, or architectural design, while largely ignoring frequency-domain transformations. We propose DCT-based methods to enhance both efficiency and expressiveness in vision transformers.

3 Methodology

3.1 Background: Multi-Head Self-Attention

To handle a 2D image tensor I ∈ RH× W×C , the multihead self-attention (MSA) module first splits it into patches X ∈ RN×M2×C , where H × W are the resolution of the image and C is the number of channels, M ×M is the resolution of each image patch and N = WH M2 is the resulting number of patches. After that, three linear layers (LQ, LK, LV ) are applied on X to compute the queries, keys, and values Q, K, V ∈ RN×M2×C :

where WQ,WK,WV ∈ RC×C and bQ, bK, bV ∈ R1×C are trainable weights and biases. Then, to perform P-head MSA, Q,K,V are split into P slices as Qi,Ki,Vi ∈ RN×M2×d for i = 0, . . . , P - 1, d = CP . The scaled dotproduct attention Ai ∈ RN×M2×d is computed as:

where σ(·) is the Softmax function. Bi ∈ RM2×M2 is the relative position bias. Since the relative position along each axis lies in [-M+1,M-1], values in Bi are taken from a parameterized smaller-sized bias matrix ˆBi ∈ R(2M-1)×(2M-1) . After that, {Ai} are concatenated to A ∈ RN×M2×C to feed into a linear layer LO to obtain the attention output Y ∈ RN×M2×C :

Finally, Y is reversed into the image shape J ∈ RH×W×C .

3.2 DCT-based Initialization for Attention

In the attention layer, the first three linear layers (LQ, LK, and LV ) extract different features of the input patches X. However, their weights WQ, WK, and WV are trained simultaneously. Such training from scratch is difficult, especially when the dataset is insufficient and small-scale. On the other hand, DCT is a well-designed feature extractor. In our proposed DCT initialization strategy, one of WQ, WK, WV ∈ RC×C is initialized using the orthogonal type-II DCT matrix D [Ahmed et al., 1974]:

D can be obtained from applying DCT on an identity matrix IC×C because DCT is a linear transform. The remaining weight matrices and other trainable parameters are still initialized regularly. We don't initialize multiple weight matrices to avoid the weight symmetry problem [Hu et al., 2019a] caused by initializing the multiple weights as the same values, and our ablation study has verified this point.

The DCT-based initialization can be treated as applying DCT instead of linear projection on X to obtain Q, K, or V. Some DCT coefficients capture low-frequency information, while others capture high-frequency details. Therefore, the DCT matrix is a good starting point for the weight matrices.

3.3 DCT-Based Compressed Attention

Data compression using DCT is common in signal, image, and video processing for energy concentration, compression efficiency, and noise reduction. Usually, higher-frequency DCT coefficients contribute less to perceptual quality and contain more noise. Preserving essential information in lower frequencies while discarding less critical details in higher frequencies results in efficient signal representation and storage. The specific truncation strategy depends on the application's requirements and the acceptable level of information loss.

We observed that the attention layer matrices are correlated with each other, so that we can compress them using DCT. Fig. 1(b) shows the method of DCT-based compression for the attention layer. First, the input patches X ∈ RN×M2×C are encoded by DCT along the channel of C with the highfrequency components truncated. If we keep the first τC coefficients (τ < 1), the essential information of X is still preserved, while the input and output dimensions of the linear layers (LQ,LK, and LV ) are reduced from RN×M2×C to RN×M2×τC . In this way, the computational cost from LQ,LK, and LV is reduced significantly. In our experiments,

τ is chosen from 25%, 50%, and 75%. After obtaining the attention A ∈ RN×M2×τC from the queries Q, keys K, and values V, we apply zero-padding along the last channel to make the dimension back to RN×M2×C . Before being fed into the linear layer LO, these padded patches are decoded by the inverse DCT (IDCT):

Because D is orthogonal, i.e. DDT = IC×C, the IDCT matrix can be obtained directly from the transpose of the DCT matrix. Although DCT and IDCT can be implemented in a fast manner with the complexity of O(C log2 C) using the butterfly algorithm [Chen et al., 1977], such an implementation is not officially supported in PyTorch currently. Therefore, in this work, we implement the DCT and IDCT via matrix multiplication between the input tensor and the truncated DCT and IDCT matrices ¯D ∈ RτC×C and ¯D-1 ∈ RC×τC . These transform matrices are obtained by truncating the DCT and IDCT matrices along rows or columns to keep τ coefficients or zero padding, respectively. Then, DCT with truncation or IDCT with zero padding is implemented as one matrix multiplication to reduce memory pressure. Furthermore, as no non-linearity exists between IDCT and the final linear layer LO, these two steps can be implemented as a single linear layer whose weight matrix is WO ¯D-1 and the bias is still bO. However, such a combination is not applied on DCT and WQ, WK, or WV . This is because it may not always reduce the computational cost for them: To compute Q, K, and V, with such a combination, 3kNM 2C 3 multiplications are needed. In contrast, if they are computed separately, there are τNM 2C 3 multiplications from DCT and τ 2NM 2C 3 multiplications from each linear layer. As a result, there are a total of (τ + 3τ 2 )NM 2C 3 multiplications. Hence, combing DCT with WQ, WK and WV brings computational cost saving only when τ > 2 3 . To maintain the consistency of the framework, we do not combine DCT with WQ, WK, or WV .

Computational cost comparison of the vanilla versus the proposed DCT-compressed attentions is presented in Table 1. We evaluate both naive and simplified implementations of the DCT-compressed attention with the vanilla attention. The simplification is implemented by fusing IDCT into the linear layer LO. Computational overhead from the Softmax function is omitted because the Softmax function is based on exponential terms. Despite this, the Softmax function in the DCT-compressed attention requires less computational cost than in the vanilla attention as the number of entries in ( QKT √d +B) in the DCT-compressed attention is τ 2 times (τ < 1) as in the vanilla attention.

3.4 Frequency DomainMotivation of the DCT-based Initialization

In traditional random initialization, all the weight matrices are initialized as random numbers that are independently generated. As a result, the initial matrices can be considered as a realization of white noise covering the entire spectrum. A good initialization should take advantage of the full bandwidth to cover the entire spectrum of the input. The DCTbased initialization covers the entire frequency domain: The set of DCT basis vectors { √2 2 , cos (2n+1)kπ 2C } is the class of discrete Chebyshev polynomials:

for n = 0, 1, . . . , C - 1. The basis member cos (2n+1)kπ 2nC is the k-th Chebyshev polynomial Tk[ξ] evaluated at the n-th zero of Tn[ξ] [Ahmed et al., 1974]. Fig. 2 draws the basis vectors in Eq. (6) for an 8×8 DCT matrix and the magnitude of their discrete Fourier transform (F{·}):

The entire spectrum is covered as √∑C-1 l=0 | ˆ Tl[k]|2 has perfect full-band coverage, and each weight vector in the DCTbased initialization covers a different bandwidth. On the other hand, in the traditional random initialization, all of the weights start with the same spectra. This is why our DCTbased initialization is superior to the traditional random initialization for the attention layer.

3.5 Motivation of the DCT-based Compression

The mechanism behind the DCT-based compression mimics the Karhunen-Loeve transform (KLT), which is also known as principal component analysis (PCA), a mathematical technique used for dimensionality reduction in signal processing [Ji et al., 2022]. Neighboring image patches are highly correlated with each other. During training, the attention matrices are inter-correlated because they are obtained using similar gradient vectors. Therefore, they can be decorrelated and compressed using DCT. When the input data is highly correlated, the covariance matrix can be approximated by a Teoplitz matrix ϕ of [1, ρ, ρ2 , · · · , ρC-1 ] with ρ < 1 close to 1 [Akansu and Torun, 2012]. This is the covariance matrix of an autoregressive model random process AR(1) with the correlation coefficient ρ [Testa and Magli, 2016]. Fig. 2 shows that DCT basis vectors approximate the KLT of a first-order Markov process with high correlation among the neighboring samples [Dony and others, 2001; Rad¨unz et al., 2022]. We observe that Q, K, and V have parameters with high correlations among themselves. We can directly use DCT in data compression since its matrix approximates the corresponding KLT matrix [Ahmed et al.,

0 1 2 3 4 5 6 7 8

0 1 2 3 4 5 6 7 8

0 1 2 3 4 5 6 7 8

0 1 2 3 4 5 6 7 8

Figure 2: Basis vectors of an 8 × 8 DCT matrix and their frequency response. The DCT basis vectors provide a good approximation to the eigenvectors of the Toeplitz matrix with ρ = 0.9 (KLT) and cover the entire frequency spectrum.

1974]. Although KLT can also be utilized in data compression, it requires massive computation for the covariance matrices and the corresponding eigenvectors of X. On the other hand, DCT is very successful in decorrelating the data without computing the covariance matrix and eigenvectors. As a consequence, it is also used in JPEG image [Wallace, 1991] and MPEG video [Ebrahimi and Horne, 2000] compression.

4 Experimental Results

We conduct experiments on the CIFAR-10 and ImageNet-1K classification tasks and COCO 2017 detection and segmentation tasks. Swin Transformers [Liu et al., 2021] were employed as the backbone network. Models were trained on a server with 8 NVIDIA A100 80GB GPUs using Python with the PyTorch framework.

4.1 Implementation Details

Image Classification Training models for the image classification tasks from scratch mostly followed the settings in [Liu et al., 2021]. We employed AdamW optimizer for 300 epochs using a cosine decay learning rate scheduler and 20 epochs of linear warm-up. We set the batch size as 1024, the initial learning rate as 0.001, and the weight decay as 0.05. Data augmentation was performed as follows: The CIFAR10 augmentation followed [He et al., 2016]. First, the training images were padded with 4 pixels. Then, they were randomly cropped to get 32 by 32. Finally, the images were randomly flipped horizontally. The images were normalized with the means of [0.4914, 0.4822, 0.4465] and the standard deviations of [0.2023, 0.1994, 02010]. The ImageNet-1K augmentation followed PyTorch Torchvision's GitHub repository [Paszke et al., 2019]. As a start, images were resized to make the smaller edge of the image 232 for Swin-T and 246 for Swin-S. Next, they were cropped to 224 by 224 images and randomly flipped horizontally. Techniques such as Autoaugment [Cubuk et al., 2018], mixup [Zhang et al., 2018], cutmix [Yun et al., 2019], and random erasing [Zhong et al., 2020] were applied. Images were normalized with the means of [0.485, 0.456, 0.406] and the standard deviations of [0.229, 0.224, 0.225]. During the training, the best models were stored based on the top-1 accuracy on the CIFAR-10 test dataset and the ImageNet-1K validation dataset.

COCO Detection and Segmentation Training models for the COCO tasks followed the settings in [Liu et al., 2021]. Models started from ImageNet-1K pre-trained weights. We applied AdamW optimizer for 36 epochs with an initial learning rate of 0.0001, weight decay of 0.05, batch size of 16, 3× schedule, and multi-scale training.

4.2 Image Classification Benchmark

CIFAR-10 To demonstrate the benefits of using our DCTbased decorrelated attention on the small-scale dataset, we conduct Swin Transformer Tiny (Swin-T) from-scratch CIFAR-10 training. Table 2 presents the performance comparison on the CIFAR-10 test dataset, where the number of parameters only counts trainable parameters. The baseline vanilla Swin-T obtained 90.51% accuracy, and its accuracy was always improved by the DCT-based initialization regardless of which attention weight was initialized as the DCT matrix. When the weights to compute keys (K) were initialized as the DCT matrices, the accuracy was improved to 90.83%. For DCT-compressed attention, when we only keep 25% of DCT coefficients, trainable parameters were reduced from 27.53M to 21.45M (22.1% off), while the accuracy only dropped 0.05%. If we kept more DCT coefficients, the accuracy even improved. 75% of DCT coefficients increased accuracy from 90.51% to 90.90%. This is because although the number of trainable parameters is reduced, transforming the input data into the frequency domain using DCT can boost the extraction of important features.

ImageNet-1K Table 3 presents the performance comparison on the ImageNet-1K dataset. Same as on the CIFAR-10

dataset, DCT-based initialization consistently improves performance on both Swin-T and Swin-S models, regardless of which attention weight was initialized as the DCT matrix. For example, when we initialized the weights to compute keys (K) as the DCT weights, the accuracy of Swin-T was improved from 81.474% to 81.728%, and Swin-S was increased from 83.196% to 83.378%, respectively. We then evaluated the DCT-compressed attention with different truncating rates and compared the results with DCFormer [Li et al., 2023], a DCT-based approach designed to reduce computational cost. When 75% of DCT coefficients were kept, we obtained a comparable accuracy result on Swin-T and a slightly better accuracy result on Swin-S, while reducing the trainable parameters from 28.29M to 25.45M (13.1% off) and from 49.61M to 44.45M (10.4% off), respectively. Flops were reduced from 4.49G to 4.18G (6.9% off) for Swin-T and 8.13G to 8.74G (7.0% off) for Swin-S. Retaining fewer coefficients yields further savings at the cost of accuracy. In contrast, DCFormer-SW-T/S with τ = 2 (4× frequency compression) substantially reduces FLOPs but suffers notable performance degradation, as it treats low- and high-frequency components equally. We further validated our methods on the ViT-B-32 backbone [Dosovitskiy et al., 2020]. DCT-based initialization of the key projection improves accuracy from 73.120% to 73.614%. Applying DCT-compressed attention with 75% coefficient retention further improves accuracy to 73.226%, while reducing parameters from 88.22M to 75.83M (14.0%) and FLOPs from 4.41G to 4.05G (8.2%).

4.3 Ablation Study

Non-Trainable DCTWeights Initialization This study investigates the effectiveness of DCT weights as a principled starting point for attention projection initialization. Specifically, we fix the projection weights as DCT matrices, under which the computation of queries, keys, and values becomes equivalent to applying fixed shifts in the DCT representation of the input features. Table 4 shows that our method achieved accuracy comparable to the baseline on the CIFAR10 dataset while requiring fewer trainable parameters. On the ImageNet-1K dataset, non-trainable DCT weights for computing Q or K led to only a marginal drop in top-1 accuracy (less than 0.3%). In this work, the DCT was implemented via matrix multiplication, incurring the same computational cost as standard attention; however, this overhead can be further reduced using butterfly-based DCT implementations.

Multiple DCT Initializations This study shows why we only initialize one instead of multiple projection weights. Table 5 presents an ablation study on initializing multiple projection matrices with the DCT matrix. Although this approach still improved top-1 accuracy for Swin-T, the gains were reduced compared to initializing only a single matrix. For example, initializing bothWQ andWK resulted in a top1 accuracy of 81.540%, lower than 81.568% by initializing only WQ and 81.728% by initializing only WK. This degradation was likely due to the weight symmetry problem [Hu et al., 2019a], which often arises when multiple weight matrices are initialized with identical values. In the context of attention mechanisms, the query, key, and value projections play distinct roles in capturing relationships across the input sequence. If they are overly similar due to identical initialization, the model's ability to attend to relevant and diverse information is compromised. One of the key strengths of self-attention lies in its capacity to model complex dependencies between different positions in the input. Therefore, when WQ,WK, andWV share the same initialization, this capacity is diminished, leading to reduced performance. We further verified this observation using Swin-S by applying the DCTbased initialization to both WQ and WK. The results confirmed the same trend, supporting our conclusion that singlematrix DCT initialization is generally more effective.

Removing DCT in Compressed Attention This study discusses the necessity of DCT in the DCT-compressed attention. Specifically, removing DCT reduces the compression mechanism to a form of direct coefficient pruning in the spatial domain, without exploiting frequency-domain decorrelation. As Fig. 3 presents, when we removed the DCT in the DCT-compressed attention and retained other components, the accuracy of the revised Swin-T models on ImageNet-1K decreased across all compression ratios. These experiments show that DCT is essential because it transforms feature representations into the frequency domain, reducing redundancy and enhancing the model's ability to capture essential information. As a result, the attention mechanism benefits from improved expressiveness and robustness, even under compression. The accuracy drop observed in Fig. 3 confirms that DCT plays a crucial role in preserving discriminative features, demonstrating its necessity in the proposed approach.

4.4 COCO 2017 Benchmark

We further evaluated the proposed methods on COCO 2017 object detection and instance segmentation tasks [Lin et al., 2014]. We took Swin Transformers with ImageNet-1K pretrained weights as the backbone and the Cascade Mask RCNN [Cai and Vasconcelos, 2018] as the pipeline. Based on the superior performance observed in the ImageNet-1K experiments when initializing WK with the DCT matrix, we adopted this initialization for the Swin-T and Swin-S backbones in our evaluation. As shown in Table 6, both models achieved slightly higher average precision (AP) scores compared to their vanilla counterparts across both tasks. We also evaluated the DCT-compressed attention variant with the 0.75 compression ratio. This modification allowed Swin-T to maintain comparable AP performance while reducing the parameter count by approximately 3 million. It also enabled Swin-S to achieve improved AP scores with around 5 million fewer parameters.

5 Conclusion

In this work, we introduced two new methods using DCT to decorrelate the input in the attention for vision transformers. First, we presented a novel DCT-based initialization approach to address the challenges associated with training attention weights. We demonstrated a significant improvement over traditional initialization, providing a robust foundation for the attention mechanism. Second, we formulated a DCTbased compression method that can effectively reduce the computational overhead by truncating high-frequency components. It resulted in smaller weight matrices that compromised considerable accuracy. We validated the efficacy of these breakthroughs, showcasing their potential to enhance the performance and efficiency of Transformer models in classification tasks. These contributions underscore the importance of thoughtful initialization strategies and compression techniques in advancing the capabilities of the Vision Transformer architectures.

Acknowledgments

H. Pan and U. Bagci were supported by NIH grants R01HL171376 and U01-CA268808. E. Hamdan, X. Zhu, and A. E. Cetin were supported by NSF 2531376.