Vision Transformers leverage DCT for improved attention and efficiency

Researchers have developed a novel approach using the Discrete Cosine Transform (DCT) to enhance Vision Transformers. This method includes a DCT-based initialization strategy for self-attention, which improves classification accuracy on benchmarks like CIFAR-10 and ImageNet-1K. Additionally, a DCT-based attention compression technique reduces computational overhead by truncating high-frequency components of input patches, maintaining performance in models like the Swin Transformer. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Introduces methods to reduce computational costs and improve accuracy in Vision Transformers, potentially enabling wider deployment.

RANK_REASON Academic paper introducing novel techniques for improving Vision Transformer efficiency and performance.

Read on arXiv cs.CV →

• Discrete Cosine Transform
• Vision Transformers
• CIFAR-10
• ImageNet-1K
• Swin Transformer

• paper
• infra