Colinearity Decay trains vision Transformers for better low-bit quantization

Researchers have developed a new training technique called Colinearity Decay (CD) to make Vision Transformers (ViTs) more amenable to low-bit quantization. This method acts as a structural regularizer, penalizing alignment within Transformer blocks to mitigate harmful activation outliers without affecting the architecture or task loss. CD aims to improve the accuracy of quantized models while maintaining or enhancing full-precision performance, offering a way to prepare ViTs for efficient deployment with no inference-time overhead. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT This technique could enable more efficient deployment of Vision Transformers on resource-constrained devices.

RANK_REASON This is a research paper introducing a novel training technique for improving model quantization. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CV →

• Colinearity Decay
• Vision Transformers
• ImageNet-1K
• COCO

• paper
• infra