Colinearity Decay

Colinearity Decay trains vision Transformers for better low-bit quantization

Researchers have developed a new training technique called Colinearity Decay (CD) to make Vision Transformers (ViTs) more amenable to low-bit quantization. This method acts as a structural regularizer, penalizing alignm…