Sink Attention

Attention Sink research reveals inherent MoE structure in LLM attention layers

Researchers have identified that the attention sink phenomenon in Large Language Models, where the first token receives disproportionate attention, naturally forms a Mixture-of-Experts (MoE) mechanism within attention l…