Sigmoid attention improves biological foundation models with faster, stable training

Researchers have developed a new attention mechanism called Sigmoid Attention, which offers significant improvements for training biological foundation models. This novel approach leads to better learned representations, achieving 25% higher cell-type separation and improved cohesion metrics compared to traditional softmax attention. Furthermore, Sigmoid Attention enables faster training, with models completing up to 10% quicker, and enhances stability by mitigating inherent issues found in softmax attention. The team has also released TritonSigmoid, an efficient GPU kernel that outperforms existing solutions on H100 GPUs. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Introduces a more stable and efficient attention mechanism for biological foundation models, potentially accelerating research in the field.

RANK_REASON Academic paper introducing a novel attention mechanism with empirical results and open-source code.

Read on arXiv cs.LG →

• Sigmoid Attention
• softmax attention
• TritonSigmoid
• H100 GPUs
• FlashAttention-2
• FlashSigmoid
• Vijay Sadashiviah
• arXiv

• paper
• infra