Researchers analyze signal propagation in normalization-free transformers

Researchers have analyzed signal propagation in normalization-free transformers using the averaged partial Jacobian norm (APJN). Their theory explains how attention mechanisms affect APJN growth in deep vision transformers. The study indicates that transformers with LayerNorm exhibit power-law APJN growth, while those using elementwise nonlinearities are subcritical, requiring careful initialization and optimization for stable training. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Provides theoretical insights into transformer training stability, potentially guiding future architecture design.

RANK_REASON Academic paper analyzing signal propagation in transformer architectures. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv stat.ML →

• arXiv
• Sergey Alekseev
• Dynamic Tanh
• DyT
• Dynamic erf
• Derf

• paper
• other