New analysis reveals how step size impacts SGD alignment phenomenon

This paper analyzes the phenomenon of "suspicious alignment" in stochastic gradient descent (SGD) when dealing with ill-conditioned optimization problems. The study focuses on how step size selection influences the alignment of gradient updates with dominant subspaces. Researchers propose a step-size condition that differentiates between alignment-decreasing and alignment-increasing regimes, and demonstrate that under certain conditions, projecting SGD updates to the dominant space can paradoxically increase loss. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT Provides a theoretical understanding of SGD behavior, potentially informing the development of more robust optimization techniques for AI models.

RANK_REASON This is a research paper published on arXiv detailing a theoretical analysis of an optimization algorithm. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.LG →

• SGD
• arXiv
• Tianyu Pang

• paper
• other