Tianyu Pang

New analysis reveals how step size impacts SGD alignment phenomenon

This paper analyzes the phenomenon of "suspicious alignment" in stochastic gradient descent (SGD) when dealing with ill-conditioned optimization problems. The study focuses on how step size selection influences the alig…