stochastic gradient descent
PulseAugur coverage of stochastic gradient descent — every cluster mentioning stochastic gradient descent across labs, papers, and developer communities, ranked by signal.
9 day(s) with sentiment data
-
New study reveals SGD noise-covariance link to loss landscape curvature
Researchers have uncovered a new relationship between the noise introduced by Stochastic Gradient Descent (SGD) and the curvature of the loss landscape in deep learning models. Their findings indicate that this noise is…
-
New method predicts neural network generalization using Fourier fractal dimension
Researchers have developed a new method to predict how well deep neural networks will generalize without needing separate validation data. This approach uses the Fourier fractal dimension of the network's weight variati…
-
Deep Neural Networks Achieve Optimal Generalization Rates
Two new papers submitted to arXiv analyze the generalization performance of gradient descent methods in deep neural networks. The research establishes minimax-optimal rates for excess population risk in deep ReLU networ…
-
Lyapunov framework analyzes stochastic algorithm convergence
Researchers have published a paper detailing a Lyapunov-based framework for analyzing the finite-time convergence of stochastic iterative algorithms. This approach uses generalized Moreau envelopes as universal Lyapunov…
-
New paper re-evaluates SGD dynamics, challenging Brownian motion analogy
A new paper challenges the common assumption that Stochastic Gradient Descent (SGD) noise behaves like Brownian motion. Researchers propose an alternative model where SGD dynamics occur within a fluctuating loss landsca…
-
Simple Random Node Sampling outperforms full-graph training for GNNs
Researchers have found that a simple Random Node Sampling (RNS) method for training Graph Neural Networks (GNNs) can match or exceed the performance of full-graph training. This surprising result holds true across numer…
-
New Bayesian Framework Optimizes Neural Network Learning Rates
Researchers have introduced a novel probabilistic framework to optimize the learning rate in neural network training, moving beyond empirical trial-and-error. This new approach develops classic Bayesian statistics into …
-
New method adds missingness to SGD to reduce bias in incomplete data
Researchers have developed a novel method called Richardson-SGD to address gradient bias in stochastic gradient descent when dealing with incomplete data. The technique involves deliberately introducing additional missi…
-
Factor Augmented SGD optimizes high-dimensional machine learning
Researchers have introduced Factor-Augmented SGD (FSGD), a novel optimization method designed for high-dimensional machine learning tasks. FSGD operates on streaming data, enabling scalability for large-scale problems w…
-
Adam optimizer corrects SGD's frequency bias in language model training
New research highlights a frequency bias in Stochastic Gradient Descent (SGD) when training language models on imbalanced token distributions. This bias causes parameters for common tokens to converge quickly, while tho…
-
New theory shows momentum enables perfect parallelization in SGD
Researchers have developed a new theory explaining how classical momentum schemes like Polyak's heavy ball can accelerate stochastic gradient descent (SGD) for large-scale machine learning. The theory applies to quadrat…
-
LLM training research explores distillation, feedback, and optimizers
New research explores methods to improve Large Language Model (LLM) training efficiency and effectiveness. One study challenges the necessity of a strong teacher model in knowledge distillation, finding that even smalle…
-
Paper analyzes SGD dynamics in high-dimensional linear networks
A new paper details the high-dimensional behavior of stochastic gradient descent (SGD) on diagonal linear networks. The research shows that in high dimensions, SGD dynamics can be accurately modeled by a stochastic diff…
-
New papers analyze gradient descent convergence in neural networks
Two new research papers explore the convergence properties of gradient descent in neural network training. The first paper, focusing on wide shallow models with bounded nonlinearities, proves that non-global minimizers …
-
New method tackles unbounded variance in variational inference
Researchers have developed a new approach to optimize Black-Box Variational Inference (BBVI) by addressing the inherent unbounded variance in its stochastic gradients. Their method, detailed in a new paper, focuses on t…
-
New research derives advanced optimizers from evolutionary principles
Researchers have developed a new method to derive advanced optimization algorithms directly from evolutionary principles, unifying previously disparate views of evolution. This approach introduces Darwinian Lineage Simu…
-
Bayesian Parameter Shift Rule enhances VQE gradient estimation
Researchers have introduced a Bayesian variant of the parameter shift rule (PSR) for variational quantum eigensolvers (VQEs). This new method utilizes Gaussian processes to estimate objective function gradients, offerin…
-
Researchers explore efficient parameter estimation for truncated Boolean product distributions
Researchers have developed a new method for estimating parameters of truncated Boolean product distributions, a problem previously unaddressed in discrete settings. The approach relies on a concept of 'fatness' for the …
-
Researchers develop novel bootstrap for SGD confidence sets
Researchers have developed a novel method for constructing confidence sets in Stochastic Gradient Descent (SGD) algorithms. This new approach utilizes the multiplier bootstrap procedure and establishes its non-asymptoti…
-
Evolutionary game theory deciphers shortcut learning in deep neural networks
Researchers have developed a new theoretical framework using evolutionary game theory to understand shortcut learning in deep neural networks. The study formally defines core and shortcut features, modeling data samples…