Sparse Mixture of Experts
PulseAugur coverage of Sparse Mixture of Experts — every cluster mentioning Sparse Mixture of Experts across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
New SSMoE framework uses eigenvectors to fix SMoE model collapse
Researchers have introduced Singular Value Decomposition SMoE (SSMoE), a new framework designed to tackle the expert collapse issue in Sparse Mixture of Experts (SMoE) models. Unlike previous methods that require extens…
-
JetBrains releases efficient Mellum2 MoE model; research advances MoE techniques
JetBrains has released Mellum2, an open-source 12-billion parameter Mixture-of-Experts (MoE) model optimized for efficient inference in text and code tasks. This model activates only a fraction of its parameters per tok…
-
LLMs explore preference alignment and failure mitigation techniques
Researchers are exploring new methods for aligning large language models (LLMs) with human preferences and mitigating specific failure modes. One approach uses Direct Preference Optimization (DPO) to reduce text degener…
-
New research optimizes Sparse Mixture-of-Experts for efficient LLM scaling
Researchers are exploring new methods to optimize Sparse Mixture-of-Experts (SMoE) models, which are crucial for scaling large language models efficiently. One paper reveals a geometric coupling between routers and expe…