Pythia
PulseAugur coverage of Pythia — every cluster mentioning Pythia across labs, papers, and developer communities, ranked by signal.
1 day(s) with sentiment data
-
Language model surprisal may not predict metaphor novelty as thought
A new paper published on arXiv suggests that language model surprisal, often used as a proxy for contextual predictability and metaphor novelty, may be misleading. The research indicates that lexical frequency is a stro…
-
New AEN-SAE architecture tackles feature starvation in LLM interpretability
Researchers have introduced Adaptive Elastic Net Sparse Autoencoders (AEN-SAEs) to address feature starvation in sparse autoencoders used for interpreting LLM representations. Traditional methods struggle with dead neur…
-
Researchers find Transformers know counts but struggle to output them
A new paper identifies a specific bottleneck in Transformer models that hinders their ability to perform counting tasks. Researchers found that while models like Pythia, Qwen3, and Mistral store count information accura…
-
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
Researchers are developing several novel methods to optimize the Key-Value (KV) cache in large language models, which is a major bottleneck for long-context processing. These approaches include training models to inhere…
-
AI model evaluations are becoming a costly bottleneck, surpassing training expenses
AI model evaluations are becoming prohibitively expensive, with recent benchmarks costing tens of thousands of dollars and consuming thousands of GPU hours. This high cost is particularly pronounced for agent-based eval…
-
Transformer architecture significantly impacts model error detection capabilities
A new paper reveals that a transformer model's architecture significantly impacts its ability to signal decision quality through internal activations, a property termed 'observability.' This observability is crucial for…
-
Transformer research probes security flaws, training dynamics, and in-context learning limits
Researchers have identified vulnerabilities in the shuffling defense mechanism used to secure Transformer models during inference, demonstrating an attack that can extract model weights by aligning permuted activations.…