New research suggests mean pooling of generated tokens improves LLM state representation

A new research paper proposes mean pooling of hidden states from generated tokens as a superior method for capturing a language model's internal state. This approach, which aggregates information distributed across multiple tokens, yields more semantically rich representations than using individual tokens alone. The study demonstrates that representations derived from generated tokens outperform those from prompt tokens, offering insights into model behavior dynamics. AI

Summary written by gemini-2.5-flash-lite from 1 source. How we write summaries →

IMPACT This research could lead to more accurate and interpretable internal representations of language models, potentially improving downstream applications.

RANK_REASON The cluster contains an academic paper detailing a novel method for representing language model states. [lever_c_demoted from research: ic=1 ai=1.0]

Read on arXiv cs.CL →

• arXiv
• language model

• paper
• model release