A new paper reveals a significant gap between the capabilities of AI models evaluated in academic research and the actual frontier models available at the time. The study found that the median research paper evaluates models that are approximately 10.85 ECI points behind the current state-of-the-art, a gap that is widening annually. This "publication elicitation gap" is attributed to factors beyond peer-review latency, with a substantial portion stemming from the use of older or less capable models and insufficient disclosure of evaluation configurations. AI
Summary written by gemini-2.5-flash-lite from 2 sources. How we write summaries →
IMPACT Highlights a systemic issue in AI evaluation, potentially misinforming policy and investment by overstating current capabilities.
RANK_REASON This is a research paper analyzing academic evaluations of AI models.