SimpleBench 揭示顶尖模型常识推理不及普通人
研究团队发布 SimpleBench 基准,包含 200 余道时空推理、社会智能与语言对抗性陷阱题。在九名普通高中知识水平受试者构成的基线达到 83.7% 的情况下,包括 Claude Fable 在内的所有受测前沿模型均未超越,最高仅 81.9%,说明记忆知识与近似推理检索仍不足以应对基础常识问题。
报道正文
We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants, outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.
For a deeper dive into our results and our methods, check out the full technical report here.
Use all of these models on the Simple Bench app - LMcouncil.ai