SimpleBench Shows Top LLMs Still Trail Ordinary Humans on Common Sense
Researchers released SimpleBench, a 200-plus-question multiple-choice benchmark covering spatio-temporal reasoning, social intelligence and linguistic adversarial trick questions. A non-specialized human baseline of 83.7% from nine participants beat every tested frontier model, with top performer Claude Fable reaching only 81.9%, showing memorized knowledge and approximate retrieval are still insufficient for basic common-sense questions.
Full Report
We introduce SimpleBench, a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants, outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9%. While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.
For a deeper dive into our results and our methods, check out the full technical report here.
Use all of these models on the Simple Bench app - LMcouncil.ai