New Benchmark Tests AI Intuition; Humans Outperform Top Models

AI can read charts, solve competition problems, recognize dense text, and even use tens of thousands of reasoning tokens to analyze a visual problem in meticulous detail.

But it might fail to notice: why the woman chasing the bus suddenly slowed down; whether another car could fit between two existing vehicles; who is most distracted in the meeting room; or what the illegible label on the last beaker in a row actually says.

For these questions, humans often have an answer at a glance.

Just now, Scale AI, in collaboration with Elorian, released a new Benchmark specifically designed to test this capability – Humanity’s Sixth Sense (HSS). The results are striking: 20 human participants scored 93.1%, while the current top model, GPT-6-astra, achieved only 53.6% even at maximum reasoning intensity; the median score across 25 models was just 30.9%.

HSS Paper: https://labs.scale.com/papers/humanitys-sixth-sense

Scale Labs Official Introduction: https://labs.scale.com/blog/introducing-humanitys-sixth-sense

HSS Live Leaderboard: https://labs.scale.com/leaderboard/hss

Dataset: https://huggingface.co/datasets/ScaleAI/HSS

This isn’t a small test of "whether models can solve problems."

What HSS truly asks is: When AI enters cars, homes, and offices, can it understand the situation like a human does—before things are explicitly stated?

HSS Initial Leaderboard. All 25 multimodal models significantly lag behind humans.

AI Has Vision

But Lacks "Situational Awareness"

The "Sixth Sense" referred to by HSS is not supernatural ability, but rather intuitive visual reasoning that humans use every day.

When looking at an image or a few seconds of video, we don't just identify "what is in the frame"; we automatically fill in information that isn't directly displayed: what happened previously, what might happen next, whether an object can pass through a gap, who knows what, who is obeying whom, and what hidden rules lie behind a scene.

This capability is astonishingly fast. Cognitive science research shows that humans can complete natural scene classification within approximately 150 milliseconds, simultaneously extracting physical dynamics, behavioral intentions, and social structures.

Existing visual Benchmarks mostly stand at two extremes: one category tests "expert-level slow thinking" such as math olympiads, charts, and academic knowledge; the other tests low-level perception such as OCR, object recognition, and detection localization.

The capability that sits in the middle—and is most important for the real world—has long gone unmeasured individually: visual intuition where humans get answers at a glance without listing formulas or being able to clearly explain the process.

HSS has formally turned this blank space into a score.

522 Questions Specifically Testing What Isn't Explicitly Shown in the Frame

HSS contains 522 open-ended tasks, corresponding to 288 images and 234 videos, with a total video duration of 17.6 hours. The questions are divided into 4 major categories and 11 sub-domains:

Temporal and Causal Dynamics. Models must infer the past from the current frame, determine the physical mechanisms at play, or predict what will happen next. For example, in a video, a woman switches a lever from her right hand to her left; how does this action change the railway switch?

Physical and Spatial Logic. Models must judge occluded objects, mass, and invisible forces, and also answer questions like "will it fit," "what would be seen from a different angle," or "can that person reach it."

Social Understanding. Models must infer people's beliefs, emotions, knowledge gaps, and intentions, and also understand social roles, norms, and power dynamics. For example, a woman chasing a bus suddenly slows down while running because the bus stopped to wait for her, so she no longer needs to sprint.

Abstract and Contextual Inference. Models must find the hidden rules that give meaning to a scene, including consequences of changes, patterns in arrangements, and pareidolia (seeing faces in objects). For example, if labels on a row of beakers are sequentially 40, 50, 60, 70, what should the blurry last label be? The answer is 80.

HSS Four Major Task Domains. Each question includes an image/video, a question, a reference answer, and scoring criteria.

The most critical design feature is: The answer cannot appear directly in the pixels.

Questions where the answer becomes visible upon high-resolution zooming were eliminated; questions requiring specialized professional knowledge were eliminated; and questions where humans could not form a consistent judgment were also excluded from the final test set.

The annotation team wrote 3,466 candidate questions, which underwent three rounds of independent review, leaving only 522—a selection rate of just 15.1%. In the first round alone, 2,302 questions were eliminated, primarily because "the answer was already in the image" or "humans could not agree on the correct answer."

Scoring is equally rigorous: models provide free-form answers with no options to eliminate; an LLM judge (Claude-Opus-5) scores each response against the criteria item by item, counting as correct only if all criteria are met. The team re-scored 5 models using judges from three different vendors, finding that rankings remained completely consistent, with no obvious "self-model bias" detected.

25 Models Collectively Fail

Top Score Less Than 60% of Human Performance

The research team tested 25 multimodal models from eight vendors.

GPT-6-astra ranked first with 53.6%, followed by GPT-6.1-sol (46.6%), Claude-Opus-5.5 (44.6%), Gemini-3.8-flash (41.6%), and Claude-Fable-5.1 (40.8%). All remaining models scored below 40%.

Humans, by contrast, achieved 93.1%.

Even more intriguing was the "blind test control": when researchers removed images and videos, forcing the strongest model to guess answers based solely on text prompts, GPT-6-astra’s performance plummeted to 6.6%. This demonstrates that HSS does not test linguistic patterns or allow for guessing via common-sense templates. The evidence determining the correct answer truly lies within the visual scene.

Video tasks proved particularly difficult. Of the 25 models, 23 performed worse on video tasks, with an average drop of 7.3 percentage points. Social understanding emerged as the weakest domain: 21 models scored lowest in this area, with an average accuracy of just 24.4%, compared to an average of 34.1% across the other three domains.

Models are not failing at reasoning;

they are failing to see.

To identify the root causes of failure, the research team conducted a detailed attribution analysis of 8,573 model failures.

The conclusion runs almost counter to the intuition that "models need stronger logical reasoning capabilities": 94% of failures stem from perception or latent information inference, while true logical reasoning errors account for only a small fraction.

The most common error was missing key clues decisive to the answer (21.3%); followed by misidentifying objects, people, or roles (19.9%); then misjudging size, distance, and fit (12.2%); misinterpreting orientation (12.0%); and mistaking 2D occlusion for 3D alignment (9.1%).

Failures across different sub-domains are highly concentrated in "misseeing" and "failing to extract implicit information from the visual scene."

Moreover, these errors are not random. In questions where at least five models failed simultaneously, the median showed that 80% of models failed for the exact same reason. This suggests that models from different vendors and architectures may share similar visual blind spots.

They can generate long chains of thought, yet might grasp the wrong critical fact from the very first glance: misidentifying a person, reversing direction, or misunderstanding occlusion relationships; then spending thousands of tokens elaborating on an incorrect premise until it becomes increasingly coherent. On average, the 25 models used approximately 4,000 reasoning tokens per question.

Increasing reasoning intensity generally helped but remained unstable. In several sub-domains, higher reasoning levels actually led to lower scores, with peak performance often occurring at intermediate levels.

When the input world model is flawed, "thinking longer" does not automatically yield the correct answer.

Equipping Models with Agents Can Help, But Cannot Instill True Spatial Awareness

The research team further integrated models into Claude Code and Codex, allowing agents to crop, zoom, re-extract frames, search, and repeatedly inspect visual materials.

The effect was significant. On a subset of 388 questions, Claude-Opus-5 improved from 30.8% to 51.3%, and Claude-Fable-5.1 rose from 41.2% to 57.7%; GPT-6-astra improved from 54.4% to 59.3% within Codex, achieving the highest score under agent settings.

Agents excelled at fixing errors discoverable through "a closer look": object and role identification errors dropped from 181 to 147; orientation errors fell from 117 to 93; size and fit judgment errors decreased from 137 to 105; and missed key clues reduced from 154 to 110.

Agents can reduce various types of errors by re-inspecting visuals, but remain largely powerless against 3D depth judgments.

However, one metric exposed the boundary: errors mistaking 2D overlap for 3D alignment barely changed, dropping only from 102 to 101 instances.

This type of failure is not due to "insufficient zooming," but rather the model's lack of stable 3D spatial representation. An agent can pick up a magnifying glass, but cannot spontaneously develop human-like spatial intuition.

More alarmingly, a few logical errors actually increased under agent settings. Tool invocation and iterative thinking can fix shallow observations but may also introduce new reasoning path deviations.

Thus, agents are not the end solution for HSS problems; they serve more as a diagnostic tool: visual tools can alleviate "not seeing clearly," but cannot solve "not understanding."

The Next Wall for Multimodal AI

In recent years, progress in multimodal models has often manifested as reading more text, solving harder problems, and processing longer videos.

HSS proposes another axis: whether a model possesses intuition about the real world.

This axis is crucial for real-world AI agents. Home robots need to notice a child moving toward stairs before he falls; autonomous driving systems must judge if a vehicle can pass, rather than merely identifying "two cars ahead"; office assistants need to understand who is giving orders, who hasn't grasped the instruction, and the true intent behind polite pleasantries.

These capabilities cannot wait for the world to caption its answers.

The significance of HSS extends beyond creating another leaderboard. It decomposes issues long vaguely termed "common sense," "world models," or "social intelligence" into measurable temporal, spatial, social, and abstract inference abilities; furthermore, it uses failure attribution to tell researchers exactly where bottlenecks occur.

Looking at the results, next-generation multimodal systems may require not just longer chains of thought, but more reliable object-role binding, more stable 3D representations, more continuous temporal modeling, and social/physical experience gained from real interactions.

Today’s models can already speak fluently about an image. But HSS reminds us: describing the world is not equivalent to understanding it; reasoning is not equivalent to possessing intuition.

Between humans’ 93.1% and models’ 53.6% lies not a harder exam question, but the essential glimpse machines must acquire before truly entering the real world.

References:

https://labs.scale.com/papers/humanitys-sixth-sense

Chasing ASI Instantly

⭐ Like, Share, and Watch – One Click Combo ⭐

Star our page to lock in Xinzhiyuan’s rapid updates!