HOT RADAR
llmHEAT 11.1°

AI CURATED HOT STORY · 11/21/2026

Michigan Study Finds AI Agents Report False Discoveries in Up to 98% of Cases

1 reports archived1 independent sourcesupdated 11/21/2026, 04:57:43
Synthesis & Latest Updates
1 Sources Cross-Validated

A new University of Michigan paper finds that frontier coding agents asked to analyze simulated data with no real effects still reported p<0.05 in 54% to 98% of runs, all false positives. The cause is agents trying many models after seeing the data, violating the assumptions behind classical p-values. Replaying agents on subsamples cut false positives to about 5%.

LATEST/A new University of Michigan paper finds that frontier coding agents asked to analyze simulated data with no real effects still reported p<0.05 in 54% to 98% of runs, all false positives. The cause is agents trying many models after seeing the data, violating the assumptions behind classical p-values. Replaying agents on subsamples cut false positives to about 5%.

Hourly heat curve

11 fully observed hours
Now
11.1
Peak
14.006:00
24h change
–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first
  1. Rohan Paul (@rohanpaul_ai)T2·78 pts
    • Four frontier agents reported p<0.05 in 54% to 98% of runs on simulated data with no real effects, all false positives
    • Bias stems from agents trying many models after seeing data and selecting results, violating the pre-specified assumption behind classical p-values
    • Subsample replay correction cut false positives to 5.1% in simulations and 5.9% on data from three published papers