Rohan Paul (@rohanpaul_ai)T2·Models
78 pts
A new University of Michigan paper finds that frontier coding agents asked to analyze simulated data with no real effects still reported p<0.05 in 54% to 98% of runs, all false positives. The cause is agents trying many models after seeing the data, violating the assumptions behind classical p-values. Replaying agents on subsamples cut false positives to about 5%.