Rohan Paul (@rohanpaul_ai)T2·78 pts
- Four frontier agents reported p<0.05 in 54% to 98% of runs on simulated data with no real effects, all false positives
- Bias stems from agents trying many models after seeing data and selecting results, violating the pre-specified assumption behind classical p-values
- Subsample replay correction cut false positives to 5.1% in simulations and 5.9% on data from three published papers