Rohan Paul (@rohanpaul_ai)T2·Models
71 pts
A new Amazon paper audited nine public rule-following benchmarks such as RuleArena and LegalBench and found that their reference answers and grading code contain errors, causing correct model responses to be marked wrong. In LegalBench's tax scorer, decimal points were stripped, turning $259,487.13 into 25,948,713; after the fix, three models' accepted answers rose from 8-9 out of 96 to 87-91. The authors urge checking scorer code and hand-reviewing failed answers before trusting low scores.