Amazon

A new Amazon paper audited nine public rule-following benchmarks such as RuleArena and LegalBench and found that their reference answers and grading code contain errors, causing correct model responses to be marked wrong. In LegalBench's tax scorer, decimal points were stripped, turning $259,487.13 into 25,948,713; after the fix, three models' accepted answers rose from 8-9 out of 96 to 87-91. The authors urge checking scorer code and hand-reviewing failed answers before trusting low scores.

Amazon has assembled a new team to develop an unprecedented home robot that can adapt to changing household environments and operate autonomously around family members. The group draws on former Astro staff and an MIT robotics expert, though product form and capabilities remain undisclosed. Astro itself shipped only to a few thousand invitees and its commercial variant ended within a year, yet Amazon is clearly not abandoning Bezos's long-term home robotics bet.