A new University of Michigan paper finds that frontier coding agents asked to analyze simulated data with no real effects still reported p<0.05 in 54% to 98% of runs, all false positives. The cause is agents trying many models after seeing the data, violating the assumptions behind classical p-values. Replaying agents on subsamples cut false positives to about 5%.
Coding Agent
编码代理
Agent systems that autonomously read and write code, run commands, and iterate on development tasks.
Curated Intelligence
A developer used coding agents to implement LLM inference on a reprogrammable FPGA, reporting that it simply worked. The post recounts a series of breakthroughs while entering hardware from scratch and argues that coding agents sharply lower the barrier to low-level hardware work, opening directions previously reserved for specialized teams.
Researchers from Northwestern, Carnegie Mellon and six other universities introduce Embodied First Exam (EFE), a unified framework that plugs general coding agents into simulated robots to test task completion under physical constraints. Fable 5.2 scores 80% in open-book mode but only 47% closed-book, revealing heavy reliance on documentation and external hints. EFE offers a reproducible map of where coding agents break down when moving from software to embodied settings.
Agent37 proposes a low-cost scheduling scheme for coding agents: since agents spend roughly 95% of their time waiting on model responses and use only about 5% of CPU, many agents can share the same physical cores, running 100 agents for $0.68 per hour. The design trades dedicated CPU for price, essentially oversubscribing resources around the I/O-wait profile of agent workloads.
Expo is rolling out Expo Simulators, cloud-hosted iOS and Android simulator environments built for coding agents, so they can install their own builds, drive the app, and record the session before requesting human review. Agents start a remote session via the eas simulator command and operate the app through agent-device, Argent, or Appium, returning screenshots, video, and a live browser link reviewers can tap through. The move closes a key loop gap for coding agents that can write mobile code but never see it run.
A team led by Jiaheng Liu at Nanjing University introduces TestPrism, a test-generation benchmark arguing that the prevailing single-reference evaluation systematically overstates Coding Agent test quality. TestPrism spans 300 test tasks from 17 sources and 3,000 candidate implementations, each task paired with a validated panel of correct and buggy implementations, and scores with a Joint Success Function (JSF) that requires tests to pass every correct implementation while rejecting every faulty one. Under this stricter criterion, Coding Agents fail collectively far more often than prior evaluations suggested, with reported pass rates inflated roughly twofold.
Terence Tao used modern coding agents to rebuild QED, his 2018 interactive logic textbook, into Lean Grimoire, which can display an in-progress Lean proof of the player's game state and adds a fantasy-themed mode. The project demonstrates coding agents' practical productivity in modernizing math visualization and teaching tools, and Tao is seeking playtesting feedback to improve logic and Lean instruction.
Chrome has released Modern Web Guidance 1.0, a stable skill set that injects expert-reviewed web platform best practices and cross-browser compatibility data directly into coding agents. Version 1.0 adds a modernization workflow that scans existing codebases, an expanded guidance database, a public evaluation dashboard, and expands into a cross-browser initiative with Firefox and Edge.
A joint Carnegie Mellon and JetBrains study finds that engineers' token spend is only weakly linked to shipped output: for the same engineer, doubling spend corresponded to roughly 10% more commits. The work linked five weeks of coding-agent token usage to commits from 404 JetBrains engineers, backed by a 412-person company survey. The authors recommend using token spend only for budgeting and judging agent-assisted work by task and outcome evidence.
GitHub head Jay Parikh revealed that coding agents now draft the first version of over 40% of pull requests on the platform, up from 17% in March, a roughly sixfold rise in half a year. He argues PRs should not be mandatory for every project and that GitHub will make collaboration steps configurable, while stressing that code review and access control remain essential for enterprise teams.