Google's PLE Benchmark Reveals Top AI Agents Score Only 50.6% on Real Engineering Tasks
Google introduces the PLE benchmark for real-world physical engineering tasks, featuring 85 jobs in electrical, civil, and other fields vetted by practicing engineers. The top-performing agent, Claude Opus 5.5, scored only 50.6%, primarily failing due to missed hard safety or feasibility constraints. The study highlights that existing LLM benchmarks favor textbook problems, whereas real engineering requires processing unstructured files while strictly adhering to codes.
Full Report
Rohan Paul (@rohanpaul_ai) · 2026-10-28 22
New Google paper finds that the best AI agent scores only 50.6% on physical engineering jobs, often by missing a hard safety or feasibility rule.
Engineering benchmarks for LLMs mostly test textbook questions or toy problems that top agents already handle well. Real engineers work from piles of PDFs, spreadsheets, and CAD files and must issue designs that meet code and stay cheap.
They built PLE, 85 electrical, civil, structural, and mechanical jobs written or checked by practising engineers. Agents must read about 61 project files per task and deliver schedules, reports, or 2D and 3D drawings.
Claude Opus 5.5 in Claude Code topped 14 models at 50.6%, using 27M tokens and 63 minutes per task. On design tasks its outputs met 91% of constraints, but only 41.3% met every hard constraint.
If you build engineering agents, gate every design on hard safety and code limits, and spend effort on model choice before tuning the agent scaffold.
