New RWE-Bench Reveals AI's Poor Real-World Engineering Design Skills
Xplore AI released RWE-Bench to evaluate LLMs on real-world engineering tasks like CAD and simulation. Results show even the top model (GPT-6 Astra) achieved only a 37% pass rate, with most models scoring near zero, highlighting significant gaps in handling physical constraints and complex design workflows.
Full Report
AI models can already outperform humans in solving math problems and writing code.
However, when the same batch of models is given access to the CAD and simulation software engineers use daily—tasked with designing crankshafts, routing memory traces, or calculating heat dissipation for turbine blades, with a maximum time limit of 24 hours per problem—even the best model, GPT-6 Astra, achieves only a 37% pass rate.
In this benchmark, other models scored nearly zero across the board.
What RWE-Bench Tests
On October 27, Xplore AI released RWE-Bench, short for Real World Engineering Bench.
The benchmark consists of 20 problems: 9 mechanical engineering tasks, 6 aerospace tasks, and 5 electronics tasks.
Each problem was created by an engineer specializing in that field within the team, who also produced a reference solution using the same set of tools.
Scoring relies neither on AI nor manual review.
Instead, a program invokes physics solvers to check submitted designs item by item; failure to meet any single criterion results in a failed score.
Take the mechanical engineering task as an example: design the crankshaft for a three-cylinder diesel engine.
The technical brief specifies limits for mass, stress, fatigue factors, and crankpin deformation, each with concrete numerical values.
Adding too much counterweight causes excess mass; adding too little leads to imbalance during rotation. The design must also withstand millions of cycles at full load under 2,400 RPM.
Models are provided only with CadQuery, Gmsh, CalculiX, and Python. They must handle modeling, mesh generation, and finite element analysis (FEA) entirely on their own.
The electronics task involves adding DDR3 trace routing to a production FPGA board.
Nearly 300 components are already placed on the board, and all existing traces are fixed and cannot be altered.
Over seventy nets must be routed through the gaps in an 8-layer PCB, subject to strict timing skew and impedance constraints. If even one net exceeds these limits, the entire board fails.
The aerospace task requires calculating heat transfer for high-pressure turbine guide vanes, from flow channel modeling to CFD convergence, ultimately delivering the heat transfer coefficient and total thermal load.
In engineering firms, such tasks are typically assigned to experienced engineers proficient in the relevant software, representing a significant barrier to entry.
Data Beyond the Scores
1. Look at the rankings first.
GPT-6 Astra: 37%, Claude Opus 5.5: 29%, GPT-6.1 Sol: 25%, Gemini 4 Argon: 12%, Claude Fable 5.1: 7%. Other models scored nearly zero in this evaluation.
2. Time spent answering reveals more than scores alone.
Although each problem allows up to 24 hours, most models submit answers within two to three hours on average, with the longest submission taking only about 4.6 hours.
Human engineers often spend far more time iteratively refining models and running simulations for such tasks.
It is likely that models believe they have completed the task correctly, when in fact they have made errors...
3. Higher cost does not guarantee higher scores.
Running GPT-6.1 Sol once costs just over $10, while Astra and Opus 5.5 both cost around $58, yet their scores are only 12 and 4 points higher, respectively.
Claude Fable 5.1 costs nearly $98 per run—the most expensive among them—but achieves a pass rate of only 7%.
4. If each problem allows 5 attempts and passing once counts, Opus 5.5 scores 70%, surpassing Astra's 60%.
Several aspects of the leaderboard warrant scrutiny
1. The benchmark contains only 20 problems, resulting in a small sample size.
According to the official 95% confidence intervals, Astra ranges from 30.7% to 43.8%, while Opus 5.5 ranges from 21.9% to 37.6%. Since these intervals overlap, it is not yet possible to definitively conclude which model is stronger between the top two.
2. The page states that all models were run with maximum reasoning intensity, but the data file lists Gemini 4 Argon as "high" rather than "max."
Of course, the newly released flagship model, Gemini 4 Carbon, did not make it into this evaluation cycle. We look forward to Xplore AI conducting supplementary tests on it.
3. It is worth noting that Xplore itself sells RL training environments.
The official website states that they help model companies conduct evaluations and build training environments, though the problems in the leaderboard are not licensed for use in training.
Lower scores on the leaderboard make their business easier to sell.
This does not necessarily mean the data is flawed—the problems and grading methods are public—but controlling problem difficulty directly impacts their business interests.
4. There are 4 problems that no model solved, but the specific questions have not been disclosed by the organizers.
Xplore plans to add problems related to biomechanics, civil engineering, and materials engineering next, and is currently hiring engineers and researchers.
When SWE-bench was first released in 2023, the best models solved less than 2% of the problems. Everyone has seen how those scores have risen since then.
Stay tuned for new models to see their scores skyrocket on RWE-Bench.
References:
Chasing ASI Instantly
⭐ Like, Share, and Watch – One-Click Triple Combo ⭐
Star us to lock in instant updates from New Intelligence Era!
