DishwasherBench Launches: Embodied AI Benchmark Based on Real Bosch Dishwasher Physics and Wash Chemistry Simulation

DishwasherBench is a dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable by people in a browser. The game inside is called Spotless, and the packages are named @spotless/*.

A player loads a 3D model of a Bosch SMV4HAX40G with rigid-body physics. A wash on its Eco 50 programme is then simulated (jets, droplets, run-off, soil and detergent chemistry), and the player gets points for every item that comes out clean. On every level after the first, not everything fits, so the player has to choose what to load and how.

This is an early release, version 0.1.0. The simulation is calibrated on one test wash of the real machine. A score shows whether a player packed well and put cups, glasses and pots in mouth down. It does not yet show whether plates and cutlery were stacked so that they wash clean. The section What a score can be trusted for says so in full.

What is solid today: loading is deterministic rigid-body physics, every episode is a record that anyone can verify by replaying it and by washing its load again, and the same game is played by people in a browser and by models through MCP. What is rough is how the wash scores plates and cutlery.

Built and maintained by Composo. Many documents under docs/ are working records, kept as they were written; docs/README.md explains how to read them.

Loading the first level in the browser. A dinner plate is in hand over the lower basket, and "Suggest spots" has numbered the free seats, best first. The panel on the right holds the exact pose a program would send.

What works today

• The machine. One Bosch SMV4HAX40G as data and geometry, from its owner's photographs and tape readings (packages/machine): both baskets with their tine rows, shelves and rests, the spray arms and the Eco 50 programme.
• Loading. A catalogue of items and soils, deterministic rigid-body loading (Jolt), and a session layer with the actions and queries of the game (place, remove, set_rack, list_slots, check_load and the rest).
• The wash. A simulation at two tiers. The official tier gives the scored result; the fast tier is a coarser preview. A cold official wash of the largest level takes 4.9 minutes on the reference machine, a MacBook Pro (Apple M3 Pro, 18 GB), best of three, and 5 to 8.5 minutes with another job running beside it, against a budget of 30.
• A course of seven levels, numbered in order of difficulty (docs/benchmark/levels.md), each with saved reference loads of three scripted players (careful, careless, random). Six pass the benchmark's gates on the reference loads; the tutorial does not (its careless load loses too little).
• Three ways to play. The page (apps/web), an MCP server for agents (packages/mcp), and a harness that runs scripted players and models and writes verified episode records (packages/bench).
• Checks. 48 loading phenomena from the literature as executable pairs of washes (docs/research/standards.md section 9), and a calibration tool that scores the simulation against a record of a real wash, fits parameters and ranks candidate test washes (the stored ranking is of an earlier parameter set and has not been run again).

Two models have played, through an OpenAI-compatible endpoint: six episodes on three levels at the official tier (Results so far). No person has played for a baseline, and the path for Anthropic's API has not been run against the live API.

What a score can be trusted for

The full account, with every number and where it comes from, is docs/validation/README.md. In short, the simulation has been checked against four things:

• One test wash on the real machine. Four egg-yolk plates on Eco 50. A prediction written before the result got the order of the plates and the form of the residue right, and was ten to thirty times too optimistic on the amount left on the two oven-dried plates. The calibration that followed moved one number: the share of the long soak during which the pump runs, from 0.5 to 0.8 (anywhere from 0.59 to 1). The record that the prediction came first is in the private development history, which is not published, so that order cannot be checked from this repository.
• Published class means of a sister machine (Bosch SMV6ECX51E, eco programme, reference detergent): cutlery within 0.1 of a score point, crockery 0.2 low, glass 0.1 to 0.5 low. This is a fit, not a prediction: the soil and chemistry defaults were set so that these numbers come out.
• 48 loading phenomena with evidence in the literature: 17 pass, 21 fail in size or sign, 1 is a faulty scenario and 9 need mechanisms that are not built.
• Its own book-keeping. Water is conserved exactly, and the same input gives the same result on one machine. The fast tier agrees with the official tier on 89 % of the verdicts of a standard 13-place load and on 97 % on the levels.

A score today shows whether a player packed well and put cups, glasses and pots in mouth down. It does not yet show whether plates and cutlery were stacked so that they wash clean.

（注：更多技术实现细节与完整 API 文档请查阅原项目 README）