Tsinghua and 16 Institutions Launch Mind2Act World Embodied Benchmark
Tsinghua mmlab@SIGS, together with Yuanxing Guangnian and 16 institutions, released Mind2Act World, a diagnostic benchmark for robotic manipulation with 30 tasks and 180 configurations that tests reasoning and physical execution separately and then jointly, attributing each failed episode to a root cause. Across 9 systems and 56,700 simulated episodes, the best agent scored 52.7% on reasoning-only and 39.0% on execution-only tasks, but just 12.0% on composed tasks, while end-to-end policies peaked at 3.6%.
Related Entities
Full Report

Unity of Knowledge and Action

Published by Machine Heart
VLA/WAM Is Dead, Long Live Foundation Models?
Over the past month, general-purpose large models, operating as Agents, have repeatedly achieved high scores in robotic manipulation benchmarks. Whether end-to-end VLA and WAM approaches should continue to be pursued has become an unavoidable topic of debate in the embodied intelligence community.
In this debate, both sides rely on success rates as evidence. Success rates can show who accomplishes more, but they cannot reveal where the failures fall short: whether a given approach has hit a dead end or is merely missing a patchable gap—scores alone cannot tell.
Take a cooking task as an example: a robot must add ingredients in sequence according to a memorized recipe, then switch to keep-warm mode within two seconds after the pot temperature reaches 80°C. Neither the World Action Model InternW0 nor the Agent powered by GPT-6 Astra succeeded, but their failure modes were very different: of InternW0's 300 episodes, 271 turned on the heat before all ingredients were added; of GPT-6 Astra's 30 episodes, 21 failed to even grasp the ingredient jar. Both failed, but the former lacks confirmation of the previous step, while the latter is stuck at grasping—what needs to be fixed is completely different.
Identifying such differences is exactly what Mind2Act World sets out to do. This general-purpose robotic manipulation diagnostic benchmark was released by Tsinghua University's mmlab@SIGS in collaboration with Yuanxing Guangnian and 16 other universities and institutions, comprising 30 tasks and 180 configurations. It separates a robot's "thinking" and "doing" into independent tests first, then tests them together, and identifies the cause of failure for each episode, compiling them into a diagnostic card for each model along with directions for capability improvement. The first batch of evaluations covers 9 representative systems and 56,700 simulated episodes, and the leaderboard remains open for ongoing participation.
Results show that the best-performing Agent achieved a 52.7% success rate on tasks testing only "thinking" (memory and reasoning), 39.0% on tasks testing only "doing" (physical execution), and only 12.0% on tasks requiring both; end-to-end policies scored at most 3.6% on such tasks.
Returning to the opening question: both approaches have their strengths—end-to-end policies excel at "doing," while Agents excel at "thinking"—and neither has mastered tasks that require thinking while doing. Rather than asking which approach will win, the more worthwhile question right now may be: at exactly which step does each model fall short?

- Paper Title: Mind2Act World: A Diagnostic Benchmark for Embodied Mind, Physical Action, and Mind-Action Composition in Robotic Manipulation
- Paper URL: https://arxiv.org/abs/2611.26444
- Leaderboard: https://mind2act-world.com/leaderboard.html
- Model Diagnostic Cards: https://mind2act-world.com/diagnosis.html
- Benchmark Code: https://github.com/Mind2Act/Mind2Act_World
- Training Data: https://huggingface.co/datasets/Mind2Act-World/mind2act-train
Mind2Act World Project Introduction Video

Mind2Act World Overview: 30 Tasks, 12 Capability Categories, Diagnostic Loop, and Current Leaderboard
Same Zero Score, Different Reasons—Finding the Cause of Every Failure First
To answer "at which step does it fall short," we first need a method for identifying the causes of failure.
Beyond determining success or failure, the Mind2Act World evaluator also records the stage progress and key events of each episode. Failures that can be determined by rules are judged by rules first; the rest are handed to a large-model judge that reviews videos and action logs, and uncertain cases are further reviewed manually. For each failed episode, only the first unrecoverable error is identified—whether it lies in "thinking," in "doing," or in the "coordination" between the two, i.e., proceeding without confirming the result of the previous step.
All failed episodes of a model, taken together, form its diagnostic card. The report generates empirical diagnostic cards for three representative systems: InternW0, GPT-6 Astra, and G0.5.
First Test "Thinking" and "Doing" Separately, Then Test Them Together
To pinpoint which step fails, the tasks themselves must first separate different capabilities. The technical report opens with a quote from Wang Yangming on the "unity of knowledge and action": "Knowledge is the beginning of action; action is the completion of knowledge." The 30 tasks are likewise divided into three groups of 10 each:
Mind (Thinking): Actions are kept as simple as possible; what is tested is memory and decision-making. For example, steamer restoration: observe the food in each layer of the steamer, then stack them back in the original order.
Act (Doing): The goal is given directly; what is tested is accuracy and timing. For example, grasping objects from a moving conveyor belt.
Mind2Act (Coordination of Thinking and Doing): Steps are interlocking—memorized information determines the current action, and the result of the action determines the next step.
Mind2Act World task overview: 10 tasks in each of three categories, covering 5 application domains and 12 capability types
Each task has three difficulty levels—Easy, Medium, and Hard—and is tested in both clean scenes and randomized scenes (with added clutter, changed materials, and altered lighting). Success is judged solely by physical outcomes: whether the object is placed stably and whether the button is actually pressed.
Getting half of the thinking right and 40% of the doing right leaves only 12% combined
The 9 participating systems fall into two categories. Three Agents (GPT-6 Astra, DeepSeek-V4.1-Flash, Gemini-4-Argon) operate robots through tool calls; four VLAs (KinRT, π0.5, G0.5, X-VLA) and two WAMs (InternW0, FastWAM) are fine-tuned on the same set of 4,500 demonstrations. The two categories differ in action interface and budget, so they are ranked separately.

Success rates (%) of each system on Mind, Act, and Mind2Act tasks, with Agents and end-to-end policies ranked separately
On the Agent leaderboard, GPT-6 Astra currently ranks first: Mind 52.7%, Act 39.0%, Mind2Act 12.0%. On the end-to-end leaderboard, InternW0 currently ranks first with an overall score of 8.68 (out of 100). These policies perform best on Act, with the four VLAs achieving success rates between 12.0% and 19.6%; on Mind, none exceeds 5.4%; on Mind2Act, the highest is π0.5 at 3.6%.
Task by task, the strengths of the two categories are exactly complementary.

Per-task success rates (%) grouped by application domain
GPT-6 Astra is strong at "thinking." On the hidden-danger-zone navigation task, it succeeds every round, and its success rate on card matching is 93%; when the tabletop material and lighting are changed, its Mind success rate only shifts from 54.7% to 50.7%. But on tasks that require correction through contact, it has no successful record yet—for example, across 30 rounds of tray carrying, it lost control every time it lifted the tray.
End-to-end policies are strong at "doing." On the timed gate-passing task, KinRT, π0.5, and X-VLA achieve success rates of 76%–79%, comparable to GPT-6 Astra's 80%; on the cloth-folding task, X-VLA reaches a 64% success rate. However, they struggle to remember task state: after scene randomization, the four VLAs retain 86%–94% of their Act successes, but only 28%–41% of their Mind successes. What the models remember is more like what the scene "looks like" than "what is in it."

Mind2Act World visualizes the leaderboard as a "road to the moon," with each model's position corresponding to its overall score
Most pipelines break in the first few steps, before "coordination" even comes into play
The low Mind2Act scores are easily interpreted as poor coordination between "thinking" and "doing." Looking round by round, the situation is far more specific.
Most rounds are interrupted at a very early stage, according to task scores. In addition to success or failure, each task awards a score of 0–100 based on how far the system got, with success recorded as 100. Across all systems, Mind2Act scores exceed success rates by at most 1.5 points—meaning failed rounds earned almost no partial-stage credit. On the four Mind2Act tasks that no system completed, the highest average score across systems is only 1.2 points; on bridge-building for vehicle passage and whiteboard restoration, no system earned even the first stage score.
The three diagnostic cards later in this article convey the same message: physical execution is the largest category of failure, and failures that first arise in the "coordination" step account for only 6%–11%. The conclusion follows: reliable coordination first requires every capability to hold up at every step of the pipeline.
The difficulty curve tells the same story. From Easy to Hard, GPT-6 Astra's Mind success rate drops from 64% to 44%, Act from 44% to 34%, and Mind2Act from 26% to 3%. What Hard difficulty adds to Mind2Act is more interdependent operations—any single misstep makes the entire pipeline impossible to complete. If each step has a 90% success rate, the probability of completing ten steps in a row is only about 35%.
From a single score to an improvement checklist
Only by knowing where the pipeline breaks can you talk about how to fix it. First, let's look at what each of the three diagnostic cards says.
Selected segment of the temperature-controlled finishing phase in dual-arm ingredient preparation and temperature-controlled cooking (CP04, Hard), a simulation demonstration by a scripted expert
InternW0: starts cooking before all ingredients are in. In the opening cooking task, the problem is not that it cannot perform these actions, but that it proceeds to the next step without confirming the result of the previous one. Among its 8,220 failed rounds, there are 761 errors of this type—"acting before preconditions are satisfied." The report notes that the training demonstrations contain only successful steps, so the model has never seen how to recover after an error; adding such demonstrations is a feasible direction.
InternW0's empirical diagnostic card, covering 8,220 failed episodes
GPT-6 Astra: It never picked up the jar. Of its 589 failed episodes, 55.3% had their first error in physical execution; 459 episodes were stopped by the model itself, and in 264 of those, more than a third of the decision budget remained when it stopped. A common pattern is that the motion planner rejected the pose it proposed, and it simply stopped trying. Finer-grained motion tools and retrying before stopping would help it more than increasing the budget.

GPT-6 Astra's empirical diagnostic card, covering 589 failed episodes
G0.5: Stuck at the first grasp. Its most common problem is grasping: 1,324 failed episodes had their first error in failing to grasp or failing to grasp stably. On the "build a bridge for the car" task, 202 of 300 episodes never managed to pick up a single block.
Breakpoints also show where data should be supplemented. On the "whiteboard restoration" task, 299 of G0.5's 300 episodes stopped before removing the whiteboard from its stand. Supplementing with complete long demonstrations at this point would place most of the data in the latter half that the model cannot yet reach; collecting demonstrations starting from the state where the episode broke down puts the data exactly where it is truly needed. For embodied data with high collection costs, this directly determines where the budget is spent.
The diagnostic report offers two directions for improvement. End-to-end policies start with data: supplement demonstrations by breakpoint, and add demonstrations of recovery after errors. Agents start with the toolchain: return the execution result of each step, check before continuing, and retry before stopping. These directions come from summarizing failure records, and their effectiveness remains to be verified by subsequent experiments.
The diagnostic card also has a half-joking "behavioral profile" section, borrowing the four MBTI letters: InternW0 and G0.5 are ENFP, and GPT-6 Astra is ISFJ. Beyond the joke, the three cards point to three concrete things: add state checks, improve motion tools, and practice grasping. Thus, a single score on the leaderboard can be broken down into a checklist that can be followed to make changes—this is information more directly usable than the ranking itself.
A ledger beyond the scores: time and cost
The above success rates have one precondition: during model inference, the simulation clock is paused.
Factoring in this time, the comparison between the two types of systems takes on a different appearance. The three Agents require an average of 11.0–35.6 seconds per call, while end-to-end policies need only 77–488 milliseconds per action prediction. GPT-6 Astra averages 9.2 seconds per call on the Act task, whereas the Hard-difficulty "press when the light turns on" task leaves the robot a time window of only 2.6 seconds. In terms of cost, GPT-6 Astra's average API cost is $25.13 per episode.
These two ledgers are invisible in the success rate, yet they are certain to be asked about before robots are deployed.
There is no universal model, only a model that happens to fit and keeps getting better
Back to the question in the title. More worth discussing than who wins and who loses is the direction a good embodied system should take. The team believes:
Specialization. There is no model suitable for all tasks, only a model that happens to suit the task at hand. For steps that require remembering before deciding, Agents are more suitable; for steps that a single well-practiced action can cover, end-to-end policies are faster and more economical. Even within the same VLA category, strengths differ: G0.5 excels at time-limited reactions, while X-VLA excels at resistance adjustment and cloth folding. So when choosing a model, look at the task first—whether this step needs memory, precision, or speed—and then find the one that is reliable in that dimension.
Self-evolution. A good system must be able to turn its own failures into the capabilities of the next round: which step broke and why, turned into data to supplement and tools to improve, then re-tested on the same set of problems. How far a robot is from being on the job depends on the weakest step in the pipeline, not the most beautiful single success. The recurring failure patterns in the diagnostic cards—such as proceeding without confirmation, or stopping after being rejected once—are exactly the places most suited to being fixed round after round.
This is also what Tsinghua University's mmlab@SIGS and Yuanxing Guangnian plan to do together next: turn the self-evolution loop of "diagnose—improve—re-test" into directly usable infrastructure.
Welcome to participate, and welcome to co-build and support
Bring your model to test. Participating models receive not only a ranking, but also sub-scores across three task categories—"think," "do," and "coordinate"—a record of which step each failed episode broke at, and a diagnostic card pointing out directions for improvement. Robot manufacturers, model teams, and research institutions are welcome to participate in evaluation and leaderboard submission with VLAs, WAMs, Agents, and hybrid systems.
Let's make it bigger together, and more support and sponsorship are welcome. The evaluation code, task definitions, and scorer are open-sourced on GitHub (MIT license). Researchers are welcome to collaborate with the team on new tasks, new model integration, and diagnostic methods, and to submit issues and improvements. Evaluating more models requires sustained compute and API investment; partners are welcome to support with GPU compute, API credits, and other means; sponsorship does not change any scores or rankings. Currently, the team is co-building evaluation infrastructure with Baidu AI Cloud, with more progress to come.
- Evaluation access and collaboration: [email protected]
- Evaluation documentation: https://mind2act-world.com/document.html
- Support and sponsorship: https://mind2act-world.com/support.html
Preview
Next, the team plans to continue advancing in three directions:
- Mind2Act Mini World: smaller in scale and fully open-source, enabling training and evaluation locally.
- Sandbox Agent evaluation: participants can bring their own Agent frameworks, integrate them in an isolated environment, and have a separate leaderboard.
- Extended diagnosis and improvement verification: diagnosis covers all participating systems; after model modifications, re-test on the same set of instances to verify whether the diagnostic recommendations are effective.
The leaderboard will be continuously updated with new models and new tasks.
About the authors
This research was conducted by Tsinghua University mmlab@SIGS in collaboration with the Velabot AI (velabot.ai) team, with collaborators from the University of Cambridge, the University of Oxford, Stanford University, Carnegie Mellon University, the University of Hong Kong, the Hong Kong University of Science and Technology, the Chinese University of Hong Kong, Peking University, Shanghai Jiao Tong University, Fudan University, Renmin University of China, Shenzhen University, as well as Shenpu Intelligence and Baidu AI Cloud. The project leaders are Huang Yuzhi and Wu Jie, and the corresponding authors are Chi Cheng, Jiang Jingyan, and Wang Zhi.
© THE END
For reprinting, please contact this official account for authorization
Submissions or report inquiries: [email protected]