谷歌发布 Gemini 4 Argon：ERQA 基准得分 87% 引领具身智能与机器人全栈工作流

There's a Diet Coke can behind you and another on the black table on your left.
Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.
Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.

[Attached: wrist_cam.png, obs.json]

Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.

## Execution Discipline
- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.

## Visual Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base
- `+Y` (LEFT in image): World left         | `-Y` (RIGHT): World right
- `+Z`: Height above table                | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Current 3D pinch center
- `Cyan [ + ] brackets`: Current jaw opening and closing line
- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis

## 闭环回合工作流
1. 检查当前状态并查看 `wrist_cam.png`：
   `curl -s http://127.0.0.1:8081/obs`（返回 `ee_pos`、`holding`、`status`、`workspace`、`wrist_cam_path`）。
2. 在第 1 回合预先选择 VLM 调用粒度：
   - 粗粒度（2–4 次调用）：`look_at` -> `grasp` -> `place` -> `done`
   - 抓取前悬停验证（4–7 次调用）：`look_at` -> `goto(z=0.04)`（验证青色 `[ + ]` 括号是否跨越目标）-> `grasp` -> `place_until_contact` -> `done`
   - 精细视觉伺服（8–20 次调用）：`look_at` -> `grasp` -> 预姿态 `goto` -> 闭环 `move_by` / `move_until_contact` -> `done`
3. 发送运动原语（`POST /plan`），并在每个回合后通过更新后的 `wrist_cam.png` 进行视觉验证。如果 `status == "grasp_failed"`，请调用 `robot.look_at(...)` 重新观察并恢复闭环控制。

## Code-as-Policies 原语 (`POST http://127.0.0.1:8081/plan`)
```python
robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)
robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robot.place(x=0.24, y=0.06, z=0.075)
robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)
robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)
robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)
robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)

Goal: {task_goal}

<omni_prompt>
[00:00.000] Character A's left hand rests at the side while the right hand reaches toward the target object.
[00:02.500] Character A's right fingertips push the object smoothly into the center of the table, leaving its starting area empty.
[00:05.000] Character A's right hand opens and retracts to the side so all placed objects remain in clear view of the camera.
</omni_prompt>

## **通过 Argon 解锁 Real2Sim**

### **1. 从机器人视角到仿真环境**

Argon 观察真实的机器人遥操作 episode，并构建一个数字孪生体；它获取或创建在接触灵巧手时外观和行为都与真实物体一致的仿真资产。Argon 在一个迭代式的视觉推理循环中调整资产和场景，使仿真的视觉效果和接触事件与真实录制数据相匹配。

抱歉，您的浏览器不支持播放此视频

左侧（真实视图）：同步的真实机器人摄像头画面；右侧（仿真视图）：由 Argon 重建并校准的 3D MuJoCo 场景。

*Prompt:*

```plaintext
[Attached: robot_rollout_episode]

Build a MuJoCo scene of the robot and reconstructed task objects. When each episode is replayed frame by frame, by driving the robot joints toward the logged observation.state with joint-space PD.

Iterate using two verification objectives:
(1) Visual match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras line up with, and look like, those real observations.
(2) Physical match: the simulated contact force/torque on each Sharpa fingertip, in the sensor frame, matches the logged tactile_force in timing, magnitude and direction.

在仿真开始前，你可以调整以下内容：
(1) 静态物体资产：逼真的几何形状、碰撞体、质量、惯性、摩擦系数和关节约束，外观需与真实物体高度一致，以及 t = 0 时的初始物体位姿。
(2) 离线规划的机器人指令，仅在真实指尖接近物体的帧中进行修改。每条指令必须保持在记录的遥操作轨迹的 0.55 rad（手臂关节）和 0.65 rad（手部关节）范围内。在其他所有情况下，原样重放记录的轨迹。

禁止仿真作弊：任务物体只能通过机器人手与环境之间的真实 MuJoCo 接触力（mj_step）发生运动。
t = 0 之后不允许的操作：修改 MjModel 参数；对物体施加外力或力矩；使用焊接或等式约束；施加超出关节空间 PD 控制的前馈力矩；设置或回滚物体的 qpos/qvel；禁用或排除碰撞；报告非来自 MuJoCo 求解器的接触力（例如，直接复制记录的触觉读数）。

立即以交互式 HTML 会话查看器形式呈现重建的场景和 rollout 结果，每个 episode 对应一个页面，显示：
- 底部：同步时间轴（播放、拖动、速度控制），驱动所有面板。
- 左侧面板：交互式 3D Blender 场景，展示仿真的机器人和物体，并在回放过程中显示接触点和力矢量箭头。
- 右侧面板：并排显示全部 3 个摄像头的真实视图与仿真视图。并排显示真实手指受力与仿真手指受力的可视化对比。

[附件: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json]

你正在构建一个物理上可解的 500 Hz MuJoCo 操作环境（`scene.xml`）以及 Blender 工作室渲染，以在配备 22 自由度灵巧五指手的双臂人形机器人上，忠实复现 `egocentric_human_video.mp4` 中展示的桌面任务：

1. 场景重建与相机标定：
   - 仅识别与操作相关的物体（被抓取的目标物体和接收容器或地标），并根据第一人称视角相机的内参/外参三角测量其 3D 桌面坐标。
   - 用复合凸分解 3D 网格替换粗糙的边界框，以便凹形容器（嵌套碗、凹槽木托盘、开放式料箱）支持物理内部放置。

2. 500 Hz 灵巧运动拟合压力测试：
   - 将 3D 人类手腕和手指关键点重定向到人形机器人的 7 自由度手臂和 22 自由度手部关节，并在零运动学焊接约束下，以 500 Hz（dt = 0.002s）在 MuJoCo 中步进轨迹。
   - 验证拇指与手指的对向力闭合、干净的物体提升（> 2 cm）以及在目标地标内部或旁边的稳定最终放置。

[附件: hand_drawn_throw_sketch.jpg]

将此手绘草图转换为 `mujoco_warp` 中经过物理验证的人形机器人仿真，以及光线追踪的 Blender Cycles 视频：

1. 基于草图的场景与运动学设置：
   - 解析 `hand_drawn_throw_sketch.jpg` 中的空间布局：将 76 自由度人形机器人（32 自由度身体配备两只 22 自由度五指灵巧手）放置在原点，面向一张桌子，桌上有一个由 6 个刚性目标箱子组成的 3-2-1 金字塔，右手五指手中握着一个投射球体。

2. 全身平衡与灵巧投掷合成：
   - 合成一个 50 Hz 的全身投掷轨迹（140 步，2.80 秒），涵盖四个阶段：(1) 稳定站立姿态及五指强力抓握闭合，(2) 躯干/肩部蓄力配合踝-髋部质心补偿，(3) 向前鞭打式加速，以及 (4) 在手部速度达到峰值时同步张开 22 自由度的五指，使球沿抛物线轨迹射入金字塔第二层方块的质心。
   - 在 `mujoco_warp` 接触物理引擎中验证人形机器人保持直立平衡（骨盆漂移 < 3 cm）并击倒全部 6 个目标方块。

[Attached: Aloha2_Origami_20250309.mp4]

Produce a time-aligned multi-granularity hindsight annotation JSON with four levels:
1. G1_episode_instruction: Concise imperative sentence describing the overall manipulation goal.
2. G2_subtasks: Contiguous semantic subtask phases specifying which arm(s) anchor vs. fold/rotate the object.
3. G3_atomic_primitives: Fine-grained contact-level primitives (3–15s) aligned with grasp closure, crease sweeps, release, or retry attempts.
4. quality_control_and_reweighting: Hold task_progress_pct flat during non-progressing slip/delamination retry loops and downweight them (0.15x–0.25x) for Behavior Cloning.

# Prompt 1 — Physics-Accurate MuJoCo Environment & Benchmark Generation
Make a MuJoCo environment where a dual-arm robot (two 7-DoF Franka FR3 arms with parallel-jaw grippers on a shared tabletop) is tasked to solve the classic 12-ball balance scale weight puzzle from https://www.mathsisfun.com/pool_balls_solution.html.
1. Create 12 visually indistinguishable pool balls in a 2x6 starting rack (11 standard 150 g, 1 odd ball either 270 g or 30 g).
2. Build a functional two-pan Roberval balance scale at the center of the table with parallelogram equality constraints (`mjEQ_JOINT`) so the Left (Gold) and Right (Teal) pans stay horizontal, plus Red (Heavy) and Blue (Light) submission trays.
3. Track distinct settled weighings (`weighing_count`) and reward `1.00` for isolating the true odd ball in the correct Heavy/Light tray within <= 3 weighings.

# Prompt 2 — Closed-Loop Vision-Only Solver
Implement an independent closed-loop policy relying strictly on stereo RGB camera inputs (`head_camera_left_rgb_image`, `head_camera_right_rgb_image`) and robot proprioception (`add_privileged_task_observations=False`):
1. Segment the 12 balls, Gold/Teal pans, and Red/Blue trays in both stereo views and triangulate their 3D coordinates.
2. Visually read the balance scale tilt after each weighing from stereo vertical pixel disparity (+1 = left heavier, -1 = right heavier, 0 = balanced).
3. Rotate the gripper wrist yaw by 90 degrees when grasping/placing inside the narrow scale pans to clear the raised rims, and isolate the odd ball into the Red or Blue tray within <= 3 weighings.

[Attached: umi_head.mp4, umi_left_gripper.mp4, umi_right_gripper.mp4]

Look at 3 videos of human wearing a gripper performing activities from head, left, right gripper views. Look at the videos thoroughly and give a detailed plot of progress at the task (with positive, neutral, negative slope), identifying important segments in the video and labeling the activities. Look at individual frames or subsegments of the video if necessary to get a better understanding.

[Attached: real_tabletop_photo.jpg]

你有一张来自单个顶部 RGB 摄像头的照片（`real_tabletop_photo.jpg`），拍摄的是一个双臂机器人工作区（两个配备平行夹爪的 7-DoF Franka FR3 机械臂和 4 个 RGB 摄像头），正对着一张摆放着照相馆派对道具的桌子。
1. 为所有 9 种派对道具重建程序化 3D 网格和复合 MuJoCo 碰撞基元，使其匹配真实世界的尺寸、颜色和静止姿态，包括用于柔软毛绒帽子的弹簧铰链式可变形冠部面板。
2. 定义一个包含 77 项任务的单臂和双臂课程（抓取放置、扶正、定向推动、空中交接、双臂舀取并推动以及流水线式桌面清理）。
3. 合成一个闭环 IK 轨迹生成器，覆盖 ±10 cm 的姿态随机化，在 MuJoCo 物理引擎中验证 `reward == 1.0`，并导出 4,500+ 个 episode 用于 VLA 微调。

### **2. 视觉抓取与放置的自我改进**

在一个简单的 6-DoF 机械臂上，Argon 通过试验和自我反思，迭代地自我提升抓取与放置任务的执行速度和放置精度。

抱歉，您的浏览器不支持播放此视频

SO-101 视觉引导抓取与放置 — 跨迭代 1 → 2 → 3 的硬件迭代自我改进：迭代 1 (58.9s)，迭代 2 (22.9s，放下时擦过盒子边缘)，以及迭代 3 (17.4s，正中目标放下)。

*提示词:*

```plaintext
# Stage 1 — Measure Objects & Dry-Run Plan (No Hardware Motion)
Stage 1: measure objects — find the objects and plan the whole motion. The arm does not move in this stage.

A. I put a LEGO brick and a box on the table in front of the arm, and the wooden box to one side. Find the brick and the wooden box in the image. And also in the real world. Show this in an annotated image.

B. A dry-run plan (no hardware). Write the planner to pick and place the lego block. Print the plan table, draw all poses onto the overhead image, and stop. Do not move the robot.
Report with evidence.

# Stage 2 — Pre-Pick Hover & Wrist-Camera Verification
Stage 2: Put the gripper above the brick without touching it, and verify with the wrist camera that the arm and the vision agree. Please record executions with both cameras (compressed video is OK).

# Stage 3 (Autonomous Pick-and-Place → Self-Improvement → Optimization):
1. Execute autonomously: Turn the Stage 1 dry run into a real grasp, lift, and place into the wooden box. Run it yourself subject to safety rules from AGENTS.md.

2. 自我改进：当尝试失败时，自行诊断问题。审查自己的日志以及手腕/头顶视角的视频素材，找出根本原因，在试运行中验证修复方案，然后重试。

3. 基于反馈优化：一旦成功，在安全限制内（规则 R5）将其速度至少提升 2 倍，除非我授权进一步提速或提供反馈。

[附件: paper.pdf, 可选 author_checkpoints/ 或 urdf_assets/]

从零开始在 MuJoCo WebAssembly 中复现论文 "<Paper Title & arXiv ID>"，并部署一个自包含的浏览器实验环境 + 视觉文章：
1. 在 MuJoCo XML 中精确匹配论文的仿真时间步长、执行器增益 (kp/kv)、扭矩限制和运动学约束，并使用 JavaScript 以论文规定的确切控制频率运行真实的神经网络推理或 Web Worker iLQR。
2. 构建 `index.html`（带有相机预设、实时遥测图表和扰动注入功能的 Spatial UI 模拟器）和 `scrollytelling.html`（交互式视觉文章）。
3. 运行无头 Chromium (Playwright) 验证循环，断言控制台错误数为 0，且在 300+ 帧内任务进度非零。