OpenAI Dactyl:仿真训练机械手实现零样本现实迁移

原标题:Learning dexterity
68AI 研判分

OpenAI 发布名为 Dactyl 的机器人系统,利用通用强化学习算法在纯仿真环境中训练类人机械手。该系统成功实现了从模拟到现实的零样本迁移,无需精确物理建模即可操控物体完成复杂姿态调整任务。这一突破证明了大规模仿真训练结合域随机化技术可解决真实世界中的精细操作难题。

报道正文

本文目录13 个章节

我们训练了一只类人机器人手,使其能够以空前的灵巧度操控物理物体。

我们的系统名为 Dactyl,完全在模拟环境中进行训练,并将知识迁移到现实世界,利用我们过去一年一直在研究的⁠技术和⁠方法来适应真实的物理规律。Dactyl 从零开始学习,使用的是与⁠OpenAI Five相同的通用强化学习算法和代码。我们的⁠结果表明,可以在模拟环境中训练智能体,并让它们解决现实世界的任务,而无需对世界进行物理上精确的建模。

任务

Dactyl 是一个使用⁠Shadow Dexterous Hand(新窗口打开)操控物体的系统。我们将立方体或棱柱等物体放置在手掌中,并要求 Dactyl 将其重新定位到不同的朝向;例如,旋转立方体以使新的面朝上。神经网络仅观察指尖坐标以及来自三个普通 RGB 摄像头的图像。

Image 1: Robot hand with cube

尽管第一只仿人手早在几十年前就已开发出来,但有效地利用它们来操控物体一直是机器人控制领域长期存在的挑战。与其他问题如⁠运动控制(新窗口打开)不同,采用传统机器人方法在灵巧操控方面进展缓慢,且⁠当前技术(新窗口打开)在操控现实世界物体方面的能力仍然有限。

在手部重新定向物体需要解决以下问题:

  • 在现实世界中工作。强化学习在模拟环境和视频游戏中取得了许多成功,但在现实世界中的成果相对有限。我们在物理机器人上测试了 Dactyl。

  • 高维控制。Shadow Dexterous Hand 拥有 24 个自由度,而典型机械臂仅有 7 个。

  • 噪声与部分观测。Dactyl 在物理世界中运行,因此必须处理带有噪声且存在延迟的传感器读数。当指尖传感器被其他手指或物体遮挡时,Dactyl 只能利用部分信息进行工作。物理系统的许多方面(如摩擦力和滑移)无法直接观测,必须通过推断获得。

  • 操作多个物体。Dactyl 的设计具有足够的灵活性,能够重新定向多种类型的物体。这意味着我们的方法不能采用仅适用于特定物体几何形状的策略。

我们的方法

Dactyl 完全在仿真环境中学习解决物体重定向任务,无需任何人类输入。经过这一训练阶段后,学到的策略可以直接应用于真实机器人,而无需进行任何微调。

用于机器人操作的机器学习方法面临一个两难困境。仿真机器人可以轻松提供足够的数据来训练复杂的策略,但大多数操作问题无法被精确建模,导致这些策略难以迁移到真实机器人上。即使是模拟两个物体接触时会发生什么——这是操作中最基本的问题——也是一个活跃的研究领域⁠(opens in a new window),目前尚无广泛接受的解决方案。直接在物理机器人上进行训练可以让策略从现实世界的物理规律中学习,但当前的算法需要数年的经验积累才能解决像物体重定向这样的问题。

我们的方法——域随机化(domain randomization)——在一个旨在提供多样化体验而非追求极致真实感的仿真环境中进行学习。这让我们兼得两种方法的优点:通过在仿真中学习,我们可以快速扩展规模以获取更多经验;通过降低对真实感的强调,我们可以解决那些仿真器只能近似建模的问题。

已有研究表明(由OpenAI⁠和其他机构⁠(opens in a new window)证明),域随机化可以应用于日益复杂的问题——域随机化甚至被用于训练 OpenAI Five⁠。在此,我们想验证扩大规模的域随机化是否能够解决远超当前机器人技术能力范围的任务。

Image 2: System Overview Diagram

我们使用 MuJoCo⁠(opens in a new window) 物理引擎构建了机器人系统的simulated version⁠。该仿真仅是对真实机器人的粗略近似:

  • 测量摩擦、阻尼和滚动阻力等物理属性既繁琐又困难,而且随着机器人的磨损,这些属性也会随时间变化。

  • MuJoCo 是一个rigid body⁠(opens in a new window)(刚体)模拟器,这意味着它无法模拟手指指尖处的可变形橡胶或肌腱的拉伸。

  • 我们的机器人只能通过反复接触来操作物体。然而,在仿真中准确复现接触力是出了名的困难。

虽然可以通过校准参数使仿真更贴近机器人行为,从而提升仿真的真实性,但许多效应在当前的模拟器中根本无法被准确建模。

因此,我们在一系列随机化物理和视觉属性的仿真环境分布上训练策略。随机化取值是一种自然的方式,用于表示我们对物理系统的不确定性,同时也能防止模型过拟合于单一仿真环境。如果策略能在所有仿真环境中完成任务,那么它在现实世界中成功的可能性也更高。

学习控制

通过构建支持迁移学习的仿真环境,我们将“在真实世界中控制机器人”这一问题转化为“在仿真中完成任务”,而后者非常适合采用强化学习方法来解决。尽管在仿真手中操纵物体的任务本身已经somewhat difficult⁠,但在所有随机化物理参数的组合下完成这一任务则难度显著增加。

为了在不同环境中实现泛化,策略需要能够根据环境动力学的差异采取不同的动作。由于大多数动力学参数无法仅凭单次观测推断出来,我们使用了LSTM⁠(opens in a new window)——一种具备记忆功能的神经网络——使网络能够学习环境动力学特性。在仿真中,配备 LSTM 的策略比无记忆策略多完成了约两倍的旋转次数。

Dactyl 使用Rapid⁠进行学习,这是为支持 OpenAI Five 解决 Dota 2 问题而开发的大规模近端策略优化(Proximal Policy Optimization)实现。尽管我们与 OpenAI Five 采用了不同的模型架构、环境和超参数,但使用的是相同的算法和训练代码。Rapid 利用 6144 个 CPU 核心和 8 块 GPU 来训练我们的策略,在 50 小时内收集了相当于约一百年的经验数据。

在开发和测试阶段,我们通过带有嵌入式运动追踪传感器的物体来验证控制策略,以隔离评估控制和视觉网络的性能。

学习视觉感知

Dactyl 的设计目标是能够操作任意物体,而不仅仅是那些经过特殊修改以支持追踪的物体。因此,Dactyl 使用常规的 RGB 相机图像来估计物体的位置和姿态。

我们使用卷积神经网络训练了一个姿态估计器。该神经网络接收来自机器人手部周围三个摄像头的视频流,并输出物体的估计位置和姿态。我们使用多个摄像头来解决歧义和遮挡问题。同样,我们采用域随机化技术,仅在仿真环境中使用Unity⁠(opens in a new window)游戏开发平台来训练此网络,该平台能够模拟比 Mujoco 更广泛的视觉现象。

通过结合这两个独立的网络——即根据物体姿态重新定向物体的控制网络,以及将相机图像映射到物体姿态的视觉网络——Dactyl 能够通过“看”来操作物体。

Image 3: Simulated Camera Grid

用于学习估计方块姿态的示例训练图像。

结果

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

Image 4: Grasp Types

We observed that for precision grasps, such as the Tip Pinch grasp, Dactyl uses the thumb and little finger. Humans tend to use the thumb and either the index or middle finger instead. However, the robot hand’s little finger is more flexible due to anextra degree of freedom⁠(opens in a new window), which may explain why Dactyl prefers it. This means that Dactyl can rediscover grasps found in humans, but adapt them to better fit the limitations and abilities of its own body.

Transfer performance

We tested how many rotations Dactyl could achieve before it dropped the object, timed out, or reached 50 successes. Our policies trained purely in simulation were able to successfully manipulate objects in the real world.

Image 5: Lab setup

Dactyl lab setup with Shadow Dexterous Hand, PhaseSpace motion tracking cameras, and Basler RGB cameras.

For the task of block manipulation, policies trained with randomization could achieve many more rotations than those trained without randomization, as can be seen in the results below. Also, using the control network with pose estimated from vision performs nearly as well as reading the pose directly from motion tracking sensors.

RandomizationsObject trackingMax number of successes****Median number of successes All randomizations Vision 46 11.5 All randomizations Motion tracking 50 13 No randomizations Motion tracking 6 0

Learning progress

The vast majority of training time is spent making the policy robust to different physical dynamics. Learning to rotate an object in simulation without randomizations requires about 3 years of simulated experience, while achieving similar performance in a fully randomized simulation requires about 100 years of experience.

Image 6: Learning Progress graph

Learning progress with and without randomizations over years of simulated experience.

What surprised us

  • Tactile sensing is not necessary to manipulate real-world objects.Our robot receives only the locations of the five fingertips along with the position and orientation of the cube. Although the robot hand has touch sensors on its fingertips, we didn’t need to use them. Generally, we found better performance from using a limited set of sensors that could be modeled effectively in the simulator instead of a rich sensor set with values that were hard to model.

  • Randomizations developed for one object generalize to others with similar properties.After developing our system for the problem of manipulating a block, we printed an octagonal prism, trained a new policy using its shape, and attempted to manipulate it. Somewhat to our surprise, it achieved high performance using only the randomizations we had designed for the block. By contrast, a policy that manipulated a sphere could only achieve a few successes in a row, perhaps because we had not randomized any simulation parameters that model rolling behavior.

  • With physical robots, good systems engineering is as important as good algorithms.At one point, we noticed that one engineer consistently achieved much better performance than others when running the exact same policy. We later discovered that he had a faster laptop, which hid a timing bug that reduced performance. After the bug was fixed, performance improved for the rest of the team.

What didn’t pan out

We also found to our surprise that a number of commonly employed techniques did not improve our results.

  • Decreasing reaction time did not improve performance.Conventional wisdom states that reducing the time between actions should improve performance because the changes between states are smaller and therefore easier to predict. Our current time between actions is 80ms, which is smaller than human reaction time of 150-250ms, but significantly larger than neural network computation time of roughly 25ms. Surprisingly, decreasing time between actions to 40ms required additional training time but did not noticeably improve performance in the real world. It’s possible that this rule of thumb is less applicable to neural network models than to the linear models that are in common use today.

  • Using real data to train our vision policies didn’t make a difference.In early experiments, we used a combination of simulated and real data to improve our models. The real data was gathered from trials of our policy against an object with embedded tracking markers. However, real data has significant disadvantages compared to simulated data. Position information from tracking markers has latency and measurement error. Worse, real data is easily invalidated by common configuration changes, making it a hassle to collect enough to be useful. As our methods developed, our simulator-only error improved until it matched our error from using a mixture of simulated and real data. Our final vision models were trained without real data.

This project completes a full cycle of AI development that OpenAI has been pursuing for the past two years: we’ve developeda new learning algorithm⁠, scaled it massively to solvehard simulated tasks⁠, and then applied the resulting system to the real world. Repeating this cycle atincreasing scale⁠is the primary route we are pursuing to increase the capabilities of today’s AI systems towards safe artificial general intelligence. If you’d like to be part of what comes next,we’re hiring⁠!

Authors

Josh Tobin, Bob McGrew, Wojciech Zaremba, Maciek Chociej, Szymon Sidor, Glenn Powell, Jakub Pachocki, Alex Ray, Marcin Andrychowicz, Bowen Baker, Arthur Petron, Matthias Plappert, Rafał Józefowicz, Jonas Schneider, Peter Welinder, Lilian Weng

Contributors

Larissa Schiavo, Jack Clark, Greg Brockman, Ilya Sutskever, Diane Yoon, Ben Barry

Feedback

Thanks to the following for feedback on drafts of this post: Pieter Abbeel, Tamim Asfour, Marek Cygan, Ken Goldberg, Anna Goldie, Edward Mehr, Azalia Mirhoseini, Lerrel Pinto, Aditya Ramesh & Ian Rust.

Cover Art

Ben Barry & Eric Haines