OpenAI 发布 Sora:视频生成模型作为世界模拟器
OpenAI 正式公布其文本到视频生成模型 Sora,提出“视频生成模型即世界模拟器”的核心论点。该模型基于扩散 Transformer 架构,在大规模视频数据上进行训练,能够生成长达一分钟的高保真、连贯且符合物理规律的视频内容。这一突破标志着多模态生成从静态图像向动态时空模拟的关键跨越。
报道正文
本文目录19 个章节
- 研究
- 产品
- 企业
- 开发者
- 公司
- 基金会(在新窗口中打开)
- 研究
- 产品
- 企业
- 开发者
- 公司
- 基金会(在新窗口中打开)
OpenAI
2024 年 2 月 15 日
视频生成模型作为世界模拟器

加载中…
分享
我们探索了在视频数据上对生成式模型进行大规模训练。具体而言,我们在具有不同时长、分辨率和宽高比的视频及图像上,联合训练了文本条件扩散模型。我们利用了一种 Transformer 架构,该架构在视频和图像潜在代码的时空补丁(spacetime patches)上运行。我们最大的模型 Sora 能够生成长达一分钟的高保真视频。我们的结果表明,扩展视频生成模型的规模是构建通用物理世界模拟器的一个有前景的路径。
加载中...
本报告重点关注:(1) 我们将所有类型的视觉数据转换为统一表示的方法,从而支持生成式模型的大规模训练;以及 (2) 对 Sora 的能力与局限性的定性评估。报告中未包含模型及实现细节。
此前已有大量研究利用多种方法对视频数据进行生成建模,包括循环神经网络,1, 2, 3 生成对抗网络,4, 5, 6, 7 自回归 Transformer,8, 9 以及扩散模型。10, 11, 12 这些工作通常聚焦于狭窄类别的视觉数据、较短的视频或固定尺寸的视频。Sora 是一个通用的视觉数据模型——它能够生成涵盖不同时长、宽高比和分辨率的视频及图像,最高可达一分钟的高清视频。
将视觉数据转化为 Patch
我们借鉴了大型语言模型(LLM)的思路,这些模型通过在互联网规模的数据上进行训练来获取通用能力。13, 14 LLM 范式的成功部分归功于 token 的使用,它优雅地统一了文本、代码、数学以及各种自然语言等多样化的模态。在本工作中,我们探讨了视觉数据的生成模型如何继承这一优势。如果说 LLM 使用文本 token,那么 Sora 则使用视觉 patches(图块)。已有研究表明,patches 是视觉数据模型的一种有效表示形式。15, 16, 17, 18 我们发现,patches 是一种高度可扩展且有效的表示方法,适用于在多种类型的视频和图像上训练生成模型。

从宏观层面来看,我们将视频转化为 patches 的过程如下:首先将视频压缩到一个低维的 latent space(潜在空间),19 然后将该表示分解为时空 patches。
视频压缩网络
我们训练了一个用于降低视觉数据维度的网络。20 该网络以原始视频作为输入,并输出一个在时间和空间上均经过压缩的 latent representation(潜在表示)。Sora 在该压缩后的潜在空间中进行训练,并随后在此空间中生成视频。我们还训练了一个相应的解码器模型,将生成的 latents 映射回像素空间。
时空潜在 patches
对于压缩后的输入视频,我们提取一系列时空 patch(图块),将其作为 Transformer 的 token。该方案同样适用于图像,因为图像本质上只是单帧的视频。基于 patch 的表示方法使 Sora 能够在具有不同分辨率、时长和宽高比的视频及图像上进行训练。在推理阶段,我们可以通过将随机初始化的 patch 排列成适当大小的网格来控制生成视频的尺寸。
面向视频生成的 Transformer 扩展
Sora 是一个扩散模型21, 22, 23, 24, 25;给定输入的含噪 patch(以及文本提示等条件信息),其训练目标是预测原始的“干净”patch。重要的是,Sora 是一个扩散 Transformer。26 Transformer 已在包括语言建模,13, 14 计算机视觉,15, 16, 17, 18 以及图像生成.27, 28, 29 在内的多个领域展现出卓越的扩展性。

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
In this work, we find that diffusion transformers scale effectively as video models as well. Below, we show a comparison of video samples with fixed seeds and inputs as training progresses. Sample quality improves markedly as training compute increases.
Loading...
Variable durations, resolutions, aspect ratios
Past approaches to image and video generation typically resize, crop or trim videos to a standard size—e.g., 4 second videos at 256x256 resolution. We find that instead training on data at its native size provides several benefits.
Sampling flexibility
Sora can sample widescreen 1920x1080p videos, vertical 1080x1920 videos and everything inbetween. This lets Sora create content for different devices directly at their native aspect ratios. It also lets us quickly prototype content at lower sizes before generating at full resolution—all with the same model.
Loading...
Improved framing and composition
We empirically find that training on videos at their native aspect ratios improves composition and framing. We compare Sora against a version of our model that crops all training videos to be square, which is common practice when training generative models. The model trained on square crops (left) sometimes generates videos where the subject is only partially in view. In comparison, videos from Sora (right) have improved framing.
Loading...
Language understanding
Training text-to-video generation systems requires a large amount of videos with corresponding text captions. We apply the re-captioning technique introduced in DALL·E 330 to videos. We first train a highly descriptive captioner model and then use it to produce text captions for all videos in our training set. We find that training on highly descriptive video captions improves text fidelity as well as the overall quality of videos.
Similar to DALL·E 3, we also leverage GPT to turn short user prompts into longer detailed captions that are sent to the video model. This enables Sora to generate high quality videos that accurately follow user prompts.
Loading...
Prompting with images and videos
All of the results above and in our landing page show text-to-video samples. But Sora can also be prompted with other inputs, such as pre-existing images or video. This capability enables Sora to perform a wide range of image and video editing tasks—creating perfectly looping video, animating static images, extending videos forwards or backwards in time, etc.
Animating DALL·E images
Sora is capable of generating videos provided an image and prompt as input. Below we show example videos generated based on DALL·E 231 and DALL·E 330 images.
Loading...
Loading...
Loading...
Loading...
Extending generated videos
Sora is also capable of extending videos, either forward or backward in time. Below are three videos that were all extended backward in time starting from a segment of a generated video. As a result, each of the three videos starts different from the others, yet all three videos lead to the same ending.
Loading...
We can use this method to extend a video both forward and backward to produce a seamless infinite loop.
Loading...
Video-to-video editing
Diffusion models have enabled a plethora of methods for editing images and videos from text prompts. Below we apply one of these methods, SDEdit,32 to Sora. This technique enables Sora to transform the styles and environments of input videos zero-shot.
Loading...
Connecting videos
We can also use Sora to gradually interpolate between two input videos, creating seamless transitions between videos with entirely different subjects and scene compositions. In the examples below, the videos in the center interpolate between the corresponding videos on the left and right.
Loading...
Image generation capabilities
Sora is also capable of generating images. We do this by arranging patches of Gaussian noise in a spatial grid with a temporal extent of one frame. The model can generate images of variable sizes—up to 2048x2048 resolution.
Loading...
Emerging simulation capabilities
We find that video models exhibit a number of interesting emergent capabilities when trained at scale. These capabilities enable Sora to simulate some aspects of people, animals and environments from the physical world. These properties emerge without any explicit inductive biases for 3D, objects, etc.—they are purely phenomena of scale.
3D consistency. Sora can generate videos with dynamic camera motion. As the camera shifts and rotates, people and scene elements move consistently through three-dimensional space.
Loading...
Long-range coherence and object permanence. A significant challenge for video generation systems has been maintaining temporal consistency when sampling long videos. We find that Sora is often, though not always, able to effectively model both short- and long-range dependencies. For example, our model can persist people, animals and objects even when they are occluded or leave the frame. Likewise, it can generate multiple shots of the same character in a single sample, maintaining their appearance throughout the video.
Loading...
Interacting with the world. Sora can sometimes simulate actions that affect the state of the world in simple ways. For example, a painter can leave new strokes along a canvas that persist over time, or a man can eat a burger and leave bite marks.
Loading...
Simulating digital worlds. Sora is also able to simulate artificial processes–one example is video games. Sora can simultaneously control the player in Minecraft with a basic policy while also rendering the world and its dynamics in high fidelity. These capabilities can be elicited zero-shot by prompting Sora with captions mentioning “Minecraft.”
Loading...
These capabilities suggest that continued scaling of video models is a promising path towards the development of highly-capable simulators of the physical and digital world, and the objects, animals and people that live within them.
Discussion
Loading...
Sora currently exhibits numerous limitations as a simulator. For example, it does not accurately model the physics of many basic interactions, like glass shattering. Other interactions, like eating food, do not always yield correct changes in object state. We enumerate other common failure modes of the model—such as incoherencies that develop in long duration samples or spontaneous appearances of objects—in our landing page.
Loading...
We believe the capabilities Sora has today demonstrate that continued scaling of video models is a promising path towards the development of capable simulators of the physical and digital world, and the objects, animals and people that live within them.
- Sora
- DALL·E
- Generative Models
- Exploration & Games
- Simulated Environments
- Language
- Learning Paradigms
References
-
1 Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhudinov. "Unsupervised learning of video representations using lstms." International conference on machine learning. PMLR, 2015.
-
2 Chiappa, Silvia, et al. "Recurrent environment simulators." arXiv preprint arXiv.02254 (2017).
-
3 Ha, David, and Jürgen Schmidhuber. "World models." arXiv preprint arXiv.10122 (2018).
-
4 Vondrick, Carl, Hamed Pirsiavash, and Antonio Torralba. "Generating videos with scene dynamics." Advances in neural information processing systems 29 (2016).
-
5 Tulyakov, Sergey, et al. "Mocogan: Decomposing motion and content for video generation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
-
6 Clark, Aidan, Jeff Donahue, and Karen Simonyan. "Adversarial video generation on complex datasets." arXiv preprint arXiv.06571 (2019).
-
7 Brooks, Tim, et al. "Generating long videos of dynamic scenes." Advances in Neural Information Processing Systems 35 (2022): 31769-31781.
-
8 Yan, Wilson, et al. "Videogpt: Video generation using vq-vae and transformers." arXiv preprint arXiv.10157 (2021).
-
9 Wu, Chenfei, et al. "Nüwa: Visual synthesis pre-training for neural visual world creation." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.
-
10 Ho, Jonathan, et al. "Imagen video: High definition video generation with diffusion models." arXiv preprint arXiv.02303 (2022).
-
11 Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
-
12 Gupta, Agrim, et al. "Photorealistic video generation with diffusion models." arXiv preprint arXiv.06662 (2023).
-
13 Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
-
14 Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.
-
15 Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv.11929 (2020).
-
16 Arnab, Anurag, et al. "Vivit: A video vision transformer." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
-
17 He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
-
18 Dehghani, Mostafa, et al. "Patch n'Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution." arXiv preprint arXiv.06304 (2023).
-
19 Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
-
20 Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv.6114 (2013).
-
21 Sohl-Dickstein, Jascha, et al. "Deep unsupervised learning using nonequilibrium thermodynamics." International conference on machine learning. PMLR, 2015.
-
22 Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.
-
23 Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." International Conference on Machine Learning. PMLR, 2021.
-
24 Dhariwal, Prafulla, and Alexander Quinn Nichol. "Diffusion Models Beat GANs on Image Synthesis." Advances in Neural Information Processing Systems. 2021.
-
25 Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in Neural Information Processing Systems 35 (2022): 26565-26577.
-
26 Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
-
27 Chen, Mark, et al. "Generative pretraining from pixels." International conference on machine learning. PMLR, 2020.
-
28 Ramesh, Aditya, et al. "Zero-shot text-to-image generation." International Conference on Machine Learning. PMLR, 2021.
-
29 Yu, Jiahui, et al. "Scaling autoregressive models for content-rich text-to-image generation." arXiv preprint arXiv.10789 2.3 (2022): 5.
-
30 Betker, James, et al. "Improving image generation with better captions." Computer Science. https://cdn.openai.com/papers/dall-e-3(opens in a new window). pdf 2.3 (2023): 8
-
31 Ramesh, Aditya, et al. "Hierarchical text-conditional image generation with clip latents." arXiv preprint arXiv.06125 1.2 (2022): 3.
-
32 Meng, Chenlin, et al. "Sdedit: Guided image synthesis and editing with stochastic differential equations." arXiv preprint arXiv.01073 (2021).
Authors
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, Aditya Ramesh
Citation
Please cite as Brooks, Peebles, et al., and use the following BibTeX for citation:https://openai.com/bibtex/videoworldsimulators2024.bib