World Labs Launches Frameboy for Real-Time Generated Explorable Worlds

Frameboy.

Edited by Neo, Zeynan

Today, people in the AI world are playing games.

Fei-Fei Li's startup World Labs has launched its first "app," Frameboy, which uses AI to generate first-person visuals in real time, turning generative AI into an experience you can directly explore.

Frameboy is an optimized demo of the Atlas world model, using streaming from cloud GPUs, with frame rates reaching up to 15 fps. In practice, the model's response time is only about 70ms, already approaching that of cloud gaming.

• Try it here: https://frameboy.worldlabs.ai/

It currently offers a curated set of scenes for users to explore, including various real-world landscapes, cyberpunk spaces, and the Stanford University campus. Some of these scenes start from a single image, letting Atlas infer other viewpoints and gradually build an explorable environment; others input multiple photos of the same location, and the model then generates new viewpoints between and beyond the photos.

World Labs has also integrated Frameboy directly into its beta web app Viewfinder as a streaming overlay rendering mode, allowing users to build scenes frame by frame with their own input and then explore and extend virtual worlds in real time.

Just like using a camera, you can navigate from different viewpoints and generate 3D-consistent views of the world. Each frame is generated by Atlas and contains color and depth information. Once ready, users can trigger the Frameboy overlay to play the generated frames in real time during exploration.

Atlas: Inferring a 3D world from a single photo

To understand the significance of this Frameboy real-time demo, we first need to look at what Atlas actually is.

On September 1 of this year, World Labs, founded by Fei-Fei Li, officially released Atlas, a general-purpose world model pretrained from scratch that can process text, images, video, and 3D spatial information. Unlike video models that mainly generate continuous footage, Atlas attempts to solve a more complex problem: if the camera moves to another position, what should the world in front of it look like?

World Labs showed a very intuitive example at launch.

Researchers first gave Atlas a photo of a garden and asked it to generate an aerial view of the area. Since the original photo contained only part of the garden, the model reproduced the existing scenery but had to infer the surrounding houses and environment on its own.

Then the researchers added a second photo showing a small house next to the garden. This time, Atlas's generated image included both the garden and the small house, but buildings farther away were still filled in by the model itself. Only when a third photo was added, showing the nearby main house, did the model reconstruct the entire scene more faithfully to the real environment.

This process demonstrates an important characteristic of Atlas: it can both reconstruct already-observed space from limited images and use world knowledge learned during training to infer what might exist beyond the camera's view.

World Labs calls its core mechanism "Spatial Context."

Simply put, Atlas doesn't just recognize which objects are in an image; it also associates each image with the camera's position and orientation in 3D space. When a user specifies a new viewing position, the model generates the image that should be seen from that position based on the existing spatial information.

This is also an important difference between Atlas and ordinary video generation models. The latter typically control camera movement through text descriptions such as "pan left" or "orbit around the person," whereas Atlas can directly receive precise camera positions and angles, allowing users to explicitly specify the viewpoint from which to observe the world.

In terms of technical architecture, Atlas uses a multimodal autoregressive diffusion Transformer. It combines ideas from language models and diffusion generative models: it organizes text, images, camera poses, and depth information into a unified spatial context, then progressively generates new visuals and spatial data.

In the initial demo, users could already pre-design camera paths, have Atlas generate videos up to one minute long and up to 1440p resolution, and maintain scene consistency across different shots as much as possible.

But Atlas's capabilities go beyond video generation.

It can also generate images and depth information simultaneously, and output 3D representations such as point clouds and 3D Gaussian Splatting. In other words, the model can not only present the view from a certain angle but also produce data usable in workflows such as 3D rendering and spatial reconstruction.

For example, researchers have used three to five ordinary phones and action cameras to film the same scene from different angles. Atlas reconstructs the space from these images, allowing users to switch viewing angles at the same moment the action occurs, achieving a "bullet time" effect similar to that in the movie The Matrix, without deploying a traditional large-scale multi-camera filming system.

Another application direction is robot simulation. Researchers can record a real environment with a phone video, then use Atlas to reconstruct the space and generate the visuals and depth data that a robot's onboard camera might observe as it moves along different routes. This provides a new way to build robot simulation environments from a small amount of real recordings.

Now, with the arrival of Frameboy, the way people create based on world models has changed: we can observe as we move, and decide the next step as we go. This experience has potential in film location scouting, spatial concept design, and game level ideation.

References:

https://www.worldlabs.ai/blog/atlas-realtime

© THE END

For reprints, please contact this official account for authorization.

Submissions or report inquiries: [email protected]

Open in WeChat