VAST Open-Sources EdenLive: Real-Time World Model Hits 24FPS on Single GPU

Reported by Tingyu from Aofeisi Quantum Bit | Official Account QbitAI

VAST’s world model has achieved a new milestone today.

The real-time world model EdenLive has been officially released and open-sourced.

According to VAST, EdenLive is the first to simultaneously achieve arbitrary reference input, arbitrary style generation, and long-duration continuous operation under real-time generation conditions.

Specifically, by providing simplified scenes or reference videos, along with a target initial frame and prompts, it can continuously generate detailed visuals in various styles—such as photorealistic or anime—following the layout and motion of the references.

Reference sources can vary, and generated styles can be specified, with all these changes reflected during the real-time generation process.

On a single consumer-grade Nvidia RTX 5090 GPU at 720P resolution, the 5B model achieves a generation speed exceeding 24 FPS, capable of sustaining minute-level video generation.

According to VAST, the current end-to-end latency from user input to corresponding visual output is on the order of hundreds of milliseconds.

While the visuals change continuously, interaction rules and world states are maintained by the upstream interaction system. The system receives player inputs, updates the state, and EdenLive then renders these changes in real time based on the corresponding reference frames.

Recently, AI capabilities in 3D scene construction and interactive creation have become increasingly intuitive.

GPT-6 Astra is a typical example: it can already build houses in Blender and transfer them into Unreal Engine 5, creating scenes that users can enter and roam through.

As AI gains the ability to construct scenes and design interactive gameplay, achieving fine-grained, real-time visual representation for this content becomes the next challenge to solve.

Against this backdrop, a technical approach has gradually become an industry consensus: the interaction system maintains underlying states and operational rules, while generative models handle visual rendering.

VAST clearly defined this division of labor and published results as early as early June this year.

The publicly released Project Eden research preview had already established "decoupling state maintenance from visual rendering" as its core design principle.

Chronologically, this strategic choice preceded the formation of the industry consensus by approximately 3–4 months.

Now, EdenLive is released as a phased achievement of this approach, with model weights and inference code opened up, alongside the publication of a technical report and an end-to-end demo.

Code: https://github.com/VAST-AI-Research/EdenLive

Technical Report: https://assets.vast-eden.ai/assets/edenlive/EdenLive-tech-report.pdf

Open Weights: https://huggingface.co/VAST-AI/EdenLive

Multiple Reference Inputs, Real-Time Generation in Various Styles

The core value of EdenLive lies in providing high-quality real-time rendering for simplified scenes and interactive content, while the underlying logic and states continue to be maintained by the upstream interaction system.

What exactly can it do? Let’s take a look at some demos.

Interactive Game Preview

First is the interactive game preview, which supports integration with both 2D and 3D scenes.

At this stage, EdenLive is better suited for low-frequency interactions such as scene roaming and Sokoban-style puzzles, allowing creators to view detailed rendering effects while operating game prototypes.

[Video cannot be inserted here, unfortunately... Please check the Quantum Bit official account~]

For example, in a lava platformer, a character uses a freeze gun to create footholds and jumps forward continuously.

To ensure game visuals update continuously with operations, generation speed must keep up.

According to data published by VAST, on a single consumer-grade RTX 5090 GPU at 720P, EdenLive’s 5B model achieves a generation speed exceeding 24 FPS, meaning more than 24 frames are generated per second.

Another demo—a two-player fighting game—specifically demonstrates how character actions coordinate with game states such as health bars and round timers.

Beyond visual update speed, the time it takes to see feedback after pressing a key directly impacts the interactive experience.

According to VAST, EdenLive currently achieves an end-to-end latency of hundreds of milliseconds from user input to corresponding visual output.

The system also supports minute-level continuous generation, providing longer expansion space for scene roaming and continuous operations.

Scene and Animation Preview

In addition to game interactions, EdenLive can be used for scene and animation previews. With scene layouts and camera movements already determined, different art styles can be swapped in.

For instance, making the "pelican riding a bicycle" meme everyone plays with come true...

Never seen such a lifelike pelican before.

The generated visuals not only provide detailed textures for feathers and seaside environments but also simulate realistic lighting and shadows.

A snowy driving scene built with simple models, when processed by EdenLive, presents more realistic car body textures, snow accumulation, and lighting effects.

The generation of these visuals is controlled by three types of inputs: reference images provide scene layout, camera, and motion; the target initial frame sets the overall appearance and style; and prompts supplement style and details.

Therefore, the same layout and camera movement can be used to generate versions in different art styles, allowing directors and concept designers to compare and choose.

Video Re-rendering and Style Transfer

Existing videos can also be fed into EdenLive for re-rendering, enabling real-time replacement of character appearances, visual content, and overall style.

For example, converting a live-action person against a blue background into a cartoon character, while maintaining consistency with the reference image in terms of pose and clothing color scheme.

This usage is similar to virtual streamers changing avatars, using live-action movements as references to generate alternative character appearances.

Another example involves converting a filmed house into a hand-drawn animation style.

The general layout of the house is preserved, while roofs, walls, and window frames are re-outlined with strokes, and colors and lighting become softer.

Embodied Data Augmentation

EdenLive can also be used for embodied data augmentation, transforming real human operation footage into robot perspectives and modifying objects and appearances in first-person videos.

For instance, in this video, a human hand uses a spray bottle and cloth to clean a countertop.

After processing by EdenLive, the hands are replaced with robotic arms, and the cloth changes from orange to yellow, yet the cleaning action remains consistent with the original video.

Beyond replacing the operating agent, objects, materials, and colors in first-person videos can also be adjusted.

For example, replacing human hands with metallic robotic arms and changing cart handles to silver metallic textures, while keeping grasping and pushing actions consistent with the original video.

This allows existing manipulation videos to be repurposed into versions with different appearances, expanding video samples for embodied AI research.

From simple 2D animations and 3D scenes to live-action footage and first-person perspective (FPV) manipulation clips, materials from these diverse sources can all serve as reference inputs for EdenLive.

The core capability demonstrated by this set of demos is the use of a single model to ingest multiple references, alter visual presentation while preserving existing layouts and motion, and output frames via real-time, minute-level continuous generation.

At this stage, EdenLive still has areas that require improvement.

High-frequency, intense interactions demand higher standards for generation speed and action fidelity. There is also room for further optimization in identity preservation when multiple characters appear simultaneously, fine-grained control over small objects, and maintaining visual consistency during long-duration generation.

However, VAST has already released model weights of various specifications and inference code, along with the technical report and the aforementioned end-to-end demos.

Officially, they will also open-source Game Prototype Turbo, a "Prompt to Proxy Game" Skill.

Users can leverage this tool to rapidly generate playable game prototypes. During gameplay, the interaction system continuously updates state and outputs corresponding simplified reference frames; EdenLive then renders these frames into high-fidelity visual content in real time and feeds them back to the player, forming a complete pipeline from game logic to real-time visual presentation.

Everyone is encouraged to try it out~

Deconstructing EdenLive's Technical Roadmap and Core Moats

Next, let’s examine the technical roadmap behind EdenLive.

Its core technology is "Decoupling World State from Visual Rendering," which separates how the world operates from how visuals are presented.

Consider this street fighting demo:

Character movement, attacks, and defense are controlled by the player pressing specific keys; the interaction system receives inputs, updates character positions and actions, and determines hit detection and health changes.

EdenLive generates visuals based on these changes, rendering the simple blocky character on the left as a photorealistic human on the right.

Players observe the visual feedback and continue inputting commands, allowing the game to progress through this loop.

Placing this division of labor within the full pipeline provides a more intuitive view:

In the diagram, simplified scenes or reference videos provide layout and motion, while the target initial frame and prompts specify appearance and details.

EdenLive combines these inputs to continuously output high-fidelity frames; upstream interaction systems handle game logic and state updates.

This illustrates EdenLive's role within Project Eden.

Project Eden is VAST’s long-term world model research initiative, with EdenLive representing its phased achievement in real-time generative rendering.

While the pipeline appears clear, actual implementation requires balancing several factors simultaneously: sufficient visual fidelity, motion alignment with references, generation speeds meeting interactive requirements, and sustained operation.

This involves data collection and pairing strategies, as well as collaboration between 3D, video generation, and systems engineering.

First is Data Accumulation.

To enable the model to generate photorealistic characters from simple blocks while maintaining motion consistency with reference frames, it is necessary to establish correspondences between references and generated outputs.

Scene layouts, camera movements, and object positions must be aligned within the data.

According to VAST, their data pipeline covers paired data from diverse sources, including 2D assets, simplified 3D scenes, engine-rendered outputs, and real-world videos.

From asset collection and scene construction to data pairing, these efforts collectively support EdenLive’s ability to process various references and scenarios.

The second layer of accumulation comes from Cross-Domain Team Collaboration.

EdenLive spans 3D, video generation, and systems engineering: it must handle spatial structures and motion constraints, generate detailed visual effects, and ensure the model runs continuously at sufficient speed.

VAST’s team composition covers these directions.

These requirements must be coordinated within a unified system. As image quality improves, can inference speed keep up? When appearances change, do character actions continue to align with references? These challenges require joint solutions from both model and engineering teams.

The third layer of accumulation is Strategic Judgment and Continuous R&D.

In previously published research previews for Project Eden, VAST explicitly proposed a "Decoupled" design:

Separating world state from visual generation, allowing underlying states to persist independently of camera perspectives, with a rendering module subsequently presenting them as visuals.

Around this concept, VAST outlined a three-layer architecture: continuously evolving structured state, an interface from state to observation, and generative neural rendering, responsible for state maintenance, observation transformation, and frame generation, respectively.

This strategic choice preceded industry consensus by approximately three to four months. This period provided the team with a window to accumulate data, refine models, and optimize systems along the established direction.

Following the 3D Path Toward Interactive Worlds

From this division of labor, EdenLive continues VAST’s original trajectory.

Building an interactive world requires character, prop, and scene assets, as well as clear definitions of object positions, scales, and spatial relationships. 3D provides an organized and operable foundation for these elements, participating in the representation of world state.

Video generation handles detailed visual presentation, transforming existing structures and motions into rich textures, lighting, and dynamic effects.

Only by combining both can creators adjust scenes and gameplay while receiving corresponding visuals.

This explains why VAST is advancing toward world model research.

Beyond generating assets, VAST’s existing 3D capabilities contribute to constructing the data required for world models.

In the Project Eden research introduction, VAST noted that the team leverages existing 3D capabilities to extract structural information such as depth, camera poses, and geometric motion trajectories from videos; simultaneously, they use game engines to obtain corresponding state information and rendered frames.

These efforts extend VAST’s prior accumulation in 3D models and data pipelines to address the structural, state, and visual correspondence challenges inherent in world models.

From a creative workflow perspective, this path gradually expands what users can accomplish: generating characters and props, building scenes, adding interaction rules, and assigning distinct visual styles to running content.

This aligns with VAST’s vision: Enabling everyone to freely create and experience interactive worlds.

Project Eden undertakes this long-term exploration, while EdenLive advances the real-time generative rendering component within it.

For creators, it offers a concrete entry point: integrating runnable scenes and gameplay into the model, experimenting with visual effects during actual operation, and iteratively refining their work based on these results.

One More Thing

Hold on—don’t scroll away just yet.

If you’re interested, VAST is also looking for new colleagues to join these research efforts.

The company has launched the V·STAR Top Talent Program, which accepts applications year-round from young algorithm and research talent worldwide. The program focuses on 3D generative foundation models, 3D understanding, agents, and world models.

VAST emphasizes an open and flat research environment, encouraging young researchers to propose new ideas. Promising technical concepts can be quickly initiated and validated. Even interns have the opportunity to participate in core model training and engage deeply in both model R&D and engineering practices.

The compensation package is genuinely attractive.

Whether full-time or intern, candidates have the chance to receive salaries that are more competitive than those at top-tier tech giants. For exceptionally outstanding individuals, there is no upper limit on compensation, along with long-term equity options.

If you want the freedom to define problems, drive a project from scratch, witness the explosive growth of a leading AI company from 1 to 100, and gain invaluable startup experience, consider applying through this channel:

Send your resume to [email protected]. Please include the following in the email subject line: VSTAR – Name – Graduation Date