Oxford Team Proposes Oneira: Using Coding Agents to Maintain World State Tables for Precise Object Interaction and Long-term Consistency in Video Generation

It significantly outperforms existing methods in terms of interaction accuracy and long-term consistency.

Recently, a team from the University of Oxford, Monash University, and the University of Bristol introduced Oneira, an interactive video world model that advances these models from open-ended generation to open-world interaction.

Existing video world models can continuously generate new scenes as the camera moves, but agents within them are typically limited to navigation or simple interactions. Oneira uses a World State Table maintained by a Coding Agent to record objects in the scene and their states. Whether dealing with objects present in the input image or newly generated ones appearing during camera movement, Oneira can precisely modify them and ensure that the altered states persist throughout subsequent video frames.

The first author of this paper is Xindi Yang, currently a PhD student at Monash University in Australia. His supervisors are Professor Tianjin Huang and Professor Jianfei Cai, and his primary research focus is on video generation models and world models.

• Project Page: https://madaoer.github.io/projects/oneira/
• Paper: https://arxiv.org/abs/2610.01614
• Code: https://github.com/Madaoer/Oneira

Introduction

Video world models have made significant progress in recent years, supporting navigation and simple interactions. However, users often find that when attempting to modify a specific object in the scene, the model frequently acts on the wrong target. Even if the correct object is identified for interaction, changes to the object often disappear after the camera pans away and returns.

The research team attributes this limitation to the fact that existing models do not maintain a separate record of the world's state. Information about which objects exist in the scene and which have been modified resides only within the generated frames and context. Consequently, the model struggles to distinguish between individual object instances in the scene and relies solely on text descriptions to infer the interaction target. Furthermore, interaction results exist only within the generated frames and are easily lost as the video progresses. While game engines and traditional simulators can accurately track the state of each object, they require pre-built scenes and objects, preventing expansion alongside generated content.

To overcome this limitation, the research team proposed Oneira, an interactive video world model that decouples "what happens in the world" from "how it is rendered."

The former is handled by the Coding Agent, which records the position and state of every object in the scene via a World State Table. Given an action or goal, the Coding Agent first reads the World State Table to identify the target object and plan the interaction, then writes the result back into the table. For new objects appearing during camera movement, the Coding Agent invokes 3D perception tools on the generated frames to obtain their 3D bounding boxes (BBoxes) and adds them to the table. Thus, the World State Table expands dynamically with the generated content, allowing direct interaction with newly appearing objects in the generated frames.

The latter is delegated to the Video Generation Model. The Engine first renders the World State Table into a simplified conditional video, while realistic appearances, motions, and interaction details are completed by the Video Generation Model.

Experimental results demonstrate that Oneira can precisely modify specified objects and maintain those changes over long video sequences. This achievement validates the feasibility of separately recording world states via a World State Table to make generated worlds truly interactive.

Methodology

In this work, the research team proposes a framework comprising four key components: the Coding Agent, the World State Table, the Engine, and the Video Generation Model.

Specifically, the Coding Agent invokes off-the-shelf 3D perception models to construct a World State Table from input images. The table records only interaction-relevant information, including walkable ground areas, as well as each object’s semantic label, 3D bounding box (BBox), and interaction state. Object appearances are not stored in the table; instead, they are generated by the Video Generation Model.

Interacting with one object may simultaneously change the states of other objects. Taking the example above, where the goal is to “pick up the apple in the bottom-left corner,” the Coding Agent combines the World State Table with the current frame to determine which specific apple to pick up and in which chunk to execute the action (each ~10-second video segment is divided into 8 chunks). It also infers that two other apples will roll away as a result, and includes these two apples along with the target in the plan. Subsequently, the Engine updates the World State Table based on the planned actions, marking all three apples as having left their original positions while keeping other objects unchanged. Only the results of each interaction are written to the table; how the apples move and where they end up are generated by the Video Generation Model based on motion priors. Once the other two apples come to rest, the Coding Agent re-registers their positions from the generated frames and adds any newly appearing pears to the table.

In the conditional video, each object is represented by a colored bounding box. Changes in box color indicate changes in object state, while the disappearance of a box indicates that the object has left its original position. After generating each video segment, the Coding Agent applies the same set of 3D perception tools to the new frames, adding newly appeared objects to the World State Table. Thus, interactable objects increase progressively during exploration. States of previously modified objects persist in the table, ensuring that when the camera returns to a prior location, the scene reflects those persistent changes.

Finally, the Video Generation Model generates the current video segment conditioned on the first frame, the conditional video, and textual descriptions for each chunk. In the conditional video, grid lines and colored backgrounds hint at camera motion, 3D BBoxes specify the objects involved in interactions and their timing, and textual descriptions detail the type of interaction. When the camera returns to a previously visited location, the model also references the corresponding frame as a memory frame. Since 3D BBoxes encode state solely through color—regardless of object category—the model can perform interactions even on object categories unseen during training.

Experimental Results

Instance-Level Interaction

Long-Horizon State Persistence

Quantitative Results

For evaluation, the research team constructed InteractionBench, comprising 100 interaction test cases and 50 memory test cases, and conducted both qualitative and quantitative comparisons against MiniMax-H3 Ref2VA, LingBot-World-V2, YUME 1.5, and AlayaWorld.

As shown in the figures and tables, on interaction test cases, Oneira achieves the best performance across three metrics: whether an interaction occurs, timing accuracy, and correct object targeting—outperforming the strongest baseline by 29, 19, and 29 percentage points, respectively. In scenes containing multiple identical objects, baseline models often interact with the wrong object or inadvertently modify multiple objects, whereas Oneira exclusively alters the specified target.

On memory test cases, after the camera leaves and returns to the same location, Oneira improves PSNR and SSIM by 93% and 83%, respectively, and increases state persistence rate by 18%. On VBench video quality metrics, Oneira also maintains performance at or near the best level.

Additional experimental results are presented below.

Open-Ended Interaction with Generated Content

Multi-Step Interactions

More Interaction Examples

Conclusion

Although video world models can already generate navigable scenes, they lack a separate record of the world state. Interactions often affect incorrect objects, and modified states are prone to being lost. The research team proposed Oneira, an interactive video world model that maintains a World State Table via a Coding Agent to record objects and their states within the scene. This enables precise modification of specified objects and persistent retention of the altered states. Experimental results demonstrate that Oneira significantly outperforms existing methods in terms of interaction accuracy and long-term consistency.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]