HKUST-GZ Open-Sources Motion Forcing for Physics-Aware Driving Video Generation

One image plus a few lines: AI-generated driving videos no longer "clip through" objects

Contributed by the Motion Forcing team

QbitAI | WeChat Official Account QbitAI

A car in the adjacent lane suddenly cuts in, and the ego vehicle quickly slows down to avoid it.

This driving footage was not captured by a dashcam—it was generated by AI.

For autonomous driving, the hardest gaps to fill are those rare moments that, once they occur, can directly cause danger.

Now, generating this kind of video requires only a single photo plus a few vehicle trajectories. How the leading car cuts in and how the ego vehicle slows to avoid it—the entire process follows the laws of physics, and the cars do not "clip through" each other.

△ Two cars cut in one after another, and the ego vehicle slows down to avoid each in turn

This is the new work Motion Forcing from the HKUST (Guangzhou) team.

In simple terms, its approach is to first figure out how things move, and then draw what things look like.

On real Waymo driving data, it ranks first on both motion coherence and physical plausibility, surpassing Seedance 2.0 and Wan 2.6, two leading video models.

Beyond driving scenes, it can also generate videos of dominoes falling in a chain reaction and a robotic arm pushing objects, all of which likewise follow the laws of physics. The model code and interactive demo have both been open-sourced.

△ Dominoes falling in a chain reaction △ A robotic arm pushing objects

Looking realistic is not enough—it must drive correctly and be controlled precisely

Today's video models produce increasingly refined imagery, but they easily "give themselves away" in complex scenes: a car drives right through another car, an occluded car vanishes into thin air, and post-collision reactions look wrong.

What is the root cause?

The team believes it is because the model has to juggle two things at once: how things move (where the car goes, how fast it drives, who yields to whom) and what things look like (body color, lighting and shadows, road texture).

For the model, rendering visual details well is far easier than getting the laws of physics right. When the two are tangled together, it naturally prioritizes appearance and sets physics aside. There is another layer of confusion: the camera is mounted on the ego vehicle, so when a car in the frame moves, it could be turning on its own, or both the other car and the ego vehicle could be moving.

Some existing methods try to provide a rough motion direction first and then have the model draw accordingly. But a few sparse trajectory lines are far removed from a full high-definition video, and once the scene gets complex, the model either ignores the guidance or makes physical errors.

Motion Forcing's approach is to decouple along two dimensions: first separate "how it moves" from "what it looks like," then separate ego-vehicle control from other-vehicle control, splitting generation into three layers: points, shape, and appearance.

It is like painting: first place a few points to fix the positions, then sketch the outlines and front-to-back relationships, and only then apply color.

In Motion Forcing, the "points" are the trajectories drawn by the user, the "shape" is a depth video with only near-far information and no color, and the "appearance" is the final color video.

This is the main pipeline of generation. Beyond the main pipeline, the team added two key designs: one used in training to force the model to truly learn physics, and one used in the input to tell the model how the ego vehicle drives. Let's look at each in turn.

△ Overall framework of Motion Forcing: points (trajectories) → shape (depth video) → appearance (color video)

Points: each vehicle is a circle

Each vehicle is simplified into a circle: the center represents where it is, and the size of the circle represents how far it is from the camera—the closer, the larger. The trajectories drawn by the user are simply how these circles move.

This makes control very flexible: hand-drawn arrows, a single written instruction, or even directly adjusting the speed at a certain moment can all be converted into point trajectories.

Shape: first sort out who is in front and who is behind

Next, the model does not draw a color image first; instead, it generates a depth video (a grayscale image where each pixel indicates how far it is from the camera).

A depth video has no color or texture, only spatial structure: who is in front, who is behind, who occludes whom, and how far apart two cars are. Physical relationships such as collisions and occlusions must be sorted out at this step.

Appearance: apply color last

Once the spatial structure is fixed, the model uses it as a blueprint to fill in color, lighting, and texture and generate the final video.

It is worth noting that these two steps use the same model. During training, the model randomly switches between the two tasks of "inferring structure" and "applying color," which amounts to a single model learning to be both a "physics engine" and a "renderer."

Having an extra depth-video layer in between also brings an additional benefit: users can inspect it before coloring, nudge the cars if their positions are wrong, and then decide whether to render.

Hiding the answer to force it to learn physics

Having covered the main pipeline, let's first look at the training design. Merely drawing according to trajectories is not enough. In practice, users often draw only the first few steps, or only a few of the vehicles.

So the team deliberately "hides the answer" during training: sometimes masking the latter half of a trajectory, sometimes removing an entire vehicle's trajectory, while still requiring the model to generate a complete and correct depth video.

To get it right, the model has to learn physical regularities such as inertia and occlusion on its own: for example, if only the first half of a trajectory is drawn, the model must infer where the car will go next based on how fast it was driving before; an occluded car must not vanish into thin air, and its position must line up when it reappears.

"Drawing" camera motion directly for the model to see

Now let's look at the input design. Everything above concerns how the surrounding cars move. But the ego vehicle is also driving, and since the camera is mounted on it, the entire frame turns and moves forward accordingly. When a car's position changes in the frame, it could be moving on its own, or both cars could be moving. So in addition to the trajectories of surrounding vehicles, the model must also be told how the ego vehicle will drive.

A common approach is to feed the model a series of numbers such as "turn how many degrees, travel how many meters," but it is hard for the model to map these numbers accurately onto the image.

Motion Forcing takes a different approach: it first "shifts" the depth map of the first frame along the route the ego vehicle will take, creating a preview of the camera motion, and then hands that to the model.

What the model sees is no longer abstract numbers but where each pixel should move, making turning, accelerating, and decelerating more precisely controlled. In this way, surrounding vehicles and the ego vehicle can be adjusted independently. The generated video can then achieve effects such as: for the same cut-in, the ego vehicle can slow down in a different way; for the same ego-vehicle driving segment, the leading car can cut in at a different moment or angle.

The visuals are not the most refined, but the physics are the most faithful

The team tested on 100 real driving videos from Waymo, using three yardsticks: whether the visuals look like a real video, whether the motion is coherent, and whether it conforms to physics.

The results are quite interesting.

Thanks to pretraining on massive data, Seedance 2.0 and Wan 2.6 do indeed perform better on the visual metric.

But when it comes to motion and physics, the situation flips: Motion Forcing's motion coherence score of 205.2 is clearly better than Wan 2.6's 316.2 and Seedance 2.0's 345.6 (lower is better); its physical plausibility score of 33.2 is also higher than their 31.2 and 30.5.

In other words, the large models render more beautifully, while Motion Forcing drives more "correctly". And for autonomous driving simulation, the latter often matters more.

△ Comparison results on the Waymo dataset: Motion Forcing ranks first in both motion coherence and physical plausibility

However, the team also admits that Motion Forcing still has shortcomings: in scenes with particularly dense pedestrians and bicycles, a few dots can hardly describe the motion of so many small targets; when multiple vehicles overlap severely, depth may also fail to distinguish front-to-back order. If the trajectory a user draws is too absurd and does not match reality, the generated result will likewise go wrong.

Team Introduction

The first author of this paper is Tianshuo Xu, who is currently pursuing a PhD at The Hong Kong University of Science and Technology (Guangzhou).

The corresponding author is Professor Ying-Cong Chen, from The Hong Kong University of Science and Technology (Guangzhou)/The Hong Kong University of Science and Technology. He is an Assistant Professor in the Artificial Intelligence Thrust at HKUST (Guangzhou) and holds a joint appointment as Assistant Professor in the Department of Computer Science and Engineering. He previously served as a postdoctoral researcher at MIT.

Currently, the team has open-sourced the code, model, and interactive demo: upload an image, click on a car in the frame, draw a trajectory, and a video can be generated.

Paper link: https://arxiv.org/abs/2603.10408 Code: https://github.com/Tianshuo-Xu/Motion-Forcing Model: https://huggingface.co/TSXu/MotionForcing_driving

One-click triple: "Like", "Share", "Heart"

Welcome to leave your thoughts in the comments!

— The End —

[Academic submissions] Please send an email on weekdays to: [email protected], with the subject marked [Submission], and tell us: who you are, where you are from, attach a project/homepage link to the submission content, and provide contact information.

🎓 We will reply to you (as promptly as possible) :)

🌟 Turn on the star 🌟

See frontier tech progress every day

Read the original article

Open in WeChat