LeCun's AMI Open-Sources H-JEPA: Hierarchical World Model Achieves 73% Success in Long-Horizon Planning

The idea has been brewing since his time at Meta four years ago.

henry reporting from Aofeisi

QbitAI | WeChat Official Account: QbitAI

AMI (Advanced Machine Intelligence), Yann LeCun’s world model startup, has just quietly delivered its first results.

In the latest paper H-JEPA, researchers from AMI, in collaboration with New York University, INRIA Paris, and Brown University, have introduced a hierarchical world model for visual planning.

H-JEPA stacks multiple JEPA models layer by layer, allowing each layer to predict states over different time horizons within its own representation space.

During planning, higher layers formulate coarse plans and pass predicted intermediate states as subgoals to lower layers. The lower layers then progressively refine these goals, ultimately generating executable actions.

Through this division of labor, the world model can both plan for long-term objectives and handle immediate actions.

In the Visual AntMaze simulation task, the three-layer H-JEPA achieved a success rate of 73%, compared to 18% for single-layer models.

Additionally, across various simulation tasks, the hierarchical model demonstrated superior performance with less computational overhead for planning.

Beyond delivering tangible benchmark results, H-JEPA continues the open-source path that LeCun emphasized from the very beginning of the startup.

The team simultaneously released the code and pre-trained model weights, allowing researchers to directly load the models and reproduce the planning experiments.

Why Do World Models Need Hierarchical Planning?

LeCun has actually been advocating for this concept for quite some time.

As early as 2022, while serving as Chief AI Scientist at Meta, he proposed the idea of hierarchical JEPA:

Enable models to make predictions at different levels of abstraction and time scales, then use hierarchical planning to translate complex goals into concrete actions.

This might sound abstract at first glance, but it becomes easy to understand through an example LeCun often cites.

Imagine you are sitting in your office at New York University, preparing to travel to Paris. When planning the entire trip, your focus is on the destination, flight schedules, and timing.

Once the itinerary is set, you begin arranging how to leave the office, go downstairs, and take a taxi to the airport.

When you are actually stepping out of the office, your attention shifts to the ground immediately in front of you, the stairs, where to place your next step, and which foot to move first.

**

These planning levels cover different information scopes. At the high level, the model focuses on long-term goals; at the low level, it handles immediate motor control.

Specifically, the state encoder is responsible for extracting representations of the current environment. The lowest layer directly reads images, while higher layers further abstract these lower-level representations, preserving information that is more critical at their respective time scales.

The action encoder describes the changes induced by actions. Lower layers represent specific actions, whereas higher layers compress sequences of continuous actions into coarser-grained changes.

The predictor combines the current state and action to predict future states. Different layers cover different time horizons; in the paper’s experiments, one step at an adjacent higher layer corresponds to two steps of prediction at the lower layer.

Regarding model training, all layers are learned jointly through end-to-end training, with different levels connected via subgoals.

The intermediate states predicted by higher layers serve as subgoals for lower layers. The lower layers map their own predicted state transitions into the higher-layer representation space, compare them against the subgoals specified by the higher layers, and adjust their actions accordingly.

However, when training world models, a classic problem must be addressed: representation collapse.

If the encoder compresses all inputs into the same vector, the predictor can still achieve very low prediction error by simply outputting this vector, but the model effectively learns no useful information.

To prevent this, H-JEPA adopts SIGReg, previously proposed by LeCun’s team in LeJEPA, which constrains the distribution of representations to avoid collapsing distinct states into identical vectors.

This year, LeWorldModel applied this methodology to world model training. H-JEPA uses it as its foundational layer, stacking additional JEPA modules on top to form a structure characterized by joint training and top-down planning.

In experiments, researchers observed distinct information trade-offs across different layers. In maze tasks, higher layers gradually de-emphasized leg posture while retaining the robot’s positional information.

However, hierarchical structures have applicability conditions. A two-layer model performed well on the Push-T task, but performance declined when extended to three or four layers.

The paper suggests that shorter task trajectories may limit the amount of training data available for higher layers.

LeCun Remains True to His Vision

Overall, the design of H-JEPA continues JEPA’s fundamental premise: performing predictions within the representation space.

JEPA stands for "Joint Embedding Predictive Architecture." While LLMs typically learn to generate content by predicting the next token, visual JEPAs first encode images and videos into internal representations, then predict the representations of target parts, without needing to generate individual image pixels.

In world models like H-JEPA, the model also incorporates actions to predict representations of future states. Planners use these predictions to compare the consequences of different actions and identify pathways to goals.

LeCun has consistently argued that understanding the world, predicting the outcomes of actions, and planning are essential capabilities for achieving human-level intelligence. His assessment of LLMs has always centered on this point.

In September of this year, during his talk at ECCV in Malmö, Sweden, his slides featured the familiar advice once again:

If you are interested in human-level AI, do not study LLMs.

After this slide circulated on social media, some questioned whether it was fake. LeCun replied:

It’s real.

He subsequently clarified that LLM-based AI tools are extremely useful and widely used. However, he believes that LLMs alone cannot lead to human-level intelligence.

Today, this research philosophy has become AMI’s startup direction.

In March of this year, AMI announced it had completed a $1.03 billion funding round at a$3.5 billion pre-money valuation.

The company is chaired by LeCun, with Alexandre LeBrun serving as CEO, Xie Saining as Chief Scientist, Pascale Fung leading Research & Innovation, and Michael Rabbat heading World Models. Rabbat is also one of the authors of the H-JEPA paper.

That said, while research achievements are valuable, external expectations for AMI still heavily focus on products.

At least based on the current official website, the company primarily showcases its technical directions, team, and job openings, with no public-facing product entry yet visible.

Meanwhile, World Labs, another company betting on world models, has reached a new turning point.

On September 28, AMD announced an all-stock acquisition agreement valued at approximately $8.2 billion with World Labs, founded by Fei-Fei Li.

One can observe that while one side is moving toward integration with chip companies, the other continues to advance along its own research trajectory.

However, with Fei-Fei Li’s world model startup achieving a milestone breakthrough, is it merely a matter of time before Yann LeCun delivers his major results?

References [1]https://venturebeat.com/technology/h-jepa-teaches-world-models-to-plan-at-multiple-levels-of-abstraction [2]https://arxiv.org/pdf/2610.06805

Like, Share, and Save 「👍 Like」「🔁 Share」「❤️ Save」

We welcome your thoughts in the comments!

— End —

🌟 Star Us 🌟

Daily updates on cutting-edge tech developments