LeCun Paper Proves Scaling Inefficiency; 100M Model Beats 72B LLM

Original title: LeCun证明了:规模从未起作用,下一token路线效率低
85AI Score

Yann LeCun's new paper mathematically proves the inefficiency of autoregressive next-token prediction and challenges the scaling hypothesis. By formalizing the Platonic Representation Hypothesis into a theorem, the study demonstrates that a 109-million-parameter world model can outperform a 72-billion-parameter LLM, providing strong theoretical backing for the JEPA architecture.

Full Report

"I trained a 100-million-parameter model that outperformed a 72-billion-parameter LLM."

Edited by: Neo, Zenan

AI pioneer Yann LeCun has long argued that "generative AI is a dead end, and world models are the true future."

He believes autoregressive generation suffers from fundamental mathematical flaws. Text cannot capture most of human knowledge, and true reasoning is not based on optimization-driven inference. Instead, he argues that intelligence should be measured by the speed at which a system learns new tasks and its ability to handle entirely new environments in a zero-shot manner.

Based on this perspective, LeCun seeks an alternative path: World Models + JEPA (Joint Embedding Predictive Architecture), a goal-driven architecture. A recent paper he submitted marks a significant step in this direction. It mathematically demonstrates the inefficiency of the next-token prediction approach and proposes his own method, where a 109-million-parameter model defeats a 72-billion-parameter LLM.

Will a vision-only model and a large text model form similar internal "concept maps"?

The Platonic Representation Hypothesis (PRH), proposed by Huh et al. in their ICML 2024 paper, suggests that as models trained on different modalities (vision, language) scale up, their internal representations become increasingly similar, eventually converging toward a shared statistical model of reality. This is akin to different people in Plato's Cave seeing the same "world of ideas." However, the original hypothesis relied only on empirical curves and intuitive arguments for an idealized contrastive learner; it provided neither a proof nor a recipe. The industry could only continue scaling up parameters.

This paper turns PRH into a theorem and provides actionable sufficient conditions.

Teaching Models to Learn the Same Scene from Different Descriptions

Every sentence read by a text model mixes two types of information: what it describes, and how it describes it. "The dog chasing the ball" belongs to the former, while choosing between "chasing" or "pursuing" belongs to the latter. If we want text models and image models to form aligned representations, we need text models to preserve more of the scene content.

But during training, how do we tell the model which content is worth preserving?

The authors' solution is to have the model simultaneously learn from different observations of the same scene. Two people describing the same photo may use different words and sentence structures, but the subjects and relationships within the scene remain largely consistent. By requiring the model to extract similar representations from these descriptions, we provide a learning signal: find the common content behind different expressions.

JEPA (Joint Embedding Predictive Architecture) offers a way to implement this idea. In this setup, an encoder first converts inputs into vectors, and training pulls the vectors of a group of inputs closer together. This means that the choice of which inputs form a group directly influences what the model is encouraged to preserve.

To help the model extract scene information from text, should training present it with different descriptions of the same scene, or different versions of the same sentence created via cropping and masking? The authors compared these two options through controlled experiments.

Assume a photo shows a dog chasing a ball on the grass. Two people write descriptions: "A dog is chasing a ball on the grass" and "A puppy on the lawn is pursuing a toy."

Group 1 ("Good Views"): Takes fragments from these two different descriptions and masks some words. For example, the model might see "dog on grass chasing [MASK]" and "puppy on lawn pursuing [MASK]." Since the sentences express things differently, forcing their vectors to be close requires the model to rely more on the underlying scene content shared by "grass/lawn" and "chasing/pursuing."

Group 2 ("Bad Views"): Takes only the first sentence and processes it into two versions, e.g., "A dog on [MASK] chasing a ball" and "A dog on grass chasing [MASK]." Both inputs retain identical phrasing like "A dog on" and "chasing," allowing the model to bring their vectors closer simply by leveraging textual similarity rather than semantic understanding.

The authors label Group 1 as "good views" and Group 2 as "bad views." Here, a "view" can be understood as an observation provided to the model during training. Both groups use the same JEPA learning mechanism, allowing us to examine whether sharing scenes versus sharing specific sentence expressions makes a difference in representation formation.

Using "good views" turns PRH into a theorem and improves the sample efficiency of the alignment process by 42 times.

This training regime also needs to solve one problem: if the model converts all text into the same vector, it technically satisfies the requirement of "close vectors." Therefore, the authors introduce constraints to prevent "representation collapse," ensuring the overall representations remain dispersed. The model must find the common content across a set of descriptions without conflating all scenes into one.

Following this line of reasoning, the connection between text and images becomes easier to understand. Wording is information unique to text, while camera angle is information unique to images. The subjects and relationships within a scene, however, may be preserved by both modalities. If models trained separately learn to extract these common structures, their internal representations have the potential to align.

Captions generated by a network that has never undergone any image-text pair training: This setup uses frozen DINOv2 and DINOv3 image encoders, applies only a single rotation, and employs a language model fine-tuned solely on text, generating new captions for each scene change.

The authors provide mathematical conditions for this. In a theoretically constructed finite world, they assume that each observation is sufficient to determine the corresponding state of the world. If training across different modalities connects the same world-state relationships, then once reduced to the world state, all models are solving the same optimization problem. Under optimal conditions—where the only difference is an overall rotation—the resulting representations differ merely by a global rotation.

It is like two people drawing maps with different orientations; if the relative positions of cities are consistent, the maps can overlap after rotation.

This leads to a comparison with next-token prediction: What learning signal does predicting subsequent words provide to the model?

To accurately continue writing, a model must utilize content as well as grammar, wording, and style. By reformulating the objective function, the authors incorporate next-token prediction into a framework similar to JEPA, pointing out that it connects the prefix and subsequent content of the same document. This connection preserves the expressive characteristics of the document itself.

Based on this, the authors propose an explanation: Good-view training directly connects different descriptions of the same scene; next-token prediction, however, needs to gradually establish connections between these descriptions from a vast array of linguistic associations. For forming representations that can align with visual models, the latter may require more data.

The experiments aim to verify how much performance gap this difference in training signals actually creates.

How much change can a different training method bring?

If the previous explanation holds, then given the same model scale and training corpus, changing the view selection should alter how closely the model’s representations align with visual representations.

The team first tested this on Flickr30k. Each image in this dataset has five human-annotated captions, which can provide different expressions of the same scene and allow multiple versions to be derived from single-sentence descriptions. This makes it suitable for comparing the differences between good views and bad views.

Researchers used the same text encoder and caption corpus to train three types of models: Good-view JEPA, Bad-view JEPA, and Next-token Prediction. They then compared these against independently trained visual encoders.

How do we judge whether they learned similar structures? One method is to compare the "neighbors" selected by each model. For the same batch of image-text samples, the visual model finds similar images for a given image, while the text model finds similar captions for a given description. If the neighbors selected by both sides frequently correspond, it indicates that they organize these scenes in a consistent manner. The mutual 10-NN metric used in the paper measures this degree of neighborhood overlap.

The results show that the text representations obtained via Good-view JEPA are more consistent with visual representations.

In the Flickr30k experiments, Good-view JEPA led in both neighborhood overlap and image-text retrieval metrics after rotational mapping.

However, Good-view training also utilized additional information: which captions belong to the same image. Could the improvement primarily stem from this layer of association?

To address this, the authors concatenated two captions from the same image and fed them to the next-token prediction model for learning. The result showed only minimal improvement. This indicates that simply providing related captions is not enough; one must also consider how to use them for training. Concatenating two captions for continuation yields different effects than directly requiring them to form similar representations.

The next question is whether this advantage can extend to larger-scale, more complex web text.

The team used Recap-DataComp data, where each image corresponds to original alternative text and two LLaVA-generated captions. These texts provide multiple views of the same image, and the text model reads the text during training.

Here, the authors further compared the amount of training samples required to reach a similar level of alignment. Using the same 109-million-parameter encoder and the same corpus source, Good-view JEPA achieved a WIT alignment score of 0.109 after processing 21.6 million training samples; Next-token Prediction achieved 0.111 after processing 900 million samples.

The scores of the two evaluated groups are close, but the sample size differs by approximately 42 times. Since Good-view training reads three texts per sample, calculating based on the actual number of text tokens read reduces the gap to about 14 times. This translates the impact of the training method into data utilization efficiency.

So, what level can Good-view JEPA reach with continued scaling?

The team scaled up the training of the 109-million-parameter model to 900 million samples using text corresponding to 940 million web images. On the WIT-1024 benchmark, using DINOv2-B as the visual reference, the average alignment score from three default-configured runs was 0.184, higher than Qwen2.5-72B's 0.154 and Llama-3-70B's 0.151.

This is the specific result behind the claim "109M parameters surpassing 72B": The textual neighborhood structure learned by the small model is more consistent with this independently trained visual model.

Control experiments at the same scale again highlight the role of the training method. Using the same encoder, same text, and same number of training samples, Next-token Prediction scored 0.111, and Bad-view JEPA scored 0.088, both lower than Good-view JEPA's 0.184. When switching the visual reference to CLIP or using the COCO test set, Good-view JEPA maintained its lead.

Performance of Good-view training in web text scaling experiments.

At this point, the experiments support the earlier explanation on three levels: changing the training approach can improve alignment, reduce the number of samples required to reach comparable performance, and enable a smaller model to achieve a representational structure closer to that of vision models than large language models.

Can this structural proximity be further translated into functionality?

The authors demonstrated this using an image captioning task. First, they trained a text decoder to reconstruct captions from vectors produced by the text JEPA. When processing images, a frozen visual encoder extracts vectors, which are then transformed into the text representation space via a linear mapping before being passed to the decoder to generate sentences.

Specifically, the mapping was fitted using paired image-text data, while the decoder was trained on text alone. By further combining components such as DINOv2 and DINOv3, the system achieved a CIDEr score of 0.838 on the COCO dataset—a metric that measures the consistency between generated captions and human-annotated descriptions.

Thus, the separately learned visual and text modules were connected: scene information provided by images is mapped into the text representation space and ultimately interpreted as a descriptive sentence.

COCO image captioning examples presented by the authors. The separately learned visual and text modules are connected via a mapping, converting scene information from images into textual descriptions.

Conclusion

For Yann LeCun’s ongoing efforts toward world models, this paper lays a foundation in the area of representation learning. It explains under what conditions different modalities might form shared structures and demonstrates how training objectives impact learning efficiency. Building on this, whether these representations can support reliable prediction and planning will determine how far this approach can go.

Yann LeCun’s research offers a promising direction for multimodal models that warrants further validation.

References:

https://x.com/ylecun/status/2116977821088096276

图片

© THE END

Please contact this official account for authorization to repost.

For submissions or media inquiries: [email protected]

Open in WeChat