RAVEL Boosts Multimodal Vision Encoder Selection Accuracy Without Training

How to Select the Right Vision Encoder for a Target LLM Before Full MLLM Training?

The vision encoder is a core component of Multimodal Large Language Models (MLLMs). Even when using the same Large Language Model (LLM), different vision encoders can lead to significantly different downstream performance. The most direct way to find the optimal vision encoder is to integrate each candidate into the LLM, perform cross-modal alignment training, and then evaluate the results. However, this process requires substantial computational resources. Therefore, accurately predicting the downstream performance of a vision encoder before full training has become an important research problem.

Existing evaluation methods fall into two main categories: unimodal methods focus primarily on the intrinsic capabilities of the vision encoder itself, such as image classification performance; cross-modal methods further consider the target LLM by measuring the relationship between visual and linguistic representations to assess how well different vision encoders align with the LLM. Intuitively, cross-modal methods account for the matching relationship between vision and language and should provide more accurate predictions.

As the diversity of candidate vision encoders increases, the performance of traditional cross-modal evaluation methods drops significantly, sometimes even falling below that of unimodal methods. In contrast, RAVEL maintains a significant advantage across different candidate pools, demonstrating greater stability and reliability.

However, researchers discovered through training and evaluating 210 MLLMs that the advantages of existing cross-modal methods are not stable. In smaller candidate pools with limited variety, cross-modal methods typically outperform unimodal methods. But as the number and diversity of candidate vision encoders increase, the evaluation performance of cross-modal methods declines sharply, often becoming weaker than unimodal methods. This finding indicates that existing cross-modal evaluation methods still have clear limitations in large-scale, diverse vision encoder selection tasks.

To address this issue, the team proposed RAVEL, a training-free evaluation method. RAVEL retains the fundamental approach of cross-modal evaluation but introduces two simple technical improvements: Stabilized PCA Whitening, which reduces interference from dominant directions in the feature space; and Patch-level Similarity Scoring, which leverages local image features to capture detailed information in visual representations more effectively. These improvements enhance the accuracy and stability of cross-modal evaluation, allowing RAVEL to maintain strong performance across candidate pools of varying scales and diversity.

Experimental results show that on three different LLMs, the correlation between RAVEL's evaluation scores and actual downstream performance is significantly higher than previous work. Without additional training, RAVEL achieves an average Spearman correlation coefficient of 0.85, surpassing the training-required Alignment Probing method (+36.1%) and significantly outperforming prior training-free methods GW (+93.5%) and MutualNN (+101.1%).

RAVEL demonstrates that without additional training, simply utilizing existing visual and linguistic representations more rationally can significantly improve the accuracy of vision encoder evaluation, offering an efficient and reliable new method for component selection in MLLMs.

• Paper Title: A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models
• Paper Link: https://arxiv.org/abs/2610.05413
• Open Source Code: https://github.com/JuntaoTang/MLLM-VisionEncoder-Eval
• Model Weights: https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo

The team also open-sourced the weights of the 210 MLLMs trained during these experiments, covering various combinations of vision encoders and LLMs. Researchers can directly use these models for studies on vision encoder selection, cross-modal alignment, and representation analysis, avoiding the need to repeat large-scale multimodal training.

I. Why Can't We Just Look at the Vision Encoder's Own Performance When Choosing One?

The most intuitive approach is to look directly at how strong the vision encoder is on its own, for example, its performance on tasks like image classification. However, in MLLMs, the vision encoder must also work in conjunction with the language model backbone: the visual information it extracts ultimately needs to be understood and utilized by the LLM backbone. Therefore, excellent standalone performance of a vision encoder does not guarantee the best performance when combined with a specific LLM.

Experimental results confirm this: when the LLM changes, the best-performing vision encoder also changes. This means that relying solely on pure visual metrics like linear probing makes it difficult to determine which vision encoder is truly suitable for the current LLM.

Figure 1: Vision encoder performance under different LLMs. Stars represent the best-performing vision encoder for each LLM.

II. Can Existing Cross-Modal Alignment Metrics Be Directly Used for Vision Encoder Selection?

Some studies have attempted to leverage both visual and linguistic information simultaneously, predicting the downstream performance of visual encoders by comparing representations across these two modalities. However, are such cross-modal metrics still reliable when applied to a larger and more complex pool of candidates?

To address this question, the team expanded the candidate pool to include visual encoders with varying architectures, pre-training methods, and input resolutions, covering both continuous visual encoders and discrete visual tokenizers. The experiments revealed that while existing cross-modal metrics outperform purely visual metrics in smaller candidate pools, their predictive correlation drops significantly as the scope expands; some methods even perform worse than purely visual metrics.

Figure 2: As the diversity of candidate visual encoders increases, the performance of traditional cross-modal evaluation methods declines significantly, sometimes falling below that of single-modality methods. In contrast, RAVEL maintains a distinct advantage across different candidate pools, demonstrating superior stability and reliability.

III. RAVEL: Do Image-Text Neighborhood Relationships Align?

RAVEL employs an evaluation framework based on cross-modal neighborhood consistency. Its core intuition is that if a candidate visual encoder and the target LLM form similar semantic structures for the same batch of samples, the neighborhood relationships between images and texts within their respective representation spaces should also be consistent.

Specifically, given a batch of images and their corresponding texts, RAVEL uses the candidate visual encoder and the target LLM to extract features separately, then retrieves neighbors within each representation space: searching for similar images on the image side and similar texts on the text side. For a pair of image-text samples, if the neighbors found on the image side correspond to texts that also appear among the neighbors on the text side, it indicates that both representation spaces have identified the same semantic neighbors. The greater the overlap in these neighborhoods, the more consistent the local structure of the visual and linguistic representations.

RAVEL aggregates the neighborhood consistency across all samples to score the candidate visual encoders. Since it only compares neighborhood relationships within their respective representation spaces, it does not require visual and text features to have the same dimensionality, nor does it need any training or downstream performance labels.

Figure 3: The evaluation framework for cross-modal neighborhood consistency.

Within this framework, to ensure that neighborhood relationships more accurately reflect semantic structure, RAVEL incorporates two additional design elements:

PCA Whitening with Stabilization Term: Mitigates the impact of uneven feature variance on similarity calculations.

Local Patch Matching: Preserves local semantic information that is often lost during global pooling.

IV. Experiments with 210 MLLMs: Is the Predicted Ranking Reliable?

The team combined 70 visual encoder configurations with Qwen3-1.7B, Qwen2.5-1.5B, and SmolLM2-1.7B, training a total of 210 Multimodal Large Language Models (MLLMs). All models adopted a unified LLaVA-style two-stage training pipeline and were evaluated on 11 downstream benchmarks, covering tasks such as Visual Question Answering (VQA), multimodal reasoning, and Optical Character Recognition (OCR).

Based on this setup, the team compared RAVEL against nine existing evaluation metrics, including purely visual metrics and cross-modal methods that either require or do not require additional training. To ensure fairness, all methods were compared under similar computational budgets (FLOPs). The results showed that across all three LLMs, RAVEL achieved the highest Spearman and Pearson correlation coefficients.

Table 1: Comparison of predictive capabilities across different evaluation metrics. ρ and r denote Spearman and Pearson correlation coefficients, respectively.

V. Why Does RAVEL Work?

Ablation studies indicate that RAVEL's improvements stem primarily from two designs: PCA whitening significantly enhances the quality of neighborhood relationships, while local matching further boosts predictive capability on top of this foundation.

Table 2: Results of the ablation study.

1) PCA Whitening with Stabilization Term

Different models often exhibit distinct feature distributions in their representation spaces: a few high-variance directions may dominate similarity calculations, and strong correlations between dimensions can cause retrieved neighbors to inaccurately reflect semantic relationships between samples. PCA whitening can mitigate this issue, but conventional whitening tends to excessively amplify low-variance directions. RAVEL addresses this by introducing a stabilization term to suppress such amplification, followed by L2 normalization after whitening. Further analysis indicates that the performance gains primarily stem from the stabilization term, with L2 normalization providing additional improvements.

Figure 4: Impact of the stabilization term and normalization on prediction performance. Baseline represents conventional PCA whitening.

2. Local Patch Matching

Two images may contain similar objects but differ in viewing angle, object position, and background. Global pooling mixes these factors, potentially diluting the similarity of the objects themselves with irrelevant background information.

To address this, RAVEL performs matching directly between local image patches: each patch searches for its most similar region in the other image, and bidirectional matching results are aggregated without requiring pre-established one-to-one correspondences. Visualization results show that high-similarity matches concentrate on semantically related regions across the two images, while matches in unrelated areas receive lower scores.

Figure 5: Red lines on the left connect highly similar local regions; brighter colors on the right indicate stronger matches.

VI. Conclusion

This work systematically investigates a practical yet under-explored question: how to select an appropriate visual encoder for a target LLM before fully training an MLLM? By training and evaluating 210 MLLMs, the team found that existing cross-modal metrics do not always maintain their advantage in more complex candidate pools. Based on these findings, they proposed RAVEL, which more reliably predicts the downstream performance of visual encoders by measuring neighbor consistency between visual and language representations, combined with stabilized PCA whitening and local matching. The team has also open-sourced the code and the weights of the 210 trained MLLMs, providing reusable baselines and model resources for visual encoder selection, cross-modal alignment, and multimodal representation research.

© THE END

Please contact this official account for authorization to repost.

For submissions or media inquiries: [email protected]