LlamaIndex 开源多模态文档检索模型 vdr-2b
LlamaIndex 团队发布并开源了 vdr-2b-multi-v1,这是目前性能最佳的多语言视觉文档检索嵌入模型。该模型基于新发布的包含 50 万高质量样本的最大开源多语言合成数据集训练,显著提升了跨语言场景下的文档理解与检索能力。
报道正文
本文目录14 个章节
视觉文档检索迈向多语言时代
发布于 2025 年 1 月 10 日 在 GitHub 上更新 点赞 78
Marco Cimolai marco llamaindex Logan Markewich cheesyFishes llamaindex
TL;DR:我们推出了
vdr-2b-multi-v1,这是目前用于视觉文档检索的最佳多语言嵌入模型。同时,我们也发布了其纯英文版本vdr-2b-v1,并开源了全新的vdr-multilingual-train数据集。该数据集包含 50 万条高质量样本,是目前规模最大的开源多语言视觉文档检索合成数据集。
隆重介绍 vdr-2b-multi-v1 (🤗),这是一款专为跨多种语言和领域进行视觉文档检索而设计的多语言嵌入模型。该模型旨在将文档页面截图编码为稠密的单向量表示,从而能够有效地搜索和查询视觉丰富的多语言文档,无需依赖任何 OCR、数据提取流程或分块处理……
vdr-2b-multi-v1 模型基于 MrLight/dse-qwen2-2b-mrl-v1,并在一个广泛的多语言查询-图像对自建数据集上进行训练。该模型是与 LlamaIndex 合作构建的,也是 mcdse-2b-v1 的下一代迭代产品。我们的 vdr-2b-multi-v1 扩展并改进了其训练所用的学习方法和策略,从而打造出一个功能更强大、性能更优越的模型。
- 基于 🇮🇹 意大利语、🇪🇸 西班牙语、🇬🇧 英语、🇫🇷 法语和 🇩🇪 德语训练: 这些语言共同构成了一个全新的大型开源多语言训练数据集,包含 50 万条高质量样本。
- 低显存占用与更快的推理速度: 在合成的视觉文档检索(ViDoRe)基准测试中,我们仅使用 768 个图像 patch 的纯英文模型表现优于使用 2560 个图像 patch 的基础模型。这使得推理速度提升了 3 倍,并显著降低了显存占用。
- 跨语言检索: 在真实场景中的表现大幅提升。例如,你可以使用意大利语查询来搜索德语文档。
- Matryoshka 表示学习(套娃表示学习): 你可以将向量维度缩减至原来的 1/3,同时仍能保留 98% 的嵌入质量。这不仅能显著提升检索速度,还能降低存储成本。
使用方法
🎲 立即体验
vdr-2b-multi-v1,可在 Hugging Face Space 上试用!
借助 SentenceTransformers 和 LlamaIndex 的直接集成,使用 vdr-2b-multi-v1 生成嵌入变得前所未有的简单。只需几行代码即可开始:
通过 LlamaIndex
pip install -U llama-index-embeddings-huggingfacefrom llama_index.embeddings.huggingface import HuggingFaceEmbedding
model = HuggingFaceEmbedding(
device="cpu",
trust_remote_code=True,
)
image_embedding = model.get_image_embedding("image.png")
query_embedding = model.get_query_embedding("Chi ha inventato Bitcoin?")from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
device="cuda",
trust_remote_code=True,
model_kwargs={
"torch_dtype": torch.bfloat16,
"device_map": "cuda:0",
"attn_implementation": "flash_attention_2"
},
)
embeddings = model.encode("image.png")训练数据集
为视觉文档检索训练高质量的单向量模型需要优质数据,但目前现成的多模态数据集非常稀缺,且缺乏多语言支持。
So, we've spent a lot of time building it from scratch. The raw dataset consists of 500k multilingual query-image samples, collected and generated from scratch using public internet PDFs. The queries associated with each image are synthetic and generated using VLMs. For comparison, our dataset has 10x more samples than the prior largest open source synthetic dataset for multimodal visual document retrieval, i.e. the scraped documents generated for the ColPali training dataset.
Data Gathering
For each language, we generate a long list of search queries covering many different topics, which are then used to search for PDFs. We use the language filtering capabilities of the search engine to scrape documents that are only in the specified language. This "search by topic" technique ensures that the model has seen a lot of diverse topics and domains, and that it performs well in real life scenarios.
The scraping process produced ~50k multilingual documents. Contrary to the method used in the previous mcdse-2b-v1 model, pages were not extracted randomly. Instead, each page of each PDF was run through a document layout analysis model to determine whether the page contained more textual or visual elements. The result is a number that classifies the page as text-only, visual-only or mixed. This labelling step was then used to sample ~100k pages, ensuring they were evenly distributed by page type.
Synthetic Generation
The queries were then generated using gemini-1.5-pro and Qwen2-VL-72B. They were tasked to come up with a specific and a general question. Only the specific question is then used to train the model, but forcing the LLM to distinguish between the two often resulted in stronger specific questions for information retrieval training.
After generation, a further cleaning step ensures that the questions are good enough for training. This includes:
- Ensuring the language is correct
- Fix formatting problems
- Remove markdown
- Ensuring that only one question is posed
- Removing grounding phrases (e.g. "according to Figure 1", "this document", ...)
Filtering and Hard-Negative Mining
This cleaning step ensures that the queries are syntactically correct and follow some strict guidelines. But it still doesn't ensure that the queries are good enough for information retrieval.
To filter out bad questions, we have embedded and indexed each broad query with the voyage-3 embedding model. For each specific question, we search the index. The query is marked as 'good' if its associated broad question appears in the top 100 results. This method removes low entropy, duplicate or too similar questions. On average, 40% of queries were removed from each language dataset.
Hard negatives were then mined using voyage-3 only on specific questions with a fixed threshold of 0.75. Experiments were also carried out using positive aware negative mining as described in nvidia/NV-Retriever-v1, but on this dataset it seems to produce too easy/distant negatives.
Download
The (vdr-multilingual-train 🤗) training dataset is now open-source and directly available on Hugging Face. The training dataset consists of 496,167 PDF pages, of which only 280,679 are associated with the filtered queries (using the method described above). The images that remain without a query are still used as hard negatives.
| Language | # filtered queries | # unfiltered queries |
|---|---|---|
| English | 53,512 | 94,225 |
| Spanish | 58,738 | 102,685 |
| Italian | 54,942 | 98,747 |
| German | 58,217 | 100,713 |
| French | 55,270 | 99,797 |
| TOTAL | 280,679 | 496,167 |
The dataset is made of 5 different subsets, each for every language. Here you can explore it directly:
Alternatively, you can download languages individually by specifying the language subset in load_dataset:
from datasets import load_datasetEvaluations
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
The model has been evaluated on the ViDoRe benchmark and on custom-built evaluation sets that allow testing its multilingual capabilities on text-only, visual-only and mixed page screenshots. The evaluation dataset is also publicly available on Hugging Face (vdr-multilingual-test 🤗).
We made sure that no page in these datasets was also present in the training set to avoid any evaluation contamination. The datasets were collected and generated using the same methods as the training dataset, but with a smaller sample size. The filtering step was all done manually: each query is evaluated, curated and improved (if necessary) to ensure high data quality.
All evaluations are performed by calculating NDCG@5 scores using 1536 dimensions vectors and an image resolution that can be represented with maximum 768 tokens.
| Avg | French (text) | French (visual) | French (mix) | |
|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 93.5 | 94.7 | 90.8 | 95.1 |
| vdr-2b-multi-v1 | 95.6 | 95.6 | 93.3 | 97.9 |
| +2.2% |
| Avg | German (text) | German (visual) | German (mix) | |
|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 93.0 | 93.4 | 90.0 | 95.5 |
| vdr-2b-multi-v1 | 96.2 | 94.8 | 95.7 | 98.1 |
| +3.4% |
| Avg | Italian (text) | Italian (visual) | Italian (mix) | |
|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 95.1 | 95.1 | 94.0 | 96.2 |
| vdr-2b-multi-v1 | 97.0 | 96.4 | 96.3 | 98.4 |
| +2% |
| Avg | Spanish (text) | Spanish (visual) | Spanish (mix) | |
|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 96.7 | 97.2 | 94.7 | 98.2 |
| vdr-2b-multi-v1 | 98.1 | 98.3 | 96.9 | 99.1 |
| +1.4% |
| Avg | English (text) | English (visual) | English (mix) | |
|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 98.0 | 98.3 | 98.5 | 97.1 |
| vdr-2b-multi-v1 | 98.1 | 97.9 | 99.1 | 97.3 |
| +0.1% |
The multilingual model outperforms the base model in every language and every page type, on average by +2.3%. On the ViDoRe benchmark, it also performs slightly better (+0.5%). Our fine-tuned vdr-2b-multi-v1 makes big leaps in performance, especially in non-English visual-only or mixed pages. See for example the +6.33% NDCG@5 improvement for German visual-only retrieval over the base model.
We also trained a version only on the English subset (vdr-2b-v1 🤗). On the full ViDoRe benchmark (evaluated with 768 image tokens), both the multilingual and English-only versions outperform the base model.
| Avg | shiftproject | government | healthcare | energy | ai | docvqa | arxivqa | tatdqa | infovqa | tabfquad | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 83.6 | 79.8 | 95.7 | 96.9 | 92.0 | 98.2 | 56.3 | 85.2 | 53.9 | 87.5 | 90.3 |
| vdr-2b-multi-v1 | 84.0 | 82.4 | 95.5 | 96.5 | 91.2 | 98.5 | 58.5 | 84.7 | 53.6 | 87.1 | 92.2 |
| vdr-2b-v1 | 84.3 | 83.4 | 96.9 | 97.2 | 92.6 | 96.8 | 57.4 | 85.1 | 54.1 | 87.9 | 91.3 |
Faster Inference
The English-only vdr-2b-v1 model also matches the performance of the base model on the ViDoRe benchmark synthetic datasets, while only using 30% of the image tokens (768 vs. 2560). This effectively results in 3x faster inference and much lower VRAM usage.
| Avg | shiftproject | government | healthcare | energy | ai | |
|---|---|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 (2560 image tokens) | 93.0 | 82 | 96 | 96.4 | 92.9 | 97.5 |
| vdr-2b-v1 (768 image tokens) | 93.4 | 83.4 | 96.9 | 97.2 | 92.6 | 96.8 |
Cross-Lingual Retrieval
Although the model was trained on each language separately, it also improves in cross-lingual retrieval. To test this ability, the German evaluation set queries were translated into Italian using DeepL. The document page screenshots remain in the original German language.
| Avg | Italian -> German (text) | Italian -> German (visual) | Italian -> German (mix) | |
|---|---|---|---|---|
| dse-qwen2-2b-mrl-v1 | 93.1 | 92.6 | 93.5 | 93.3 |
| vdr-2b-multi-v1 | 95.3 | 95.0 | 95.8 | 95.1 |
| +2.3% |
The model is significantly better across all document types, with an average improvement of +2.3%. These retrieval capabilities are essential for real-world use cases, especially in linguistically fragmented continents such as Europe. For example, it enables language-independent searches on complex multilingual sources such as European binding decisions, instruction manuals, financial asset KIDs, pharmaceutical package leaflets and many more...
MRL and Binary Embeddings
This model is trained using Matryoshka Representation Learning (MRL). The loss function used during training is calibrated to track performance across all these dimensions, leading the model to frontload the most important identifying information. This effectively allows you to shrink the embedding dimensions according to your scale and budget. To learn more about MRL, this blog post by Hugging Face explains it very well.
To test the model retrieval capabilities with different vector dimensions, evaluations are performed in the Italian->German cross-lingual benchmark.
NDCG@5 (float)
| Avg | Italian -> German (text) | Italian -> German (visual) | Italian -> German (mix) | |
|---|---|---|---|---|
| 1536 dimensions | ||||
| dse-qwen2-2b-mrl-v1 | 93.1 | 92.6 | 93.5 | 93.3 |
| vdr-2b-multi-v1 | 95.3 | 95.0 | 95.9 | 95.1 |
| +2.3% | ||||
| 1024 dimensions | ||||
| dse-qwen2-2b-mrl-v1 | 92.2 | 90.9 | 92.3 | 93.5 |
| vdr-2b-multi-v1 | 94.6 | 93.1 | 95.7 | 95.1 |
| +2.5% | ||||
| 512 dimensions | ||||
| dse-qwen2-2b-mrl-v1 | 89.8 | 87.9 | 89.4 | 92.2 |
| vdr-2b-multi-v1 | 93.0 | 91.1 | 93.4 | 94.5 |
| +3.4% |
NDCG@5 (binary)
| Avg | Italian -> German (text) | Italian -> German (visual) | Italian -> German (mix) | |
|---|---|---|---|---|
| 1536 dimensions | ||||
| dse-qwen2-2b-mrl-v1 | 89.8 | 88.2 | 90.3 | 90.8 |
| vdr-2b-multi-v1 | 92.3 | 89.6 | 94.1 | 93.3 |
| +2.8% | ||||
| 1024 dimensions | ||||
| dse-qwen2-2b-mrl-v1 | 86.7 | 84.9 | 88.2 | 86.9 |
| vdr-2b-multi-v1 | 90.8 | 87.0 | 92.6 | 92.8 |
| +4.6% | ||||
| 512 dimensions | ||||
| dse-qwen2-2b-mrl-v1 | 79.2 | 80.6 | 81.7 | 75.4 |
| vdr-2b-multi-v1 | 82.6 | 77.7 | 86.7 | 83.3 |
| +4.0% |
1024 dimension float vectors offer a very good balance between quality and size. They are ~30% smaller but still retain 99% of the retrieval performance. This is also true for the 1536 dimensions binary vectors, which have 10x fewer bytes per vector but still retain 97% of their retrieval quality. It's also interesting to see that 1536 binary vectors almost match the performance of the base model 1536 float vectors.
Conclusions and Next Steps
We believe that vdr-2b-multi-v1 and vdr-2b-v1 will prove useful to many users.
Our multilingual model is the first of its kind, it significantly improves performance in multilingual and cross-lingual scenarios, and thanks to MRL and Binary Quantization, retrieval is more efficient and faster than ever. We believe this will unlock new use-cases and opportunities, especially in linguistically fragmented continents such as Europe.
Its English-only twin represents a considerable improvement over the base model, now being able to embed documents 3x faster, with much less VRAM and with the same (or better) retrieval quality.
All this is possible thanks to the new vdr-multilingual-train dataset. With 500k high quality samples, it is the largest multilingual open source synthetic dataset for visual document retrieval.
Future work will explore how our models perform when adapted to new and specific domains. This is still in the early stages of development and more work needs to be done before results are published, but early tests already seem to suggest impressive retrieval gains with very minimal data and computational resources.
Stay tuned for future updates!



