Hugging Face 发布 SmolVLM 256M/500M:全球最小视觉语言模型
Hugging Face 推出 SmolVLM-256M 和 SmolVLM-500M,其中 256M 版本号称是全球最小的视觉语言模型。新模型在保持多模态性能的同时大幅降低资源占用,支持 Transformers、MLX 和 ONNX 直接加载,并提供了基础版与指令微调版共四个检查点。
报道正文
本文目录13 个章节
SmolVLM 进一步瘦身——推出 256M 与 500M 模型!
发布于 2025 年 1 月 23 日 在 GitHub 上更新 点赞 196
Andres Marafioti andito Miquel Farré mfarre merve merve
本文也有中文版本:简体中文。
TLDR
我们很高兴地宣布 SmolVLM 家族新增两位成员:SmolVLM-256M 和 SmolVLM-500M。没错,只有 256M 参数,使其成为目前世界上最小的视觉语言模型(Vision Language Model)!
我们在 SmolVLM 2B 积累的经验基础上,专注于提升效率、优化数据混合策略以及探索新的设计权衡。我们非常兴奋地向大家介绍这一对模型,它们以极小的体积保留了强大的多模态性能。
本次发布包含四个检查点(checkpoints):两个基础模型和两个经过指令微调的模型,参数量分别为 256M 和 500M。这些模型可以直接加载到 transformers、MLX 和 ONNX 中,并且我们提供了基于 transformers 和 WebGPU(使用 ONNX)的演示。你可以在这里找到所有模型及本次发布的演示。

目录
概述
- SmolVLM-256M – 全球最小的 VLM!
- SmolVLM-500M – 拥有五亿参数的姊妹模型,在保持极致轻量的同时,性能显著提升。
- 全新的视觉编码器选项 – 我们将 SigLIP 400M SO(用于 SmolVLM 2B 及许多其他大型 VLM)与更小的 SigLIP base patch-16/512 进行了对比。令人惊讶的是,更大的编码器仅带来微乎其微的效果提升,因此我们在这些新版本中选择了参数量为 93M 的 SigLIP base patch-16/512。
- 更高的图像分辨率 – 我们较小的视觉编码器能够以更高的分辨率处理图像(灵感来自 Apple 的 VLM 研究和 Google 的 PaliGemma)。这在几乎不增加开销的情况下,实现了更清晰的图像理解。
- 训练优化 – 一种新的分词技巧显著提升了实际基准测试的表现,尽管它在理论上使训练损失看起来更差。
目前,我们的模型规模已与 SmolLM2 系列(135M、360M、1.7B)实现对齐,因此你拥有一套完整的小型 LLM + VLM 组合可供探索。
为什么选择更小的模型?
当我们发布 SmolVLM 2B 时,社区反响热烈:该模型非常轻量、开源且许可宽松,易于集成到现有工作流中。但我们希望将这一理念推向极致,以满足那些使用受限设备、消费级笔记本电脑,甚至可能基于浏览器进行推理的用户的需求。这正是我们推出全新 256M 和 500M 模型的初衷。另一方面,对于需要处理海量数据的用户而言,这些模型的运行成本仅为 2B 模型的一小部分。
在过去一年中,我们训练了两个 80B 的 VLM,并将其压缩至 8B。随后在 SmolVLM 项目中,我们挑战了进一步将其压缩至 2B。我们从中学到的经验是:我们可以将前沿技术推向更远的边界!我们很高兴地展示,即使在 256M 和 500M 参数规模下,依然能取得出色的性能。我们的全新 256M 模型是有史以来发布的最小 VLM,但其性能却超越了仅仅 17 个月前发布的 Idefics 80B 模型。

认识这位 256M 参数的“巨人”
凭借仅 2.56 亿的参数,这款模型成为了有史以来最小的 VLM。尽管体积小巧,它却蕴含着惊人的实力。在许多多模态任务上,它都表现出色,包括:
- Captioning: Describing images or short videos.
- Document Q&A: Answering questions about PDFs or scanned text.
- Basic Visual Reasoning: Answering questions about charts or diagrams.
A Step Up: 500M
If you need more performance headroom while still keeping the memory usage low, SmolVLM-500M is our half-billion-parameter compromise. It’s significantly smaller than the previous 2B release yet manages to push scores on tasks like DocVQA and MMMU closer to the bigger models. We also found this model to be more robust to prompting, which makes it out-of-the-box better fitted for production. But both models do great when fine-tuned.
We have visualized the throughput gains across different batch sizes in below graph. Below numbers are throughput benchmarks ran on A100.

What Changed Since SmolVLM 2B?
1. Vision Encoder Choices Previously, we used the standard SigLIP 400M SO vision backbone, the same one found in many VLM architectures. For these smaller models, we experimented with two setups:
- SigLIP 400M SO: Higher capacity, great performance.
- SigLIP base patch-16/512 (93M): Much smaller, surprisingly close performance.
We found the performance gap wasn’t big enough to justify the heavier encoder for our 256M and 500M models. So, we decided to go small on the vision encoder, too. As a bonus, the smaller encoder processes images at a larger resolution, which (per research from Apple and Google) can often yield better visual understanding without ballooning parameter counts.
2. Data mixture update
Similarly to our previous release, we rely on The Cauldron and Docmatix with the addition of MathWriting to the mix.

The proportions of the datasets were adjusted to place a stronger emphasis on document understanding (41%) and image captioning (14%), while still maintaining a balanced focus on other essential areas such as visual reasoning, chart comprehension, and general instruction following. With this update the model is built on a strong document understanding basis and lets the door open to fine-tunes that will adjust its understanding of specific tasks.
3. Tokenization optimizations
We increased the pixel shuffle even more! Our new models encode images at a rate of 4096 pixels per token, compared to 1820 pixels per token in the 2B model.
To optimize the tokenization even more, we added special tokens to represent our sub-image separators in a more efficient way. This means that now instead of a string like <row_1_col_1> being mapped to 7 tokens, it is mapped to a single token. We did the same for strings up to <row_6_col_6>. This led to a sizeable improvement in the model's stability during training and quality of the results. More details were documented in this LinkedIn post.
4. Completing the SmolLM2-SmolVLM family
SmolLM2 came in three sizes: 135M, 360M, and 1.7B. With the two models we are releasing today, we now have a complete set of smaller LLM + VLM combos to play with.
Smaller Multimodal Retrieval: ColSmolVLM 256M & 500M
We also found that it's surprisingly easy to fine-tune and experiment. The team behind the ColBERT-like retrieval models have trained ColSmolVLM, delivering SOTA multimodal retrieval speeds with performance rivaling models 10x their size. SmolVLM makes it faster and cheaper to build searchable databases. We think the 256M model can become a great specialized model for many tasks. Find the link on how to use the new ColSmolVLM with the new SmolVLM models in Next Steps.

SmolDocling
We partnered with IBM to build models for Docling. Their early results with the 256M model are impressive. Below are some early examples they shared with us. Stay tuned for more updates on this!
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]


Using Smaller SmolVLM
Newer SmolVLMs are working out-of-the-box with the old SmolVLM code, so you can use transformers and MLX for inference and fine-tuning, and TRL for alignment 🚀 Moreover, this release also comes with ONNX checkpoints.
Get started with SmolVLM using transformers like below.
import torch
from transformers import AutoProcessor, AutoModelForVision2Seq
processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")
model = AutoModelForVision2Seq.from_pretrained(
"HuggingFaceTB/SmolVLM-500M-Instruct",
torch_dtype=torch.bfloat16,
_attn_implementation="flash_attention_2" if DEVICE == "cuda" else "eager",
)
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "Can you describe this image?"}
]
},
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt")
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(
generated_ids,
skip_special_tokens=True,
)Use SmolVLM with MLX by running the following CLI command:
python3 -m mlx_vlm.generate --model HuggingfaceTB/SmolVLM-500M-Instruct --max-tokens 400 --temp 0.0 --image https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vlm_example.jpg --prompt "What is in this image?"
You can play with the WebGPU demos for the SmolVLM-256M-Instruct and SmolVLM-500M-Instruct.
Find links to fine-tuning and multimodal RAG with ColSmolVLM on the Next Steps.
Citation information
You can cite us in the following way:
@article{marafioti2025smolvlm,
title={SmolVLM: Redefining small and efficient multimodal models},
author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},
journal={arXiv preprint arXiv:2504.05299},
year={2025}
}Next Steps
- We are looking forward to ways you will be using SmollerVLMs! Get started here.
- Learn more in-depth about SmolVLM here.
- Fine-tuning and QLoRA SmolVLM using transformers
- Direct Preference Optimization on SmolVLM using TRLSmolVLM using direct preference optimization (DPO with TRL on a consumer GPU)
- Smol Multimodal RAG: Building with ColSmolVLM and SmolVLM on Colab’s Free-Tier GPU
We would like to thank ViDoRe team for training ColSmolVLM: Tony Wu, Manuel Faysse, and Joshua Lochner for the ONNX conversion and WebGPU demo and Vaibhav Srivastav for his help on this release.