Hugging Face 优化 vLLM 后端:原生速度免代码

原标题:Native-speed vLLM transformers modeling backend
68AI 研判分

Hugging Face 宣布其 transformers 库的 vLLM 后端现已实现与自定义 vLLM 实现相当甚至更快的推理速度。模型作者无需编写任何优化代码,即可直接利用现有的 transformers 实现获得极速推理体验。这一更新大幅降低了高性能推理部署的技术门槛。

报道正文

本文目录5 个章节

返回文章列表

原生速度的 vLLM transformers 建模后端

发布于 2026 年 7 月 8 日 在 GitHub 上更新 点赞 77

Harry Mellor hmellor Lysandre lysandre

TL;DR: transformers 的 vLLM 后端现在在许多 LLM 架构上的速度已经与 vLLM 自定义实现持平(甚至更快)。模型作者可以自动利用其 transformers 实现,免费获得极速的 vLLM 推理性能。

BASH

uv pip install --upgrade vllm --torch-backend auto

transformers 库已成为机器学习领域的参考建模库。它通过一致的 API 支持 450+ 种架构,其设计核心目标是确保模型实现自包含且易于理解。阅读 transformers 代码有助于贡献者了解架构的工作原理,并将其移植到其他框架中,如 vLLM、SGLang、MLX、llama.cpp 等。

我们已全面拥抱这一生态系统角色,并投入大量精力使其更加便捷。去年将 transformers 集成为 vLLM 的建模后端是这一方向上的重要一步。这使得模型作者能够在 vLLM 中运行 transformers 模型(包括 LLM 和 VLM),而无需进行任何移植工作。Transformers 提供建模代码,vLLM 则提供连续批处理(continuous batching)和自定义注意力内核(custom attention kernels)等极致优化的推理技术。

如今,这一集成变得更好了 🚀!

展示

我们将 vLLM 的 transformers 建模后端与 vLLM 手写的原生实现在三个截然不同的 Qwen3 模型上进行了正面比较:

  • 单 GPU 上的 4B 稠密模型
  • 张量并行(tensor parallelism)下的 32B 稠密模型
  • 同一 8×H100 节点上采用数据 + 专家并行(data + expert parallelism)的 235B 参数 FP8 混合专家模型(MoE)
Pre and Post PR benchmarks with trasnformers vllm backend
结果:transformers 建模后端在所有测试场景中均达到或超越了原生吞吐量。

Running any* Hugging Face model through the transformers modeling backend is a single flag — --model-impl transformers. It composes with the usual parallelism options, so nothing about your serving setup changes:

BASH

vllm serve Qwen/Qwen3-4B --model-impl transformers

vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2

vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers --data-parallel-size 8 --enable-expert-parallel

*Models that use linear attention are not currently supported, but they will be soon! Custom models where the code lives in a Hub repo are unlikely to work as they will not have been written compliantly.

How we measured

Each model is compared under three conditions that are identical in every way except the code path:

  1. native — --model-impl vllm, vLLM's hand-written model (the bar to match)
  2. after — --model-impl transformers with the PR
  3. before — --model-impl transformers without the PR

The full, reproducible runner is available as a gist: benchmark.sh

So, what's new?

The transformers modeling backend for vLLM used to focus on attention as the bottleneck for inference. By plugging vLLM’s attention implementation at runtime, we could make a transformers model run efficiently inside the vLLM engine. But there are many dimensions to deployments that only a custom port can target to extract maximum inference performance. Parallelization across GPUs, compilation, fused kernels, and many more, all contribute to leveraging your hardware to achieve ultra-fast inference.

New model integration to transformers and vLLM
A new model used to be integrated once for transformers, and once for vLLM with custom optimizations

When model authors wanted the absolute best performance, they were still writing custom vLLM implementations.

New model integrates to transformers, and is immediately available to vLLM
A new model once integrated to transformers, can now be immediately used in vLLM with native vLLM implementation speed

The latest iteration of the transformers modeling backend for vLLM dynamically applies inference specific layer fusions at runtime to match the speed of custom code implementations, for compatible architectures.

How does it work?

The transformers modeling backend for vLLM now uses torch.fx to perform static analysis on the model’s graph. This process searches for known patterns that can be optimised. After any patterns have been identified, it uses ast (abstract syntax tree) to manipulate the source code and rewrite some of the operations in place.

What can we achieve with this?

  • Fused operations that are many-to-one mapped to (ultra) optimized vLLM kernels, such as the ones used for Expert Parallelization (EP) in Mixture-of-Experts (MoE) models.
  • The main other fused operations are vLLM's MergedColumnParallelLinear and QKVParallelLinear. These blocks allow us to infer parallel plans for TP (tensor-parallel). PP (pipeline-parallel) plans can also be inferred if the decoder block list is easily identifiable.
  • The manipulated models are still fully (torch) compilable, being passed through torch.compile and CUDA Graphs, just the same as a dedicated vLLM model implementation.
  • Unlike vLLM model implementations, Transformers model implementations can be used in training. So you can use the same model code for training/evals/RL rollouts.

As shown above, this results in native vLLM inference speed for compatible models, without having to write a single line of code to optimize the model for inference.

We are in the process of writing a detailed blog post to dive deep inside these optimized inference methods and explain in detail how we manipulate the model to adapt to them.