Hugging Face Jobs 支持一键部署 vLLM 服务

原标题:Run a vLLM Server on HF Jobs in One Command
48AI 研判分

Hugging Face 推出通过 `huggingface_hub` CLI 在 HF Jobs 上一键启动私有、OpenAI 兼容的 vLLM 推理端点的新功能。该方案无需配置服务器或 Kubernetes,按秒计费,适合快速测试、评估和批量生成场景,与面向生产环境的 Inference Endpoints 形成互补。

报道正文

本文目录10 个章节

返回文章列表

一条命令在 HF Jobs 上运行 vLLM 服务器

发布于 2026 年 6 月 26 日 在 GitHub 上更新 点赞 14

Quentin Gallouédec qgallouedec

只需一条命令,即可在 Hugging Face 基础设施上启动一个私有的、兼容 OpenAI 的 LLM 端点——无需配置服务器,无需 Kubernetes,按秒计费。一旦启动,你就可以从笔记本电脑、笔记本环境或任何地方对其进行查询。

这是为测试、评估或批量生成快速部署模型的最快方式。(如果你需要的是托管式、生产就绪的服务,那正是 Inference Endpoints 的用途——关于如何选择两者的更多信息,请参阅文末的 HF Jobs 还是 Inference Endpoints。)

以下是完整的端到端流程。

前提条件

  • 一种支付方式或正值的预付费余额(Jobs 按硬件使用量每分钟计费)。
  • huggingface_hub >= 1.20.0:pip install -U "huggingface_hub>=1.20.0"。
  • 已在本地登录:hf auth login。

启动服务器

hf jobs run 相当于针对 HF 基础设施的 docker run。我们使用官方的 vllm/vllm-openai 镜像,通过 --flavor 指定 GPU,并通过 --expose 暴露 vLLM 端口:

BASH
hf jobs run --flavor a10g-large --expose 8000 --timeout 2h \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000

--expose 8000 将容器的端口路由到 HF 的公共作业代理(完整参考请参见 Serve Models 指南)。该命令会打印出你的服务器可访问的 URL:

YAML
✓ Job started
  id: 6a381ca1953ed90bfb947332
  url: https://huggingface.co/jobs/qgallouedec/6a381ca1953ed90bfb947332
Hint: Exposed ports are reachable at (requires an HF token with read access to the job):
  https://6a381ca1953ed90bfb947332--8000.hf.jobs

6a381ca1953ed90bfb947332 是你的作业 ID。请记下它,后续我们需要用到。在本文其余部分中,我们将使用 <job_id> 作为其占位符。

给它几分钟时间下载权重并启动。当日志显示 Application startup complete 时,你就已经上线了。

从任何地方进行查询

vLLM 支持 OpenAI API,每个请求只需要将你的 HF token 作为 bearer token 即可。最快的访问方式是使用 curl:

BASH
curl https://<job_id>--8000.hf.jobs/v1/chat/completions \
  -H "Authorization: Bearer $(hf auth token)" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

which returns the usual OpenAI-style JSON, with choices[0].message.content holding "Hello! How can I assist you today? 😊".

Or, from Python, point the OpenAI client at the exposed URL and pass the token as the API key:

PYTHON
from huggingface_hub import get_token
from openai import OpenAI

client = OpenAI(
    base_url="https://<job_id>--8000.hf.jobs/v1",
    api_key=get_token(),
)
resp = client.chat.completions.create(
    model="Qwen/Qwen3-4B",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)
文本
Hello! How can I assist you today? 😊

Quick health check before you start: curl https://<job_id>--8000.hf.jobs/v1/models -H "Authorization: Bearer $(hf auth token)" should list the model.

🔐 The endpoint is gated, not public. Every request must carry an HF token with read access to the job's namespace. A plain browser visit will be rejected. In effect, the jobs proxy is your API gate: access is scoped to you (and your org). That's fine for private use, but treat the URL accordingly: don't share it expecting it to be open, and don't paste your token into untrusted places. If you need finer-grained or public access, put a proper gateway in front instead. Or see HF Jobs or Inference Endpoints? below.

Clean up

Jobs are billed per second, so stop the server when you're done:

BASH
hf jobs cancel <job_id>

The --timeout you set is a safety net (it'll auto-stop), but cancelling explicitly is cheaper. An a10g-large runs at $1.50/hour — check hf jobs hardware for the full price list and pick the smallest flavor that fits your model.

Going further: bigger models

The same command scales to much larger models — pick a beefier --flavor and tell vLLM to shard the model across the GPUs with --tensor-parallel-size. For example, the 122B Qwen3.5 mixture-of-experts model on 2× H200:

BASH
hf jobs run --flavor h200x2 --expose 8000 --timeout 2h \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3.5-122B-A10B \
  --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \
  --max-model-len 32768 --max-num-seqs 256

--tensor-parallel-size should match the number of GPUs in the flavor (h200x2 → 2, h200x8 → 8). Run hf jobs hardware to see what's available and give bigger models a longer --timeout, since they take longer to download and load. For large models, H200 flavors are usually the best value.

The --max-model-len 32768 --max-num-seqs 256 flags are specific to this model: Qwen3.5-122B is a hybrid Mamba/attention architecture with a 256K-token default context, which doesn't leave enough memory for vLLM's default batch settings. Capping the context length and concurrent-sequence count keeps it within the GPUs' memory. If a model fails to start with an out-of-memory or cache-block error, dialing these two down is the first thing to try. Everything else (the exposed URL, the OpenAI client, the token auth) stays exactly the same.

Going further: Chat with it in a UI

Prefer a chat window over curl? A few lines of Gradio point at the same endpoint. Add --reasoning-parser deepseek_r1 to the vllm serve command so Qwen3's thinking comes back as a separate field (not necessary, but helpful), then run this code locally (you'll just need the job ID):

PYTHON
import gradio as gr
from gradio import ChatMessage
from huggingface_hub import get_token
from openai import OpenAI

client = OpenAI(base_url="https://<job_id>--8000.hf.jobs/v1", api_key=get_token())

def chat(message, history):
    messages = [{"role": m["role"], "content": m["content"]} for m in history if not m.get("metadata")]
    messages.append({"role": "user", "content": message})
    stream = client.chat.completions.create(model="Qwen/Qwen3-4B", messages=messages, stream=True)

thinking, answer = "", ""
    for chunk in stream:
        delta = chunk.choices[0].delta
        thinking += delta.model_extra.get("reasoning", "")
        answer += delta.content or ""
        out = []
        if thinking.strip():
            status = "done" if answer.strip() else "pending"
            out.append(ChatMessage(role="assistant", content=thinking, metadata={"title": "💭 Thinking", "status": status}))
        if answer.strip():
            out.append(ChatMessage(role="assistant", content=answer))
        yield out

gr.ChatInterface(chat).launch()

Run it, open http://127.0.0.1:7860, and chat — reasoning streams into the collapsible panel, the answer below.

Going further: SSH into the running server

Need to debug a startup failure, watch GPU memory, or tail logs interactively? You can open a shell straight into the running job. Launch it with --ssh and make sure your public key is registered at huggingface.co/settings/keys:

BASH
hf jobs run --flavor a10g-large --expose 8000 --timeout 2h --ssh \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000

then connect with the job ID:

BASH
hf jobs ssh <job_id>

You're now inside the container, where you can run nvidia-smi, inspect the process, or poke at the model directly — which makes debugging and monitoring much easier than reading logs from the outside. SSH support requires huggingface_hub >= 1.20.0.

Going further: Use it as a coding-agent backend with Pi

The same endpoint can back a terminal coding agent. Pi is a provider-agnostic agent harness. Point it at the job and you get a Read/Write/Edit/Bash agent running on your own self-hosted model.

One thing to set up first: agents drive the model through tool calls, and vLLM only accepts those if the server is launched with tool calling enabled. So relaunch with --enable-auto-tool-choice and a --tool-call-parser matching the model family (hermes for Qwen3). Agents also benefit from a stronger model, so this is a good place to bring in the bigger one:

BASH
hf jobs run --flavor h200x2 --expose 8000 --timeout 2h \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3.5-122B-A10B \
  --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \
  --max-model-len 32768 --max-num-seqs 256 \
  --reasoning-parser deepseek_r1 \
  --enable-auto-tool-choice --tool-call-parser hermes

Then add the job as a custom provider in ~/.pi/agent/models.json:

JSON
{
  "providers": {
    "hf-jobs": {
      "baseUrl": "https://<job_id>--8000.hf.jobs/v1",
      "api": "openai-completions",
      "apiKey": "!hf auth token",
      "models": [
        { "id": "Qwen/Qwen3.5-122B-A10B" }
      ]
    }
  }
}

Then launch the agent against it:

BASH
pi

The model you spun up a couple of commands ago, now driving an interactive coding agent in your terminal.

HF Jobs or Inference Endpoints?

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

HF Jobs isn't the only way to serve a model on Hugging Face. Inference Endpoints are our managed product for the same job, and which one fits depends on what you're after.

Reach for HF Jobs when you want maximum flexibility and control: it's just docker run on HF infrastructure, so you pick the image, the exact vllm serve flags, and the hardware, and you pay per second for as long as the job runs. That makes it a great fit for experiments, one-off evals, batch generation, or kicking the tires on a model before committing to anything.

Reach for Inference Endpoints when you want something more production-ready. They add the operational niceties a long-lived service needs: finer-grained access control (an endpoint can be public, protected, or private), and scale-to-zero, so you're not billed during periods of inactivity. If you're standing up a durable endpoint rather than running a job, that's the tool to grab.