Spinf-serve 开源发布 单卡批量决策推理引擎

spinf-serve

Bulk classification and decision inference on one GPU.

spinf-serve reads a piece of content once and answers many questions about it in the same pass. Each answer is a probability over the options you give, not generated text. It runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API, the same API as the hosted spinf service. Apache 2.0.

Why it exists

We built this engine for our own work: a daily market index read from millions of news articles, each article asked dozens of narrow, calibrated questions. A serving engine built for chat re-reads the content for every question and generates one token at a time, which made that job slow and expensive. spinf-serve lays every question out as a branch of one read of the content and packs many contents into one forward pass. Bulk classification is where it started and where it is strongest.

The same mechanism is what is now called a decision model: a model that returns a probability over a fixed set of answers (yes or no, a label, a level) instead of a paragraph. Any open instruction model works this way in spinf-serve, and dedicated decision models run as they are. The hosted spinf API grew out of this engine; we are publishing it because people asked to run it on their own hardware.

It is not a chat server. Nothing is generated unless you ask for it, and it is built for throughput rather than for the latency of a single request.

Install and run

Linux, an NVIDIA GPU with 24 GB and fp8 tensor cores (Ada, Hopper or Blackwell: RTX 4090 / 5090, L4, L40S, RTX 6000 Ada, H100), a driver supporting CUDA 13, Python 3.12. Ampere cards run with int8 weights instead (configuration).

pip install "spinf-serve[server]"
spinf-serve serve spinf/gemma-4-12b-fp8          # downloads the checkpoint on first start; serves on port 8000

Or the image, with the NVIDIA Container Toolkit installed (the volume keeps the model and the compile caches; arguments after the image name are those of spinf-serve serve):

docker run --gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve

Then ask. Two classifications and one extraction from a single read of the content:

curl http://localhost:8000/v1/score \
  -H "Content-Type: application/json" \
  -d '{
    "model": "spinf/gemma-4-12b-fp8",
    "messages": [
      { "role": "user",
        "content": "Customer support message:\nI was charged twice for September, please refund one of the charges." }
    ],
    "scoring": {
      "queries": [
        { "id": "topic",
          "template": "\n\nTopic (billing, technical, account, shipping, or cancellation):{?}",
          "options": [" billing", " technical", " account", " shipping", " cancellation"] },
        { "id": "refund",
          "template": "\n\nIs the customer asking for a refund?\nAnswer:{?}",
          "options": [" yes", " no"] },
        { "id": "month",
          "template": "\n\nQ: Which month was charged twice?\nA: \"{*}\"" }
      ]
    }
  }'

For each scored question the response gives every option's probability, and the same probability on empty content (real numbers from the 12B model):

{ "id": "refund", "combinations": [ { "values": {}, "options": [
    { "text": " yes", "logprob": -0.90, "p": 0.72, "floor_p": 0.60 },
    { "text": " no",  "logprob": -1.86, "p": 0.28, "floor_p": 0.40 } ],
    "residual": 0.44 } ] }

p is the probability among your options. floor_p is the same question with no content, so p - floor_p is what the content added (here topic moves " billing" from 0.47 to 0.91). residual is the probability that went to words outside your options. For month the response lists the most likely answers with their probabilities ("September" first). The same body, with an API key, runs against api.spinf.com; the OpenAI SDKs work through /v1/chat/completions with the scoring block in extra_body (quickstart).

What you get

• One read, many questions. The content is read once; every question is a branch of the same pass and costs its own tokens only. Placeholders turn one template into a grid of questions.
• Scores you can threshold. A probability per option with its floor and log-probability; a decision is a threshold you choose on your own data (calibration).
• Extraction from the same read: {*} text, {n} numbers and {$} amounts decoded with their probabilities, or free text generated into a slot.
• The same score whatever else is in the batch. With the fp8 checkpoints, a content's scores are bit-identical alone or packed with thousands of others, across calls, days and servers. The system_fingerprint says when two responses are not comparable.
• Images, audio and video frames in the content, for the models that read them.
• Prompt packs. Ready-made, calibrated question sets for a job (content moderation, triage, product listings, reviews, brand, entertainment), preloaded and named in a request (below).
• Hosted or self-hosted. A client switches between this server and api.spinf.com by changing the base URL and the key. spinf-client talks to both; the agent skills teach a coding agent the API, this server and the pack format.

（注：更多技术实现细节与完整 API 文档请查阅原项目 README）