IBM 开源 Granite 4.2:详解推理模型构建全流程

原标题:Granite 4.2 LLMs: How They're Built
68AI 研判分

IBM 发布并开源了首个密集解码器架构的 Granite 4.2 推理模型家族,涵盖 3B、8B 和 30B 三种参数规模。该文章详细披露了从 Granite-4.1 基座模型出发,通过合成数据生成(Code Alchemy)及多阶段后训练技术构建高性能推理模型的完整工程路径。

报道正文

本文目录43 个章节

返回文章列表

Granite 4.2 LLMs:构建之道

企业级文章 发布于 2026 年 8 月 25 日 点赞 128

Yousaf Shah yousafshah ibm-granite Swanand Kadhe kswanand1 ibm-granite Riddhiman Moulick rmoulick ibm-granite Ashish Sunil Agrawal ashish23 ibm-granite santosh borse sanborse ibm-granite

关于我们如何构建 Granite 4.2 推理模型系列的技术详解。

作者: IBM Granite 团队


TL;DR: Granite 4.2 是我们首个稠密、仅解码器(decoder-only)的推理 LLM 家族,提供 3B、8B 和 30B 三种规格。这些模型基于 Granite-4.1 基础模型进行后训练而成。Granite-4.1 基础模型从零开始预训练了约 15T tokens,采用五阶段策略将上下文窗口扩展至 512K tokens,随后在思维链(chain-of-thought)、推理及智能体轨迹数据上进行监督微调,最后通过多阶段强化学习流水线进行后训练。该流水线包含智能体 RL,其中 8B 和 30B 模型学会了在真实的沙盒环境中使用工具采取行动。所有模型均具备思考 / 非思考切换开关、一种针对简单问题分配少量推理预算的低开销思考模式,以及原生工具调用能力。所有 Granite 4.2 模型均以 Apache 2.0 许可证发布。

链接:


概览

Granite 4.2 是 Granite 语言模型家族中专注于推理的版本。早期的 Granite 版本在指令遵循方面表现出色,而 Granite 4.2 则引入了显式推理能力。每个模型都能在给出答案前生成思维链(chain of thought),并根据任务所需的思考深度,在 thinking(思考)或 non-thinking(非思考)模式之间切换。low-effort(低开销)模式介于两者之间,针对简单问题分配较短的推理预算。

三种尺寸(3B、8B 和 30B)共享相同的架构设计,并遵循相同的训练流程(基础模型、SFT,随后是多阶段 RL),各自在其规模上独立运行。这三种模型均具备强大的推理能力和指令遵循能力。最显著的能力差异体现在后训练阶段。8B 和 30B 模型额外经过了一个 agentic RL(智能体强化学习)模块的训练,使其能够作为智能体运作:调用工具、编辑和运行代码、操作终端以及在真实环境中进行网络搜索。所有模型均支持原生工具调用。通过 OpenAI 兼容端点提供服务时(例如使用 vLLM),模型以 OpenAI function-calling 格式输出工具调用,无需额外适配即可接入智能体框架。Granite 4.2 也支持 SGLang,请参阅 SGLang cookbook 获取现成的部署方案。

本文后续部分将详细介绍构建过程:架构、预训练、监督微调、多阶段 RL 流程以及实验结果。


模型架构

Granite 4.2 模型基于仅解码器(decoder-only)的稠密 Transformer 架构构建,包含以下核心组件:

  • Attention: 分组查询注意力机制(Grouped Query Attention, GQA),拥有 40 个注意力头和 8 个 KV 头
  • Position Embedding: 旋转位置编码(Rotary Position Embedding, RoPE),θ = 10,000,000
  • Feed-Forward: 采用 SwiGLU 激活函数的 MLP
  • Normalization: RMSNorm (ε = 1e-5)
  • Embeddings: 独立的输入/输出嵌入层(不共享权重)
  • Precision: bfloat16
Component3B Dense8B Dense30B Dense
Embedding size256040964096
Number of layers404064
Attention head size64128128
Number of attention heads403232
Number of KV heads888
MLP hidden size81921280032768
MLP activationSwiGLUSwiGLUSwiGLU
Sequence length131072131072131072
Position embeddingRoPERoPERoPE
# Parameters3B8B30B

预训练

Granite 4.2 models are post-trained from Granite-4.1 base models. Granite-4.1 base models were trained from scratch on approximately 15 trillion tokens using a five-phase training strategy. Phases 1–2 focus on foundational pre-training, phases 3–4 perform mid-training with progressively higher-quality data annealing, and phase 5 introduces long-context training, extending the context window to 512K tokens. Each phase uses a distinct data mixture and learning-rate schedule, gradually shifting from broad web-scale data toward more curated, high-quality sources.

The pre-training recipe closely follows the previous generation; for a detailed treatment of the data blend, phase schedule, and long-context extension, see the Granite 4.1 blog.


SFT: Data Preparation & Quality Control

Supervised fine-tuning (SFT) turns the base model into a reliable instruction-following, reasoning, and tool-using assistant. The SFT data mixture combines agentic (31.6%) and non-agentic (68.4%) data, totaling approximately 7.2 million samples, or roughly 100B tokens, of which about 65B are trainable.

The agentic corpus covers a broad range of domains, including software engineering (SWE, 69%), tool calling (12.1%), terminal use (8.0%), math (3.5%), search (0.8%), and action (0.2%). These samples and trajectories are generated using a diverse set of agent scaffolds and harnesses, including OpenHands, OpenCode, Terminus-2, SWE-agent, OpenResearcher, MiniSWE, OpenSeeker, EnvScaler, Gemini CLI, Hermes, Codex, and Goose. The agentic data combines samples from both open-source datasets and our own synthetically generated RL environments, spanning a variety of agent–harness combinations.

The non-agentic corpus consists of several major categories: instruction following (18.8%), coding (18.8%), math (14.6%), multilingual (7.0%), science (5.4%), reasoning (3.0%), and safety (0.8%).

Data Quality Control

We apply multiple stages of quality control before a sample enters the final SFT mixture. First, data from different sources is normalized and reformatted into a consistent OpenAI Chat format, making the conversation structure and tool interactions uniform across datasets and scaffolds.

We then use GPT-OSS-120B and Gemma 4 as LLM-based judges to assess sample quality. Low-scoring samples are removed, as are samples containing hallucinated or fabricated information, invalid tool interactions, or tool calls to functions that are not defined in the corresponding tool list. Several targeted, dataset-specific heuristic rules are also applied where appropriate to further improve quality and remove known sources of noise.

Finally, we perform both local and global deduplication. Deduplication is based on SHA-256 hashes computed over the combination of the tools and messages fields, removing duplicate samples both within individual data sources and across the overall SFT mixture.

SFT Training Details

The complete corpus is first globally shuffled to reduce ordering effects and ensure that samples from different domains are well mixed during training. The shuffled corpus is then partitioned into equally sized .parquet shards, which are tokenized using the model's tokenizer and chat template and prepared for large-scale distributed training.

Before launching the final large-scale runs, we tune hyperparameters on representative configurations, sweeping learning-rate schedules, initial learning rates, and warm-up ratios to find settings that train stably across model sizes. The final training configuration is summarized below:

ParameterValue
Compute32–128 nodes (by model size), 4× Grace/GB200 per node
Sequence length (packed)131,072 (128K)
Global batch size128
Learning rate1.0e-5, constant after warm-up; 3.0e-6 for Phase 2
LR warm-up2.5% of train_iters
Training duration~2 epochs
ParallelismTP=2, PP=1, CP=4 or CP=2

Phase 2 SFT for the 30B Model

For the 30B model, we additionally perform a second phase of SFT focused specifically on agentic coding. In this phase, agentic, SWE, and coding data are upsampled to increase their effective contribution to the training distribution, while approximately 16% of the mixture is retained as replay data from the original SFT corpus.

The 30B model is then fine-tuned for roughly one additional epoch at a lower learning rate of 3.0e-6. This targeted second phase increases the model's exposure to agentic coding trajectories without discarding the capabilities acquired during the initial SFT phase.


Reinforcement Learning: A Multi-Stage, Multi-Environment Pipeline

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

After SFT, we apply a multi-stage, multi-environment reinforcement learning pipeline. Rather than a single RL pass, we run a chain of focused stages spanning many environments: math, code, science, instruction following, tool use, and structured output, then software engineering, terminal use, and web search. Each stage is an independent RL run that targets one capability and warm-starts from the previous stage's checkpoint.

Granite 4.2 staged RL curriculum

Figure 1. The staged RL curriculum. Foundational RL (verifiable rewards + skill boosters) runs for all sizes; the agentic RL block (SWE → Terminal → Search) runs for 8B and 30B only. Every model finishes with RLHF. Each stage is a separate GRPO run that warm-starts from the previous checkpoint.

Training Methodology

Every stage trains with asynchronous GRPO (Group Relative Policy Optimization), so the generator and trainer halves of the loop never block on each other. A pool of generation workers keeps sampling responses and dropping the finished trajectories into a shared buffer; once the buffer holds a full step's worth, the trainer pulls that batch, takes an optimizer step, and streams the updated parameters back to the generator workers without pausing them. A refresh can land partway through a rollout, leaving a single trajectory stitched together from two adjacent policy versions. We allow this instead of paying to prevent it: the workers reuse their existing KV cache rather than rebuilding it after each refresh, and the one guardrail is a limit that keeps them from drifting more than a single update behind the trainer, which bounds how off-policy any sample can get. Whatever mismatch survives that limit is handled in the objective by truncated importance sampling, which clamps the train-versus-generation log-probability ratio to a fixed ceiling so a handful of stale tokens cannot dominate an update.

Advantages are group-relative with a leave-one-out baseline: each response is judged against the mean reward of the other samples drawn for the same prompt, which removes the need for a separate value network. To make this concrete, take RLVR, the first and longest-running stage: each step pairs 256 prompts with 16 sampled responses apiece for a 4,096-example batch, which the trainer consumes in a single optimizer step before the next rollout begins. Later stages keep this machinery unchanged and adjust only the per-stage shape, shown next.

RL training configuration

The pipeline keeps a common backbone of hyperparameters across every stage, which makes the curriculum easier to run and compare. A handful of knobs are fixed everywhere:

ParameterValue (shared across stages)
AlgorithmGRPO (no value network; group-relative advantages)
Training stackNeMo-RL (Megatron-Core + vLLM) with NeMo-Gym environments
Ratio clip (min / max)0.2 / 0.28
Micro-batch size1
Parallelismtensor-parallel 2–4; no pipeline- or context-parallelism

What changes from stage to stage is the shape of each run: how many prompts and generations per step, how long the context is, whether the agent loop runs, and how hard we pull back toward the reference policy. The table below gives the exact settings for the 30B chain, stage by stage:

StagePrompts/stepGens/promptMax seq lenRollout turnsKLLR
RLVR (×3)2561664K105e-7
IF booster2561664K105e-7
Code booster641664K10.055e-7
SWE 16416128K10.015e-7
SWE 23216128K12805e-7
Terminal83264K640.011e-6
Search3216128K640.015e-7
RLHF1281648K10.055e-7

Parameters shown for the 30B model. Global batch size = prompts/step × generations/prompt (e.g. 256 × 16 = 4096 for RLVR). The 3B and 8B models use the same recipe and hyperparameters with fewer stages (see How the Three Sizes Differ); the stage list is what changes, not the knobs.

The KL schedule follows the reward type: explore freely where the reward is objective and verifiable (RLVR and SWE 2 run at KL 0), and stay close to the reference where the objective is preference, safety, or a narrow skill graft (RLHF and the code booster use KL 0.05). The rollout-turns column counts the environment interactions GRPO itself sees per rollout. In every case the model is trained on complete, real-environment trajectories.

The Staged Curriculum

Each stage is a separate RL run with a single objective and its own reward signal. When it finishes, its policy is exported to Hugging Face format and becomes the base model for the next stage, so the pipeline is a sequence of warm-starts:

文本
SFT ─▶ RLVR ─▶ Skill boosters ─▶ SWE agent ─▶ Terminal ─▶ Search ─▶ RLHF
      └──────── foundational RL ────────┘   └──────── agentic RL (8B / 30B) ────────┘

The 8B and 30B models follow the full ladder. The 3B model takes a shortened path: foundational RL and alignment, without the agentic block.

Reward Signals

A stage is defined mostly by how it is rewarded. Across the pipeline there are three reward types, and a single stage can use more than one:

Reward typeWhat it measuresUsed in
VerifiableExact-match, unit tests, format checkers, rule-based checkers on ground truthRLVR · boosters · SWE
Reward model / LLM judgeOpen-ended quality, preference, safety, answer correctnessRLVR · Search · RLHF
Agentic outcomeDid the model actually solve a task in a real environment?SWE · Terminal · Search

Verifiable rewards are objective and hard to game, so the pipeline front-loads them. Judge- and preference-based rewards handle open-ended qualities that no checker can express. Agentic-outcome rewards are the sparsest: often a single bit at the end of a long tool-use trajectory.

Foundational RL: Build the Skills

RLVR: verifiable-reward RL

RLVR is the foundational stage and the broadest data mix in the pipeline: a single blended dataset spanning many verifiable domains.

  • Math: chain-of-thought with boxed-answer checking, plus formal proving in Lean
  • Competitive coding: solutions checked against hidden tests in a sandbox
  • STEM / graduate-level science MCQA and general knowledge
  • Instruction following: structured-output and inverse-instruction tasks
  • Tool / function calling: single-step tool use
  • Reasoning puzzles and abstention (knowing when to refuse)

Each task type carries its own verifier, so the reward is grounded per example. RLVR runs for two rounds on 3B and 8B, and three on 30B. Each round is a fresh warm-started run on a re-weighted mix of public and internally curated RL data.

Skill boosters: targeted lifts

After RLVR, a few short booster stages sharpen specific capabilities that benefit from concentrated training focusing on the following domains:

  • Instruction following (IF): multi-turn chat, inverse-IFEval, structured outputs
  • Code: competitive coding only

Boosters are small, focused runs. A light KL penalty keeps the model close to its current behavior while nudging one skill.

Agentic RL: Learning to Act (8B / 30B)

In the agentic stages the model learns to act: call tools, observe results, and iterate inside a real environment, rewarded on whether the task was actually solved. These stages share the same shape: multi-turn tool use, real (not simulated) environments, sparse outcome rewards, and GRPO, warm-started from the coding-boosted checkpoint. They run in order: SWE → Terminal → Search.

Agentic RL environments

Figure 2. The three agentic-RL environments. Each pairs a real harness with a real environment and a sparse, outcome-based reward. The 3B model runs none of these.

  • SWE agent (software engineering). Each task is a real repository in its own sandbox. Driven by the OpenHands harness, the model reads code, edits files, and runs the test suite over many internal turns. The reward is verifiable: do the hidden tests pass? Tasks are drawn from open-source SWE datasets, each instance backed by a per-repo container image.
  • Terminal agent (terminal / OS operation). Multi-step tasks in a live shell, run through the Harbor / Terminus-2 agent harness. The model plans a sequence of commands, observes their output, and recovers from errors. Reward is assigned when the task completed successfully. This is the one stage that drives its multi-turn agent loop at the GRPO level, with rollouts spanning up to 64 environment turns.
  • Search agent (deep research). The model answers hard, multi-hop questions using live web-search tool calls inside a browsing agent loop: gather evidence across hops, reason over it, and produce an answer. Because correctness here is open-ended, the reward is an LLM judge on the final answer.

Alignment: RLHF

The final stage of every model is RLHF for human preference and safety. It optimizes against a generative reward model (GenRM) for preference, plus a safety reward covering jailbreak resistance and appropriate refusals. This stage uses the highest KL penalty in the pipeline, aligning tone and safety without eroding the capabilities the earlier stages built. In addition to human preference and safety alignment, this stage also applies a reasoning-length penalty to discourage overly verbose reasoning behavior acquired during earlier stages.

How the Three Sizes Differ

Same method and infrastructure; the difference is how far up the ladder each model goes.

Stage3B8B30B
RLVR (verifiable)×2×2×3
Skill boosterscodeIF · GPQA · codeIF · code
SWE agent—✓✓
Terminal agent—✓✓
Search agent—✓✓
RLHF (preference + safety)✓✓✓

3B is a strong foundational-RL model; 8B and 30B add the agentic-RL block on top, learning to act with tools in real environments.


Agentic AI Infrastructure for Scalable RL

Reinforcement learning at this scale needs infrastructure that can drive a training loop and a fleet of live environments at the same time. This matters most in the agentic stages, where every training example is a multi-turn rollout that edits code, runs commands, or browses the web. Granite 4.2's RL runs on two open components: NeMo-RL on the training side and NeMo-Gym on the rollout side.

NeMo-RL + NeMo-Gym system architecture

Figure 3. The RL system. NeMo-RL drives the GRPO loop (Megatron-Core training backend, vLLM generation, and Megatron-Bridge for HF⇄Megatron weight conversion). NeMo-Gym orchestrates rollouts and hosts the tools, sandboxes, and reward/verifier calls as pluggable Resources.

The division of labor:

  • NeMo-RL (training side). Megatron-Core is the training backend; vLLM generates rollouts; Megatron-Bridge converts weights between Megatron and Hugging Face formats, so each stage can export a clean HF checkpoint for the next one.
  • NeMo-Gym (rollout side). It exposes each environment as a set of Resources (verifiers, tools, sandboxes, and reward models) behind a uniform interface. This is the plug point for the agentic stages: the SWE repo sandboxes, the terminal harness, and the web-search tools all attach here, and to the training loop they look the same as a simple math verifier.

That uniformity is what makes the staged curriculum above practical: a booster's rule-based checker and a full SWE sandbox present the same interface to GRPO.

This split is also what makes the asynchronous training loop described above physically possible: generation and policy updates live on separate GPU pools, so the expensive generation fleet — including the live agentic environments — stays busy instead of idling through optimizer steps.


Results

Granite 4.2 was evaluated across agentic coding, general agentic and tool use, reasoning, chat and instruction following, and long context. The full benchmark table is below, followed by charts that break out the headline results by model size.

Task3B Dense8B Dense30B Dense
Agentic (Coding)
SWE Bench MultilingualNA30.7841.89
SWE Bench ProNA19.1133.29
SWE Bench VerifiedNA47.6757.00
Terminal-Bench 2.1NA20.5629.24
Agentic (General)
τ³-bench45.7858.0662.00
BFCL (v4)52.4150.2961.39
ProfBench32.1041.2042.90
BirdBenchNA41.0741.85
GDPvalNA1189.001225.00
Reasoning
AIME2578.3386.6789.17
HMMT Feb2566.6778.3389.17
GPQA54.8064.1466.41
LiveCodeBench v669.7173.2475.77
SciCode24.1136.0938.76
Chat & Instruction Following
MMLU-Pro67.8474.0477.60
MMLU-ProX lite (IBM)27.7861.0666.64
Arena-Hard-V234.9665.1967.93
IFBench (prompt)74.3379.3377.17
Long Context
RULER 64K67.5280.9989.96
RULER 128K55.3071.4181.38

Supported languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.

The charts below break these results out by capability area.

Reasoning benchmarks

Figure 4. Reasoning (pass@1). Scores rise consistently with model size across math (AIME25, HMMT), science (GPQA), and code reasoning (LiveCodeBench, SciCode).

Agentic coding benchmarks

Figure 5. Agentic coding resolve rates. The agentic-RL block is trained only for 8B and 30B; the 30B model leads across SWE-Bench variants and Terminal-Bench.

General agentic and tool-use benchmarks

Figure 6. General agentic and tool-use benchmarks, reported for all three sizes.


Quantization

We also released four quantized variants of the Granite 4.2 models for inference with vLLM. The models are converted to FP8, NVFP4, and MXFP4 using LLM Compressor, and to the GGUF format using the llama.cpp framework for reduced-memory deployment.

FP8

The FP8 version is quantized with dynamic per-channel weights and per-token activations. No calibration is used.

FP4

The NVFP4 and MXFP4 versions are quantized using GPTQ calibrated on 2K samples drawn from the SFT dataset. Max context length is 2K during calibration.

GGUF

Conversion of the Granite 4.2 models to GGUF is done with the canonical llama.cpp tool as described in https://github.com/IBM/gguf#gguf-conversion--quantization.

Several GGUF formats are provided:

  • Q8_0
  • Q6_K
  • Q5_K_S
  • Q5_K_M
  • Q5_1
  • Q5_0
  • Q4_K_S
  • Q4_K_M
  • Q4_1
  • Q4_0
  • Q3_K_S
  • Q3_K_M
  • Q3_K_L
  • Q2_K

Infrastructure

Hardware

We trained the Granite 4.2 language models on an NVIDIA GB200 NVL72 cluster hosted by CoreWeave, featuring:

  • A 72-GPU NVLink domain for high-speed intra-rack communication
  • A non-blocking Fat-Tree NDR 400 Gb/s InfiniBand fabric for full-bandwidth inter-rack connectivity
  • Thousands of GPUs operating at cluster scale

This infrastructure delivers the high-bandwidth, low-latency communication required for efficient large-scale distributed training.

Software Stack

The training software stack is packaged into .sqsh container images, each giving a run a reproducible, portable environment with its SBSA-compatible CUDA targets, Linux aarch64 Python wheels, and GPU-specific binaries pinned. The large-scale SFT runs build on an NGC PyTorch base image (Ubuntu 22.04, CUDA 12.8, Python 3.12); the RL stack runs in its own NeMo-RL container.


Getting Started (Transformers)

Installation

BASH
pip install torch
pip install accelerate transformers

Basic Inference (Thinking Mode)

PYTHON
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "ibm-granite/granite-4.2-30b"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="cuda", torch_dtype=torch.bfloat16)
model.eval()

messages = [
    {"role": "user", "content": "How many r's are in the word 'strawberry'?"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=8192, temperature=1.0, top_p=0.95, do_sample=True)

print(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))

Example Output

文本
<think>
Okay, let's see. The problem is to find how many 'r's are in the word 'strawberry'.

First, I need to write out the word: s t r a w b e r r y.

Now, I need to count the number of 'r' letters. Let's list each letter and check for 'r'.

1. s – not r
2. t – not r
3. r – yes, that's one
4. a – no
5. w – no
6. b – no
7. e – no
8. r – yes, that's two
9. r – yes, that's three
10. y – no

Total r's = 3.
</think>
There are **3** r's in the word "strawberry".<|im_end|>

Non-Thinking Mode

PYTHON
messages = [
    {"role": "user", "content": "What is the capital of France?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=2048, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))

Example Output

文本
<think></think>The capital of France is Paris.<|im_end|>

Low-Effort Thinking

PYTHON
messages = [
    {"role": "user", "content": "What is 2 + 2?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                                     enable_thinking=True, low_effort=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))

Example Output

文本
<think>
Simple answer.
</think>
2 + 2 = 4.<|im_end|>

Tool Calling

Granite models support tool calling with integrated reasoning: the model reasons about which tool to call and why before calling it. Tools are defined with the OpenAI function definition schema.

Basic Tool Calling

PYTHON
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather for a specified city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "Name of the city"}
                },
                "required": ["city"]
            }
        }
    }
]

messages = [
    {"role": "user", "content": "What's the weather like in Boston right now?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools,
                                     add_generation_prompt=True, enable_thinking=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))

Example Output

文本
<think>
Okay, the user is asking for the weather in Boston right now. Let me check the tools
available. There's a function called get_current_weather that takes a city parameter.
I need to call that with the city set to Boston.
</think>
<tool_call>
<function=get_current_weather>
<parameter=city>
Boston
</parameter>
</function>
</tool_call>
<|im_end|>

Multi-Turn with Tool Response

PYTHON
messages = [
    {"role": "user", "content": "What's the weather like in Boston right now?"},
    {"role": "assistant", "content": "<think>\nThe user wants to know the current weather in Boston. I should call get_current_weather.\n</think>",
     "tool_calls": [{"function": {"name": "get_current_weather", "arguments": {"city": "Boston"}}}]},
    {"role": "tool", "content": '{"temperature": "72°F", "condition": "Partly cloudy", "humidity": "65%"}'},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, tools=tools,
                                     add_generation_prompt=True, enable_thinking=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=4096, temperature=1.0, top_p=0.95, do_sample=True)
print(tokenizer.decode(output[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))

Example Output

文本
<think>
The tool returned the weather data for Boston: temperature 72°F, partly cloudy, humidity 65%.
I need to present this information clearly to the user.
</think>
The current weather in Boston is 72°F, partly cloudy, with 65% humidity.<|im_end|>

Multi-Turn Conversations

History Thinking Truncation

PYTHON
messages = [
    {"role": "user", "content": "What is 15 * 37?"},
    {"role": "assistant", "content": "<think>\nLet me calculate 15 * 37.\n15 * 37 = 15 * 30 + 15 * 7 = 450 + 105 = 555\n</think>\n15 * 37 = 555"},
    {"role": "user", "content": "Now divide that by 5"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                                     enable_thinking=True, truncate_history_thinking=True)

text_full = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                                          enable_thinking=True, truncate_history_thinking=False)

Parsing Thinking vs. Final Answer

PYTHON
import re

def parse_model_output(text):
    """Separate thinking content from final answer."""
    think_match = re.search(r'<think>(.*?)</think>', text, re.DOTALL)
    if think_match:
        thinking = think_match.group(1).strip()
        answer_start = text.find('</think>') + len('</think>')
        answer_end = text.find('<|im_end|>', answer_start)
        answer = text[answer_start:answer_end].strip() if answer_end != -1 else text[answer_start:].strip()
    else:
        thinking, answer = "", text.strip()
    return thinking, answer

thinking, answer = parse_model_output(output_text)

Using with Agentic Coding Harnesses

Granite models can serve as the backbone for agentic coding tools. Because they support reasoning and tool calling through the OpenAI-compatible API, they integrate with popular agentic harnesses without extra adapters. Start the vLLM server, then follow the harness-specific instructions below.

OpenCode

OpenCode is an AI coding agent that runs in your terminal.

Install:

BASH
curl -fsSL https://opencode.ai/install | bash

Configure ~/.config/opencode/opencode.json:

JSON
{
  "$schema": "https://opencode.ai/config.json",
  "model": "local/granite-4.2-30b",
  "provider": {
    "local": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "vLLM (local)",
      "options": {
        "baseURL": "http://localhost:8000/v1",
        "apiKey": "EMPTY"
      },
      "models": {
        "granite-4.2-30b": {
          "name": "Granite 4.2 30B",
          "limit": {
            "context": 131072,
            "output": 8192
          }
        }
      }
    }
  }
}

Run:

BASH
opencode
opencode run "your task description"

For full documentation, see opencode.ai/docs.

Pi

Pi is a minimal agent harness for AI-powered coding that runs in your terminal. It supports custom providers via a models.json configuration file.

Install:

BASH
curl -fsSL https://pi.dev/install.sh | sh

Configure ~/.pi/agent/models.json:

JSON
{
  "providers": {
    "vllm": {
      "baseUrl": "http://localhost:8000/v1",
      "api": "openai-completions",
      "apiKey": "EMPTY",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [
        {
          "id": "granite-4.2-30b",
          "name": "Granite 4.2 30B",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 131072,
          "maxTokens": 8192,
          "samplingParams": {
            "temperature": 1.0,
            "top_p": 0.95
          },
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
        }
      ]
    }
  }
}

Run:

BASH
pi

Then select the granite-4.2-30b model with /model or Ctrl+L in the interactive session.

For full documentation, see pi.dev/docs.

OpenHands

OpenHands is an AI software engineer that can plan, write code, and execute commands.

  1. Install and launch OpenHands following the official installation guide.
  2. Configure the LLM in the OpenHands settings with:
  • Model: granite-4.2-30b
  • Base URL: http://localhost:8000/v1
  • API Key: your vLLM --api-key value

Note: The openai/ prefix is required when connecting to OpenAI-compatible endpoints like vLLM. Refer to the OpenHands local LLM documentation for detailed setup instructions, troubleshooting, and alternative installation methods.


Resources:


Last updated: September 11, 2026 to clarify base models.