350M 小模型 GRPO 微调:100 步提升结构化输出
LiquidAI 与 Hugging Face 联合展示了一套低成本微调方案,利用 TRL 库和 GRPO 算法在仅 100 个训练步骤内显著提升了 LFM2.5-350M 模型的结构化输出能力。该实验基于 IFStruct 基准测试,全程仅需约 500 条样本,可在免费 GPU 资源上复现,为小模型对齐提供了高效范式。
报道正文
本文目录11 个章节
仅需 100 步 GRPO,微调 350M 模型以提升结构化输出质量
发布于 2026 年 9 月 3 日 在 GitHub 上更新 点赞 147
Leonie Monigatti iamleonie LiquidAI ben burtenshaw burtenshaw Sergio Paniego sergiopaniego
本指南提供了一套完全公开且低成本的方案,旨在显著提升小型模型在结构化输出合规性方面的表现。我们使用 TRL 库通过组相对策略优化(Group Relative Policy Optimization, GRPO)对 LFM2.5-350M 进行微调,并在 IFStruct 基准测试上进行评估。整个训练过程仅涉及约 500 个样本和 100 个训练步骤,规模极小,足以在免费层级的 Colab 或 Kaggle GPU 上运行,相关代码已在 GitHub 上开源。结果显示,即使采用轻量级的微调流程,也能将模型在 IFStruct 基准测试上的性能从 22.6% 提升至 29.7%。
结构化输出是大语言模型(LLM)最常见的实际应用场景之一,然而大多数基准测试往往将其混入更广泛的推理或信息抽取评分中,而非单独衡量。模型能否可靠地以请求的格式和结构返回有效、可解析的输出——即模式合规性(schema compliance)——通常决定了它是否能够被接入下游系统。
请注意,此处描述的训练流程并非用于训练 IFStruct 博客中所述强化学习模型的流程。本笔记本的目标并非复现 IFStruct 基准测试得分,而是展示针对特定任务的小模型微调如何提升性能,并达到与更大模型相当的效果。
前置条件
本指南分为两部分,分别在各自的环境中运行:
- 微调需要在 GPU 上运行。配套的笔记本已针对免费层级的 Colab 或 Kaggle GPU 进行了优化。
- 评估可以在本地 MacBook 上运行(例如配备 Apple M5 Max 芯片和 36 GB 统一内存的 MacBook Pro),通过
llama.cpp实现。llama.cpp会暴露一个兼容 OpenAI API 的服务端,供 IFStruct 评估器调用。
我们需要 uv 来处理 Python 工具链,并使用 llama.cpp 进行模型服务。请参照 Liquid AI llama.cpp 部署文档,通过 Homebrew 安装 llama.cpp,并验证 llama-server 是否可用:
brew install llama.cpp
llama-server --versionLFM2.5-350M(基础模型)的 IFStruct 评估
在开始之前,让我们先在 IFStruct 基准测试上评估 LFM2.5-350M,看看我们能否复现其报告的 21.1% 得分。
IFStruct 是一个用于测试 LLM 输出有效性及 schema 遵循度的基准测试。该基准测试已在 Liquid4All/ifstruct 开源,公开基准数据集可在 Hugging Face 上的 LiquidAI/ifstruct-v1.0 获取。
git clone https://github.com/Liquid4All/ifstruct.git为了进行比较评估,我们在 MacBook 上使用 llama.cpp 本地部署模型。我们将使用 BF16 GGUF 格式(LiquidAI/LFM2.5-350M-GGUF)。
然后,使用以下命令启动基础模型服务器:
llama-server \
-hf LiquidAI/LFM2.5-350M-GGUF:BF16 \
-c 32768 \
-np 4 \
-ngl 99 \
--alias LiquidAI/LFM2.5-350M \
--host 127.0.0.1 \
--port 8080--alias:IFStruct 向 OpenAI 兼容端点发送时使用的模型名称-ngl 99:请求llama.cpp在可用时将所有层卸载到 GPU-np 4:并行处理四个请求-c 32768:提示词上下文窗口大小
服务器运行后,我们可以使用包含 2000 个样本的完整基准测试进行评估:
uv run ifstruct-eval \
--model LiquidAI/LFM2.5-350M \
--base-url http://localhost:8080/v1 \
--api-key dummy \
--dataset data/test.jsonl \
--results-file results/lfm2.5-350m-llamacpp-base.json \
--n-threads 4 \
--max-tokens 2048 \
-v============================================================
Model: LiquidAI/LFM2.5-350M
============================================================
Overall: 452/2000 passed (22.6%)
Average latency: 1453ms
By format:
JSON: 180/1000 passed (18.0%)
YAML: 272/1000 passed (27.2%)
By top-level structure:
Wrapper key 288/1011 passed (28.5%)
Bare list 164/989 passed (16.6%)
By entity type:
test__camera_review 6/83 passed (7.2%)
test__clinical_trial 20/104 passed (19.2%)
test__conference_schedule 7/87 passed (8.0%)
test__escaping__bug_report_batch 24/89 passed (27.0%)
test__escaping__config_snippet_audit 15/85 passed (17.6%)
test__escaping__customer_email_thread 5/73 passed (6.8%)
test__escaping__dialogue_sample 14/95 passed (14.7%)
test__escaping__interview_transcript_segment 21/80 passed (26.2%)
test__escaping__log_parser_examples 21/72 passed (29.2%)
test__escaping__pr_discussion 22/87 passed (25.3%)
test__escaping__repro_steps_batch 16/73 passed (21.9%)
test__escaping__screenplay_scene 16/92 passed (17.4%)
test__escaping__short_story_chapter 15/84 passed (17.9%)
test__escaping__support_ticket_batch 27/73 passed (37.0%)
test__escaping__terminal_session_notes 20/70 passed (28.6%)
test__event_ticket_booking 49/107 passed (45.8%)
test__gpu_review 6/94 passed (6.4%)
test__invoice 28/86 passed (32.6%)
test__job_posting 25/85 passed (29.4%)
test__real_estate_listing 31/82 passed (37.8%)
test__recipe 3/70 passed (4.3%)
test__rental_car_booking 27/79 passed (34.2%)
test__scientific_experiment 13/69 passed (18.8%)
test__travel_itinerary 21/81 passed (25.9%)
Common errors:
7228x required field missing
738x wrong item count
540x type mismatch
317x Unclosed code block
190x extraneous field 'notes'
181x extraneous field 'path'
175x extraneous field 'constraints'
170x extraneous field 'type'
170x missing code block
100x expected bare list, got wrapperThe IFStruct release blog reports 21.1% for LFM2.5-350M. Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.
GRPO Fine-tuning with TRL on Structured Outputs
The full, runnable pipeline lives in the accompanying notebook. We will cover only the relevant pieces in this section.
Training data
We use nvidia/Nemotron-RL-instruction_following-structured_outputs, which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training.
Because the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two gaps between them:
- 40% get a "return the output inside a fenced code block" instruction appended, so the model learns to follow the format instruction rather than always emitting raw JSON.
- A disjoint 20% are converted into top-level-array tasks (the schema is wrapped in an
arraywith a required item count), which trains bare-list output and item-count compliance.
Model and LoRA
We load LiquidAI/LFM2.5-350M and attach a LoRA adapter. Because LFM2.5 uses a hybrid attention/convolution architecture, we target the LFM-specific module names:
lora_config = LoraConfig(
r=16,
lora_alpha=32,
bias="none",
task_type="CAUSAL_LM",
target_modules=[
"q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
"w1", "w2", "w3",
],
)This trains ~6M parameters, about 1.66% of the model.
Reward functions
Then we define three reward functions, each on a [0, 1] scale, which score every completion on whether the extracted structure is correct:
json_format_reward: Is the output parseable, and in the requested form? Full credit (1.0) for the requested form (fenced vs. raw),0.2for the wrong-but-parseable form,0.0for unparseable output.field_count_reward: Does the object have the expected number of top-level fields? An exact match earns1.0, and the score decays linearly with the miss.schema_validation_reward: Does the output validate against the row's JSON Schema? It counts every constraint violation and gates partial credit on required-key coverage.
We combine the three as a weighted sum with reward_weights=[1.0, 0.5, 2.0].
Training
We train for 100 steps with 8 generations per prompt group, sized for a free-tier 16 GB GPU:
from trl import GRPOConfig
> *[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]*
training_args = GRPOConfig(
output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",
learning_rate=5e-5,
max_steps=100,
warmup_steps=10,
num_generations=8,
per_device_train_batch_size=4,
gradient_accumulation_steps=8,
steps_per_generation=2,
max_completion_length=1024,
mask_truncated_completions=False,
temperature=1.1,
beta=0.01,
reward_weights=[1.0, 0.5, 2.0],
logging_steps=1,
save_steps=100,
)As you can see in the notebook, over the run, all three reward components climb, the KL from the reference model lifts off zero after warmup, and the truncated-completion fraction stays near zero.
Merging and saving the model
Finally, we merge the LoRA adapter back into the base weights and save it as a single self-contained checkpoint, ready to convert to GGUF for serving:
MERGED_DIR = f"{training_args.output_dir}-merged"
merged_model = trainer.model.merge_and_unload()
merged_model.save_pretrained(MERGED_DIR)
tokenizer.save_pretrained(MERGED_DIR)IFStruct Evaluation on GRPO Tuned LFM2.5-350M
After GRPO fine-tuning, we rerun the IFStruct evaluation. For this, we need to convert the merged model checkpoint into a BF16 GGUF. The converter script ships with the llama.cpp source, so we clone the repo once and install the converter's gguf package.
git clone --depth 1 https://github.com/ggml-org/llama.cpp
pip install ./llama.cpp/gguf-py
mkdir -p models
python llama.cpp/convert_hf_to_gguf.py \
PATH_TO_YOUR_MERGED_MODEL \
--outfile ./models/lfm25-350m-grpo-bf16.gguf \
--outtype bf16Then we serve the merged model with the following command:
llama-server \
-m ./models/lfm25-350m-grpo-bf16.gguf \
--alias lfm25-350m-grpo-structured-output \
-c 32768 \
-np 4 \
-ngl 99 \
--host 127.0.0.1 \
--port 8081Then, we will run the full IFStruct evaluation again with the fine-tuned model:
uv run ifstruct-eval \
--model lfm25-350m-grpo-structured-output \
--base-url http://localhost:8081/v1 \
--api-key dummy \
--dataset data/test.jsonl \
--results-file results/lfm25-350m-grpo.json \
--n-threads 4 \
--max-tokens 2048 \
-v============================================================
Model: lfm25-350m-grpo-structured-output
============================================================
Overall: 594/2000 passed (29.7%)
Average latency: 1518ms
By format:
JSON: 319/1000 passed (31.9%)
YAML: 275/1000 passed (27.5%)
By top-level structure:
Wrapper key 300/1011 passed (29.7%)
Bare list 294/989 passed (29.7%)
By entity type:
test__camera_review 5/83 passed (6.0%)
test__clinical_trial 31/104 passed (29.8%)
test__conference_schedule 11/87 passed (12.6%)
test__escaping__bug_report_batch 32/89 passed (36.0%)
test__escaping__config_snippet_audit 24/85 passed (28.2%)
test__escaping__customer_email_thread 9/73 passed (12.3%)
test__escaping__dialogue_sample 17/95 passed (17.9%)
test__escaping__interview_transcript_segment 13/80 passed (16.2%)
test__escaping__log_parser_examples 33/72 passed (45.8%)
test__escaping__pr_discussion 26/87 passed (29.9%)
test__escaping__repro_steps_batch 23/73 passed (31.5%)
test__escaping__screenplay_scene 34/92 passed (37.0%)
test__escaping__short_story_chapter 24/84 passed (28.6%)
test__escaping__support_ticket_batch 36/73 passed (49.3%)
test__escaping__terminal_session_notes 23/70 passed (32.9%)
test__event_ticket_booking 62/107 passed (57.9%)
test__gpu_review 7/94 passed (7.4%)
test__invoice 36/86 passed (41.9%)
test__job_posting 33/85 passed (38.8%)
test__real_estate_listing 32/82 passed (39.0%)
test__recipe 7/70 passed (10.0%)
test__rental_car_booking 37/79 passed (46.8%)
test__scientific_experiment 14/69 passed (20.3%)
test__travel_itinerary 25/81 passed (30.9%)
Common errors:
7331x required field missing
890x wrong item count
555x type mismatch
102x expected bare list, got wrapper
62x extraneous field 'metadata.tone'
55x 6 is greater than maximum 5
49x extraneous field 'speaker_labels'
47x extraneous field 'tone'
44x 'cups' not in allowed values ['mg', 'g', 'kg', 'oz', 'lb', 'ml', 'l', 'cl', 'dl'
44x extraneous field 'notes'Comparing the two runs on the identical serving stack:
| IFStruct group | base | GRPO-tuned | Δ |
|---|---|---|---|
| Overall | 22.6% | 29.7% | +7.1 |
| JSON | 18.0% | 31.9% | +13.9 |
| YAML | 27.2% | 27.5% | +0.3 |
| Wrapper key | 28.5% | 29.7% | +1.2 |
| Bare list | 16.6% | 29.7% | +13.1 |
The gains land exactly where the training aimed: the JSON pass rate rises by nearly 14 points (18.0% → 31.9%), while YAML stays mostly the same. While this is still below the Qwen3.5-2B score of 33.15%, it shows that even light task-specific fine-tuning can bring a small model close to a larger one.
Conclusion
A short GRPO run with about 500 samples and 100 steps can lift a small 350M parameter model from 22.6% to 29.7% on IFStruct. The takeaway is that a cheap, task-specific reward signal can make a small model substantially more reliable about form, closing much of the gap to models several times its size.
To reproduce or extend this work, see the original IFStruct v1.0 blog post, the Liquid4All/ifstruct benchmark repo, and the LiquidAI/ifstruct-v1.0 dataset.