350M 小模型 GRPO 微调:100 步提升结构化输出

原标题:Fine-tuning a 350M Model for Better Structured Outputs in 100 GRPO Steps
78AI 研判分

LiquidAI 与 Hugging Face 联合展示了一套低成本微调方案,利用 TRL 库和 GRPO 算法在仅 100 个训练步骤内显著提升了 LFM2.5-350M 模型的结构化输出能力。该实验基于 IFStruct 基准测试,全程仅需约 500 条样本,可在免费 GPU 资源上复现,为小模型对齐提供了高效范式。

报道正文

本文目录11 个章节

返回文章列表

仅需 100 步 GRPO,微调 350M 模型以提升结构化输出质量

发布于 2026 年 9 月 3 日 在 GitHub 上更新 点赞 147

Leonie Monigatti iamleonie LiquidAI ben burtenshaw burtenshaw Sergio Paniego sergiopaniego

本指南提供了一套完全公开且低成本的方案,旨在显著提升小型模型在结构化输出合规性方面的表现。我们使用 TRL 库通过组相对策略优化(Group Relative Policy Optimization, GRPO)对 LFM2.5-350M 进行微调,并在 IFStruct 基准测试上进行评估。整个训练过程仅涉及约 500 个样本和 100 个训练步骤,规模极小,足以在免费层级的 Colab 或 Kaggle GPU 上运行,相关代码已在 GitHub 上开源。结果显示,即使采用轻量级的微调流程,也能将模型在 IFStruct 基准测试上的性能从 22.6% 提升至 29.7%。

结构化输出是大语言模型(LLM)最常见的实际应用场景之一,然而大多数基准测试往往将其混入更广泛的推理或信息抽取评分中,而非单独衡量。模型能否可靠地以请求的格式和结构返回有效、可解析的输出——即模式合规性(schema compliance)——通常决定了它是否能够被接入下游系统。

请注意,此处描述的训练流程并非用于训练 IFStruct 博客中所述强化学习模型的流程。本笔记本的目标并非复现 IFStruct 基准测试得分,而是展示针对特定任务的小模型微调如何提升性能,并达到与更大模型相当的效果。

前置条件

本指南分为两部分,分别在各自的环境中运行:

  • 微调需要在 GPU 上运行。配套的笔记本已针对免费层级的 Colab 或 Kaggle GPU 进行了优化。
  • 评估可以在本地 MacBook 上运行(例如配备 Apple M5 Max 芯片和 36 GB 统一内存的 MacBook Pro),通过 llama.cpp 实现。llama.cpp 会暴露一个兼容 OpenAI API 的服务端,供 IFStruct 评估器调用。

我们需要 uv 来处理 Python 工具链,并使用 llama.cpp 进行模型服务。请参照 Liquid AI llama.cpp 部署文档,通过 Homebrew 安装 llama.cpp,并验证 llama-server 是否可用:

BASH
brew install llama.cpp
llama-server --version

LFM2.5-350M(基础模型)的 IFStruct 评估

在开始之前,让我们先在 IFStruct 基准测试上评估 LFM2.5-350M,看看我们能否复现其报告的 21.1% 得分。

IFStruct 是一个用于测试 LLM 输出有效性及 schema 遵循度的基准测试。该基准测试已在 Liquid4All/ifstruct 开源,公开基准数据集可在 Hugging Face 上的 LiquidAI/ifstruct-v1.0 获取。

BASH
git clone https://github.com/Liquid4All/ifstruct.git

为了进行比较评估,我们在 MacBook 上使用 llama.cpp 本地部署模型。我们将使用 BF16 GGUF 格式(LiquidAI/LFM2.5-350M-GGUF)。

然后,使用以下命令启动基础模型服务器:

BASH
llama-server \
  -hf LiquidAI/LFM2.5-350M-GGUF:BF16 \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --alias LiquidAI/LFM2.5-350M \
  --host 127.0.0.1 \
  --port 8080
  • --alias:IFStruct 向 OpenAI 兼容端点发送时使用的模型名称
  • -ngl 99:请求 llama.cpp 在可用时将所有层卸载到 GPU
  • -np 4:并行处理四个请求
  • -c 32768:提示词上下文窗口大小

服务器运行后,我们可以使用包含 2000 个样本的完整基准测试进行评估:

BASH
uv run ifstruct-eval \
  --model LiquidAI/LFM2.5-350M \
  --base-url http://localhost:8080/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-llamacpp-base.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v
YAML
============================================================
Model: LiquidAI/LFM2.5-350M
============================================================
Overall: 452/2000 passed (22.6%)
Average latency: 1453ms

By format:
  JSON: 180/1000 passed (18.0%)
  YAML: 272/1000 passed (27.2%)

By top-level structure:
  Wrapper key 288/1011 passed (28.5%)
  Bare list   164/989 passed (16.6%)

By entity type:
  test__camera_review                 6/83 passed (7.2%)
  test__clinical_trial                20/104 passed (19.2%)
  test__conference_schedule           7/87 passed (8.0%)
  test__escaping__bug_report_batch    24/89 passed (27.0%)
  test__escaping__config_snippet_audit 15/85 passed (17.6%)
  test__escaping__customer_email_thread 5/73 passed (6.8%)
  test__escaping__dialogue_sample     14/95 passed (14.7%)
  test__escaping__interview_transcript_segment 21/80 passed (26.2%)
  test__escaping__log_parser_examples 21/72 passed (29.2%)
  test__escaping__pr_discussion       22/87 passed (25.3%)
  test__escaping__repro_steps_batch   16/73 passed (21.9%)
  test__escaping__screenplay_scene    16/92 passed (17.4%)
  test__escaping__short_story_chapter 15/84 passed (17.9%)
  test__escaping__support_ticket_batch 27/73 passed (37.0%)
  test__escaping__terminal_session_notes 20/70 passed (28.6%)
  test__event_ticket_booking          49/107 passed (45.8%)
  test__gpu_review                    6/94 passed (6.4%)
  test__invoice                       28/86 passed (32.6%)
  test__job_posting                   25/85 passed (29.4%)
  test__real_estate_listing           31/82 passed (37.8%)
  test__recipe                        3/70 passed (4.3%)
  test__rental_car_booking            27/79 passed (34.2%)
  test__scientific_experiment         13/69 passed (18.8%)
  test__travel_itinerary              21/81 passed (25.9%)

Common errors:
  7228x required field missing
  738x wrong item count
  540x type mismatch
  317x Unclosed code block
  190x extraneous field 'notes'
  181x extraneous field 'path'
  175x extraneous field 'constraints'
  170x extraneous field 'type'
  170x missing code block
  100x expected bare list, got wrapper

The IFStruct release blog reports 21.1% for LFM2.5-350M. Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.

GRPO Fine-tuning with TRL on Structured Outputs

The full, runnable pipeline lives in the accompanying notebook. We will cover only the relevant pieces in this section.

Training data

We use nvidia/Nemotron-RL-instruction_following-structured_outputs, which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training.

Because the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two gaps between them:

  • 40% get a "return the output inside a fenced code block" instruction appended, so the model learns to follow the format instruction rather than always emitting raw JSON.
  • A disjoint 20% are converted into top-level-array tasks (the schema is wrapped in an array with a required item count), which trains bare-list output and item-count compliance.

Model and LoRA

We load LiquidAI/LFM2.5-350M and attach a LoRA adapter. Because LFM2.5 uses a hybrid attention/convolution architecture, we target the LFM-specific module names:

PYTHON
lora_config = LoraConfig(
    r=16, 
    lora_alpha=32, 
    bias="none", 
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
        "w1", "w2", "w3",
    ],
)

This trains ~6M parameters, about 1.66% of the model.

Reward functions

Then we define three reward functions, each on a [0, 1] scale, which score every completion on whether the extracted structure is correct:

  • json_format_reward: Is the output parseable, and in the requested form? Full credit (1.0) for the requested form (fenced vs. raw), 0.2 for the wrong-but-parseable form, 0.0 for unparseable output.
  • field_count_reward: Does the object have the expected number of top-level fields? An exact match earns 1.0, and the score decays linearly with the miss.
  • schema_validation_reward: Does the output validate against the row's JSON Schema? It counts every constraint violation and gates partial credit on required-key coverage.

We combine the three as a weighted sum with reward_weights=[1.0, 0.5, 2.0].

Training

We train for 100 steps with 8 generations per prompt group, sized for a free-tier 16 GB GPU:

PYTHON
from trl import GRPOConfig



> *[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]*


training_args = GRPOConfig(
    output_dir="./outputs/lfm25-350m-nemotron-schema-grpo",
    learning_rate=5e-5,
    max_steps=100,
    warmup_steps=10,
    num_generations=8,              
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,  
    steps_per_generation=2,
    max_completion_length=1024,     
    mask_truncated_completions=False,
    temperature=1.1,                
    beta=0.01,                      
    reward_weights=[1.0, 0.5, 2.0], 
    logging_steps=1,
    save_steps=100,
)

As you can see in the notebook, over the run, all three reward components climb, the KL from the reference model lifts off zero after warmup, and the truncated-completion fraction stays near zero.

Merging and saving the model

Finally, we merge the LoRA adapter back into the base weights and save it as a single self-contained checkpoint, ready to convert to GGUF for serving:

PYTHON
MERGED_DIR = f"{training_args.output_dir}-merged"

merged_model = trainer.model.merge_and_unload()
merged_model.save_pretrained(MERGED_DIR)
tokenizer.save_pretrained(MERGED_DIR)

IFStruct Evaluation on GRPO Tuned LFM2.5-350M

After GRPO fine-tuning, we rerun the IFStruct evaluation. For this, we need to convert the merged model checkpoint into a BF16 GGUF. The converter script ships with the llama.cpp source, so we clone the repo once and install the converter's gguf package.

BASH
git clone --depth 1 https://github.com/ggml-org/llama.cpp
pip install ./llama.cpp/gguf-py

mkdir -p models
python llama.cpp/convert_hf_to_gguf.py \
  PATH_TO_YOUR_MERGED_MODEL \
  --outfile ./models/lfm25-350m-grpo-bf16.gguf \
  --outtype bf16

Then we serve the merged model with the following command:

BASH
llama-server \
  -m ./models/lfm25-350m-grpo-bf16.gguf \
  --alias lfm25-350m-grpo-structured-output \
  -c 32768 \
  -np 4 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8081

Then, we will run the full IFStruct evaluation again with the fine-tuned model:

BASH
uv run ifstruct-eval \
  --model lfm25-350m-grpo-structured-output \
  --base-url http://localhost:8081/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm25-350m-grpo.json \
  --n-threads 4 \
  --max-tokens 2048 \
  -v
YAML
============================================================
Model: lfm25-350m-grpo-structured-output
============================================================
Overall: 594/2000 passed (29.7%)
Average latency: 1518ms

By format:
  JSON: 319/1000 passed (31.9%)
  YAML: 275/1000 passed (27.5%)

By top-level structure:
  Wrapper key 300/1011 passed (29.7%)
  Bare list   294/989 passed (29.7%)

By entity type:
  test__camera_review                 5/83 passed (6.0%)
  test__clinical_trial                31/104 passed (29.8%)
  test__conference_schedule           11/87 passed (12.6%)
  test__escaping__bug_report_batch    32/89 passed (36.0%)
  test__escaping__config_snippet_audit 24/85 passed (28.2%)
  test__escaping__customer_email_thread 9/73 passed (12.3%)
  test__escaping__dialogue_sample     17/95 passed (17.9%)
  test__escaping__interview_transcript_segment 13/80 passed (16.2%)
  test__escaping__log_parser_examples 33/72 passed (45.8%)
  test__escaping__pr_discussion       26/87 passed (29.9%)
  test__escaping__repro_steps_batch   23/73 passed (31.5%)
  test__escaping__screenplay_scene    34/92 passed (37.0%)
  test__escaping__short_story_chapter 24/84 passed (28.6%)
  test__escaping__support_ticket_batch 36/73 passed (49.3%)
  test__escaping__terminal_session_notes 23/70 passed (32.9%)
  test__event_ticket_booking          62/107 passed (57.9%)
  test__gpu_review                    7/94 passed (7.4%)
  test__invoice                       36/86 passed (41.9%)
  test__job_posting                   33/85 passed (38.8%)
  test__real_estate_listing           32/82 passed (39.0%)
  test__recipe                        7/70 passed (10.0%)
  test__rental_car_booking            37/79 passed (46.8%)
  test__scientific_experiment         14/69 passed (20.3%)
  test__travel_itinerary              25/81 passed (30.9%)

Common errors:
  7331x required field missing
  890x wrong item count
  555x type mismatch
  102x expected bare list, got wrapper
   62x extraneous field 'metadata.tone'
   55x 6 is greater than maximum 5
   49x extraneous field 'speaker_labels'
   47x extraneous field 'tone'
   44x 'cups' not in allowed values ['mg', 'g', 'kg', 'oz', 'lb', 'ml', 'l', 'cl', 'dl'
   44x extraneous field 'notes'

Comparing the two runs on the identical serving stack:

IFStruct groupbaseGRPO-tunedΔ
Overall22.6%29.7%+7.1
JSON18.0%31.9%+13.9
YAML27.2%27.5%+0.3
Wrapper key28.5%29.7%+1.2
Bare list16.6%29.7%+13.1

The gains land exactly where the training aimed: the JSON pass rate rises by nearly 14 points (18.0% → 31.9%), while YAML stays mostly the same. While this is still below the Qwen3.5-2B score of 33.15%, it shows that even light task-specific fine-tuning can bring a small model close to a larger one.

Conclusion

A short GRPO run with about 500 samples and 100 steps can lift a small 350M parameter model from 22.6% to 29.7% on IFStruct. The takeaway is that a cheap, task-specific reward signal can make a small model substantially more reliable about form, closing much of the gap to models several times its size.

To reproduce or extend this work, see the original IFStruct v1.0 blog post, the Liquid4All/ifstruct benchmark repo, and the LiquidAI/ifstruct-v1.0 dataset.