Hugging Face 开源高效知识蒸馏实现,大幅降低显存占用
Hugging Face 联合 Multiverse Computing 发布并开源了一种新的知识蒸馏损失函数实现(Chunked KL Loss)。该方法通过分块计算显著降低了训练过程中的峰值显存占用,使大规模模型蒸馏在普通硬件上更具可行性。相关论文已公开,代码已在 GitHub 上线。
报道正文
本文目录6 个章节
让知识蒸馏的成本低到足以规模化运行
团队文章 发布于 2026 年 8 月 10 日 点赞 42
Antonio Tiene AntonioTN MultiverseComputingCAI Iker García-Ferrero Iker MultiverseComputingCAI Ali Hashemi ali-hashemi MultiverseComputingCAI Bakbergen Ryskulov bryskulov-mc MultiverseComputingCAI
知识蒸馏,即训练一个较小的学生模型以匹配较大教师模型的性能,是机器学习中一种广为人知的技术。随着近期开源大语言模型(LLM)的浪潮,例如 gpt-oss、Qwen、GLM 或 Kimi,它再次成为主流的研究热点。部署这些超大型模型成本高昂:近期的 Kimi-K3 模型拥有 2.8 万亿参数,仅加载就需要大约 3TB 的显存(VRAM)。因此,将它们压缩为更小的模型并通过知识蒸馏恢复原始能力已成为标准做法,Nvidia(Nemotron 3 Puzzle 75B)或 Multiverse Computing(Hypernova 60B)等公司最近都发布了高质量的压缩模型。
蒸馏步骤决定了最终质量的大部分内容,但通常也是整个流程中最昂贵的部分。同时加载教师模型和学生模型,并为每个 token 生成覆盖整个词表的概率分布,需要巨大的显存(VRAM),这通常只有在拥有数百个 GPU 并采用精细的张量并行策略时才能实现。我们最新的论文 Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss 通过两项系统级改进解决了这一问题:一次性缓存教师模型的 Top-K logits,从而避免教师模型与学生模型同时驻留内存;以及一种新的、内存高效的 KL 散度损失函数,该函数避免了实例化完整的“词表大小 × 序列长度”矩阵,从而将显存使用量大幅降低至远低于 PyTorch 或 NVIDIA Megatron-Bridge 等库中默认实现所需的水平。这两项改进共同显著降低了训练成本,使得在单个 GPU 上进行长上下文修复成为可能,并且成本低廉到足以支持大规模实验。
为什么蒸馏恢复如此昂贵
标准设置是使用 Kullback-Leibler 散度损失(KL 损失)进行在线蒸馏,这会同时保持教师模型和学生模型处于加载状态。在每个训练步骤中,教师模型执行一次完整的前向传播以生成其输出分布,而学生模型则被训练以匹配该分布。这是最具表达力的设置,因为可以获得完整的教师分布,但它也是最消耗内存和计算资源的:每个 token 位置都需要保留两个全词表大小的张量,而且即使教师模型的行为在整个训练过程中保持不变,也必须在每一步重新计算教师模型。
作为一个实际例子,gpt-oss-120b 的词表大小为 201,088 个 token。在序列长度为 32K、批次大小为 4 的情况下,仅教师概率张量的形状就是 4 × 201,088 × 32,768;在 bfloat16 精度下,这单个张量就已经占用约 50GB 的显存。再加上梯度、激活值、模型权重和优化器状态,单次蒸馏训练迭代的显存峰值可达约 250GB,这甚至超过了 H200 或 B200 GPU 所能提供的容量。在本文中,我们将展示如何通过重构 KL 损失以分块处理数据,从而将这一成本几乎降至零。
Dense KL spikes to roughly 250GB, above a single H200's 141GB capacity. The fused chunked loss never forms that spike and peaks at about 128GB. Source: paper Figure 1.
Two systems changes
Offline distillation. Instead of recomputing the teacher at every step, we compute its output once, cache the top-100 most likely tokens per position, and train the student against that cache. The teacher never has to sit in memory during training and does not need to be run again once the cache exists, so the same cache can be reused across many ablations.
A fused, chunked KL loss. To see why the loss itself is expensive, picture what it actually builds: for every token position in a sequence and every word in the vocabulary, the loss needs a number describing how much the student's prediction disagrees with the teacher's. Laid out as a grid, that's one row per vocabulary entry and one column per sequence position, for a vocabulary of 100K+ words and a long sequence, that grid is enormous, and the default way of computing a KL loss builds the whole thing before it can produce a single number.
We compare three ways of computing this same loss, all mathematically equivalent:
- Dense KL is the textbook approach. It rebuilds a full, dense teacher-probability grid from the cached top-100 logits and compares it against the student's own dense grid of log-probabilities. This is the version closest to how online distillation already works, so we use it as our correctness baseline, but it holds the full vocabulary × sequence grid in memory, twice over.
- Forward-chunked KL keeps the teacher sparse (only its cached top-100 logits per position, never expanded into a dense grid) and computes the loss piece by piece, one slice of sequence positions at a time. This removes the dense teacher and the dense comparison, and turns out to be the fastest of the three methods in our benchmarks. It still has one blind spot, though: the student's own logits, the grid produced by the model's output layer, are still computed in full and kept around for the backward pass, so memory still grows steeply with sequence length.
- Fused chunked KL, our main contribution, goes a step further and fuses the model's output projection directly into the loss computation. It never produces the student's full logits grid at all: it processes one chunk of the sequence at a time end to end, projecting hidden states to logits for that chunk, folding the result into the running loss, and discarding the chunk before moving to the next one. The backward pass recomputes each chunk on the fly instead of storing it. The cost is doing that projection twice, once forward, once in backward, but in exchange, peak memory grows only linearly with sequence length instead of spiking with the full vocabulary × sequence size.
The GIF below shows the difference between the dense and fused-chunked approaches: one builds the whole comparison grid and holds onto all of it, the other builds and discards one slice at a time, so memory never grows beyond a single chunk.
We have open-sourced the chunked-loss implementation: github.com/CompactifAI/Full-Chunked-KL-Loss
What this changes in practice
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
The table below puts all four setups head to head: online distillation, and the three offline loss implementations just described. Comparing them on a single H200 GPU with Llama 3.1 8B Instruct as teacher and a 3.2B Llama model as student at an 8K token context, all four reach near-identical training loss, even though the offline runs train against only the cached top-100 logits per token.
| Method (8K context, single H200) | Peak memory | Iteration time | Throughput |
|---|---|---|---|
| Online distillation | 102.8 GB | 25.9 s | 237 TFLOP/s |
| Offline, dense KL | 78.3 GB | 18.5 s | 331 TFLOP/s |
| Offline, forward-chunked KL | 61.8 GB | 18.4 s | 335 TFLOP/s |
| Offline, fused chunked KL | 58.3 GB | 20.2 s | 304 TFLOP/s |
The loss curves overlap almost exactly across all four methods, confirming offline distillation with top-100 cached logits is lossless relative to online distillation. Source: paper Figure 2. At this sequence length, the fused chunked loss is not yet the fastest option, its extra backward-pass projection costs a bit of speed, but its real advantage only shows up as context length grows, which the next section demonstrates.
Scaling to long context lengths
To see the scaling pattern more starkly, we ran an isolated benchmark on a toy output-projection network (no transformer body, just the loss kernel). At 32K tokens, peak memory falls from 85.2 GiB with the dense loss to 5.45 GiB with the fully chunked version, a 15.6× reduction, and the dense loss fails outright from 64K tokens onward. At 256K tokens, the fully chunked loss uses 11.6 GiB against 134.2 GiB for the next-best chunked variant, and is about 3.3× faster per iteration at that length.
Distilling a GPT-OSS 20B model at a 32,768-token context, the memory freed by the fused loss let the setup shrink from four GPU nodes down to one. Step time fell from 57.0 to 12.23 seconds, about 5× faster, and throughput per GPU rose from 74.2 to 345.7 TFLOP/s.
The resulting student
The efficient offline setup is what made a large-scale distillation campaign affordable in the first place. The resulting compact student, distilled from Llama 3.1 8B Instruct down to about 3.2B parameters, retains most of the teacher's accuracy on BoolQ and HellaSwag, stays within about nine points of it on MMLU, at less than half the parameter count.
The student retains most of the teacher's short-context accuracy at less than half the size. Source: paper Figure 6.
This work is part of Multiverse Computing's ongoing research into making distillation and healing practical to run at scale, not just as a one-off recipe, but as something teams can iterate on cheaply. The paper also covers additional ablations, such as how the choice of loss function and sequence packing affect recovery quality.
Want the full technical details, including the closed-form gradient behind the fused chunked loss and the complete training configuration? Read the full paper, or get in touch with our team to talk about applying this to your own distillation pipelines.
We have also open-sourced the chunked-loss implementation: github.com/CompactifAI/Full-Chunked-KL-Loss


