Today for AI
HOT RADAR
llmHEAT 16.2°

AI CURATED HOT STORY · 11/8/2026

Salesforce Study Finds External Teacher Distillation Costs More Compute Than RL

1 reports archived1 independent sourcesupdated 11/8/2026, 08:17:53
Synthesis & Latest Updates
1 Sources Cross-Validated

A new Salesforce paper compares on-policy distillation against GRPO using held-out accuracy per training FLOP. It finds that distilling from a large external post-trained teacher generally costs more compute than plain RL and tends to inflate response length, whereas training per-domain RL experts and distilling them into a single student matches joint GRPO on math and code with 42% fewer total FLOPs.

LATEST/A new Salesforce paper compares on-policy distillation against GRPO using held-out accuracy per training FLOP. It finds that distilling from a large external post-trained teacher generally costs more compute than plain RL and tends to inflate response length, whereas training per-domain RL experts and distilling them into a single student matches joint GRPO on math and code with 42% fewer total FLOPs.

Hourly heat curve

2 fully observed hours
Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first
  1. Rohan Paul (@rohanpaul_ai)T2·74 pts

    Salesforce Study Finds External Teacher Distillation Costs More Compute Than RL

    • On-policy distillation requires a teacher forward pass per training token, so dense signals are not automatically cheaper
    • Measured by held-out accuracy per FLOP, distilling from an external post-trained teacher generally loses to GRPO and inflates response length
    • Training per-domain RL experts and distilling them into one student matches joint GRPO on math plus code with 42% fewer FLOPs