斯坦福论文：不存在完全与分词器无关的评估指标
斯坦福一项新研究发现，没有任何指标能在所有意义上做到与分词器无关，被广泛当作中立标准的 bits-per-byte（BPB）也不例外。未训练模型在不同词表下 BPB 差异明显，且 BPB 与任务准确率的相关性仅在 +0.02 到 −0.30 之间。作者建议比较不同分词器时不要只看单一分数，应同时核对同一文本上的任务准确率。
报道正文
Rohan Paul (@rohanpaul_ai) · 2026-11-05 20
New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly.
To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic.
They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.
Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30.
When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.
