A new Stanford paper finds that no metric is tokenizer-agnostic in every sense, including bits-per-byte (BPB), which is widely treated as neutral. Untrained models showed different BPB across tokenizers, and BPB correlated with task accuracy only between +0.02 and −0.30. The authors advise checking task accuracy on the same text rather than trusting a single score when comparing tokenizers.
Termsslug: vocabulary
Vocabulary
词汇表
Set of all possible input tokens defined by the model
Curated Intelligence
Rohan Paul (@rohanpaul_ai)T2·Models
72 pts
Hugging Face BlogT1·Tools
58 pts
Hugging Face Transformers v5 significantly refactors the tokenizer architecture by decoupling tokenizer design from trained vocabularies, mirroring PyTorch's separation of network structure and weights. This change makes tokenizers more modular, clear, and customizable while retaining essential model-aware features like context lengths and chat templates.