Termsslug: vocabulary

Vocabulary

词汇表

Set of all possible input tokens defined by the model

Curated Intelligence

2 items · Page 1/1
Rohan Paul (@rohanpaul_ai)T2·Models
72 pts
Stanford Paper Finds No Metric Is Fully Tokenizer-Agnostic

A new Stanford paper finds that no metric is tokenizer-agnostic in every sense, including bits-per-byte (BPB), which is widely treated as neutral. Untrained models showed different BPB across tokenizers, and BPB correlated with task accuracy only between +0.02 and −0.30. The authors advise checking task accuracy on the same text rather than trusting a single score when comparing tokenizers.

Hugging Face BlogT1·Tools
58 pts
Transformers v5 Refactors Tokenizers: Decoupling Design from Vocabulary

Hugging Face Transformers v5 significantly refactors the tokenizer architecture by decoupling tokenizer design from trained vocabularies, mirroring PyTorch's separation of network structure and weights. This change makes tokenizers more modular, clear, and customizable while retaining essential model-aware features like context lengths and chat templates.