A new Stanford paper finds that no metric is tokenizer-agnostic in every sense, including bits-per-byte (BPB), which is widely treated as neutral. Untrained models showed different BPB across tokenizers, and BPB correlated with task accuracy only between +0.02 and −0.30. The authors advise checking task accuracy on the same text rather than trusting a single score when comparing tokenizers.
Tokenizer
分词器
A tool that converts raw text into numerical sequences (Token IDs) understandable by models
Curated Intelligence
A new study reveals that AI models consume up to 17 times more tokens for certain languages compared to others, creating a hidden 'tokenizer tax.' This disparity results in higher costs, slower responses, and reduced context windows for low-resource language users. Experts advocate for improved tokenizer designs and better data quality to address these linguistic inequalities.
The developer tool ttok has released version 1.0, updating its default tokenizer from GPT-4 to GPT-5/GPT-6. Although OpenAI has not officially confirmed the tokenizer consistency for new models, the author relied on third-party experimental data showing identical token counts across GPT-5 and GPT-6 fixtures to make this switch, aiming to provide more accurate cost estimation support for modern LLMs.
Anthropic has released Claude Haiku 5.5, a new lightweight model priced at $0.10/$0.50 to directly match OpenAI's GPT-6 Luna, addressing the high cost and aging performance of Haiku 4.5. However, the new model utilizes a less generous tokenizer, increasing token consumption by approximately 25% for identical prompts, which results in hidden cost escalations for long-context workloads. This move signals a shift in small-model competition from pure price wars to deeper battles over inference efficiency and billing mechanics.
Anthropic released Claude Haiku 5.5, cutting average costs by 75% and input prices by up to 90% for short prompts. The company also slashed pricing for Sonnet 5.5 and updated SDKs to support computer and browser use.
Hugging Face has released tokenizers v1, specifically targeting the bottleneck of tokenization in large-scale training and high-concurrency inference. The update focuses on deep performance optimizations to ensure tokenization does not starve models of data as compute speeds increase and workloads scale. It also previews a prototype for GPU-parallel decoding to further accelerate batch processing.
Hugging Face Transformers v5 significantly refactors the tokenizer architecture by decoupling tokenizer design from trained vocabularies, mirroring PyTorch's separation of network structure and weights. This change makes tokenizers more modular, clear, and customizable while retaining essential model-aware features like context lengths and chat templates.
Intel and Hugging Face introduced Universal Assisted Generation (UAG), a technique that accelerates LLM decoding using any assistant model without requiring strict tokenizer matching. By optimizing the draft token verification mechanism, UAG significantly improves inference throughput and is planned for integration into the Transformers library.
Hugging Face released a tutorial on building CodeParrot, a code generation model similar to GitHub Copilot, from scratch. It covers dataset extraction from BigQuery, custom tokenizer design, and GPT-2 architecture training. The resource primarily serves learners interested in Transformer principles and data engineering practices.
Hugging Face released a tutorial demonstrating how to train a small language model (EsperBERTo) from scratch using the Transformers and Tokenizers libraries. The 84M-parameter model is pre-trained on the low-resource language Esperanto and fine-tuned for part-of-speech tagging.