Hugging Face 发布 tokenizers v1:聚焦大规模训练与推理性能优化
Hugging Face 正式推出 tokenizers v1,核心目标是解决在大规模数据集训练和高并发推理场景下,分词器成为数据供给瓶颈的问题。新版本通过深度性能优化,确保在模型计算速度提升和工作负载扩展时,分词过程不再拖慢整体 ML 流水线。此外,还预告了基于 GPU 的并行解码原型,旨在进一步加速大批量处理。
关联知识实体
报道正文
本文目录9 个章节
tokenizers v1:编码、解码与扩展性实测
发布于 2026 年 9 月 21 日 在 GitHub 上更新 点赞 88
Arthur Zucker ArthurZ Simon Brandeis sbrandeis Luc Georges mcpotato Lysandre lysandre
从历史上看,分词器(tokenizer)并非机器学习工作流中的瓶颈。就计算量而言,与流水线后续阶段繁重的模型训练相比,分词操作非常轻量。然而在某些场景下,它已迅速成为加速(或拖慢)机器学习任务的关键因素。
随着模型速度提升和工作负载规模扩大,这种平衡开始发生转变。在海量数据集上进行训练、处理大量并发请求,或反复处理长输入文本,都可能给分词器带来巨大压力,导致其无法及时向模型提供数据,从而造成 GPU 闲置等待。
正因如此,我们在即将发布的 tokenizers v1 版本中重点优化了性能。分词应当保持轻量,并能够随你的工作流灵活扩展。GPU 绝不应因等待 CPU 完成分词而处于空闲状态。
在本文中,我们将探讨 v1 版本为何比 v0.23 版本快得多——通常能实现数十倍的性能提升。
这一成果完全得益于整个开源生态系统的协作。分词是一个极为活跃的开源领域,诸如 gigatoken、tiktoken、kitoken、tokie、fastokens、wordchipper 和 ai-tokenizer 等库,以及其他众多项目,各自推动了高性能分词器的发展边界。我们深入研究了这些工作,下文提到的若干优化思路之所以被采纳,正是因为其他项目证明了它们的价值。
在此次重构之前,tokenizers 远未达到其应有的性能水平,因此参与贡献可能显得缺乏吸引力。通过这次重构,我们希望明确传达一个信号:tokenizers 将成为一个值得社区投入精力去贡献的库。
此外,我们要感谢 IBM、NVIDIA 以及 ExecuTorch 团队,他们不仅提交了补丁代码,还协助我们在多种硬件平台上进行测试,从而拓宽了对不同平台的支持范围。
结果
我们展示了 tokenizers v1 发布候选版本与其他广泛使用的替代方案的性能对比。涵盖内容包括:单线程性能、多线程性能、线程扩展性、按模型对比、按语言对比、延迟、解码吞吐量、内存堆占用以及 crate 大小。
这些基准测试运行自 tokbench 仓库,并提供了在您的硬件上重新运行基准测试的命令,供您按需使用。
V1 是什么
v1 将生成与 v0.23 相同的 Token ID。我们的目标是保持输出、API、词汇表(vocabulary)和合并等级(merge ranks)不变,并改进所有可以优化的方面,其中包括功能的广度。该库在不同分词器家族之间保持通用性,而非仅针对 BPE 进行特化,因此 v1 能够加载 v0.23 支持的所有内容。
分词器将文本转换为模型读取的整数列表。tokenizers 通过四个阶段执行这一转换过程。规范化(Normalization)对原始文本应用小写转换或 Unicode 规范化等操作。预分词(Pre-tokenization)将文本分割成更小的片段,称为预 Token(pre-tokens)。模型阶段将每个预 Token 转换为 Token,并将其映射到词汇表中的 ID。后处理(Post-processing)则添加模型所需的任何特殊 Token。
本文描述的大部分工作都发生在模型阶段。本文测量的十个模型家族中,有八个使用字节对编码(Byte Pair Encoding, BPE)。BPE 从预 Token 的字节开始,反复合并排名最高的相邻字节对,直到没有可合并的排名对为止。这种排名是在训练分词器时学习得到的,并随分词器一起发布,因此相同的文本始终产生相同的 ID。合并操作永远不会跨越预 Token 边界。另外两个家族使用 WordPiece 和 Unigram,这是该库支持的另外两种模型类型。
分词流水线页面详细记录了这四个阶段。分词算法文档介绍了 BPE、WordPiece 和 Unigram。
我们对每个阶段都进行了优化。以下是关键改动:
| 变更 | 作用 |
|---|---|
| workspace 拆分 | 单个 crate 变为一个 workspace:tk-encode 是必需的运行时,而 tk-serialize、tk-convert 和 tk-train 仅在应用程序需要时才链接 |
| 无分配模型 | 合并工作集位于调用方拥有的 scratch buffer(暂存缓冲区)中;循环从不触及分配器 |
| bitcannon | 拆分模式变为对比特流执行布尔运算,使用 SIMD 指令查找分割点,而非依赖正则表达式引擎 |
| 合并循环重写 | 待合并的片段在一个预分配缓冲区内形成侵入式双向链表,因此一次合并只需更新两个索引,而不是移动数据 |
| 词缓存 | 线程本地的备忘录,将预 token 字节映射到最终 ID,从而确保重复单词只被合并一次 |
| 原生并行性 | 一个共享 tokenizer 可同时从多个线程进行编码;每个线程从其专属子池中获取 scratch buffer 和词缓存,因此线程不再因单一锁而排队 (#2365) |
拆分:用比特流代替正则表达式
BPE 模型使用正则表达式将输入文本拆分为更小、更易处理的块,称为 pre-tokens。合并操作发生在 pre-token 内部,绝不会跨越两个 pre-token 之间的边界,因此这种拆分决定了流水线后续阶段看到的内容。
该正则表达式是模型的固定参数。它随 tokenizer 一起发布,且在运行时永不改变,因此无需在每次编码时都让通用正则引擎来解释它。可以针对给定模型实际使用的模式,一次性手写实现等效的拆分函数。
手写函数随后可以利用现代 CPU 的 SIMD 指令(单指令多数据),这些指令同时对多个字节应用同一操作,非常适合处理 UTF-8 文本。bitcannon 将输入的字节视为并行的比特流,使得边界通过整个寄存器上的布尔运算自然得出,而不是逐字符扫描。它每次寄存器操作可决定 64 个字节。同样的思路也驱动了用于文本处理的 Parabix 和用于 JSON 处理的 simdjson。
这依赖于识别出特定的模式。少数几种语法涵盖了大多数 byte-level BPE 模型,而对于模式不在其中的 tokenizer,则保留正则路径,无法获得上述加速效果。这就是为什么前文提到的性能提升幅度差异如此之大。
The Word Cache
Real text contains many repeated words. Because BPE always produces the same token IDs for a given pre-token, v1 can save the result after processing it once. A thread-local cache maps each pre-token's bytes to its token IDs, allowing later occurrences to skip the merge process.
Naturally, as the input grows, the number of unique words can grow more slowly than the total number of words. Repeated words then account for an increasing share of the input. New words still appear, which accounts for the occasional misses in the animation below.
Reproduce the shared-prefix result with:
tokbench measure prefix-sharing \
--engine pipeline \
--engine hf-tokenizers \
--compare-to pipeline-no-cache \
--corpus agentic_sweCaching works best when the input contains repeated pre-tokens. Input with few repeated pre-tokens can pay for lookups without receiving many hits.
The Merge Loop
The next major cost comes from the BPE merge loop. For each pre-token, the loop repeatedly finds the highest-priority adjacent pair and merges it. The previous implementation allocated new memory for every call and built a new priority queue for every pre-token.
v1 reuses a scratch buffer owned by the caller, removing those repeated allocations. It stores symbols in a flat array and links adjacent symbols by their positions in that array, which makes updates during merging cheaper. It also processes a batch of pre-tokens in a single model call.
Each candidate pair is also packed into a single 64-bit value, with the merge rank in the high bits. Comparing two candidates is then just comparing two integers, and "no merge here" is the largest possible value, so the loop finds its next merge without a branch.
Method
Small differences in benchmark design can produce large differences in tokenizer performance. We used the following rules to keep the comparison consistent across engines.
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
| rule | why |
|---|---|
| one timing loop | every engine runs the identical loop; no per-engine fast path |
| load excluded | vocabulary load is timed separately, never inside encode |
| id-hash verified | FNV-1a over the output ids must match the baseline exactly |
| common cells only | medians are over cells every engine ran and verified |
| complete sweep per process | each repeat starts in a new process and retains every cell |
| physical-core pinning | workers are pinned to eight distinct physical cores, never sibling SMT threads |
| independent Jobs | separate Jobs measure host-to-host variation |
Repeatedly encoding one document can be faster than encoding a stream of distinct documents on the same build. The first approach measures performance when the entire document is already represented in the cache. The second measures performance on new input while allowing previously seen pre-tokens to remain cached.
Both conditions are sometimes described as "warm," even though they measure different workloads. Our headline results use distinct documents, and the complete corpus is too large to fit in the cache. Tokenizer benchmarks should identify which workload they use because the choice can dominate the result.
What This Adds Up To
Across the ten model families v1's encode path covers, it encodes text 3 to 30 times faster than v0.23 with one thread on an Apple M4 Max. The low end is t5-base, the high end gpt2. It scales at 76% of linear across eight workers. Throughout these changes, v1 produces exactly the same token IDs as the released library.
The overall improvement comes from several changes working together: a hand-written splitter in place of a regex engine, a cache that answers a repeated word without merging it again, a merge loop that never touches the allocator, and one model call per batch of pre-tokens instead of one per pre-token. Each reduces the work done at a different point in the pipeline.
The next priority is support for more model families. We will move additional models onto the new merge loop before 1.0.0. Once the release candidates stabilize, the next step will be bringing about the improvements within the transformers library and the rest of the ecosystem which depend on the tokenizers library.
This post is generated from tokbench results and will be updated as support expands.
Getting It
A release candidate for v1 is on crates.io. The API you call is the one you already call, so the only thing that changes is which build you install.
It is the ordinary install:
cargo add tokenizers --preTraining is behind a default-on feature that pulls a C++ dependency with it. If you only need to encode, turn it off to exclude the training implementation:
cargo add tokenizers --pre --no-default-features --features httpEncoding is unchanged: same call, same ids.
use tokenizers::tokenizer::{Result, Tokenizer};
fn main() -> Result<()> {
let tokenizer = Tokenizer::from_pretrained("deepseek-ai/DeepSeek-V4-Flash", None)?;
let encoding = tokenizer.encode("The tokenizer is no longer the bottleneck.", false)?; println!("{:?}", encoding.get_ids());
Ok(()) } ```
For a batch, `encode_batch` is what scales across cores. It is the call the scaling view above measures.
let encodings = tokenizer.encode_batch(documents, false)?;
Every figure in this post was measured against this crate. The Python bindings wrap the same code and are built from `bindings/python`, but they add per-call overhead that none of these measurements include.
## Progress Towards V1
The benchmarks in this post cover the completed release-candidate work listed first. The remaining sections show what is still required for `1.0.0` and what we plan to explore afterward.
### Release Candidate: Implemented
This work is in the Rust pre-release on crates.io:
cargo add tokenizers --pre
- workspace split: divide the single crate into `tk-encode`, `tk-serialize`, `tk-convert` and `tk-train`, so an application links only what it uses
- bitcannon: replace regex splitting on the encoding path with bitstream operations covering GPT-2, cl100k, o200k, Tekken and DeepSeek. This replaced the finite-state machines that shipped first [#2201](https://github.com/huggingface/tokenizers/pull/2201) [#2317](https://github.com/huggingface/tokenizers/pull/2317)
- WordCache: reuse the token IDs of previously processed pre-tokens [#2262](https://github.com/huggingface/tokenizers/pull/2262), `af5a3e3`
- faster lookup and merging structures: add FlatCache, MPHF RankStore, incremental merging, and BucketVocabStore [#2190](https://github.com/huggingface/tokenizers/pull/2190) [#2188](https://github.com/huggingface/tokenizers/pull/2188)
- reusable model memory: move temporary model state into scratch buffers so tokenization does not allocate on each call [#2175](https://github.com/huggingface/tokenizers/pull/2175) [#2183](https://github.com/huggingface/tokenizers/pull/2183)
- pipeline post-processing: expose post-processing as the `STAGE_POST` pipeline stage [#2182](https://github.com/huggingface/tokenizers/pull/2182)
- batched model calls: process multiple pre-token spans in one call [#2304](https://github.com/huggingface/tokenizers/pull/2304)
- faster decoding: write decoded bytes directly into a reusable buffer, avoid intermediate strings and copies, accelerate token lookup, support buffered streaming, and decode batches in parallel
- `role_to_token` support [#2343](https://github.com/huggingface/tokenizers/pull/2343)
- Node.js bindings [#2281](https://github.com/huggingface/tokenizers/pull/2281)
### 1.0.0
- one encoding implementation: use `tk-encode` during training validation so training and inference cannot produce different tokenization results
- optional offsets and masks: compute this metadata only when requested, keeping it off the token-ID-only path
- rework normalizers
- bitnorm support, building on atomnorm [#2209](https://github.com/huggingface/tokenizers/pull/2209)
- spm precompiled
- simpler Python bindings: reduce locking, wrapper types, and handwritten dispatch code while preserving subclassing, serialization, custom decoders, mutation behavior, and support for free-threaded CPython
- inference-only C and C++ bindings for ExecuTorch and llama.cpp, with possible JVM, Swift, and Go bindings to follow
### After 1.0.0
- tok-devices: explore GPU encoding and batch decoding while keeping text and token IDs on the device. The decoder would upload the vocabulary once, calculate output positions in parallel, and gather the corresponding bytes on the GPU. This would be an optional component intended for large batches, subject to further prototyping and measurement.