Hugging Face tokenizers v1 runs up to 30× faster

Hugging Face’s tokenizers v1 release candidate encodes text 3–30× faster than v0.23 on one Apple M4 Max while producing identical token IDs. In its tokenizers v1 benchmark, Hugging Face reports 76% of linear scaling across eight workers and offers the release candidate through cargo add tokenizers --pre; the headline numbers exclude Python binding overhead. For teams running high-concurrency inference or bulk preprocessing, this is a cheap CPU-side throughput test before buying GPU capacity, alongside the archive’s agent-search cost benchmark.