What sort of setups do people have that are bounded by the speed of the tokenizer?
rhdunn•Jul 22, 2026
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
charcircuit•Jul 22, 2026
But are those bounded on the speed of tokenization?
andersa•Jul 22, 2026
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
fwip•Jul 22, 2026
That's fair, I just figure there are useful scenarios as well. Apologies if I came off as dismissive!
ac2u•Jul 22, 2026
It didn’t come off as dismissive to me. I was curious as well as to where such optimizing helps and knew that the answers to your question would help me discover use cases I didn’t think of
marcelroed•Jul 22, 2026
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
You can, but this usually results in sequences with padding/truncation, since you won't know how many tokens your inputs map to before you actually tokenize them. This also makes shuffling difficult.
In practice every training project I've worked on does tokenization in a separate data processing phase.
fwip•Jul 22, 2026
Very cool, thanks.
imperio59•Jul 22, 2026
Pre-training data is pre-tokenized ahead of time before being used to not waste any GPU compute.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
janalsncm•Jul 22, 2026
If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
avereveard•Jul 22, 2026
I've data where i cannot store metadata that i need to search semantically so i embed it on the fly at every search with static embedding and tokenizing was more than 99% of the cpu time. Granted that was due the naive implementation of the default tokenizer which was o^2 with document length and just switching to a proper scanner solved most of it without going to simd and whatnot, but still.
maxdo•Jul 22, 2026
Interesting :
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast?
No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
sashank_1509•Jul 22, 2026
This is really cool, great work!
0xnyn•Jul 22, 2026
I had to stare at that chart for a minute just to let the numbers sink in. It's genuinely mind-bending, incredible ship OP
onlyrealcuzzo•Jul 22, 2026
This is awesome, but tokenization is typically <0.1% of total inference time.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
GenerocUsername•Jul 22, 2026
Always good to make it 0.001%
pipsterwo•Jul 22, 2026
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload
The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
scottcha•Jul 22, 2026
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
semiinfinitely•Jul 22, 2026
quite excellent software
zerolines•Jul 22, 2026
wow, best release all week.
dmezzetti•Jul 22, 2026
Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?
marcelroed•Jul 22, 2026
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.
marcelroed•Jul 22, 2026
I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.
9 Comments
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
[0] https://github.com/crusoecloud/fastokens
In practice every training project I've worked on does tokenization in a separate data processing phase.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...