What Is a Token? Train a Tokenizer and Watch One Form
A language model never sees your letters. Before anything else happens, your text is chopped into tokens — chunks that are usually bigger than a character and smaller than a word — and the model only ever works with those chunks and their numeric ids.
That single fact explains a surprising amount: why you are billed per token rather than per word, why a model can write flawless code but miscount the letters in strawberry, and why the same paragraph costs noticeably more in Hindi or Thai than in English.
The tokens are not chosen by a human. The vocabulary is learned from data, and you can watch it happen.
Train one yourself
The trainer below implements byte-pair encoding, the algorithm behind the tokenizers in GPT-2, Llama, Gemma and Qwen2. It runs on whatever you type. Press the button and watch the most frequent adjacent pair get fused into a new token, over and over.
With the corpus above, the two most frequent adjacent pairs are e+s
(9 occurrences, across newest and widest) and then es+t
(9). Merging those two gives the subword est — a real English suffix, discovered
by counting alone. Let it run to six merges and the corpus tokenizes as:
- ×5
low - ×2
lower - ×6
newest - ×3
widest
Six merges take this corpus from 79 tokens to 35, with a vocabulary of 8.
No pair repeats any more, so training stops here. A real tokenizer keeps going for tens of thousands of merges because its corpus is the size of the internet, not four words.
- Merges learned
- 0
- Vocabulary
- 0
- Tokens
- 0
- Chars per token
- 0
How the corpus splits right now
Candidate pairs, most frequent first
Merge rules learned so far
Edit the corpus and it retrains from scratch. Try adding a word ten times and watch it become a single token.
The corpus it starts with is the worked example from the paper that introduced BPE for language
models, Sennrich, Haddow & Birch (2015): low five times,
lower twice, newest six times, widest three times. Run two merges and something interesting
appears: est. Nobody told the tokenizer that English has a superlative suffix. It merged e
with s because that pair was the most common, then merged es with t for the same reason,
and a real piece of English morphology fell out of pure frequency counting.
That is the whole idea. Common sequences become single tokens; rare ones stay in pieces.
The algorithm, in about thirty lines
Here is the same thing in Python. It runs in your browser — press Run.
from collections import Counter
corpus = "low " * 5 + "lower " * 2 + "newest " * 6 + "widest " * 3
words = {w: c for w, c in Counter(corpus.split()).items()}
splits = {w: list(w) for w in words}
def count_pairs(splits, words):
pairs = Counter()
for word, symbols in splits.items():
for a, b in zip(symbols, symbols[1:]):
pairs[(a, b)] += words[word]
return pairs
def merge(splits, a, b):
for word, symbols in splits.items():
out, i = [], 0
while i < len(symbols):
if i + 1 < len(symbols) and symbols[i] == a and symbols[i + 1] == b:
out.append(a + b)
i += 2
else:
out.append(symbols[i])
i += 1
splits[word] = out
for step in range(4):
pairs = count_pairs(splits, words)
(a, b), n = pairs.most_common(1)[0]
if n < 2:
break
merge(splits, a, b)
print(f"merge {step + 1}: {a!r} + {b!r} seen {n} times -> {a + b!r}")
print()
for word, symbols in splits.items():
print(f"{word:<7} {' | '.join(symbols)}")merge 1: 'e' + 's' seen 9 times -> 'es' merge 2: 'es' + 't' seen 9 times -> 'est' merge 3: 'l' + 'o' seen 7 times -> 'lo' merge 4: 'lo' + 'w' seen 7 times -> 'low' low low lower low | e | r newest n | e | w | est widest w | i | d | est
Three functions: count every adjacent pair, find the most frequent, fuse it everywhere. Repeat until the vocabulary is the size you wanted. A production tokenizer differs in scale and engineering, not in concept — GPT-2 ran this to 50,000 merges.
What the vocabulary size actually means
Per the Hugging Face tokenizer documentation, GPT-2’s vocabulary of 50,257 breaks down as 256 byte tokens, plus 50,000 learned merges, plus one end-of-text token. The original GPT used 40,478 — that is 478 base characters plus 40,000 merges.
So a vocabulary size is really a budget: base alphabet plus however many merge rules you chose to learn. Set it low and common words get chopped into fragments. Set it high and the embedding table grows.
The 256 is worth pausing on. Early BPE used characters as its base, which is a problem when your base alphabet has to cover all of Unicode. Byte-level BPE uses the 256 possible byte values instead, which guarantees every conceivable input can be tokenized without an “unknown” token — at the cost that one unusual character may take several tokens on its own.
Three things this explains
Why models are bad at spelling. Ask a model how many r’s are in strawberry and it is working with two or three opaque chunks, not ten letters. The letters were destroyed before the model saw anything. It is being asked to report on something it structurally cannot see — closer to asking you the pixel count of a word you just read than to asking you to spell it.
Why non-English text costs more. Merges are learned from the training corpus. If that corpus is mostly English, the frequent English sequences win the merge slots, and text in a under-represented language falls back toward byte-level fragments. Same meaning, same paragraph, more tokens — and tokens are the unit of both billing and context limits.
Why token counts never match word counts. There is no rule tying them together. A common word is often one token; a rare name might be four. Any “roughly 4 characters per token” figure is a statistical average over English prose, not a conversion rate — and the tool above prints the real ratio for whatever you paste into it.
What this demo simplifies
Being straight about the gap between the toy and the real thing:
- No end-of-word marker. The original paper appends
</w>to each word so the tokenizer can distinguish word-finalestfromestin the middle of a word. Merges here cannot cross whitespace, so at this scale the distinction never arises. - Characters, not bytes. The base vocabulary is the characters in your text, not the 256 byte values. Byte-level is what makes a real tokenizer total over all of Unicode.
- Ties are broken by first-seen. When two pairs tie on frequency, this picks the one encountered first scanning left to right. Real implementations vary; the runners-up are shown so you can see when a tie happened.
- No pre-tokenization rules. Production tokenizers apply regex rules first, handling punctuation and leading spaces. This splits on whitespace and nothing else.
None of these change the mechanism. They change how the mechanism behaves at the edges.
Where to go next
Byte-pair encoding is a frequency-counting algorithm over strings, and the Python above is mostly dictionary and string work. If any of it looked unfamiliar, the linked lessons below cover exactly those pieces — and they run in the browser the same way this page does.
Sources
- Sennrich, Haddow & Birch — Neural Machine Translation of Rare Words with Subword Units (2015)
The paper that introduced byte-pair encoding as a subword segmentation method for language models, and the source of the low/lower/newest/widest worked example. - Hugging Face Transformers — Tokenization algorithms
The step-by-step description of BPE training, the byte-level variant, and the stated vocabulary sizes for GPT (40,478) and GPT-2 (50,257), plus the list of current model families that use BPE.