How Many R's in Strawberry? Why That Question Is Unfair

Ask a model that writes flawless code how many times the letter r appears in strawberry and you may well get two. The failure is so reliable it became a meme, and it is usually offered as evidence that these systems only pretend to reason.

It is better evidence of something else: the model was never shown the letters.

The word arrives already broken

Before any thinking happens, your text is split into tokens — chunks that are usually bigger than a character and smaller than a word. By the time the model receives strawberry, it is holding a handful of opaque pieces.

# An ILLUSTRATIVE split. Real tokenizers differ, and we do not ship a GPT vocabulary, so this
# is not a claim about what any particular model produces. What every tokenizer DOES do is
# replace the letters with a smaller number of opaque chunks — and that is the whole problem.
chunks = ["str", "aw", "berry"]
word = "strawberry"

print("you typed :", word)
print("it receives:", " | ".join(chunks))
print()
print("Letters you can see:", len(word))
print("Pieces it receives :", len(chunks))
print()

for c in chunks:
    print(f"  chunk {c!r:>9} -> {c.count('r')} r")

print()
print(f"Correct answer: {sum(c.count('r') for c in chunks)}")
print()
print("Note what reaching that required: knowing how each chunk is SPELLED.")
print("The model cannot look - it has to recall the spelling as a fact, then add.")
print("That is arithmetic over memorised trivia, not counting.")

you typed : strawberry
it receives: str | aw | berry

Letters you can see: 10
Pieces it receives : 3

  chunk     'str' -> 1 r
  chunk      'aw' -> 0 r
  chunk   'berry' -> 2 r

Correct answer: 3

Note what reaching that required: knowing how each chunk is SPELLED.
The model cannot look - it has to recall the spelling as a fact, then add.
That is arithmetic over memorised trivia, not counting.

That split is illustrative — real tokenizers differ, and we will not claim a specific one we cannot verify. What is not illustrative is the shape: ten letters go in, a few chunks come out, and the letters are no longer individually addressable.

Watch the letters disappear

You do not have to take that on faith. The trainer below is a real byte-pair encoding implementation, the algorithm from Sennrich, Haddow & Birch (2015) that GPT-2 and Llama use. It starts with individual characters and repeatedly merges the most frequent adjacent pair.

Run it to the end and strawberry is one token. Ask that token how many r’s it contains and there is nothing to count: the question is about characters, and characters are not what the model is holding.

The comparison that makes it land

Imagine being handed three cards reading str, aw and berry, then being asked how many r glyphs are printed on them — without being allowed to look at the cards. You would have to remember what each card says, recall its spelling, and add up.

You could do it. You would get it wrong sometimes, especially at speed, and especially for unusual words you had rarely written out.

That is precisely the model’s situation, and it explains the pattern of failures better than “it cannot reason” does:

• Common words are easier — their spelling appears constantly in training text.
• Rare and made-up words are harder — the model has less evidence about how they are spelled.
• Asking it to spell the word out first often fixes the count — once the letters exist as separate tokens in the conversation, they become countable objects rather than recalled facts.

That last one is the tell. If the failure were a reasoning failure, spelling the word out first would not help. It helps because it changes what the model can see.

Two questions people run together

Can the model see letters? Mostly no. It sees tokens, and the mapping from token to letters is knowledge it has absorbed, not something it can inspect.

Can the model reason about characters? Often yes — given the characters. Split the word up, or ask for code that does the counting, and the same model handles it easily.

Conflating those two makes the model look worse at reasoning and better at perception than it is. It is the reverse: the perception is genuinely missing, and the reasoning is mostly fine once you supply what perception would have given it.

What this does and does not excuse

It is an explanation, not a defence. A system that confidently answers two has still given a wrong answer with no signal that it was working from a guess, and confident wrongness is the real problem — not the miscount.

The useful lesson is about the shape of the tool. Character-level work — counting letters, reversing strings, spotting a typo, rhyming precisely — sits in a blind spot created before the model starts. For those tasks, ask it to write code that operates on the characters instead. The code sees the letters. The model never did.

What this page simplifies

• The split shown in the code is not a real tokenizer’s output. It is illustrative, and the page says so where it appears.
• The trainer learns from your corpus, not the internet. It is genuine BPE on a small vocabulary; a production tokenizer runs the same algorithm to tens of thousands of merges over an enormous corpus.
• Byte-level tokenizers add a wrinkle. Per the tokenizer summary, GPT-2 uses 256 byte values as its base vocabulary rather than characters, so an unusual character may itself span several tokens.
• Models have improved at this. Some now handle the strawberry question correctly, generally because the pattern is common enough in training data to be recalled directly. That is memory, not perception — the blind spot is unchanged.