Tokens are not words

❓ Is a piece a word? If the model continues English so well, why did the strawberry count fail?

That “piece” is not always a word. It is a token: an integer ID from a cut-up of your string. Letter-counting fails when the cut-up hides the letters.

Computers want numbers. The model’s first step is arithmetic. Arithmetic needs numbers. “Hello” is not a number.

So before anything clever happens, a separate program — the tokenizer — cuts your string into pieces and maps each piece to an ID. After the model emits an ID, the tokenizer glues the piece back into text. Two functions are the whole contract: encode() string → tokens, decode() tokens → string.

This stage is not the model. It has its own training data and its own recipe, usually Byte Pair Encoding (BPE) or a close relative: start from letters or bytes, repeatedly glue the pair that appears most often into a new token, until you have a fixed list of allowed pieces. After that list is chosen, it is frozen. The model never sees characters. It sees a sequence of integers from that list. The list is often tens of thousands to a few hundred thousand IDs.

You already pay for “tokens.” That billing unit is not a synonym for “word.”

What a token actually is

A token might be Paris, ing, a space, a newline, a whole Chinese character, or a fragment like straw + berry. One running example splits cleverest into cle|ve|rest. That is not a bug in the diagram. That is the representation.

A surprising fraction of “weird model behavior” is this stage: counting letters, rhyming, reversing strings, arithmetic on numbers that got split into pieces, handling trailing spaces, the difference between 2 and 2. The model can only be a genius about structure it can see.

What that implies for your apps

• Compression is uneven. “attention” might be one token. A name, a hex id, or a snippet of JSON might explode into many. That is why two prompts with similar characters have different window cost.
• The model has no privileged access to spelling. The strawberry quiz from the last page is this fact: if strawberry is one or two tokens, not nine letters, then “count the r’s” is a bad interface. You asked a next-piece machine a question about a representation it does not have.
• Leading spaces matter. Many tokenizers treat Hello and Hello as different IDs. Prompt templates that wrap values with inconsistent whitespace change the integers, not just the ink.
• You cannot mix tokenizers. A Llama tokenizer on a GPT model is a different language. The integers mean other pieces.

The size of the tokenizer’s list — call it (V), for vocabulary size — is also the width of the model’s last step. The model ends by producing (V) scores, one per token ID. A later math page turns those scores into chances that add to 1.

When a task is about letters, bytes, or exact strings, do not romanticize the model. Give it a tool, or show the spelling in the prompt as separated characters, or do the operation in code. When a task is about meaning, tokens are usually fine: BPE’s merges track frequency, and frequency tracks usefulness.

When you debug a surprising failure, print the tokens. If you cannot see the integers, you are guessing.

What this still cannot do

An ID is an integer. Integers have no nearness.

How can the machine store that cat is closer to dog than to hypotenuse?

Arithmetic needs something richer than a lookup table of IDs.

The next page puts each ID into a list of real numbers.