The whole job is the next piece

❓ If there is no retrieved answer, what exactly is being chosen after Send?

After Send there is no retrieved answer. There is a next piece, then another. This page is that piece: a pile of chances, then a pick, then the loop.

English has patterns. After “peanut butter and” you expect “jelly” more than “hypotenuse.” After “Once upon a” you expect “time.” You do this without writing a grammar.

A language model is a machine that has been forced to get extremely good at that one trick: given the text so far, write a number for every possible next chunk, so the numbers are positive and add to one. Then, usually, pick.

When a chatbot writes an essay, it is asking over and over “given the text so far, what should the next word be?” — and each time adding a word. More precisely, adding a token: the machine’s chunk — a word, a splinter of a word, a space, or punctuation.

Why a small job can look like thought

If that sounds too small to explain coding, legal memos, or fake citations, you are in the historical majority. It is a small objective. It is also an objective that, on enough text, secretly includes almost every other language task.

To guess the next token well, the model must notice nearby grammar (“the cats are”), longer grammar (“the keys to the cabinet are”), facts stated earlier in the window, facts that showed up often in the text it was trained on, the style of the text so far, that a function should close its braces, that a proof should not contradict line 3.

None of that is programmed as a rule. It is pressure on one score of wrongness: make the true next token more likely than the others. Chapter 6 is how that pressure turns knobs. Here you only need the job.

The surprise of the last decade is not that next-piece guessing exists. People used to do it by counting phrases in tables. The surprise is that a function with enough knobs, fitted on enough text, estimates those chances well enough that the continuations look like thought.

Why you cannot just count

The obvious method is a table. Count how often each word follows each phrase.

That dies immediately. Even a modest English word-list of about 40,000 words already gives 1.6 billion possible pairs and 60 trillion triples. An “essay fragment” of twenty words has more possible sequences than you can ever see written down.

So the model cannot be a lookup table of phrases. It has to be a compressed guesser: a function that assigns a chance to a sentence it has never seen, using patterns it extracted from sentences it has.

That function is a giant pile of knobs. People call the knobs weights: the numbers inside the function that training turns. A later page is the recipe those knobs sit in.

A pile of chances, not a word

The model does not emit “the answer.” It emits a pile of chances. The math name is a probability distribution: a list of numbers, one per possible next token, each between 0 and 1, adding up to 1.

Suppose the prompt is:

The capital of France is

A trained model might put most of its chance on Paris, a little on paris, a little on the, a trickle on nonsense. Your app then samples — it picks.

• Always take the top token. People call this greedy. In many APIs this is temperature 0. Safe, repetitive, sometimes stuck in loops.
• Pick in proportion to the chances, maybe sharpened or flattened. The usual knob is temperature: a number that peaks the pile (low temperature, safer, duller) or flattens it (high temperature, more variety, more chance of a worse token).
• Only consider the top few tokens, or the smallest set whose chances add up to enough. People say top-k and nucleus / top-p. A practical compromise.

Temperature is not creativity in the soul of the machine. It is how violently you peak or flatten that pile before you roll the dice. At temperature 0 you are reading off the model’s favorite. At high temperature you are wandering the tail. For essay-like generation, people often land near 0.8 — lore, not a law of physics.

This is why two calls with the same prompt can differ, and why “the model said X” is incomplete unless you also say how you picked.

Scratch work has to be written

Because the only way the model can “work on” a problem is by producing tokens, scratch work has to be written. If you demand a one-word answer to a multi-step question, you are asking it to hide the reasoning inside its knobs and a single output slot. If you let it write steps, those steps become extra text in the window for the later tokens — working memory in the only place working memory exists.

That is why “think step by step” and the newer reasoning models all rhyme. They are not a second brain. They are more tokens in the loop.

Take a prompt you actually send. Name three different next tokens the model could emit, and what your loop would do with each. You are no longer picturing an oracle. You are picturing a pile of chances plus a loop.

A quiz that went around the internet

In 2024 a lot of people typed the same tiny question into ChatGPT and its cousins:

How many r’s are in the word strawberry?

Look at the word. There are three: one in straw, two together in berry. A child with a pencil can mark them. The chatbots often said two. Sometimes they said three and then listed the wrong letters. Then they argued, politely, that they were right.

That clip spread because it was short, checkable, and rude to the “it understands English” story. The same model could write a recipe that used strawberries. It could rhyme strawberry with merry. It still failed a count you can do by pointing.

This page already said the job is next piece, not “look at a word as a row of letters.” If the machine never sees s-t-r-a-w-b-e-r-r-y as nine letters, a letter-count is a different game than continuing a sentence. The next page is that cut-up.

What this still cannot do

Is a piece a word?

“Hello world” might not be two pieces. Strawberry might be one piece, or a few, and those pieces might not be letters.

If the model continues English so well, why did the strawberry count fail?

The next page is the cut-up that comes before the pile.