Stack the blocks, pick a piece

❓ One floor is not a city. How many times is the block stacked, and how does the last list become a next piece?

One transformer block is not a chat model. A chat model stacks the same block many times, turns the last list into a pile of chances over the tokenizer, picks, and loops.

A GPT-style model is:

1. Token lists plus position (chapter 4).
2. The same block stacked (N) times (12 copies in a small 2019 model called GPT-2; 96 copies in a 2020 giant called GPT-3).
3. A final re-center of the numbers.
4. A mix from the list-length (d) to the tokenizer’s list size (V). This is the unembedding: the reverse of the first lookup, turning a list of numbers back into a score per possible next token. Often the first lookup table is reused, turned on its side.
5. Softmax → a pile of chances (Math: softmax).
6. Sample. Append. Repeat.

Those knobs are not a zoo of unrelated inventions. There are a handful of types of table — embed, query, key, value, output mix, feed-forward up, feed-forward down, unembed — and then copies of them, layer after layer.

Why the last list

During generation, you only need a pile of chances for the next token. That pile is read from the numbers sitting on the last position. Every attention head in every layer has had a chance to write into that position. If the prompt is a mystery novel ending in Therefore the murderer was, the list sitting on was has to have become, by layer (N), a summary of everything that predicts the name.

That is a staggering request. It is also why a long window is both powerful and expensive: the last position may look at tens of thousands of earlier labels.

During training, as chapter 6 said, the model predicts every position in parallel. During inference — running the trained model to produce text — you usually care about one new token at a time. Implementations cache the keys and values of past tokens (KV cache) so you do not recompute the whole prompt for every new word. That cache is why the first token of a long prompt is slow (the “prefill”) and the later tokens are cheaper (the “decode”), and why a 128k window has a memory bill even when you generate one word.

You can ignore the systems details and still keep the conceptual fact: inference is the stacked blocks, run again, with one more token in the sequence.

If you deleted all but the last transformer block, what information would the last list still be missing?
It would only have had one chance to talk. Early facts would not have been rewritten, then rewritten again, into that last position.

Picking sits outside the knobs

The knobs end at raw scores. Everything you think of as “personality of this call” — temperature, top-p, stop sequences, max tokens, JSON mode — is either extra tokens in the prompt or a policy on top of the pile of chances. None of it changes the query tables.

Greedy picking repeats itself because the highest-chance continuation of a cliché is often more cliché. Nucleus sampling was a response to that boredom. Chat products pick defaults that hide this. Your app should pick them on purpose.

What “one forward pass” costs

Roughly, compute scales with how many layers, how long the window, and how long the lists are. The expensive looking-table grows with the square of the window. That is why attention is the window-length villain. Fat models are expensive even at short windows because the mixes grow with the square of the list-length.

You do not need a spreadsheet to use this. When a vendor charges more for long context, they are selling you that square and a larger cache. When they charge more for a bigger model, they are selling you more stacked mix-clip-mix steps.

What this still cannot do

A stack trained to continue the internet is a document simulator.
Why does the API talk like a colleague — helpful, brief, refusing some things — rather than like a crawl?

The next page is the three rooms after that first downhill walk.