Why looking-at-everyone beat reading-one-at-a-time

❓ We can train mix-clip-mix on one next piece. A sentence is ordered. Does the machine read token 1, forget, read token 2 — or look at everyone at once?

Loss can train mix-clip-mix. It cannot decide how information travels along a sentence.

A sentence is ordered. The word after not can flip the meaning of everything that follows. Any model of language has to let information travel along that order.

For years the default was: read token 1, update a running summary, read token 2, update again. The usual name is a recurrent neural network (RNN): mix-clip-mix plus a running summary carried forward. In theory the summary could carry a fact from the first word to the last. In practice two things went wrong.

Distance. Signals had to survive many multiplies. A fact from word 1 had a hard time still being alive at word 200.

Time. You cannot compute token 50’s summary until token 49 is done. The chips these labs buy want huge table-multiplies, not a long chain of waiting. As sequences grew, the one-by-one design left the hardware idle.

Drop the running summary

Looking had already been used as a patch: when translating, let the writing side look at the reading side’s lists instead of trusting one summary.

In 2017 a paper titled Attention Is All You Need made a blunter claim. Drop the running summary. Drop the other old building block (a sliding local mix). Let every position look at every other position directly, with a handful of table-multiplies, and stack that.

They named the design Transformer because each block would transform its input lists. The looking step that mattered was self-attention — looking where the questions, the labels, and the payloads all come from the same sequence. The next page defines those three views by hand. Here you only need the job: mix information across positions by learned relevance.

Cost: relating two tokens is a constant number of sequential steps, but the table of all pairs grows with the square of the window length (n). That tax is why windows were 2,000 tokens, then 4,000, then 128,000 only as people paid for it, cached it, or approximated it. It is also why “just paste the whole company wiki into the prompt” has a physics.

Benefit: on a GPU, those pairwise scores are a table-multiply. Waiting stopped being the bottleneck. Training could swallow the internet.

If the window is twice as long, about how much more looking-work is there?
About four times. Pair every token with every token.

What “look at” means, before the formula

The animal didn’t cross the street because **it** was too tired.

When the model processes it, self-attention is how it can pull information from animal rather than street. Each word writes a “looking for” checklist and an “I am” checklist, and checklists that align share information.

Of the enormous parts list, two jobs make the car run. Attention mixes across positions. Mix-clip-mix still thinks inside one position. Everything else is how you make that run longer and faster.

That split is the whole transformer. The next page is the first job, in arithmetic. The page after the math sidecar is both jobs in a block.

Reading versus writing

The 2017 paper’s diagram had two stacks. One read the whole source sentence (they said encoder). One wrote the translation, looking left at what it had already written, and also looking at the reader (they said decoder). That is the right picture for translation.

The chat models you call almost always throw away the separate reader. They are a writer stack with a “do not peek ahead” rule: each token may look only at itself and the past. Your prompt and the model’s reply live in the same sequence. “Understanding the question” and “writing the answer” are the same knobs, used twice: first on your tokens, then, one generated token at a time, on the growing reply.

If the model can seem to “already know what it will say” while it is still reading your prompt: during the first pass it is already running the looking step over your tokens. It just has not picked an output yet.

Why must a next-piece model hide the future during training?
Because the future tokens are sitting in the batch. Without a hide rule, the model could “predict” a word by reading it.

What this still cannot do

Looking-at-everyone is still a slogan until you can compute it.
What are the four steps, by hand, on three tokens?

The next page is that looking step with a pencil.