Training is downhill on a measure of wrongness

❓ Softmax turns scores into a pile of chances. Who turns the knobs so the pile puts mass on the true next piece?

The last page turned raw scores into a pile of chances. That pile still sits on whatever knobs you happen to have. Someone has to turn those knobs.

If you can measure “wrong,” you can reduce “wrong.” Training a language model is that slogan at industrial scale.

Show a snippet, grade the next piece

You show the model a snippet of text. It produces a pile of chances for the next token — the same pile chapter 2 already described.

The true next token in the file was, say, ID 15496. You wanted that ID’s chance high.

A loss is a single number that says how unhappy you are with the pile. The usual loss here is: take the chance the model put on the correct token, then take the negative log of that chance. Sure-and-right is a small number. Almost-no-chance-on-the-truth is a large number. Why that formula, and what the numbers feel like, is Math: loss.

Average the loss over a handful of snippets. Then change every knob a little to make that number smaller.

The algorithm that computes “which way is down” for every knob is backpropagation: the chain rule from calculus, run backward through mix-clip-mix. The algorithm that takes a step is gradient descent (in practice a cousin named AdamW): move each knob opposite its slope. Gradient here just means “the slope of the loss with respect to that knob.”

There is no third magic. The model is the function. The data is the pressure. The slope is the instruction.

If the pile already put chance 1 on the true next token, which way is downhill?
Nowhere. The loss is already 0. Real language never quite gets there.

Facts are a side effect, not a filling

A tempting myth: training “fills the model with facts,” like pouring water into a jug.

What training actually does is reshape a function so that, on the text it was shown, the next-token chances get better. Facts appear because text that contains them was part of that pressure. Associations appear because they helped prediction. Style appears because it helped prediction. The model will also absorb copyrighted phrasing, stereotypes, and errors for the same reason.

If a fact is rare, phrased many ways, or never written, this downhill walk has little to work with. If a fact is common, it may be easy to predict even if the model has no clean “slot” you could read as a database row.

Does training store “Paris is the capital of France” as a row you could look up?
No. It makes Paris a high-chance continuation of The capital of France is. That is a different object.

Predict every next piece at once

During training, the model does not generate a whole essay and then get a grade. It takes a sequence of tokens and, in parallel, tries to predict every next token along the sequence: given tokens (1..t), predict (t+1), for all (t) at once.

That is why a later page will need a mask: token (t) must not be allowed to look at (t+1), or it would cheat.

This is also the main reason these models train efficiently on GPUs. One sequence is many graded examples.

After the first, huge downhill walk you have a base model: an internet-document simulator. It is not trying to be helpful. It is trying to continue whatever distribution it saw — Wikipedia, code, papers, lists, dialogue, garbage.

Helpfulness is extra training (a later page). So is “I am an AI made by X.” So is refusing a category of request. Those behaviors are not sitting in the architecture diagram. They are sitting in later downhill walks, plus whatever you put in the system prompt.

Why this got good only recently

Three curves moved together: compute, data, and knob count. Empirical scaling laws said that loss keeps falling, smoothly, as you scale those, in a range we have not obviously exhausted. That is why labs buy clusters instead of only inventing cleverer layers. A lot of “emergence” is what a smooth loss curve looks like when you only measure the tasks you care about at the end.

The quantity being reduced during the first training stage is next-token wrongness. A perfect loss of zero would mean the model always assigned chance 1 to the true next token — neither possible on messy language, nor desirable: it would memorize, not guess unseen sentences.

If we trained on one sentence forever, what would go wrong?
The knobs would fit that sentence. The next-piece guesser would stop being a guesser of unseen sentences.

What this still cannot do

Training needs one number that means “how wrong.”
If the model put chance p on the true next token, what is that number as a formula?

The next page is the loss.