Math: loss as “how wrong was that chance?”
❓ Training needs one number that means “how wrong.” If the model put chance p on the true next token, what is that number?
Training needs a single number that is large when the model’s guess is bad and small when it is good. That number is the loss.
For next-piece guessing the usual loss is cross-entropy. If the model put chance (p) on the token that actually came next,
[
\text{loss}=-\log p.
]
(\log) here is the natural logarithm. (-\log p) is (0) when (p=1) (the model was sure and right) and grows as (p) shrinks toward (0) (the model put almost no chance on the truth).
A feel for the numbers
| Chance on the true token | (-\log p) (about) | Reading |
|---|---|---|
| (1.0) | (0) | Perfect. Impossible on messy language. |
| (0.37) | (1) | A decent guess. |
| (0.05) | (3) | Pretty wrong. |
| (0.001) | (6.9) | The model barely considered the truth. |
Averaging this over a batch of positions is the number training walks downhill.
If the model put chance (0.5) on the true next token, is the loss zero?
No. (-\log 0.5\approx 0.69). Zero only at chance (1).
Why not “1 minus p”?
(1-p) would also be small when (p) is near 1. Cross-entropy punishes confidently wrong guesses much harder: putting (0.001) on the truth is not a little worse than (0.05); it is a lot worse. That matches the job. A next-piece model should not be calmly sure of the wrong token.
The slope
Gradient descent means: compute how the loss changes if you nudge each knob, then move the knob the other way. Gradient is that list of slopes. You do not need to compute one by hand for this course. You need the picture: loss is a hill of wrongness; training takes a step downhill.
Does a loss of zero mean the model understands the sentence?
No. It would mean it assigned chance 1 to whatever token happened to be next in the training file — often memorization, not understanding.
Who averages the loss?
Usually every next-token position in the batch, then a mean. One sequence is many graded examples, as long as later tokens cannot peek (the mask in the attention chapter).
What this still cannot do
We can train mix-clip-mix on one next piece. A sentence is ordered.
Does the machine read token 1, forget, read token 2 — or look at everyone at once?
The next page is why looking-at-everyone beat the running summary.