Math: softmax, or turning scores into a pile of chances

❓ A mix can spit out any real numbers, including negatives. How do scores become chances that add to 1?

Scores can be any real numbers: (2), (-3), (0.1), (100). A pile of chances cannot. Each chance must be positive, and they must add to (1).

Softmax is the usual conversion. For a list of scores (\mathbf{z}),

[
\mathrm{softmax}(z)_i=\frac{e^{z_i}}{\sum_j e^{z_j}}.
]

(e^{z}) is the exponential: always positive, and it grows fast. Bigger scores steal more of the pile, exponentially.

A three-number example

Scores: ([0.707,\; 0,\; 0.707]).

(e^{0.707}\approx 2.028), (e^{0}=1), (e^{0.707}\approx 2.028). Sum (\approx 5.056).

Chances: ([2.028/5.056,\; 1/5.056,\; 2.028/5.056]\approx [0.401,\; 0.198,\; 0.401]).

They add to (1). The middle score was smallest, so it got the smallest share.

The Wolfram bars use scores ([2, 0, -1]). After softmax at (T=1), A takes most of the pile, B a little, C almost none. At (T=0.3) A takes nearly everything.

If all three scores are equal, what is softmax?

Equal chances: ([1/3, 1/3, 1/3]).

Temperature

If you divide every score by a number (T) before softmax, you change the peak.

• (T<1): scores look more extreme, the pile peaks (safer, duller).
• (T=1): leave the scores alone.
• (T>1): scores look closer together, the pile flattens (more variety, more junk).

Temperature (0) in APIs usually means: skip the dice and take the largest score. The scores before softmax are often called logits. That word only means “raw scores, not yet chances.”

Where this course uses it

Twice. At the end of the model, softmax turns one score per vocabulary token into the next-piece pile. Inside attention, softmax turns one row of match-scores into mix-weights that sum to 1. Same function, two jobs.

If one score is much larger than the others, what happens?

Almost all the pile goes to that one entry. Training then sees a very flat slope, which is part of why attention scales scores first.

Can softmax output a negative chance?

No. (e^{z}) is always positive.

What this still cannot do

Softmax does not choose the knobs.

Who turns the table W so the pile puts mass on the true next piece?

The next page is that turning: measure wrong, walk downhill.