Math: why divide by (\sqrt{d_k})

❓ Raw match-scores grow as the lists get longer. Softmax then spikes. Why divide before we mix?

Attention’s compare step is a grid of dot products. If each query and each key has (d_k) slots, a raw score is a sum of (d_k) little products.

If those little pieces are noisy and about size (1), the sum has variance about (d_k). Variance is “how spread out the random number is.” A sum of many independent pieces spreads out like (\sqrt{d_k}) in typical size.

So as (d_k) grows (64, 128, …), raw scores get huge. Softmax then puts almost all the pile on one entry (the math-softmax page: one huge score starves the rest). The slope used in training dies. The paper’s fix:

[
\text{score}=\frac{\mathbf{q}\cdot\mathbf{k}}{\sqrt{d_k}}.
]

After this divide, the typical score stays near size (1) even when the lists get longer. It is numerical hygiene, not a new idea about meaning.

The Wolfram bars take the same three scores, blow them up like a long list would, then divide by (\sqrt{64}=8). Left: a spike. Right: still a mix.

A cartoon with numbers

Suppose each of 100 slots contributes a random (\pm 1). The sum is often in the tens, sometimes over (50). Divide by (\sqrt{100}=10) and you are back to scores of a few units, which softmax can still turn into a soft mix instead of a single spike.

In the napkin example of the attention chapter, (d_k=2), so we divided by (\sqrt{2}\approx 1.414). Tiny on purpose. The same button is pressed for real heads with (d_k=64) ((\sqrt{64}=8)).

If we skipped the divide, what would go wrong first?

Softmax would saturate: mix-weights near (0) and (1), almost no in-between, and training would struggle.

Does the divide change which token wins if one score is clearly largest?

Often not the winner, but it changes how much the others still get. Attention is a soft mix. The divide keeps it soft.

Is (\sqrt{d_k}) learned?

No. It is a fixed recipe from the 2017 paper. The learned knobs are the tables that build (\mathbf{q}) and (\mathbf{k}).

What this still cannot do

Attention is half a floor. Mix-clip-mix is the other half.

How do look, think, and a skip around each become one transformer block?

The next page is that floor.