Math: the dot product
❓ Later steps must compare two lists. What single number means “these two point the same way”?
The last chapter left you with lists of numbers standing for tokens. The rest of the course needs one cheap question: do these two lists point the same way?
That question has a name. The dot product of two lists of the same length is: multiply matching slots, then add.
[
\mathbf{a}\cdot\mathbf{b}=a_1b_1+a_2b_2+\cdots+a_db_d=\sum_i a_i b_i.
]
If you already remember this from a linear algebra class, you are not behind. This page is here so the formula has a job before chapter 8 uses it.
Three tiny cases
Keep both lists length 2 so you can do them in your head.
They agree. ([1,0]\cdot[1,0]=1\times1+0\times0=1). Positive.
They are sideways. ([1,0]\cdot[0,1]=1\times0+0\times1=0). No agreement.
They oppose. ([1,0]\cdot[-1,0]=1\times(-1)+0\times0=-1). Negative.
What if one list is longer? ([2,0]\cdot[1,0]=2). Same direction as the first case, bigger number, because ([2,0]) is a longer arrow. The score cares about direction and length.
The angle picture
Call the angle between the two arrows (\theta). Then the same score is also:
[
\mathbf{a}\cdot\mathbf{b}=|\mathbf{a}|\,|\mathbf{b}|\cos\theta.
]
The bars mean length — how long that arrow is:
[
|\mathbf{a}|=\sqrt{a_1^2+a_2^2+\cdots}.
]
And (\cos\theta) is (1) when they point the same way, (0) at right angles, and (-1) when they point opposite.
Same story as the three cases, with lengths allowed to vary.
What the number will mean in this course
When two tokens look at each other, the model builds two short lists (a question and a label) and takes their dot product.
- A large positive score: this token’s question matches that token’s label. Mix in more of its payload.
- A score near zero: no match. Mix in almost nothing.
- A negative score: mismatch. After the next math page (softmax), that usually becomes a tiny mix-weight, not a “subtract meaning” button.
The machine is not searching a dictionary. It is scoring alignment.
Questions
If (\mathbf{q}=[1,2]) and (\mathbf{k}=[3,0]), what is (\mathbf{q}\cdot\mathbf{k})?
(1\times3+2\times0=3).
If two lists are at right angles, what is the dot product, no matter how long they are?
Zero. (\cos 90^\circ=0).
Does a dot product of (4) mean “more related” than a dot product of (1)?
Usually yes if the lists are the same length. If one list is just a stretched copy of the other, the score grows with length even when the angle is unchanged. That is why chapter 8 will later scale the scores before turning them into mix-weights.
Is this how the model “knows” that cat is near dog?
Nearness of the first lookup lists is about where the points sit. The dot product is how a later step asks whether two rewritten lists agree. Related tools, different jobs.
What this still cannot do
A score of agreement is not a new list.
Context has to slide “bank” toward money or river. How do knobs rewrite a list?
The next page is that rewrite: mix, clip, mix.