Math: mixing with a table of knobs

❓ Each outgoing slot of a mix is a blend of the incoming ones. What is that as a formula, and as two loops?

A mix takes an incoming list (\mathbf{x}) of (n) numbers and builds a new list (\mathbf{y}) of (m) numbers. Each outgoing number is a weighted sum of the incoming ones, plus a constant.

The compact name for that job is:

[
\mathbf{y}=W\mathbf{x}+\mathbf{b}.
]

That line is useless until you can expand it. Here is the same job, slot by slot.

The general formula

Index the outgoing slots by (i = 1,\ldots,m) and the incoming slots by (j = 1,\ldots,n).

[
y_i = b_i + \sum_{j=1}^{n} W_{ij}\, x_j.
]

Read it left to right:

• (y_i) — the number that lands in outgoing slot (i).
• (b_i) — a constant added to that slot. People call the whole list (\mathbf{b}) the bias. It is still just knobs.
• (W_{ij}) — the knob in row (i), column (j) of the table. It says how much incoming slot (j) is allowed to affect outgoing slot (i).
• (x_j) — the number already sitting in incoming slot (j).
• The sum — do a dot product of row (i) of (W) with the whole of (\mathbf{x}).

So (W) is a table with (m) rows and (n) columns. (m) and (n) need not match. A mix can shorten a list or lengthen it. The last mix of a language model lengthens: from a list a few thousand long to one score per vocabulary token.

If (\mathbf{x}) has 3 slots and (\mathbf{y}) has 2, how big is (W)?

2 rows, 3 columns. Six knobs, plus two bias knobs.

Write (y_1) with the general formula, no matrix shorthand.

(y_1 = b_1 + W_{11}x_1 + W_{12}x_2 + \cdots + W_{1n}x_n).

The same job as pseudocode

# x has n slots. y will have m slots.
# W has m rows and n columns. b has m slots.

for i in 1 .. m:                  # each outgoing slot
    total = b[i]
    for j in 1 .. n:              # each incoming slot
        total = total + W[i][j] * x[j]
    y[i] = total

The inner loop is the dot product. The outer loop is “do that once per outgoing slot.” A GPU runs many of those inner products at once. That is all “matrix multiply” means here.

There is no search, no if-then, no dictionary lookup in this step. Nested addition and multiplication.

What changes if we skip the inner loop and set y[i] = b[i]?

The incoming list is ignored. Context cannot rewrite meaning. The table of knobs is the whole point.

Three slots by hand — the picture

The plate uses (\mathbf{x}=[2,1,0]) and

[
W=\begin{bmatrix}1 & 2 & 0\0 & 1 & 3\1 & 0 & 1\end{bmatrix}.
]

Bias is (0) here so you can see only the multiplies.

Same color means this (x) slot multiplies this column of (W). Blue (2) hits every knob in column 1. Peach (1) hits column 2. Green (0) hits column 3.

The inner loop for (j) walks across a row. On row (i=1) the picture writes the three products:

[
y_1 = 1\times 2 + 2\times 1 + 0\times 0 = 4.
]

The outer loop for (i) walks down the rows and repeats that job:

[
\begin{align}
y_2 &= 0\times 2 + 1\times 1 + 3\times 0 = 1,\
y_3 &= 1\times 2 + 0\times 1 + 1\times 0 = 2.
\end{align}
]

So (\mathbf{y}=[4,1,2]). Compact form: (\mathbf{y}=W\mathbf{x}) when (\mathbf{b}=\mathbf{0}).

If (W) is the identity table (1s on the diagonal, 0s elsewhere) and (\mathbf{b}=\mathbf{0}), what is (\mathbf{y})?

(\mathbf{y}=\mathbf{x}). The mix copied.

Can this step alone model “if both names are present, write basketball”?

No. The general formula is linear: each (y_i) is a sum of knobs times inputs. Two such mixes stacked are still one mix. Language needs a bend. That is why the last chapter puts a clip between mixes.

Why a table, not one number

One knob could only stretch the whole list by the same factor. A table can rotate, stretch some directions, shrink others, and pour slot (j) into slot (i). That is how a later mix-clip-mix can slide “generic bank” toward “financial bank”: it adds a particular direction, stored as a row of (W).

Where do the numbers in (W) come from?

They start random. Training (the downhill chapter) turns them. Your prompt never edits (W).

What this still cannot do

A mix can spit out any real numbers, including negatives.

A next-piece pile cannot. How do scores become chances that add to 1?

The next page is that conversion.