Mix, clip, mix

❓ A score of agreement is not a new list. How do the other words change the numbers sitting on bank?

The last page gave you one number: do these two lists point the same way? That number is a score. It does not rewrite bank.

The token bank can mean a place with money, or the edge of a river. The surrounding words decide which one.

Chapter 4 gave bank one first list of numbers — a row in a table. Same ID, same starting list, every time. If that were the whole story, “I deposited money at the bank” and “I sat on the river bank” would start from identical numbers.

They cannot stay identical. The other words have to change those numbers.

The machine does not write if money is nearby, use finance. It does arithmetic on the list. This page is that arithmetic, with two numbers, by hand.

Two slots on a card

Pretend the list for bank is a card with two numbers. Call the slots money-ish and river-ish, just so the toy has a face. Real lists are thousands of slots long, and the slots do not come with English labels. Two is enough to do the job in your head.

Start with this card:

[
[1,\ -4]
]

Those are not “the meaning of bank.” They are two numbers we can multiply without a calculator.

The surrounding words have to turn this card into a different card.

Mix: each new number is some of the old ones, added up

A mix takes the incoming card and builds a new card. Each outgoing number is: take some of slot 1, some of slot 2, add. The “some of” amounts are knobs. They sit in a little table. People call that table a matrix, and the knobs weights. You do not need those names to do the step. You need the job: every new slot is a blend of the old slots.

Here is one mix, with a very small table of knobs:

• new first slot = (1\times 1 + 0\times(-4) = 1)
• new second slot = (1\times 1 + 1\times(-4) = -3)

The card is now ([1,\ -3]). Two weighted sums. No if-then.

If every mix were just “copy the card,” what would the surrounding words do?
Nothing. bank would stay ([1,\ -4]). The knobs have to be uneven so some of the old card grows and some of it shrinks.

Where do the knobs come from on this page?
They are given, so we can watch the arithmetic. A later page is who turns them. Your prompt does not.

Clip: negatives become zero, and they cannot come back

After the mix, look at each number one at a time.

• If it is positive, leave it.
• If it is zero, leave it.
• If it is negative, write 0.

That is a clip. Negatives die. Think of a one-way valve, or a hinge that only swings one way.

Our card ([1,\ -3]) becomes ([1,\ 0]).

The (-3) is gone. A later mix cannot get it back. That throwaway is the trick.

The usual name for this clip is ReLU (rectified linear unit). You can forget the expansion. The job is: negatives become zero.

What does this clip do to (5)? To (-2)? To (0)?
(5) stays (5). (-2) becomes (0). (0) stays (0).

Why throw the (-3) away?
Language needs a corner. The word not has to flip a meaning, not merely slide it. A mix can slide numbers around. It cannot throw information away. The clip can.

Imagine, in the toy, that slot 2 was river-ish. After the words “deposited money,” a mix made river-ish negative. The clip kills it. From here on, later mixes only see the money-ish slot. That is not an if-then about banks. It is arithmetic that happens to be able to forget.

Mix again

Different knobs this time.

• new first slot = (2\times 1 + 0\times 0 = 2)
• new second slot = (0\times 1 + 5\times 0 = 0)

The card is now ([2,\ 0]).

Put them in order:

1. Mix — blend with knobs. ([1,\ -4]) → ([1,\ -3])
2. Clip — kill negatives. ([1,\ -3]) → ([1,\ 0])
3. Mix again — different knobs. ([1,\ 0]) → ([2,\ 0])

That is one pass.

The machine does not stop after one pass. It runs the same three beats again, with a new table of knobs, on the new card. Then again. Each pass can slide bank a little farther toward money, or toward river.

Why you cannot skip the clip

Suppose we mixed twice and never clipped.

First mix: ([1,\ -4]) → ([1,\ -3])
Second mix, say “double the first slot, copy the second”: ([1,\ -3]) → ([2,\ -3])

You can write one table of knobs that goes ([1,\ -4]) → ([2,\ -3]) in a single mix. Stretch, then stretch again, is still one stretch. You have not gained a new kind of rewrite. You have one bigger mix.

The clip is what you cannot fold into a mix. Once (-3) became (0), no later table can put the (-3) back. That one-way door is the new power.

On those plots, a circle of many tiny cards is the incoming lists. Two mixes and nothing else leave a stretched circle — an ellipse. You can get an ellipse with a single mix. Mix-clip-mix folds the shape. The fold is the clip.

If we deleted every clip and left only mixes, what would be left?
One giant mix. Language is not a straight line, so that function would be a bad next-piece guesser.

Why not a smoother curve than this clip?
You can use one. The idea is the same: the list must be allowed to turn a corner, not only slide. This clip is the corner you can do in your head.

What people named this

The recipe is mix, clip, mix, repeated.

The official name for that repeating recipe is neural network. In this course the name means only that: mix, clip, mix, stacked.

Deep learning is not a second recipe. “Deep” only means many copies stacked. Depth is how many times the list gets rewritten. Width is how long each list is.

Chapter 1’s model — the giant numerical function behind Send — is this recipe plus the knobs. When someone says “a 70 billion parameter model,” they are counting knobs. Parameter means knob. It is not 70 billion facts in a cabinet.

This page rewrites one list. Later, tokens will also look at each other. That is a different job.

Two piles of numbers

Keep these apart or every product bug looks like “the AI.”

Your prompt never turns the knobs. Pasting a PDF never turns the knobs. A tool result never turns the knobs. They all rewrite the list in flight. Training is what turns knobs.

Does a long conversation “teach” the function permanently?
No. It only changes the lists in the window. Close the window, the knobs are what they were. To change knobs you train. To fake memory you write the lesson back in as tokens, or into a store you re-inject.

What this still cannot do

Each outgoing slot of a mix is a blend of the incoming ones.
What is that as a formula, and as two loops — so “mix” is not a metaphor?

The next page writes the mix.