Meaning as a list of numbers

❓ An ID is an integer. Integers have no nearness. How can cat be closer to dog than to hypotenuse?

An ID is just an integer. Integers have no nearness. So each ID is stored as a list of real numbers, long enough that “cat is nearer dog than hypotenuse” can be a fact arithmetic can use.

“Cat” is closer in meaning to “dog” than to “hypotenuse.” A language model needs that fact in a form arithmetic can use. So it stores, for each token ID, a list of real numbers. People call that list a vector. Here it is not a physics arrow. It is an ordered list, like [0.2, -1.1, 0.4, …], long enough that many kinds of similarity can live in it at once.

You cannot picture twelve thousand numbers. You can picture the list as a point. The token is still a chunk with an ID. The list is what sits in the space. Training moves those lists so that tokens used in similar ways land near each other.

Cities already work this way in two dimensions. We store Washington as [38.9, 77] and New York as [40.7, 74]. Nearby cities have nearby coordinates. The city is not the pair of numbers. The pair is how we do arithmetic on it. Token lists get the same treatment, except the axes are not latitude and longitude. They are learned.

The table of these lists is the embedding. As a verb, embedding means: put the token into that number-space. The first step of the model is a lookup. Token ID 15496 becomes row 15496 of the table.

The first list is only a typical meaning

That lookup cannot be the whole story. The token mole is an animal, a chemical amount, and a skin growth. The token bank is money or a river. The first list for mole is the same every time it appears, because it is just a row in a table.

The rest of the machine exists to rewrite that list using the other tokens in the window. After several rewrites, the list sitting on mole in “one mole of carbon dioxide” is no longer the same object as the list sitting on mole in “a biopsy of the mole.” Researchers sometimes call those evolving lists hidden states. The name is fancier than the object. It is still a list of numbers per token, being overwritten as it climbs.

The leftover slots in a long list are also scratch paper. An early rewrite can park “this bank is financial.” A later rewrite can park “John is the customer, not the mechanic.” That is not English in a variable. It is a pattern of numbers that later tables of knobs have learned to read.

The machine will need to compare two lists

Soon the machine will need a yes-ish / no-ish answer to a simple question: do these two lists point the same way?

Not “are they equal.” Not “which city is this.” Just: do they agree? When two tokens later “look at” each other, that agreement number is how much one should affect the other.

The arithmetic for that comparison is a dot product. The next chapter is only that piece of math, with tiny examples, before we use it for meaning-shifts and for attention.

A shift in the space can stand for a feature

If nearby points mean similar use, then a consistent jump from one cluster to another can stand for a relationship.

The old classroom picture is king − man + woman landing near queen: start at king, subtract the “male person” jump, add the “female person” jump. Paris − France + Germany landing near Berlin is the same idea with “capital of.”

Treat that as a cartoon of the space, not as how a chat model thinks. The pictures are older than today’s chat models, they are a bit too clean, and they inherit the biases of the text they were built from. Some of those lists made doctor − man + woman land near nurse.

The load-bearing moral is smaller than the cartoon: a direction can mean a feature, so later steps can add a direction, subtract one, or measure agreement with one. That measurement is the dot product.

The lookup table does not know order

dog bites man and man bites dog use the same three tokens. If the first lists were only “what this token usually means,” those two sentences would be the same bag of lists, with no order. The machine would have no idea who bit whom.

So the first list is not only the lookup row. The model mixes in where the token sits — first chunk, third chunk, last chunk. After that mix, two copies of man in two places are slightly different lists. Order has somewhere to live.

You do not need the particular recipe (sines, rotations, learned tables). You need the job: the first list is typical meaning plus place in the window. Later rewrites add the sense this passage needs.

After this page, three facts should stick. A token becomes a list of numbers. Two lists will be compared by a dot product. The first list is a generic meaning plus a position, and later steps overwrite it.

What this still cannot do

Later steps must compare two lists.

What single number means “these two point the same way”?

The next page is that comparison. Rewriting the list (sliding “bank”) waits until we have the number.