Document simulator, then assistant, then practice

❓ A stack trained to continue internet text is a document simulator. Why does the API talk like a colleague?

A person who has only read the internet is not yet a colleague. They have statistics about how documents continue. They have not agreed to be helpful, brief, or honest.

A useful map of how chat models are made is three rooms.

1. Pre-training. This is the first, huge downhill walk from chapter 6. Tokenize a huge filtered crawl. Train the transformer to predict the next token. Months, most of the compute, most of the data. Output: a base model. It can complete a Wikipedia paragraph, a Python file, or a rant with equal professionalism, because those all appear in the distribution. Pre-trained just means: the knobs were fitted on that pile of text before anyone chatted with it.
2. Supervised fine-tuning (SFT). Show it conversations: user, assistant, user, assistant, written or chosen by people. Same next-token loss, narrower data. Output: something that plays the role of an assistant. This does not install a truth module. It aims the dreams toward helpful-assistant dreams.
3. Reinforcement learning. Let the model produce answers, score them, and increase the chance of the better ones. When the score is a human preference — people pick which of two answers they like — that recipe is RLHF (reinforcement learning from human feedback). In 2022 OpenAI published InstructGPT: a base model, then example chats, then human preference picks. When the score is a checkable test (unit tests, a math checker), you get the newer “reasoning” training. Think of this room as practice after lectures.

The architecture in chapters 8–10 does not change between these rooms. The knobs do.

If you skip rooms 2 and 3, what does a base model do with a question?
It often continues the question, as if your prompt were the start of a webpage: more questions, a forum thread, a listicle.

A document simulator is not a colleague

If you talk to a base model like a chatbot, that continuation is not the model being stupid. That is next-token prediction on internet documents.

The assistant you pay for is a base model plus the later rooms plus a system prompt that is itself more tokens. When a vendor ships a new “personality,” they are rarely inventing a new transformer. They are changing room 2, room 3, and the default prefix.

Fluent invention is in the objective

A next-token machine is rewarded for fluent continuation, not for abstaining. If the prompt looks like a question that would have an answer in the training distribution, the assistant-trained model will often emit an answer-shaped continuation. Do not trust what these models say from memory alone; trust them more when the relevant text has been pulled into the window (browsing, retrieval, a file you pasted).

That is not a moral failing. It is the loss function. Mitigation lives in your loop: retrieval, tools, citations you check, asking for uncertainty, sampling less freely on factual tasks.

Tools are tokens too

When a model “uses a calculator,” it does not grow a calculator. It emits a special pattern of tokens that your loop parses, executes, and pastes back as more window. Same for web search, code interpreters, and function calling. The model still only predicts next tokens. The extra power is a loop you wrote around it.

This is why “agent” in the useful sense is not a bigger model. It is a model, a prompt that tells it when to emit a tool pattern, tools with clear contracts, and a loop that actually runs them.

Reasoning models and extra tokens

Later training that rewards multi-step traces does not escape next-token prediction. It buys more tokens of working memory and a policy that uses them. If you then hide the trace, you are hiding the scratchpad, not deleting the need for one.

Take a failure from your own app. Classify it: tokenizer, missing window, picking, base-model ignorance, assistant role-play, or your loop (tool / retrieval) mistake. If you cannot classify it, you are still in the black box.

What this still cannot do

The rooms explain the weights. They do not tell you which knobs are yours when you ship.
Context, sampling, tools, and the harness: where does a failure actually live?

The last page is that map.