Article 2 of 6 · September 2026
How it is built, told as school
Article 2 of 6 · from a copy of the internet to something that answers you · after Andrej Karpathy
This article retells the middle of Andrej Karpathy's Deep Dive into LLMs like ChatGPT, the part where he walks through how one of these models is actually made. The factory is his. So is the school comparison, which is the single most useful picture in this whole series, and so this article opens with it, then walks the factory with that picture in hand. We only reorder and shorten, and we point at the two ideas you will need later if you ever want a model trained on your own material.
1. School, first
Open any textbook. Karpathy points out that it contains three kinds of material, and a model is trained in three stages that match them.
Most pages are exposition: background, explanation, the meat of the subject. Reading the exposition is the first stage. The model reads a cleaned copy of the internet. That is called pretraining, and it is where almost all the cost is.
Then there are worked examples: a problem, and the author's full solution beside it. Studying worked examples is the second stage. The model reads conversations in which people wrote ideal answers, and learns the shape of a good answer. That is how it becomes an assistant.
Then there are practice problems, at the end of the chapter, with only the answer in the back of the book. You are not shown how. You try, you check, you keep what worked. That is the third stage, reinforcement learning, the newest and least settled of the three.
His summary sentence: we train these models very much the way we teach children. Read the material, study the worked examples, do the practice problems. Hold that page in mind. The rest of the article is the first stage in detail, then the second, then the third.
2. Start with the internet, boiled down
The raw material is text from the web. Not "the whole internet": a cleaned copy of it.
Karpathy points at a public example, a dataset called FineWeb, so that anyone can look. It starts from Common Crawl, an organisation that has been following links and saving web pages since 2007, and had indexed about 2.7 billion pages by 2024. That pile is then filtered hard. Block-listed sites go (spam, malware, adult, and so on). The page furniture goes, the menus and code, so that only the text remains. Pages are kept mostly in one language, English in FineWeb's case, which is a design choice with a consequence: a model trained on it will be weaker in the languages that were filtered out. Personal data is removed. Duplicates are removed.
What is left is smaller than people expect. Karpathy's number: about 44 terabytes. His scale for it: you can buy a one-terabyte USB stick anywhere. The cleaned reading material for a production-grade model fits in a drawer.
3. Cut it into chunks
A model cannot read letters. It needs a fixed alphabet of symbols and a long sequence of them. Text as raw bits would be two symbols and an absurdly long sequence. Bytes give 256 symbols and a sequence eight times shorter. Then a trick called byte-pair encoding keeps gluing the most common neighbouring symbols into new ones, again and again, until the alphabet is about a hundred thousand symbols. GPT-4's alphabet has 100,277.
These symbols are the tokens from article 1. "hello world" is two of them. The 44 terabytes become a sequence of about 15 trillion tokens. Every model in this series, from here on, only ever sees that sequence: numbered chunks, one after another.
4. The guessing game
Now the actual learning, which is one game played an astronomical number of times. This is the reading of the exposition.
Take a window of tokens from the sequence, say a few thousand. Ask the model: what comes next? It answers with a probability for every one of its hundred thousand symbols. Since the text is already known, the right answer is known too. If the model gave the true next token a 3 percent chance, nudge it so that next time it gives a little more, and everything else a little less. Do this for every position in the window, for millions of windows at once, for days or months depending on the size of the model.
What is being nudged? Karpathy's picture is a DJ desk. The model is a giant mathematical expression with knobs on it, and the knobs are called parameters. Billions of them. Turning them changes what the model predicts. Training is nothing more than finding a setting of the knobs that fits the statistics of the text.
There is one number on the desk that the people training the model watch, and only one. It is called the loss, and it says how badly the model is guessing right now. Training makes it go down. Karpathy describes the job from the inside: you look at the loss, you twiddle your thumbs, you drink coffee, and you make sure the number keeps falling.
The scale is the part to remember. He reproduces GPT-2, a 2019 model with 1.6 billion knobs, trained on 100 billion tokens. In 2019 that cost about forty thousand dollars. In his re-run it took about a day and about six hundred dollars on a rented machine with eight top-end GPUs at three dollars per GPU per hour, and he says a hundred dollars is reachable with effort. Hardware got faster, the data got cleaner, the software got better. Meanwhile the frontier moved the other way: the models people use today have hundreds of billions of knobs and read trillions of tokens.
5. What you get: a document dreamer
At the end of that stage you have what Karpathy calls a base model. It is not an assistant. Ask it "what is 2 plus 2" and it does not answer; it continues the text as if it were a web page that happened to start that way, wandering wherever the statistics take it. His blunt description: "a very, very expensive autocomplete".
Two things about the base model are worth keeping.
First, what it stores. Its knobs are a compression of everything it read, and Karpathy's comparison is a zip file, but a lossy one. Famous pages it can recite almost word for word, because it saw them many times; he shows a base model reciting the opening of a Wikipedia article from memory. Rare things it half-remembers. Ask it about events after its training cut-off and it will simply continue in a parallel universe, confidently. This is the "month-old book" from article 1, seen from the factory side.
Our own demo, in the shape of his. We took a small open base model, Qwen2.5-1.5B from Alibaba's Qwen team, ran it on a laptop processor, and typed the opening of a sentence about an event after its training cut-off: "The 2026 FIFA World Cup final was won by". It finished the sentence five times and named four different winners: Italy; England; Spain, with Lionel Messi scoring the winner against France in Paris; the host nation, the United States, two nil, on a Sunday in November. None of it happened. Nothing in the answers hesitated. Then we asked the same way about the 2022 final, which did happen, and this small model dreamed there too: Brazil in a penalty shoot-out, then Spain, then Italy, then Croatia. At this size even the past is fog. Karpathy's demonstration uses a base model with 405 billion knobs, and that one recites a famous Wikipedia page word for word; a bigger book remembers its famous pages and still dreams the pages that were never written. Asked for Israel's minimum wage as of April 2026, our small model gave five different figures, each a fraction of the real one. Neither the dream nor the fog is a lie. Each answer is the most likely continuation of a page the model never got to read, or read too rarely to remember. Every completion is in the receipt beside this article, word for word.
Then we put the same three questions to thirteen finished assistants, the products people actually use, to see what the worked-examples stage does to the dreamer. That afternoon became article 4, "The test you can run yourself".
Second, what it can already do if you are clever. Give it ten pairs of English words with their Korean translations and an eleventh English word, and it continues the pattern and translates. Write a fake transcript of a helpful assistant talking to a person, and it plays the assistant. It is not an assistant. It is a very good continuer of documents, and a document that looks like a conversation gets continued as one.
6. The worked examples: how it becomes an assistant
Back to the textbook. The second stage is the worked examples, and here is what they are in practice.
The company hires people to write ideal answers to questions, hundreds of thousands of them, and the model reads those conversations instead of the internet for a while. It learns the shape of a good answer and starts behaving like an assistant. This is the "labeler" from article 1. The technical name is supervised fine-tuning, and one fact about it matters more than any other: it is the same guessing game, with the same knobs, run on a different pile of text. Nothing else changes. Karpathy is explicit that it is the exact same algorithm with the dataset swapped out.
The costs are wildly different, and that tells you where the value sits. He gives rough figures: pretraining can run for about three months on thousands of computers; the conversation stage can run for about three hours. Reading the internet is expensive. Learning good manners from a few hundred thousand conversations is cheap by comparison.
7. Practice problems, and the move no human would make
A little more on the third stage, because it explains the "thinking" models in your menu.
For a maths problem with a known answer, you can let the model try thousands of solutions, check which ones reach the answer, and nudge the knobs toward the ones that worked. Karpathy calls it guess and check. Nobody writes the solutions; the model discovers what works for it. What emerged, when a lab called DeepSeek published this properly in early 2025, is that the model started to write long inner monologues: "wait, wait, wait", let me re-check, let me try another way. Its answers got longer and got better. That inner monologue is what a thinking model does before it writes you a clean answer.
Why it can go past its teachers: he reaches for AlphaGo, the program that beat the world's best Go players. A version trained only to imitate expert moves topped out at expert level. The version that practised against itself went past them, and famously played a move that a human would have played about one time in ten thousand. In hindsight it was brilliant. Practice can find what imitation cannot.
Two cautions come with it, and he is careful about both. It only works where the answer can be checked. For a joke or a poem there is no answer key, so the labs train a second model to imitate human taste and practise against that, and the imitation can be gamed: run it too long and the "best joke" degrades into nonsense that the judge inexplicably loves. And it is early. The recipe is simple to state and hard to get right.
What to do tomorrow
- When you hear "trained on the internet", hear "guessed the next word on a cleaned copy of the web". It is a statistics machine over text, not a library.
- When it recites something famous perfectly, that is the zip file. When it talks about you or your niche, that is the fog. Put your material on the desk.
- The assistant you talk to is the worked-examples stage. Its manners were learned from a few hundred thousand human-written conversations on top of the internet.
- The "thinking" models are the practice-problems stage. Reach for them on hard maths, code, and multi-step reasoning. Karpathy himself uses the plain model for most everyday questions and the thinking model for the hard ones.