A Deep Dive into How LLMs Are Built

If you have ever typed a question into ChatGPT and wondered what the thing on the other end actually is — not the marketing answer, the mechanical one — this post is the long version of that answer. We are going to follow a single model from a blank slate to a working assistant, through the three training stages that every frontier model goes through, and stop along the way to look at the strange consequences each stage has for how the finished product behaves.

This is a companion to my earlier Intro to Large Language Models post. That one was the conceptual map — what an LLM is, the OS analogy, the security surface. This one goes a layer down into the machinery: how the numbers actually get made, why the model knows what it knows, and why it fails in the specific ways it does. You don’t need to have read the intro first, though it won’t hurt.

The structure and many of the framings here — the three-stage pipeline, “going to school” as an analogy for training, the idea that the model needs tokens to think — come from Andrej Karpathy’s superb Deep Dive into LLMs like ChatGPT. It’s about three and a half hours long and worth every minute. This is my own retelling, reorganised around the questions I most wanted answered.

What you’ll learn

By the end you should be able to:

Walk through the three training stages — pretraining, supervised fine-tuning, reinforcement learning — and say what each one contributes.
Explain tokenization and why it’s the root cause of a surprising number of model failures.
Reason about where a model’s knowledge lives, and why that explains hallucinations.
Understand why models need tokens to think, and why reasoning models are the natural next step.

The whole pipeline in one picture

Before we zoom in, here is the entire journey. Three stages, each with its own data, its own cost, and its own purpose.

The cleanest analogy — and the one I’ll keep coming back to — is going to school:

Pretraining is the textbooks. The model reads an enormous amount of background material and absorbs how the world is described in writing. This is the exposition.
Fine-tuning is the worked examples. Experts demonstrate, step by step, how a good answer is produced. This is imitation.
Reinforcement learning is the practice problems. The model is handed problems with known answers, told to find its own way to them, and rewarded when it succeeds. This is where it discovers rather than imitates.

Hold that frame. Everything below is just detail hung on it.

Stage 1: Pretraining

Step one is a giant pile of text

Pretraining starts by downloading a large chunk of the public internet. Everyone does some version of this; a good open example is FineWeb, a cleaned-up dataset distilled from Common Crawl (a non-profit that has been scraping the web since 2007). The raw crawl is petabytes. After the cleaning pipeline runs, you’re left with something on the order of tens of terabytes of text — roughly 15 trillion tokens. It would fit on a single hard drive.

“Cleaned up” is doing a lot of work in that sentence. The pipeline is a gauntlet:

URL filtering — drop known spam, malware, and adult-content domains using blocklists.
Text extraction — pull the actual prose out of raw HTML, throwing away navigation, markup, and ads.
Language filtering — keep documents above a confidence threshold for the languages you care about (English-heavy, for many models).
Deduplication — remove near-identical copies of the same page.
PII removal — best-effort stripping of things like addresses and account numbers.

What survives is the textbook the model will read. Its quality and composition silently determine almost everything downstream — including, as we’ll see, what the model will later confidently make up.

Tokenization: the model’s strange alphabet

The model does not read text as letters, or even as words. It reads tokens. Understanding tokenization is unreasonably useful, because a startling number of “why is the model so dumb about this?” moments trace straight back to it.

Why not just feed it words? Or letters? Each extreme breaks in its own way, and tokens are the compromise that threads the needle:

Whole words don’t work. The model needs a fixed list of symbols, decided once before training. But language has effectively unlimited words — names, typos, slang, code, new coinages every day. Any word not on the list would arrive as an unreadable blank, and the list could never be complete.
Single letters don’t work either. You could spell everything out with ~26 letters plus punctuation, which never runs out of symbols. But then even a short paragraph becomes thousands of symbols, and the model has to grind through far more steps to read the same amount of text. Sequences get painfully long.
Tokens are the middle ground. A token is a chunk — usually a piece of a word. Common words (the, and) become a single token; rare words get split into a few familiar pieces. You get a fixed, manageable vocabulary that never hits an unreadable blank, while keeping sequences short.

The recipe for building that chunk list is Byte Pair Encoding (BPE): start from individual bytes and repeatedly glue the most frequent adjacent pair into a new symbol, until you reach a target vocabulary size. GPT-4’s tokenizer lands on 100,277 tokens — a vocabulary of sub-word chunks where common words are one token and rare words are split into pieces.

Term to know: A token is one entry in the model’s vocabulary — usually a sub-word fragment, sometimes a whole common word, sometimes a single character. The model only ever sees sequences of these. Tools like Tiktokenizer let you paste text and watch it get chopped up.

A few quirks that fall directly out of this and will matter later:

Case and leading spaces create different tokens. "hello", " hello", "Hello", and "HELLO" can all be distinct tokens. The model has to learn they’re related.
The model can’t see letters. It sees ["straw", "berry"], not s-t-r-a-w-b-e-r-r-y. This is the real reason it fumbles “how many R’s in strawberry.”
Numbers tokenize inconsistently. "127" might be one token while "677" is two. Arithmetic is hard partly because the digits aren’t cleanly laid out.

Aha moment: The model’s “alphabet” isn’t our alphabet. It’s ~100,000 sub-word chunks, and it has no direct access to the characters inside them. Half the model’s silliest failures are spelling and counting tasks that we find trivial because we operate on letters and it doesn’t.

Training: predict the next token, a quadrillion times

Now the actual learning. We slide a context window across the token stream — say, up to 8,000 tokens at a time — and at every position ask the network one question: given everything so far, what token comes next?

The network outputs a probability for every one of the ~100,000 possible next tokens. At the start of training those guesses are random noise. We compare the guess to the token that actually came next, measure how wrong it was (the loss), and adjust every parameter a tiny amount so that next time it would lean a little more toward the right answer. Do this across trillions of tokens and the guesses sharpen from noise into something uncannily good.

That loop — guess, check, nudge — is the entire engine:

The “nudge every parameter” step is backpropagation — the math that works out which direction to turn each dial. You don’t need the formula; you just need the picture: every chunk of tokens leaves every parameter a hair closer to predicting good text.

Tokens and parameters are not the same thing

It’s easy to blur these two, so let’s pin them down — because training is essentially a machine for turning one into the other:

Tokens are the input. They’re the chopped-up text — 15 trillion of them — that flows through the model. A token is data passing by.
Parameters are the model. They’re the billions of dials inside the network. They are the only thing that changes during training, and the only thing kept afterward.

Term to know: The parameters (or weights) are the billions of numbers being tuned. They start random and end up encoding everything the model “knows.” A modern open model like Llama 3 has 405 billion of them, trained on 15 trillion tokens.

And here’s the part people find surprising: the training tokens are thrown away. Once a chunk has been shown to the model and used to nudge the dials, it’s discarded — the model never keeps a copy of its training data. What survives is only the effect those tokens had on the parameters. Fifteen trillion tokens of internet text get compressed down into a few hundred billion numbers; the tokens produced the parameters, and then the tokens are gone. (It’s also why you can’t ask a model to “show you” a page it trained on — there’s no page stored in there, only the residue it left in the weights.)

So what is this “network” the tokens flow through? Picture a giant stack of those dials, wired together in layers. A token goes in at one end, passes through every layer, and a prediction falls out the other. The specific wiring frontier models use is called a Transformer, and its one trick worth naming is attention: at each layer, every token gets to look back at every other token before it and pull in whatever context it needs. (The intro post has the 60-second tour if you want it.) For this post, the wiring doesn’t matter — what matters is what training does to those dials:

Aha moment: There is no database inside the model. We never tell it a single fact. We only ever play “guess the next token” — and to get good at that game across the entire internet, the network is forced to compress facts, grammar, reasoning patterns, and world knowledge into its weights. The knowledge is a side effect of compression, not something we stored.

This is also why training is so expensive: every one of those hundreds of billions of parameters gets updated, over and over, across the whole dataset. The 2024-era runs took thousands of GPUs and weeks; frontier 2026 runs cost tens to hundreds of millions of dollars.

Inference: the model dreams

Once trained, using the model — called inference — is cheap by comparison. The dials are now frozen; the model just runs the same next-token loop, except this time it feeds on its own output:

You hand it a prefix of tokens.
It produces a probability over every possible next token.
You sample one from that distribution (the temperature knob controls how adventurous the pick is).
You append it to the prefix and repeat.

Same engine in both phases — what differs is what changes and what’s kept:

	Training	Inference
The dials (parameters)	nudged every step	frozen
Where the text comes from	real human text	the model’s own samples
What you keep	a better model	the generated tokens
Cost	weeks, millions of dollars	milliseconds per token

Because inference samples rather than looking anything up, the output is a kind of controlled hallucination — the model produces text that is statistically plausible given its training, not text it is reciting. Sometimes that plausible text happens to be a memorised fact; sometimes it’s a confident fabrication in exactly the same shape. The model cannot tell the difference, and neither stage so far has given it any way to. Park that thought — it’s the seed of the hallucination problem.

What you have now: a base model

Karpathy demonstrates this stage with a released base model (a pretrained network with no further training), and the behaviour is genuinely weird the first time you see it:

It’s a document simulator, not an assistant. Ask it a question and it might reply with more questions, because on the internet a question is often followed by other questions. It autocompletes; it doesn’t converse.
It can regurgitate. Prompt it with the opening of a famous Wikipedia article and it will often recite the next paragraph nearly verbatim — memorisation, not understanding.
It confabulates fluently. Ask about a real person and it will produce a biography that is partly true and partly invented, with no signal telling you which is which.
It can already do new tasks from examples alone. Give it a few English: ... → Korean: ... pairs and a fresh English word, and it’ll continue the pattern. This is in-context learning — the model picking up a task from the prompt without any weight updates — and it’s the first real glimmer of intelligence in the pipeline.

A base model is a fascinating artifact and almost useless as a product. It’s a savant that only knows how to continue documents. Turning it into something that answers you is the job of the next stage.

Stage 2: Supervised fine-tuning

The trick to making an assistant is disarmingly simple: keep training the exact same model with the exact same next-token-prediction algorithm, but swap the data. Instead of random internet documents, we train on a curated dataset of ideal conversations.

These conversations are written by humans — originally contractors following detailed labelling guidelines (be helpful, be truthful, be harmless), and increasingly with heavy AI assistance. Each one looks like:

User: Can you help me write a haiku about autumn?
Assistant: Crisp leaves drift downward / a cool wind carries the smell / of woodsmoke and rain

There might be hundreds of thousands to millions of these. The model trains on them just like before — but now it’s absorbing the pattern of being a helpful assistant. The factual knowledge was already baked in during pretraining; fine-tuning is teaching behaviour and format, not knowledge.

Conversations are just tokens too

For the model to learn turn-taking, conversations have to be flattened into a single token sequence, with special tokens marking who’s speaking. Different models use different schemes, but the idea looks like:

<|im_start|>user<|im_sep|>What is 2 + 2?<|im_end|>
<|im_start|>assistant<|im_sep|>2 + 2 = 4<|im_end|>

Aha moment: When you talk to ChatGPT, you are not talking to a being that learned to converse. You are sampling from a model that is imitating how a helpful human labeler — following a company’s guidelines — would have written the next turn. The assistant is a statistical simulation of a well-trained labeler. Its “personality” is a data artifact.

That reframing — the assistant as a simulated labeler — explains almost everything odd about model behaviour. Let’s use it.

The psychology of an LLM

A fine-tuned model is useful, but it has a consistent set of cognitive quirks that look bizarre until you remember how it was made. This section is my favourite part of the deep dive, because each quirk has a clean mechanical cause.

Why it hallucinates

Picture the fine-tuning data. It is full of confident, knowledgeable answers, because the human labelers writing it usually did know the answers. So the model learns a dangerous meta-lesson: the assistant always responds with a confident answer. When you then ask about something it has no real knowledge of — a made-up person, an obscure fact — it does what its training showed it: it generates a confident, fluent, plausibly-shaped answer. There is no fact there, just the style of an answer. That’s a hallucination.

The model isn’t lying. It genuinely has no internal “I don’t know this” signal — nothing in training taught it to recognise the edge of its own knowledge. There are two fixes, and modern models use both:

Teach it to say “I don’t know.” You can probe the model to find questions it consistently gets wrong, then add training examples where the correct response to those questions is an admission of ignorance. This builds a learned sense of its own boundaries. Mechanistically, models seem to have internal features that “light up” when they’re uncertain; this training connects that signal to the act of refusing.
Give it a way to look things up. Let the model emit a search query, run the search, and paste the results back into its context. Now it can answer from retrieved text in front of it rather than from hazy memory — the difference between recalling and reading.

That second fix points at the single most important distinction in the model’s head.

Knowledge in the weights vs. knowledge in the context

Aha moment: Knowledge baked into the parameters is a vague recollection — like something you read a year ago. Knowledge in the context window is working memory — like something you’re reading right now. The second is far more reliable, and almost every good LLM technique is really about getting the right information into the context window.

Side by side:

	Weights (parameters)	Context window
Like…	something you read a year ago	something you’re reading right now
Reliability	fuzzy, easily confabulated	sharp, near-verbatim
How it got there	compressed in during training	put there by you, just now
Capacity	the whole internet, lossily	a few hundred thousand tokens, exactly

This one distinction explains a huge amount:

It’s why giving the model the relevant document and asking it to answer from that beats asking it to recall the same content from memory.
It’s why retrieval-augmented generation (RAG) is the dominant production pattern — it’s a machine for stuffing the right pages into working memory.
It’s why a model summarising a chapter you paste in is sharp, while the same model recalling that chapter “from memory” is fuzzy and error-prone.

When you can choose, put knowledge in the context, not in a prompt that asks the model to remember.

The model needs tokens to think

Here’s a subtle and genuinely important one. The model does a fixed, finite amount of computation per token. It cannot pause, think hard for a while, and then emit a brilliant answer in one shot — each token gets the same compute budget. The only way it can do more reasoning is to spread that reasoning across more tokens.

This has a concrete, almost shocking implication for how answers should be structured. Compare two fine-tuning examples for “what is 3 oranges plus 2 apples times 4?”:

Bad:  "The answer is 11. We compute 2 × 4 = 8, then 8 + 3 = 11."
Good: "First, 2 apples × 4 = 8. Then 8 + 3 oranges = 11. The answer is 11."

The “bad” version forces the model to produce the final answer (11) before it has generated the tokens that do the arithmetic — it has to compute the whole thing in a single forward pass, which it’s bad at. The “good” version lets it work left to right, using each token of the explanation as a scratchpad, so the hard computation is distributed across many tokens. Same answer, wildly different reliability.

Aha moment: “Show your work” isn’t a politeness for the human reading along. The intermediate tokens are the computation. A model that blurts the answer first has no room to think; a model that reasons step by step is literally using the page as external memory. This is the entire reason chain-of-thought prompting works.

The same principle is why you should offload exact computation to tools. Mental arithmetic is unreliable; instead, good models emit code and run it:

User: What's 1234 * 5678?
Assistant: <emits Python: print(1234 * 5678)>  → 7006652

Using the code interpreter turns a flaky mental-math guess into a deterministic correct answer. The model’s job becomes deciding to use the tool, not doing the arithmetic in its head.

The letter-counting and spelling failures

Two of the most-mocked LLM failures — “count the R’s in strawberry” and “how many words in this sentence” — come straight back to tokenization. The model sees chunks, not characters, so it has no direct access to the letters it’s being asked to count. And as we saw, it can’t reliably count in a single pass anyway. The fix, again, is tools: ask it to write code that counts, and the problem evaporates.

”Who are you?”

If you ask a fresh base model “who built you?” it’ll often claim to be ChatGPT, regardless of who actually built it — because the internet is saturated with text about ChatGPT, so that’s the statistically likely completion. A model has no innate identity. Labs hand-author fine-tuning examples (“You are [name], built by [lab]…”) or inject a system prompt at the start of every conversation to give it a consistent persona. The sense of self is, like everything else, written in.

Stage 3: Reinforcement learning

We’ve done the textbooks (pretraining) and the worked examples (fine-tuning). The last stage is the practice problems — and it’s where the most exciting recent progress lives.

Why imitation isn’t enough

In supervised fine-tuning, the model imitates expert solutions. But the expert’s way of solving a problem may not be the model’s best way. The token sequence a human finds natural might be a terrible path for the model’s particular internal representations. Imitation caps the model at “as good as the demonstrations.”

Reinforcement learning removes the cap. The setup, for a problem with a checkable answer (most of math, code that must pass tests, logic puzzles):

The model takes many independent cracks at the same problem. Some reach the right answer; most don’t. We then train the model to do more of what the winning attempts did — without anyone specifying how it should solve the problem. The model discovers its own most effective reasoning paths, in its own token-language, guided only by the reward signal “did you get it right?”

Term to know: This is reinforcement learning — learning from outcome rewards rather than from imitating demonstrations. Pretraining and fine-tuning are both imitation; only RL lets the model exceed its teachers.

The thing that emerges is reasoning

The headline result, made vivid by DeepSeek’s R1 work (which published openly what frontier labs had been doing quietly), is what happens to solution length during RL training. As the model gets better, its answers get longer — entirely on their own. Nobody told it to. It discovers that it solves more problems when it re-checks its work, tries a different approach, backtracks, considers a case it missed. You see it generate things like “wait, let me reconsider…” mid-solution.

Aha moment: Chain-of-thought reasoning wasn’t programmed in. It emerged from reinforcement learning as a discovered strategy for getting more answers right. The model figured out, on its own, that thinking longer helps — exactly the “needs tokens to think” principle from earlier, but learned rather than imposed. These long-thinking models are what we now call reasoning models.

This is the same shape of result as AlphaGo. AlphaGo trained first by imitating human games (capped at human level), then by self-play RL with a clean win/loss reward (which blew past human level). Its famous “move 37” against Lee Sedol was a play no human would make — a strategy discovered through RL, not imitated. Reasoning models are LLMs finding their own “move 37”s in the space of how to think through a problem.

What about problems with no checkable answer?

RL is clean when you can mechanically verify the answer. But most of what we want from an assistant — write a good poem, summarise this kindly, explain this clearly — has no automatic grader. You can’t write if answer == correct. Having a human score every one of millions of attempts is far too slow and expensive.

The workaround is RLHF — Reinforcement Learning from Human Feedback:

Collect a modest amount of human preference data: show people pairs of model outputs and ask which is better.
Train a separate reward model — a neural network that learns to predict those human judgments, i.e. to imitate human taste.
Run RL against the reward model’s score instead of against real humans. Now you have an automatic (if imperfect) grader you can query millions of times.

RLHF genuinely improves outputs in fuzzy domains, and it’s what gives modern assistants their polished, pleasant feel. But it has a sharp limitation worth naming:

The catch: The reward model is just another lossy neural network, not a real human. Optimise against it hard enough and the RL process will eventually find adversarial nonsense that fools the reward model into a high score while being gibberish to a person — the AI equivalent of finding the exact pixel pattern that makes an image classifier scream “ostrich.” So RLHF can’t be run indefinitely the way verifiable-domain RL can; you tune for a while, then stop before the model learns to game its grader. It’s more “a finishing polish” than “open-ended self-improvement.”

The two regimes pull apart cleanly:

	Verifiable domains	Unverifiable domains
Examples	math, code, logic puzzles	poems, summaries, explanations
Grader	automatic check — did it match?	a trained reward model imitating human taste
How long RL can run	almost indefinitely	briefly, then it games the grader
Net effect	capabilities racing ahead	a finishing polish

That asymmetry — RL can run nearly forever in verifiable domains but only briefly in unverifiable ones — is one of the most important dividing lines in the field right now, and a big part of why math and code capabilities are racing ahead of fuzzier skills.

Where this is going

A few threads from the deep dive worth keeping an eye on:

Multimodality. Audio and images are just more tokens. The same three-stage pipeline absorbs them; you’ll increasingly talk to and show things to models, not just type.
Agents. Longer, multi-step tasks where the model acts over time, uses tools, and recovers from its own mistakes — stitching many model calls into a coherent job rather than one-shot answers.
Verifiable-domain RL keeps compounding. Anywhere you can write an automatic grader, reasoning models will keep getting sharply better. The frontier is figuring out how to manufacture good reward signals for messier domains.

And the practical note: this field moves weekly. To keep up, LMArena ranks models by head-to-head human votes, an AI News style newsletter will catch you up on releases, and the open-weights models (Llama, DeepSeek, Qwen, Mistral) can be downloaded and poked at directly — often via hosting providers if you don’t have the hardware.

What to take away

If the news cycle around AI feels like an undifferentiated firehose, these are the load-bearing ideas that turn it into a story:

Three stages, one engine. Pretraining, fine-tuning, and RL all run the same next-token-prediction machine on different data. Textbooks, then worked examples, then practice problems.
Knowledge is compressed, not stored. Facts live in the weights as a side effect of compressing the internet — which is exactly why recall is fuzzy and hallucination is the default, not a bug bolted on.
Weights are vague memory; context is working memory. Almost every effective LLM technique — RAG, “here’s the document,” tool use — is a way to move knowledge from the first into the second.
Tokens are thought. The model computes by generating; “show your work” gives it room to reason, and reasoning models are what you get when RL lets the model discover that for itself.

Once those four click, the rest — agents, reasoning models, RAG, the open-vs-closed race — stops being a list of buzzwords and starts being a set of consequences you can predict.