In A Deep Dive into How LLMs Are Built, I kept waving at the thing inside the model and saying you don’t need the wiring diagram. We talked about tokens flowing through “a giant stack of dials,” about parameters getting nudged, about loss and backpropagation — all as black boxes you could take on faith. This post opens those boxes.
It’s the missing middle layer. The deep dive was the story of how a model is built; the math-for-ml series is the full derivation with worked numerical examples. This post is the conceptual machinery in between — enough to actually picture what a neuron computes, why the Transformer won, what “loss” really measures, and how reinforcement learning is the same machine pointed at a different goal. No heavy math; just the mechanisms, with a diagram or table wherever a picture beats a paragraph.
What you’ll learn
By the end you should be able to:
- Explain what a single neuron computes and why stacking them into layers produces intelligence.
- Say precisely what loss is, and how gradient descent and backpropagation use it to teach the network.
- Understand what problem the Transformer solves, and how attention does it.
- See how reinforcement learning is the same gradient machine optimising a reward instead of matching a token.
The neuron and the network
Everything in a neural network is built from one tiny, almost embarrassingly simple unit: the neuron. It does three things in order.
That’s the whole unit:
- Weight each input. Every incoming number is multiplied by a weight — how much this neuron cares about that input.
- Sum them, add a bias. Combine the weighted inputs into a single number, plus a constant nudge (the bias) that shifts when the neuron “fires.”
- Apply an activation. Pass that sum through a simple non-linear function (like ReLU, which just clamps negatives to zero). Without this step, stacking neurons would collapse into one big linear equation — the activation is what lets depth add power.
Term to know: The weights and biases are the parameters — the billions of dials from the deep dive. A neuron is just
output = activation(inputs · weights + bias). Learning means finding good numbers for those dials.
One neuron is nearly useless. The magic is in the arrangement: neurons are grouped into layers, and the output of each layer becomes the input to the next. Early layers learn crude features; later layers combine them into abstract ones. Stack dozens of layers and you can represent staggeringly complex functions — including, it turns out, “what word comes next.”
Here is the forward pass — what happens every single time the model reads tokens, whether in training or when answering you:
Tokens become vectors, vectors flow through the layers, and out the other end comes a probability for each of the ~100,000 possible next tokens. That’s it. The network is a pure function: same dials, same input → same output. Learning is the separate question of how those dials got their values — which is the next two sections.
Loss: measuring how wrong
The network starts with random dials, so its first predictions are nonsense. To improve it, we need a single number that says how wrong this prediction was. That number is the loss.
For next-token prediction the loss is cross-entropy, and its intuition is simpler than the name: it looks only at the probability the model assigned to the token that actually came next, and punishes it for not being confident enough.
| Probability the model gave the correct token | Loss |
|---|---|
| 0.99 — confident and right | tiny |
| 0.50 — unsure | medium |
| 0.01 — confident and wrong | huge |
The asymmetry is the whole point: being confidently wrong is punished savagely, while being right earns almost zero loss. Averaged over trillions of tokens, “minimise the loss” turns out to mean “predict text really well.”
So how do we lower it? Picture the loss as a landscape, where every possible setting of the dials is a location and the height is the loss there. Training is just walking downhill.
Term to know: The gradient is the slope of the loss landscape — it points in the direction the loss increases fastest, for every parameter at once. Step the opposite way and the loss goes down. This is gradient descent, and it’s the engine under every “training” you’ve ever heard about. (The calculus, in full.)
Aha moment: “Training a model” is not mystical. It is: guess, measure how wrong (loss), figure out which way is downhill (gradient), take a small step, repeat a few trillion times. The intelligence is an emergent side effect of relentlessly minimising one number.
Backpropagation: sharing out the blame
Gradient descent needs the gradient — but the network has hundreds of billions of parameters, and the loss only appears at the very end, after the prediction. How do we know how much each individual dial contributed to the error?
That’s the job of backpropagation. The loss flows backward through the network, and at each step the chain rule splits the blame: every parameter learns exactly how much it pushed the prediction in the wrong direction.
You don’t need the calculus to hold the idea: forward pass produces the error, backward pass distributes responsibility for it. Once every parameter knows its share of the blame, gradient descent nudges it accordingly. (If you want the chain rule worked out by hand through a real network, that’s the backprop post.)
This loop — forward, loss, backward, step — is training. Pretraining and fine-tuning from the deep dive are the exact same loop on different text.
The Transformer: why this architecture won
So far “the layers” have been a black box. What makes them a Transformer specifically? To see why it’s special, look at what came before.
Older sequence models (RNNs) read text the way you might read aloud: one token at a time, carrying a running summary in memory. That summary gets blurry over distance — by the thousandth word, the first word is a faint memory — and because each step depends on the last, you can’t process the sequence in parallel.
| RNN (the old way) | Transformer | |
|---|---|---|
| Reads the sequence | one token at a time, in order | all tokens at once |
| Link between distant tokens | fades with distance | direct, any token to any token |
| Training speed | slow (inherently sequential) | fast (all positions in parallel) |
| Recall of an early token | blurry | intact, via attention |
The Transformer’s breakthrough is attention: a mechanism that lets every token look directly at every other token and pull in whatever it needs, no matter how far away. The way I find it easiest to picture is as a tiny search engine running inside each layer:
- The current token emits a Query — “I’m a verb looking for my subject.”
- Every token offers a Key — “I’m a noun, here’s what I am.”
- The query is matched against all keys to produce relevance scores (high for the tokens that matter).
- The output is a blend of every token’s Value, weighted by those scores — so the token pulls in mostly from the words that are actually relevant to it.
Two refinements make it work in practice, both worth knowing by name:
- Multi-head attention. Run several of these searches in parallel, each free to focus on a different kind of relationship (grammar, topic, long-range references). The deep dive’s “dozens of layers” each contain a stack of these heads.
- Positional encoding. Attention by itself is order-blind — it sees a set of tokens, not a sequence. So we add position information to each token’s vector, giving the model a sense of word order.
That’s the conceptual tour. The Query/Key/Value math with real numbers is in the attention post, and the full assembled GPT block is in the Transformer post.
Aha moment: Attention is why an LLM can keep a 100-page document coherent. There’s no fading memory to lose — every token has a direct line to every other token in the context window. This is the mechanical reason “knowledge in the context” is so much sharper than “knowledge in the weights.”
Reinforcement learning: the same machine, a different goal
Everything above is supervised learning: for every token there’s a known right answer, so we can compute a loss and walk downhill. But the deep dive’s third stage — reinforcement learning — has no labelled next token. Nobody wrote down the “correct” way for the model to reason through a hard problem. So what replaces the loss?
A reward. Instead of grading each token against a known answer, the model produces a whole attempt, and then we score the result: did the code pass the tests? Did the math reach the right number? That single score, after the fact, is the only signal.
| Supervised (pretrain + fine-tune) | Reinforcement learning | |
|---|---|---|
| Signal arrives | at every token | once, after a full attempt |
| Question asked | ”right next token?" | "did the whole answer work?” |
| Source of truth | human-written text | a checker, or a reward model |
| Best it can become | as good as its data | can exceed its data |
The training loop has the same shape as the supervised one, but the “loss” is replaced by “how much reward did this attempt earn”:
Term to know: Nudging the dials to make rewarded behaviour more probable is a policy gradient. “Policy” is just RL’s word for the model’s strategy — the same parameters, viewed as “what it tends to do” rather than “what token it predicts.”
The crucial mental shift: gradient descent never went away. We’re still computing a gradient and stepping the dials. We’ve only swapped the objective — from “minimise the loss against a known token” to “maximise the expected reward.” Same engine, aimed at a different target. That’s why a model that learned to predict text can be taught, with the very same machinery, to reason.
The catch from the deep dive falls right out of this framing. When there’s no automatic checker — “write a kind summary” — you can’t compute a reward directly, so you train a separate reward model to imitate human preferences and use its score as the reward. It works, but the reward model is itself a lossy network, so push the optimisation too hard and the model learns to game it. (Why that asymmetry matters.)
What to take away
Pull the black boxes apart and the whole thing reduces to a few moving parts:
- A network is just weighted sums and squashes, stacked. A neuron multiplies, adds, and squashes; layers of them compose into arbitrarily complex functions. The parameters are the dials.
- Loss is the one number that drives everything. Training = measure how wrong (loss), find which way is downhill (gradient via backpropagation), step the dials, repeat. Intelligence is the side effect of minimising it.
- The Transformer’s superpower is attention. Every token gets a direct line to every other token, which is why long contexts stay coherent and why the architecture trains in parallel.
- Reinforcement learning is the same engine, new objective. Swap “match the known token” for “maximise reward” and the identical gradient machine teaches the model to discover its own reasoning — exceeding the data it was trained on.
With the wiring diagram in hand, the deep dive’s black boxes — “nudge the parameters,” “loss,” “attention,” “reward” — stop being magic words and become a single, traceable mechanism.