Write Ahead

Postgres Concurrency Internals: Connections, Locks, and MVCC

Sun, 17 May 2026 00:00:00 GMT

You run ALTER TABLE on the busy orders table and every query in the cluster freezes for forty seconds. A Python script hangs with one open transaction overnight, and the next VACUUM runs for an hour. The dashboard that’s fast for you takes three seconds for the user sitting next to you, on the same data. Most of the day-to-day weirdness in a production database comes from many clients talking to it at once, and someone holding something they shouldn’t be.

The previous post covered the semantics of concurrency — which isolation level to pick, what each one guarantees, where the anomalies come from. This post is the mechanics: what actually happens when your connection hits Postgres, where the locks live, why MVCC lets readers and writers coexist, and how to figure out who’s blocking whom at 3 AM. Same database, different layer.

Part 8 of the Database Internals series.

1. The Connection Model

Every Postgres client connection is a separate OS process on the database server. There is no thread pool inside Postgres. There is no event loop multiplexing connections onto a small number of workers. When psql connects, an entire postgres process is forked for it, lives until the client disconnects, and dies when the client goes away.

The postmaster is the supervisor process. It listens on the TCP port, accepts an incoming connection, authenticates the client, then fork()s a new backend process to serve that client for its lifetime. The backend executes queries, owns its own local memory (work_mem, temp_buffers), and reaches into shared memory (shared_buffers, the WAL buffers, the lock table) for anything cluster-wide.

A backend uses roughly 5–15 MB of resident memory on its own, depending on Postgres version and what the connection has done (cached plan trees, prepared statements, large work_mem for sorts). Multiply by max_connections, add shared_buffers, and you have a hard ceiling on how many backends a box can hold before you swap.

Why this matters

Spawning isn’t free. Forking takes single-digit milliseconds. Most short-lived clients (HTTP request handlers, lambdas, CLI scripts) spend more time on connection setup than on the actual query if they reconnect every time.
Idle backends still cost RAM. A connection holding open a transaction overnight still pins memory and prevents max_connections from being available to someone else.
There’s no in-process query cache. Each backend builds its own plan cache. Two clients running the same query don’t share a parsed plan unless they’re behind a connection pool that shares backends.
Killing a backend is the unit of cancellation. pg_terminate_backend(pid) sends SIGTERM; pg_cancel_backend(pid) cancels the current query but keeps the session.

This is unusual. MySQL InnoDB uses thread-per-connection (one OS thread, shared address space — lower memory per connection, but a bug in one thread’s code path can corrupt another’s state). MS SQL Server uses fiber-style scheduling on a thread pool. Postgres’s choice was made in the early 1990s for isolation and portability, and it has shaped almost every operational pattern around the database since.

The practical consequence: max_connections caps the number of backends, not the number of clients you can serve. Each backend is a process tied to one CPU core at a time, so useful work is bounded by your cores — not by how many backends you allow. But most clients spend most of their time not executing a query: they’re waiting on the network, on application code, on a user clicking a button. A small pool of backends, kept busy and shared across many clients, serves far more clients than the pool’s size. That gap — many clients, few backends — is the gap connection pooling exists to close.

2. The Vocabulary

Before any code, the words. Concurrency discussions go sideways quickly when these terms are used loosely.

Backend — the OS process that serves one client connection. One backend per connection, period. Visible in ps as postgres: user dbname host(port) idle.

postmaster — the parent supervisor process. Accepts connections, forks backends, restarts crashed background workers. PID 1 of the cluster.

max_connections — hard upper bound on simultaneous backends. Changing it requires a restart. Default 100. Includes superuser-reserved slots.

Connection pool — a layer between clients and the database that maintains a small pool of long-lived backends and multiplexes many client connections over them. Implementations: PgBouncer, pgcat, Supabase Supavisor, application-side pools (HikariCP, pgx’s built-in pool).

Pool mode — how aggressively the pooler hands out backends. Session = backend held for the whole client session; Transaction = backend assigned per transaction; Statement = backend per statement (rarely used).

Heavyweight lock (or regular lock) — a lock on a database object (a table, a row, an index, an advisory key) that Postgres takes on your behalf when you run a statement. You don’t pick the mode; the statement does. Tracked in the cluster-wide lock table, visible in pg_locks, released at transaction end.

Lightweight lock (LWLock) — an internal lock protecting shared-memory data structures inside Postgres (buffer pool slots, the WAL insert position, the lock table itself). Two modes: SHARED and EXCLUSIVE. Held for microseconds. You don’t take these directly — they’re acquired by Postgres internals on your behalf.

Spinlock — the lowest-level mutex, used when a lock is held for nanoseconds. Busy-waits in a tight loop. Below LWLocks in the stack.

Lock mode — for heavyweight locks, the kind of lock: from the gentlest ACCESS SHARE (a SELECT) to ACCESS EXCLUSIVE (an ALTER TABLE). Eight modes total. A lock conflict matrix says which pairs of modes can coexist on the same object and which block each other.

MVCC (Multi-Version Concurrency Control) — the trick that lets Postgres readers and writers ignore each other. Instead of locking a row while you read it, Postgres keeps multiple versions of the row on disk and serves whichever version was committed when your transaction started. A long-running SELECT doesn’t block UPDATEs, and ongoing UPDATEs don’t block readers — they each see their own consistent snapshot. The cost: outdated versions pile up until vacuum reclaims them. Postgres, Oracle, and MySQL InnoDB all use MVCC. Section 5 unpacks the on-disk mechanics.

Tuple — Postgres’s term for a row. Specifically, a physical row version on a heap page. An UPDATE produces a new tuple; the old one stays around for MVCC visibility until vacuum reclaims it.

xmin / xmax — hidden columns on every tuple. xmin = the transaction ID that inserted it. xmax = the transaction ID that deleted or updated it (zero if neither). Together they decide whether a tuple is visible to a given snapshot.

XID (Transaction ID) — a 32-bit number assigned to every write transaction. Read-only transactions get a “virtual” XID that costs nothing. Wraparound is real, autovacuum manages it.

Snapshot — a frozen record of “which transactions had already committed at the moment I started.” Every transaction takes one. Every row the transaction reads is filtered through it, so the transaction sees a consistent point-in-time view of the database even as other transactions commit changes around it.

Wait event — what a backend is blocked on right now. Column in pg_stat_activity. Values include Lock (waiting on a heavyweight lock), LWLock, BufferPin, IO, Client, Timeout, IPC. The first thing to check when “the database is slow.”

Deadlock — two or more transactions waiting on locks held by each other, forming a cycle. Postgres detects this periodically (every deadlock_timeout, default 1 second) and aborts one transaction with SQLSTATE 40P01.

Advisory lock — a database-hosted mutex that callers compete for using an integer key they pick themselves. Postgres doesn’t connect the key to any row, table, or object — it just guarantees that two callers asking for the same key can’t both hold the lock at once. Used by application code to serialize work across processes (“only one instance runs this cron job,” “elect a leader”) when you already have a database and don’t want to stand up a separate coordinator like Redis or ZooKeeper.

3. Connection Pooling

A backend can run one query at a time, and the OS can only execute as many processes in parallel as the box has cores. So however many backends you allow, only a handful are actually doing work in any given moment — roughly 2 × cores + effective_io_concurrency, give or take, depending on how much your queries wait on disk. Stack more backends on top of that and they start fighting each other for the same CPU and the same internal lock table; throughput stops climbing and starts dropping:

The trick: most applications have thousands of clients but only need dozens of backends at any instant. A web server with 200 worker threads doesn’t issue 200 simultaneous queries — most threads spend most of their time in application logic, I/O to other services, or idle. A connection pooler exploits this.

PgBouncer pool modes

PgBouncer is the industry-standard Postgres pooler. Its three modes differ in how long it holds a backend on behalf of a client.

Session pooling. A client connects, PgBouncer assigns it a backend, the backend stays bound to that client until the client disconnects. PgBouncer is essentially a connection multiplexer for the count, not the time. Useful when you have many short-lived clients (CLI scripts, lambdas) but ordinary session semantics inside each one.

Transaction pooling. A client checks out a backend at BEGIN and returns it at COMMIT/ROLLBACK. Between transactions, the same backend can serve a different client. This is the high-leverage mode — a pool of 50 backends easily serves 5000 clients if average transactions are a few milliseconds. Most production deployments use transaction pooling.

Statement pooling. Backend returned after every statement. Breaks multi-statement transactions entirely. Rarely useful outside read-only pipelines.

Transaction-pooling caveats (the real-world list)

The cost of transaction pooling is that anything that relies on session state between transactions breaks. The exhaustive list:

Server-side prepared statements. A prepared statement lives on the backend that prepared it. If the next transaction lands on a different backend, the prepared name doesn’t exist. The fixes: tell your driver to disable server-side prepares (prepareThreshold=0 in JDBC; binary_parameters=yes + prefer_simple_protocol in some Go drivers), or use a pooler that emulates prepared statements (recent PgBouncer ≥ 1.21 and pgcat both do this).
SET (without LOCAL). Session-scoped SET timezone = 'UTC' is sticky on the backend. The next client lands on a backend with whatever the previous client did. Use SET LOCAL, which is transaction-scoped and cleared at COMMIT.
Cursors declared with DECLARE CURSOR ... WITH HOLD. Cursors don’t survive backend hand-off. Use server-side cursors only in session pooling.
LISTEN / NOTIFY. The subscription is on the backend. Once the backend is returned to the pool, you stop receiving notifications.
Temporary tables. CREATE TEMP TABLE lives on the backend; the next transaction won’t see it.
Advisory session locks (pg_advisory_lock). Released only when the backend session ends. Under transaction pooling, sessions don’t really end on the client’s clock — use pg_advisory_xact_lock instead.

When PgBouncer’s transaction pooling caveats bite, the symptom is usually “works in dev with one client, breaks in prod under load.” The fix is either to disable the offending session-scoped feature or to put PgBouncer in session-pooling mode for the specific path that needs it (two pools, two ports).

Pool sizing

The starting formula for the backend pool:

pool_size = (cpu_cores × 2) + effective_io_concurrency

For a 16-core box on NVMe: 16 × 2 + 4 ≈ 36. Round to 40. The remaining max_connections budget pays for admin sessions, replication slots, and headroom.

The starting formula for the client side (default_pool_size × max_db_connections):

max_client_conn = number of app instances × average client pool size

Set this generously — PgBouncer holds idle clients cheaply (a few KB each).

What’s after PgBouncer

PgBouncer is single-process, single-threaded (which is fine for hundreds of thousands of clients on modern CPUs, but caps single-instance throughput in the millions-of-qps range). The newer options:

pgcat — Rust, multi-threaded, supports sharding and read/write splitting natively.
Supabase Supavisor — Erlang/Elixir, designed for cloud-scale fan-in.
Odyssey (Yandex) — multi-threaded, transaction-pooling pgbouncer drop-in.

For the median application: PgBouncer in transaction-pooling mode, in front of a pool of 30–60 backends, is the right starting point.

4. The Lock Manager

Two layers, very different jobs.

Heavyweight locks — the user-visible layer

Heavyweight locks protect database objects: tables, rows, indexes, advisory keys. They’re tracked in a cluster-wide hash table in shared memory, indexed by (database, relation, object). Every entry records which transaction holds it, in which mode, and which transactions are queued waiting for it.

There are two questions to keep straight, and the post is going to answer them with two separate tables.

Acquisition — given a SQL statement, which mode does it take?
Compatibility — given two locks on the same object, can they coexist or does one wait?

The first is a one-way lookup. The second is a symmetric relation between every pair of modes, and it’s the actual conflict matrix Postgres consults on every request.

Acquisition: which statement takes which mode

Postgres defines eight table-level lock modes. Statements take whichever they need automatically, weakest to strongest:

Mode	Acquired by
`ACCESS SHARE`	`SELECT`
`ROW SHARE`	`SELECT FOR UPDATE / FOR SHARE`
`ROW EXCLUSIVE`	`INSERT`, `UPDATE`, `DELETE`
`SHARE UPDATE EXCLUSIVE`	`VACUUM` (non-FULL), `ANALYZE`, `CREATE INDEX CONCURRENTLY`
`SHARE`	`CREATE INDEX` (non-concurrent)
`SHARE ROW EXCLUSIVE`	rarely explicit
`EXCLUSIVE`	`REFRESH MATERIALIZED VIEW CONCURRENTLY`
`ACCESS EXCLUSIVE`	`ALTER TABLE`, `DROP TABLE`, `TRUNCATE`, `REINDEX`, `VACUUM FULL`

The important rule: you don’t pick the lock — the statement does. SELECT always takes ACCESS SHARE. ALTER TABLE always takes ACCESS EXCLUSIVE. You don’t get a vote; Postgres just does whatever the statement requires before executing it. The three ways to ask for a specific lock yourself:

SELECT ... FOR UPDATE / FOR SHARE / FOR NO KEY UPDATE / FOR KEY SHARE — picks a row-level lock mode by name. Covered in the next subsection.
LOCK TABLE orders IN ACCESS EXCLUSIVE MODE — takes a table-level lock at exactly the mode you specify, no statement attached. Useful when you’re about to run several ALTER TABLEs and want one acquisition at the start of the transaction instead of each statement re-acquiring it.
pg_advisory_lock(key) and friends — application-defined locks keyed by an integer. Covered in section 8.

Once a mode is held on an object, the conflict matrix decides whether the next request is granted or made to wait. Rows are what’s already held; columns are what’s being requested. Blank means “compatible — grant immediately”; ✗ means “conflict — wait in queue.”

Held ↓ \ Requested →	AS	RS	RE	SUE	S	SRE	E	AE
`ACCESS SHARE` (AS)								✗
`ROW SHARE` (RS)							✗	✗
`ROW EXCLUSIVE` (RE)					✗	✗	✗	✗
`SHARE UPDATE EXCLUSIVE` (SUE)				✗	✗	✗	✗	✗
`SHARE` (S)			✗	✗		✗	✗	✗
`SHARE ROW EXCLUSIVE` (SRE)			✗	✗	✗	✗	✗	✗
`EXCLUSIVE` (E)		✗	✗	✗	✗	✗	✗	✗
`ACCESS EXCLUSIVE` (AE)	✗	✗	✗	✗	✗	✗	✗	✗

How the two tables connect: the first tells you which mode your statement asks for; the second tells you whether you get it now or queue. A SELECT (asks for AS) running on a table where someone holds AE (a slow ALTER TABLE) — table 1 says ACCESS SHARE is the requested mode, table 2’s AE row says every requested mode is blocked, so your SELECT waits. The pg_locks view is essentially this matrix in motion: at any moment it shows you what’s held, what’s requested, and whether each request was granted (granted = true) or queued.

A few entries in the matrix worth committing to memory:

The bottom-right cell — ACCESS EXCLUSIVE blocks ACCESS SHARE — is why a careless ALTER TABLE freezes every SELECT on the table.
SHARE UPDATE EXCLUSIVE conflicts with itself, which is why two VACUUMs on the same table can’t run in parallel. CREATE INDEX CONCURRENTLY exists at exactly this mode — gentle enough not to block reads or writes, but two CONCURRENTLY builds on the same table still serialize.
SHARE and SHARE don’t conflict, so two non-concurrent CREATE INDEX statements on different indexes of the same table can run in parallel (they both just read the heap).

Row-level locks (a different mechanism)

Postgres does not keep per-row entries in the cluster-wide lock table. There would be far too many of them. Instead, row-level locks live in the row’s own tuple header — specifically, the xmax field gets stamped with the locker’s XID, plus a flag indicating “this is a lock, not a delete.” This is brilliantly cheap (free in storage) but has a consequence: you can’t tell, from the lock table, who is row-locking what without joining against the tuple header. The view pg_locks shows row-level locks only as tuple rows on rows that are currently being waited on; non-contended row locks don’t appear at all.

Four row-lock modes, from weakest to strongest:

FOR KEY SHARE — taken by foreign-key checks. Blocks UPDATE of indexed columns.
FOR SHARE — explicit shared lock.
FOR NO KEY UPDATE — taken by UPDATE that doesn’t touch the primary key.
FOR UPDATE — the strong one. Blocks everyone, including FK checks.

Most application code reaches for SELECT ... FOR UPDATE (covered in the ACID post). The weaker modes are mostly internal but visible when foreign keys cause unexpected contention — an UPDATE on the parent row competes with FK checks on every child.

Lightweight locks — the internal layer

Heavyweight locks are transactional — held until COMMIT. LWLocks are operational — held for microseconds while a backend reads or writes a shared-memory structure. Examples:

Pinning a buffer in shared_buffers (so it isn’t evicted while you read it).
Inserting a record into the WAL.
Reading the cluster-wide lock table (yes, the lock table itself is LWLock-protected).
Updating the visibility map.

LWLocks have only two modes — SHARED and EXCLUSIVE — and don’t deadlock-check (they’re held for so short a time that detection would cost more than the deadlock). If a backend is spending real time blocked on an LWLock, something is wrong further up: too many backends contending on one buffer, an XLogInsert bottleneck, or a hot lock-table partition.

The wait events distinguish them clearly. In pg_stat_activity.wait_event_type:

Lock — waiting for a heavyweight lock. Probably your application code or a long transaction.
LWLock — waiting on an internal lock. Probably a Postgres configuration or workload issue.
BufferPin — waiting to pin a buffer that something else is preventing eviction of. Rare.
IO — waiting on disk or the OS page cache. Not a lock at all.
Client — waiting on the client to send the next command (idle in transaction).

Different wait_event_types, completely different fixes.

5. MVCC Under the Hood

The semantics of MVCC live in the ACID post: every transaction sees a snapshot, readers don’t block writers, snapshot isolation. The mechanics are simpler than they sound.

The tuple header

Every row in Postgres carries hidden columns alongside its user-defined columns. You can see them:

SELECT xmin, xmax, ctid, * FROM accounts WHERE id = 42;

xmin — the XID of the transaction that inserted this tuple.
xmax — the XID of the transaction that deleted or updated it (zero if neither).
ctid — the physical address (page, offset) of the tuple on disk.

When you UPDATE a row:

The current tuple’s xmax is set to your XID.
A new tuple is written, with xmin = your XID and xmax = 0.
The new tuple’s ctid points back to where the row physically lives now.

The old tuple is still on disk. It will be visible to any transaction whose snapshot was taken before your commit, and invisible to any started after. Once no snapshot can see the old tuple anymore, vacuum reclaims the space.

Snapshots

A snapshot is taken at the start of a transaction (under READ COMMITTED, also at the start of each statement). Conceptually:

snapshot = {
  xmin: lowest active XID at snapshot time,
  xmax: next XID that will be assigned,
  active: list of XIDs that were in-progress at snapshot time,
}

The visibility rule, in one sentence: a tuple is visible to a snapshot if the transaction that inserted it had committed before the snapshot was taken, and the transaction that deleted it (if any) hadn’t. That’s the whole of MVCC visibility. Everything else — repeatable reads, read-your-writes, why long transactions cause bloat — falls out of that single rule.

In code, this means checking the tuple’s xmin against the snapshot (“did the inserter finish in time?”) and its xmax against the snapshot (“did the deleter, if any, finish in time?”). The Postgres source for the check lives in HeapTupleSatisfiesMVCC; you almost never need to look at it.

HOT updates (the optimization that matters)

If an UPDATE doesn’t change any indexed column, Postgres can perform a HOT (Heap-Only Tuple) update: the new tuple is written to the same page if there’s room, and the indexes don’t need updating because they still point to the old ctid, which forwards to the new one. HOT updates massively reduce index bloat on update-heavy tables.

Concrete consequence: adding an index on a frequently-updated column can dramatically slow down updates, because every update on a row now mutates an indexed column and forces a full index entry for the new tuple. This is one of the silent reasons “we added an index and writes got worse.”

Vacuum: the garbage collector

Dead tuples (xmax < oldest active snapshot’s xmin) can be reclaimed. Autovacuum is the background daemon that does this periodically per table, based on dead-row thresholds. The user-facing consequences:

Long-running transactions delay vacuum. A transaction that started two hours ago has a snapshot two hours old. Vacuum can’t reclaim anything newer than that. Dead tuples pile up. Tables bloat.
VACUUM (non-FULL) is online. It acquires SHARE UPDATE EXCLUSIVE, which conflicts only with schema changes and other vacuums. Reads and writes continue.
VACUUM FULL is offline. It rewrites the entire table, taking ACCESS EXCLUSIVE. Don’t run it on a live production table without a maintenance window. Use pg_repack (or its bundled extensions) instead — it does the same job online, locking only for the final swap.

XID wraparound

XIDs are 32 bits — 4.3 billion of them. Postgres uses a circular comparison: at any moment, half the space is “past” and half is “future.” If a tuple’s xmin would be older than the past half (>2 billion XIDs old), it becomes invisible to everyone — a silent data-loss condition.

Autovacuum prevents this by freezing old tuples: rewriting their xmin as a special “frozen” sentinel that’s visible to every future snapshot. The vacuum_freeze_max_age setting controls how aggressive freezing is. If autovacuum can’t keep up, you eventually see warnings like database "x" must be vacuumed within N transactions. At 1 million transactions remaining, Postgres refuses new writes and enters single-user mode until you run a manual emergency vacuum.

This used to be the most embarrassing Postgres outage class — entire shops taken offline by XID exhaustion. Postgres 14+ defaults plus monitoring of pg_database.datfrozenxid have mostly retired it, but it’s still the reason “autovacuum is healthy” is not a question you skip on a production cluster.

6. Deadlocks and How They’re Resolved

A deadlock is the most boring kind of concurrency bug: two transactions, each waiting on a lock the other one is holding, neither willing to give up. They’d wait forever if nothing intervened.

Postgres notices. Whenever a backend has been waiting on a lock for longer than deadlock_timeout (default 1 second), Postgres walks the “who’s waiting on whom” graph looking for cycles. If it finds one, it picks a victim and kills it:

ERROR:  deadlock detected
DETAIL:  Process 1234 waits for ShareLock on transaction 5678; blocked by process 4321.
         Process 4321 waits for ShareLock on transaction 5679; blocked by process 1234.
HINT:  See server log for query details.
SQLSTATE: 40P01

The other transaction proceeds. The aborted transaction’s client should retry.

The classic cause: inconsistent lock ordering

Both transactions tried to do the same thing — lock two accounts in a money transfer — but each locked them in a different order. The fix is canonical lock ordering: always lock the row with the lower ID first.

-- safe pattern
BEGIN;
  SELECT * FROM accounts WHERE id IN ('A1', 'A2')
   ORDER BY id  -- guarantees consistent lock order
   FOR UPDATE;
  -- transfer logic
COMMIT;

This works because Postgres acquires row locks in row-fetch order. Two transactions hitting the same pair always lock in the same order, so no cycle is possible.

When deadlocks are inevitable

Some workloads will deadlock no matter how you order locks — for example, UPDATE on rows whose order can’t be predicted by the application. The right pattern is: catch the 40P01 error, sleep with jitter, retry the transaction. Most ORMs (and Postgres clients) make this trivial. Treating deadlocks as “fatal application bugs” instead of “expected concurrency outcomes” is a common over-correction.

`deadlock_timeout` tuning

Lowering deadlock_timeout makes the detector run more often — fewer false-wait stalls before resolution, but more CPU spent checking. The default of 1 second is right for almost all workloads; raise it only if your monitoring shows the detector running frequently on contended tables. Lowering below 100 ms is rarely productive.

7. Observing It Live

When the database “feels slow,” the answer almost always lives in three views.

`pg_stat_activity` — what every backend is doing

SELECT pid, state, wait_event_type, wait_event,
       now() - xact_start AS xact_age,
       now() - query_start AS query_age,
       LEFT(query, 60) AS query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY xact_age DESC NULLS LAST;

state — active (running a query), idle in transaction (the danger zone), idle.
wait_event_type, wait_event — what’s blocking, if anything.
xact_age — how long the transaction has been open. Anything over a few seconds on an OLTP system is suspicious; anything over minutes is almost certainly a bug.

idle in transaction is the single most common cause of mystery contention. The application opened a transaction, took some row locks, then went off to do something slow (an external HTTP call, a Thread.sleep) without committing. Every other transaction touching those rows piles up behind it.

`pg_locks` — every lock in the cluster

SELECT locktype, relation::regclass, mode, granted, pid
FROM pg_locks
WHERE NOT granted;

Ungranted locks (granted = false) are the things waiting. Join pg_locks to pg_stat_activity by pid to see which queries are stuck and on what.

`pg_blocking_pids()` — the one-shot diagnostic

SELECT pid,
       pg_blocking_pids(pid) AS blocked_by,
       wait_event_type, wait_event,
       query
FROM pg_stat_activity
WHERE state = 'active' AND pg_blocking_pids(pid) <> '{}';

This returns, for every backend currently waiting, the list of PIDs blocking it. The output reads like a directed graph: “PID 1234 is blocked by 5678 and 5679.” Trace upwards to find the root culprit — usually one idle in transaction session holding everything else hostage.

If you don’t memorize any other query in this post, memorize this one. It is the single highest-leverage observation in Postgres operations.

`pg_stat_statements` — workload accounting (different but related)

Not concurrency-specific, but the natural neighbour: query-level totals over time (calls, total time, mean time, rows). Extension, needs CREATE EXTENSION pg_stat_statements; and shared_preload_libraries. When pg_blocking_pids shows a query holding everyone else up, pg_stat_statements shows how often that query runs and how slow it is on average.

8. Advisory Locks

Postgres exposes its lock manager directly to application code via advisory locks. They have no semantic meaning to Postgres — they’re keyed by a 64-bit integer (or a pair of 32-bit integers) that you assign and interpret. Two clients that agree on what key 42 means can use it to serialize work.

-- session-scoped: released when the session ends or you release explicitly
SELECT pg_advisory_lock(42);
-- ... critical section ...
SELECT pg_advisory_unlock(42);

-- transaction-scoped: released automatically at COMMIT/ROLLBACK
SELECT pg_advisory_xact_lock(42);
-- ... critical section ...
COMMIT;

There are blocking and non-blocking variants:

pg_advisory_lock(key) — blocks until acquired.
pg_try_advisory_lock(key) — returns true if acquired, false if held by someone else. The “I’ll try, but skip if busy” pattern.

When to reach for them

Idempotent batch jobs. A nightly cron is invoked by two replicas of your scheduler at the same time. Wrap the job in pg_try_advisory_xact_lock(job_id); the second invocation no-ops.
Leader election when you already have a Postgres. One worker holds pg_advisory_lock(leader_key). Whichever connection acquires it is the leader. Drop the connection and another worker picks it up.
Coordinating expensive, non-transactional side effects. Sending an email, calling an external API, regenerating a cache — pg_try_advisory_xact_lock ensures only one instance does the work.

What NOT to use them for

Replacing row-level locks. Use SELECT ... FOR UPDATE for that. Advisory locks bypass MVCC and don’t protect against your own concurrent transactions writing the same rows.
Cross-cluster coordination. Advisory locks live in one Postgres cluster’s lock table. They mean nothing to replicas, other clusters, or anything outside that one shared memory segment.
Anything that should survive a network partition. A session that loses its TCP connection has its session-scoped lock released by the postmaster — but only when the backend actually exits, which can take tcp_keepalives_idle + tcp_keepalives_interval × tcp_keepalives_count (minutes by default).

The session-pool footgun

pg_advisory_lock is session-scoped. Under PgBouncer transaction pooling, your “session” is one transaction. The lock is released the moment that transaction ends. Use pg_advisory_xact_lock instead — its lifetime matches the transaction, which is exactly what transaction pooling guarantees you own. This is the most common subtle bug when wiring advisory locks into a pooled application.

9. Buffer Pool Concurrency (Briefly)

shared_buffers is one large block of shared memory divided into 8 KB buffers, each of which can hold one page from a table or index. Every backend reads and writes through this pool — there’s no per-backend page cache.

Concurrency lives at two layers:

The buffer mapping table — which buffer holds which page. A central hash table protected by 128 LWLock partitions. Contention here shows up as BufferMapping waits, usually triggered by very high churn between sets of hot pages and rarely-visited ones.
Per-buffer state — a pin count, a content lock (SHARED for read, EXCLUSIVE for write), an “is dirty” flag. Pinning a buffer prevents eviction while a backend is reading from it. Content locks let multiple readers share a page; writers must be alone.

The everyday consequence: two backends reading the same hot page don’t block each other. Two backends writing to the same page do — for the duration of writing one row, which is microseconds. Most “lock contention” stories in Postgres ops chats are actually buffer-content contention, which is a different fix (sort the inserts, partition the table, add a partial index).

Checkpoints are the other moment when buffer-pool concurrency matters operationally. A checkpoint flushes all dirty buffers to disk. If too many buffers are dirty at once, the I/O storm during the checkpoint causes a latency spike across every query. checkpoint_completion_target (default 0.9) spreads checkpoint writes over 90% of the interval to smooth this.

10. Decision Tree

Scenario	Reach for	Why
Many idle clients, few active transactions at any moment	PgBouncer in transaction pooling, backend pool sized to `~2× cores`	Multiplexes thousands of clients onto tens of backends. The default modern Postgres deployment shape.
ORM that relies on server-side prepared statements	Session pooling OR PgBouncer ≥ 1.21 with prepared-statement support OR disable server-side prepares in the driver	Transaction pooling breaks server-side prepares unless the pooler emulates them.
Read-heavy analytics that must not block writes	Read replica + async streaming (see replication post)	Keeps the primary’s snapshot horizon young, prevents long-running readers from blocking vacuum.
Periodic batch job that must not run twice	`pg_advisory_xact_lock(job_id)` at the start of the job’s transaction	Single-cluster mutual exclusion that auto-releases on commit/rollback.
Schema migration on a busy table	`SET lock_timeout`, then `ALTER TABLE ... NOT VALID` / `VALIDATE CONSTRAINT`, `CREATE INDEX CONCURRENTLY`	Avoid acquiring `ACCESS EXCLUSIVE` for more than the timeout; do the heavy work under weaker locks.
Repeated `40001` serialization failures under `SERIALIZABLE`	Retry with backoff, shorten transactions, narrow the rows each transaction touches	SSI assumes you retry. See the ACID post for the retry pattern.
Repeated `40P01` deadlocks	Canonical lock order + retry on conflict	Order all multi-row locks the same way across the codebase.
Hot-row contention (a single counter, an inventory total)	Atomic `UPDATE counter = counter + 1` at `READ COMMITTED`, or shard the counter	Avoid `SELECT FOR UPDATE` on a row every request hits.
`idle in transaction` sessions in `pg_stat_activity`	`idle_in_transaction_session_timeout` + fix the client code	Open transactions just before the writes; close them immediately after.
`LWLock` waits dominate `wait_event_type`	Look upstream: too many backends, hot buffer, WAL bottleneck	Not an application fix — usually `max_connections` is too high or `wal_buffers` is too small.

The pattern: most concurrency problems are upstream of the database. Pool well, hold transactions briefly, lock in consistent order, retry on conflict, and Postgres handles the rest.

11. Common Mistakes

12. Closing

The number of concurrent clients your database supports isn’t max_connections. Backends are bounded by your CPU cores; clients are bounded by how briefly each one actually needs the database. Pool well and a handful of backends serves thousands of clients. The lock manager keeps writers correct; MVCC keeps readers fast; vacuum keeps the past garbage-collected; deadlock detection keeps cycles short; pg_stat_activity tells you which of those failed.

The skill is recognizing which layer a slowdown lives in. Connection-saturation, heavyweight-lock contention, internal-LWLock contention, MVCC bloat, plain old disk I/O — these look identical to the application (“the database is slow”) and have completely different fixes. The diagnostic queries in section 7 are the difference between “we’re rolling back the deploy” and “we know exactly which transaction is holding everyone up.”

The next post in the Database Internals series steps back to the operational layer: configuration, observability, and the production checklist — the settings, dashboards, and alerts that turn the everything-above into something you can run at 3 AM.

ACID, Read Phenomena, and Isolation Levels — the semantic layer this post implements. Read first if you’re new to isolation.
How Databases Actually Store and Find Your Data — what a tuple physically is on disk; the substrate for MVCC.
Database Replication — concurrency across nodes; read replicas as a release valve for long-running readers.
PostgreSQL Indexing Deep Dive — CREATE INDEX CONCURRENTLY and the lock modes it acquires, in depth.
Postgres Partitioning — partition-level locking and how it interacts with autovacuum on huge tables.

Database Replication: Strategies, Topologies, and What to Use When

Fri, 15 May 2026 00:00:00 GMT

Every serious database supports replication. The configuration options sound similar across systems — primary, replica, sync, async, logical, physical, quorum — but the choices you make under those names determine how your application behaves when things break. Replication is where database internals leak into application design: the difference between a write being “durable” and “durable enough to survive the primary going down” is the difference between losing one transaction and losing fifteen minutes of orders.

This post walks through how replication actually works in PostgreSQL, the topology and consistency choices you’ll have to make, and a decision tree for picking what to use when. We’ll stick to Postgres for code, but the patterns transfer to MySQL, MongoDB, and most other systems with minor adjustments.

This is Part 2 of the Distributed Systems series. Part 1 covered sharding, and assumed you understood replication well enough to design the migration path. If that section felt fast, this is where it slows down.

1. Why Replicate

Five reasons, in roughly the order they show up as a system grows.

High availability. The primary dies — the replica is promoted and serves traffic. Without a replica, you’re restoring from backup and apologizing to users for an hour.
Read scaling. Analytics dashboards, reporting jobs, and bulk exports run against replicas instead of competing with transactional traffic on the primary.
Geographic distribution. A replica in another region serves local reads at 5ms instead of 150ms across the ocean.
Disaster recovery. A replica in another data centre (or another cloud) means a fire, a misconfiguration, or a deletion in one location doesn’t end your business.
Major-version upgrades. Logical replication lets you stream from a Postgres 14 primary to a Postgres 17 replica, then cut over with seconds of downtime. The alternative is pg_upgrade with hours of downtime, and a much scarier rollback story.

Most teams reach for replication for HA first, then add read replicas as they grow, then geography when international users complain.

2. The Vocabulary

Before any setup, the words. A lot of replication confusion comes from these terms being used loosely.

Primary (or leader, master in older docs) — the node that accepts writes. Postgres has exactly one primary per cluster. The modern term is primary; master is being phased out across most projects.

Replica (or standby, follower, secondary) — a node that receives the primary’s changes and applies them locally. May or may not serve reads, depending on configuration.

Write-Ahead Log (WAL) — Postgres writes every change to the WAL before applying it to the heap. Replication, at its core, is “ship the WAL from primary to replica, and replay it there.”

Replication lag — how far behind the replica is, measured either in bytes (how much WAL hasn’t been applied yet) or time (how long ago the latest applied transaction happened on the primary). Lag is almost always non-zero in async setups.

Synchronous vs asynchronous — does the primary wait for the replica to acknowledge each commit before returning success to the client? Sync = wait (safer, slower). Async = don’t wait (faster, you may lose committed data on a primary crash).

Physical replication — replicas are byte-for-byte copies of the primary at the storage level. The WAL is shipped and replayed as-is. Same Postgres version on both sides required.

Logical replication — replicas receive row changes (INSERT, UPDATE, DELETE) decoded from the WAL, not raw byte changes. The replica can be a different Postgres major version, a different schema (with caveats), or even a different database system.

Topology — the shape of the replication relationships. Single-leader (one primary, many replicas), multi-leader (multiple primaries replicating to each other), or leaderless (no designated primary).

Quorum — a write or read is considered successful if it’s acknowledged by a majority (or some configured threshold) of nodes. Used in leaderless systems and in Postgres synchronous-with-N configurations.

Consistency model — what a client can observe across reads after a write. Strong / linearizable means “as if there’s one machine.” Eventual means “you’ll see it eventually.” More nuance in Section 6.

3. The Three Real Replication Strategies

There are technically four ways databases replicate. Three of them matter.

Statement-based (historical, mostly avoided)

The primary writes a log of every SQL statement; replicas replay them. MySQL’s original replication worked this way.

Problem: non-deterministic statements. INSERT INTO logs (id, created_at) VALUES (DEFAULT, NOW()) produces different results on every machine. Same with RAND(), UUID(), statements with LIMIT and no ORDER BY, and anything depending on table contents. You end up with replicas that silently diverge.

Postgres has never used statement-based replication. MySQL moved away from it as the default in 5.6+. If your database advertises “SQL replication,” investigate carefully — you’re almost certainly looking at this category.

Physical / WAL Streaming (Postgres default)

The primary streams its raw WAL records to the replica. The replica writes them to disk and replays them, producing a byte-for-byte identical copy of the primary’s storage.

# postgresql.conf on the primary
wal_level = replica            # WAL detail level (replica is enough for streaming)
max_wal_senders = 10           # how many concurrent replicas can stream
max_replication_slots = 10     # slots reserved for replicas

# On the new replica machine: clone the primary's data directory
pg_basebackup -h primary.internal -U replicator -D /var/lib/postgresql/data \
  -P -R -X stream

# pg_basebackup with -R creates standby.signal and configures
# primary_conninfo, so the replica starts up and immediately
# starts streaming.

When to use: the default for HA and read replicas. It’s fast, low-overhead, and supports streaming replication of every operation including DDL (Data Definition Language — ALTER TABLE, CREATE INDEX, etc.).

When not to use:

Replica needs to be a different Postgres major version → use logical instead.
Replica needs a different schema (some tables only, transformed columns) → use logical.
Replicating between Postgres and a different database system → not possible with physical.

Logical Replication (publication / subscription)

The primary’s WAL is decoded into a stream of row-level changes (row INSERTED into orders, column updated on users.id=42) which subscribers consume. Subscribers can be different Postgres versions, different schemas, or external systems.

-- on the primary (publisher)
ALTER SYSTEM SET wal_level = 'logical';   -- requires a restart
SELECT pg_reload_conf();

CREATE PUBLICATION app_pub FOR TABLE users, orders, payments;
-- or: CREATE PUBLICATION all_pub FOR ALL TABLES;

-- on the replica (subscriber)
CREATE SUBSCRIPTION app_sub
  CONNECTION 'host=primary.internal dbname=app user=replicator password=...'
  PUBLICATION app_pub;

Once the subscription is created, Postgres performs an initial table copy, then streams ongoing changes. Lag is visible in pg_stat_subscription.

When to use:

Major-version upgrades (PG 14 → PG 17 with no downtime).
Replicating a subset of tables (instead of the whole cluster).
Streaming changes to non-Postgres systems (Debezium reads logical replication slots and emits to Kafka).
Cross-database migrations.

When not to use:

HA — you want physical streaming for that.
Replicating DDL changes — logical replication doesn’t carry schema changes. You apply them on both sides manually or with tooling.
Replicating sequences — they’re not replicated. You manage them out-of-band during cutover.

Trigger-based (last-resort, application-level)

Triggers on every table capture changes into a queue, which an application process consumes. Slonik and Bucardo are the classic examples. Useful before logical replication existed (pre-Postgres 10); rarely the right answer today.

4. Sync Modes — The Latency / Durability Trade

Replication is a stream by default — the primary doesn’t wait for replicas to acknowledge anything. That’s fine for most workloads. When zero data loss matters (financial transactions, audit logs), you make it synchronous.

Postgres exposes this via synchronous_commit and synchronous_standby_names.

# postgresql.conf on the primary

# Which replicas count as synchronous, and how many must ack
synchronous_standby_names = 'ANY 2 (replica1, replica2, replica3)'

# What "ack" means at commit time
synchronous_commit = 'on'         # default

synchronous_commit has five levels, ordered from fastest-and-riskiest to safest-and-slowest:

Level	Primary returns success when…	You lose on a primary crash
`off`	WAL is in memory, not yet on disk	Last few transactions, even on local disk
`local`	WAL is fsync’d on the primary	Nothing locally; replicas may lag
`remote_write`	WAL is received by the replica’s memory	Tiny window if replica also crashes
`on` (default)	WAL is fsync’d on the replica’s disk	Effectively zero data loss on failover
`remote_apply`	WAL is applied on the replica (rows visible)	Zero — replica is fully caught up

The cost of moving up this ladder is latency. Every commit now includes a round-trip to the replica. A remote_apply setup with a replica in another region adds ~10–50ms to every write transaction. That’s fine for a billing system; catastrophic for a high-write event ingestion service.

The pragmatic answer most teams settle on:

Default to async (synchronous_commit = local) for typical workloads.
remote_write with one nearby replica when you want HA with low write latency.
on or remote_apply with a quorum (ANY 2 (...)) for genuinely-can’t-lose-this data, with multiple replicas so any one can ack without blocking.

Two-replica gotcha: synchronous_standby_names = 'FIRST 1 (r1, r2)' means “always wait for r1 specifically.” If r1 dies, all your writes block. Use ANY 1 (r1, r2) to allow either replica to ack — much better availability profile.

5. Topologies — Single-Leader, Multi-Leader, Leaderless

The shape of who-replicates-to-whom. Three families.

Single-Leader (95% of systems)

One primary accepts all writes. N replicas receive changes. Reads can come from either, with consistency caveats.

              ┌──────────┐
              │ Primary  │  ← all writes
              └────┬─────┘
            WAL ───┼───────┐
              ┌────▼─────┐ │ ┌──────────┐
              │ Replica1 │ └─► Replica2 │
              └──────────┘   └──────────┘
                  ▲              ▲
              reads          reads

Strengths: simple to reason about, no write conflicts, vanilla Postgres supports it natively, every transactional guarantee you’re used to still holds.

Weaknesses: the primary is a single point of failure for writes. Failover takes seconds to minutes (until a replica is promoted). Writes can’t be served from multiple regions simultaneously.

This is what your Postgres setup is using today unless you’ve actively configured otherwise. Use it until you have a specific reason not to.

Multi-Leader

Two or more primaries, each accepting writes, replicating to each other.

   ┌──────────┐         ┌──────────┐
   │ Primary  │◄───────►│ Primary  │
   │   us     │  bi-dir │   eu     │
   └──────────┘         └──────────┘
        ▲                    ▲
     writes               writes

Useful when writes need to be served close to users in multiple regions, or for active-active disaster recovery.

The hard part is conflict resolution. If us and eu both update users.email for the same row at roughly the same time, which one wins? Standard strategies:

Last-write-wins — keep the update with the latest timestamp. Simple, but you silently drop one of the writes.
Application-defined merge — application sees both versions and picks. Works for additive structures (sets, counters via CRDTs).
Region-pinned writes — each row is “owned” by one region, only that region writes to it. Avoids conflicts by design but loses some of the locality benefit.

Postgres in this space: core Postgres doesn’t support multi-leader replication. The mature options are extensions:

pgEdge (formerly EDB BDR) — commercial distributed Postgres with built-in multi-master and conflict resolution.
pgactive — AWS’s open-source fork of BDR, used inside RDS Multi-AZ for active-active.
Citus — primarily a sharding extension but supports some multi-master patterns through its coordinator.

The honest read: multi-leader Postgres is a niche, expert-only configuration. Most teams that think they need it actually need single-leader with regional read replicas — writes go cross-region to the primary, reads stay local.

Leaderless

No designated primary. Any node can accept writes and any node can serve reads. Writes are sent to multiple nodes in parallel; reads are also sent to multiple nodes and the most recent wins.

This is the Dynamo model — Cassandra, ScyllaDB, DynamoDB, Riak all use variants of it.

       ┌─────────┐  ┌─────────┐  ┌─────────┐
       │ Node A  │  │ Node B  │  │ Node C  │
       └─────────┘  └─────────┘  └─────────┘
            ▲            ▲            ▲
            └────────────┼────────────┘
                  any node accepts
                   reads and writes

The consistency knob is the R + W > N rule:

N = total replicas of each piece of data.
W = how many replicas must acknowledge a write.
R = how many replicas must respond to a read.

If R + W > N, the read and write quorums overlap by at least one node — so any read is guaranteed to see at least one node that has the latest write. That’s how leaderless systems give you “tunable consistency” without a primary.

Common configurations:
  N=3, W=2, R=2  →  strong-ish reads, can tolerate 1 node down
  N=3, W=3, R=1  →  fast reads, slower writes, can't tolerate write-side failures
  N=3, W=1, R=1  →  fastest possible, eventual consistency only

Conflict resolution still has to happen. Leaderless systems usually attach a version vector or timestamp to every write and use last-write-wins (sometimes vector clocks for more correctness).

Postgres in this space: Postgres does not natively do leaderless replication. The model is fundamentally incompatible with Postgres’s strict serializability and ACID transactions. If you need leaderless, you’re using Cassandra, DynamoDB, ScyllaDB, or similar — not Postgres.

Worth knowing about anyway because:

Production systems often pair Postgres for transactional data with a leaderless store for high-write event data (session state, feature flags, telemetry). Knowing both models lets you pick the right tool.
The R+W>N idea is portable — it shows up in custom consensus protocols, in CRDTs, and in some edge-caching strategies.

6. Consistency Models in Plain Language

Replication is when consistency models stop being academic and start being a bug in your application. Five worth knowing.

Strong consistency / linearizability. Reads always see the most recent committed write, as if there’s only one machine. Single-node Postgres gives you this for free. Across replicas, you get it only if you always read from the primary, or use synchronous replication with remote_apply, or use a system with consensus (Raft) baked in. You’d notice this when: you don’t notice it. Things just work.

Eventual consistency. Reads may return stale data, but if writes stop, all replicas eventually converge. Default for async replication. You’d notice this when: a user updates their profile, refreshes the page, and sees the old value because the page hit a replica that hasn’t caught up.

Read-your-writes consistency. A given user always sees their own writes immediately, even if other users might see stale data. Stronger than eventual, weaker than strong. You’d notice this when: you implement this poorly and a user’s “Save” button appears to do nothing on refresh — the most common cause of users repeatedly clicking Save and producing duplicates.

Monotonic reads. Once a user has seen a particular write, they will not see any earlier state of that data on a subsequent read, even if they hit a different replica. Prevents the “looks like it went backwards” experience. You’d notice this when: a user sees their order in the order list, refreshes, and it disappears because the second read hit a less-caught-up replica.

Bounded staleness. Reads may be stale, but never by more than X seconds or Y transactions. Useful when “freshness within 5 seconds” is acceptable but “minutes-old data” is not. You’d notice this when: you intentionally configure it to keep dashboards responsive without making them outright wrong.

Strong consistency is the safest default. Every weakening is something the application has to handle. Most production systems land on eventual consistency with read-your-writes and monotonic reads — the application is allowed to see slightly stale data from other users’ updates, but always sees its own writes correctly.

7. Replica Reads — The Rules

Reading from replicas is the most common way teams hurt themselves with replication. The lag is real, the application usually ignores it, and the bugs are subtle.

When stale data is fine:

Analytics dashboards (a 10-second-old aggregate is indistinguishable from a current one).
Recommendation systems (the model doesn’t update that often anyway).
Search results.
Public-facing read-only views (blog posts, product catalogs, documentation).

When stale data is not fine:

Anything immediately after a write by the same user (“did my comment post?”).
Authorization checks (“does this user still have admin access?”).
Inventory or financial decisions (“does this user have funds?”).
Any read that drives a subsequent write decision.

Four patterns for handling this:

Pattern 1: Read primary after write

The simplest correct approach. After a write, the same request reads from the primary, not a replica. The application explicitly tracks which connection to use.

# pseudocode
def update_profile(user_id, changes):
    with primary_db.transaction() as tx:
        tx.execute("UPDATE users SET ... WHERE id = ?", user_id, ...)
        # Read from primary for the rest of this request
        return tx.execute("SELECT * FROM users WHERE id = ?", user_id)

Strength: rock solid. Weakness: offloads no reads to replicas during write-paths.

Pattern 2: Sticky sessions

A user’s session is bound to a specific replica (or to the primary) for some window after a write. Used in larger systems with many replicas — keeps things consistent for that user without sending all their traffic to the primary.

Pattern 3: LSN tracking (read-your-writes)

After a write on the primary, you record the Log Sequence Number (LSN) — Postgres’s monotonically-increasing position in the WAL. On the next read, you pass the LSN; the replica delays the query (or routes to primary) until it has replayed up to that LSN.

-- after a write on the primary, record the LSN
SELECT pg_current_wal_lsn();          -- e.g., '0/3001028'

-- on the replica side, wait until that LSN has been applied
SELECT pg_wal_replay_wait('0/3001028', timeout => '500ms');
SELECT ...                            -- now safe to query

This is the closest thing to “always see your own writes” without sending all reads to the primary. It’s slightly more code but cleaner than sticky sessions.

Pattern 4: Read-only replica with a freshness contract

Document explicitly which workloads are allowed to use the replica (analytics, exports, reports) and which must use the primary (anything user-facing immediately after a write). Enforce in code via separate connection pools — primary_pool and replica_pool — and require any new query to be reviewed against the contract.

This is what most production teams settle on. It’s not magic; it’s just “be deliberate.”

8. Postgres Replication, End to End

Putting it together. A canonical HA setup looks like this:

                ┌──────────────────────────┐
                │     Application tier     │
                └──────────────┬───────────┘
                           pgbouncer
                ┌──────────────┴───────────┐
                │  Patroni-managed Postgres │
                │                          │
                │   ┌──────────────────┐   │
                │   │  Primary         │   │
                │   │  (accepts writes)│   │
                │   └────────┬─────────┘   │
                │            │ WAL stream  │
                │   ┌────────▼─────────┐   │
                │   │  Standby #1      │   │
                │   │  sync replica    │   │
                │   └──────────────────┘   │
                │   ┌────────▼─────────┐   │
                │   │  Standby #2      │   │
                │   │  async replica   │   │
                │   │  serves reads    │   │
                │   └──────────────────┘   │
                └──────────────────────────┘

The pieces:

Patroni (or Stolon) — automated failover manager. Watches the primary’s health, promotes a standby when needed, reconfigures pgbouncer or DNS so the application reconnects.
pgbouncer — connection pooler. Important not just for connection economy but because it handles the brief disconnect during failover gracefully.
Sync replica (#1) — close-network replica with synchronous_commit = on. Guarantees zero data loss on failover.
Async replica (#2) — further from the primary, lag is allowed. Handles read traffic, also a candidate for promotion in cross-region disaster scenarios.

Monitoring lag is non-negotiable. On the primary:

SELECT
  application_name,
  state,
  pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)   AS pending_bytes,
  pg_wal_lsn_diff(sent_lsn, write_lsn)              AS write_lag_bytes,
  pg_wal_lsn_diff(write_lsn, flush_lsn)             AS flush_lag_bytes,
  pg_wal_lsn_diff(flush_lsn, replay_lsn)            AS replay_lag_bytes,
  write_lag,
  flush_lag,
  replay_lag
FROM pg_stat_replication;

Alert on replay_lag exceeding your SLA. A common threshold: anything over 30 seconds on a standby that serves reads triggers a page; anything over 5 minutes on any standby triggers an investigation.

Failover testing is the difference between a system that fails over and a system that advertises it can fail over. The expected discipline:

Do a controlled failover every quarter.
Time it. Verify the application reconnects.
Verify replica lag was zero at the moment of promotion (otherwise you’ve lost data).
Document the recovery time and any rough edges.

Teams that don’t do this discover during the actual outage that their Patroni config was subtly wrong, or that pgbouncer didn’t refresh DNS, or that one application server didn’t reconnect cleanly.

9. Decision Tree

What you need	Use
HA against primary loss, single region	Single-leader, async streaming, Patroni for failover
HA with zero-data-loss tolerance	Single-leader with `synchronous_commit = on`, `ANY N` quorum across ≥ 2 nearby replicas
Read scaling for reports / analytics	Async streaming replicas, route OLAP traffic via separate connection pool
Major-version upgrade with seconds of downtime	Logical replication PG-old → PG-new, application cutover
Replicate a subset of tables to another system	Logical replication + Debezium / CDC
Cross-region active-active writes	Multi-leader (pgEdge / pgactive) — or rethink with regional read replicas + cross-region primary
High-write event data, no transactions needed	Leaderless DB (Cassandra / DynamoDB) — not Postgres
Strict read-your-writes after a user action	Read from primary, or LSN tracking via `pg_wal_replay_wait`
Globally-consistent reporting	Read primary only, or use synchronous quorum + read from sync replica

10. Common Mistakes Checklist

11. Closing

Replication is one of those topics where the building blocks are straightforward but every combination has a real-world implication. Single-leader async streaming is the right answer 95% of the time. Logical replication is the right answer for version upgrades and CDC. Multi-leader and leaderless are real but rare, and worth understanding mostly so you can recognize when someone is reaching for them prematurely.

The skill isn’t memorizing which knob does what. It’s developing the instinct for “where would lag show up here?” before you ship the feature — because every replicated database has lag, and every async system can lose its most recent writes, and the application either accounts for that or doesn’t.

Next in the Distributed Systems series: consensus protocols. When you need stronger guarantees than single-leader async can give you — distributed transactions that actually commit atomically, or leader election that doesn’t split-brain — you reach for Raft or Paxos. We’ll walk through what they actually do, why they’re hard, and where you find them in production (etcd, Consul, CockroachDB, every modern distributed system you can name).

Sharding Postgres: When One Database Stops Being Enough — Part 1 of this series; the architectural step that often precedes thinking about replication topology.
Postgres Partitioning: Strategies, Advantages, and Pitfalls — how partitioning interacts with WAL streaming and logical replication.
ACID, Read Phenomena, and Isolation Levels — the local-transaction guarantees that replication is trying (or not trying) to preserve.
PostgreSQL Indexing Deep Dive — relevant when designing replica-only read workloads and partial-index reads against followers.

Postgres Partitioning: Strategies, Real Advantages, and Pitfalls

Wed, 13 May 2026 00:00:00 GMT

The events table is now 4 TB. Queries that used to be milliseconds are seconds. autovacuum (the background process that reclaims space from dead row versions left behind by UPDATE and DELETE) is permanently behind. The primary-key index doesn’t fit in shared_buffers (Postgres’s in-memory cache for table and index pages, sized at startup). The nightly DELETE FROM events WHERE created_at < ... runs into morning traffic and barely reclaims any disk space anyway.

You haven’t outgrown one machine. The CPU is fine, the RAM is fine, the disk has headroom. The table has outgrown the assumptions Postgres makes about a single relation. This is the problem partitioning is for.

This post walks through how partitioning works in Postgres, the advantages it actually delivers (some non-obvious), and the pitfalls that bite teams in production. When you’re done, you should know whether your workload needs partitioning, which strategy to pick, and which mistakes to plan around.

This is Part 7 of the Database Internals series. Earlier parts covered indexing, B-trees, and EXPLAIN — all relevant context here.

1. What Partitioning Actually Is

Partitioning is horizontal decomposition of one logical table into multiple physical tables that live in the same Postgres cluster. The planner treats the parent table as the query target; behind it, rows are routed to child tables (partitions) by a partition key. Same cluster, same Write-Ahead Log (WAL), same shared_buffers, same connection pool — none of the physical resources are duplicated. What changes is the size of the things the planner operates on.

Since Postgres 10, this is declarative partitioning — a first-class feature. The planner natively understands partition bounds, can prune (skip) partitions that can’t match a query, and can do partition-wise joins and aggregates with the right configuration.

A quick refresher on the two terms that show up repeatedly throughout this post. shared_buffers is Postgres’s in-memory cache for table and index pages, sized at startup (typically ~25% of system RAM) and never grows during runtime. Every read first checks shared_buffers; a miss falls through to the OS page cache and then to disk. autovacuum is the background process that reclaims space from dead row versions — the leftovers from every UPDATE and DELETE, which Postgres can’t free in place because of its MVCC (Multi-Version Concurrency Control) model. When autovacuum can’t keep up, tables bloat (grow on disk faster than their live row count would suggest) and query plans degrade.

2. The Three Partitioning Strategies

RANGE — each partition owns a contiguous range of values. The textbook case is time-series.

CREATE TABLE events (
    id          bigserial,
    user_id     bigint      NOT NULL,
    payload     jsonb,
    created_at  timestamptz NOT NULL
) PARTITION BY RANGE (created_at);

CREATE TABLE events_2026_01 PARTITION OF events
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE events_2026_02 PARTITION OF events
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

LIST — each partition owns an explicit set of discrete values. Useful for low-cardinality categorical dimensions: region, tenant tier, country.

CREATE TABLE customers (id bigserial, region text, ...)
    PARTITION BY LIST (region);

CREATE TABLE customers_us   PARTITION OF customers FOR VALUES IN ('us-east','us-west');
CREATE TABLE customers_eu   PARTITION OF customers FOR VALUES IN ('eu-west','eu-central');
CREATE TABLE customers_apac PARTITION OF customers FOR VALUES IN ('ap-south','ap-northeast');

HASH — Postgres hashes the key and assigns rows by hash(key) % N. Used when the goal is even distribution, not range pruning. Equality lookups on the key still prune to one partition (the planner computes the hash); queries without the key fan out to all partitions.

CREATE TABLE messages (id bigserial, channel_id bigint, body text, ...)
    PARTITION BY HASH (channel_id);

CREATE TABLE messages_p0 PARTITION OF messages FOR VALUES WITH (modulus 8, remainder 0);
-- ... through p7

You can also nest these — RANGE-by-time at the outer level with HASH-by-tenant within each month, for example. This is sub-partitioning, and it’s powerful but easy to overuse (more on that in the pitfalls section).

3. Partition Pruning — The Mechanism That Makes It Worth Anything

All three strategies do the same physical thing: split one logical table into smaller child tables. But splitting alone doesn’t make a single query faster — if Postgres scans every child table looking for matching rows, you’ve turned one big scan into many smaller scans and gained nothing. The strategies pay off only when the planner can skip the partitions that can’t contain matching rows for a given query. That skipping is called partition pruning, and it’s the connection between “I defined some partitions” and “my queries got faster.”

With twelve monthly partitions, a one-month query touches one partition, not all twelve. Pruning happens in two phases:

Plan-time pruning inspects literal WHERE predicates and rules out partitions whose bounds can’t satisfy them.
Execution-time pruning kicks in when the partition key arrives at runtime — from a JOIN, a parameterized statement, or a subquery. Look for Subplans Removed: N in EXPLAIN ANALYZE.

Pruning is fragile in exactly one way: it requires the column to appear plainly in the predicate. Wrapping it in a function defeats it.

-- ❌ no pruning — function on partition column blocks it
WHERE date_trunc('month', created_at) = '2026-02-01'

-- ✅ pruning works
WHERE created_at >= '2026-02-01' AND created_at < '2026-03-01'

The single most common partitioning bug: writing the query in the form that defeats pruning and not noticing because the result is still correct, just slow.

4. Local Indexes — and the Absence of Global Indexes

The single most Postgres-specific detail in the partitioning story, and it routinely surprises people coming from Oracle or SQL Server: Postgres has no global indexes. Every index on a partitioned table is implicitly local — it exists per-partition. CREATE INDEX ON events (user_id) creates that index on every partition individually, and the parent holds metadata tying them together as one “partitioned index.”

Consequences:

UNIQUE constraints can only be enforced if the partition key is part of the uniqueness columns. A globally unique email on a hash-partitioned users table isn’t possible without (email, partition_key) — usually not what you want.
Primary keys are enforced per-partition. A bigserial id won’t collide because the sequence is global, but the uniqueness check is local.
Queries without the partition key in the predicate still use per-partition indexes — but the planner scans them all in turn.

Workarounds exist (managed-uniqueness extensions, externally-issued IDs), but the right move is usually to design the schema so global uniqueness isn’t needed, or so the partition key is implicitly the prefix of the unique key.

5. The Real Advantages (with concrete examples)

This is where partitioning earns its place. The standard one-line summary — “it makes large tables manageable” — undersells it. Here’s what you actually get.

5.1 The active working set fits in `shared_buffers`

Postgres caches heap and index pages in shared_buffers. If the hot data plus its indexes are larger than the buffer pool, every random lookup is a cache miss and a disk read. Disk reads are roughly 1000× slower than buffer hits.

Concrete example. A 4 TB unpartitioned events table with a 600 GB primary-key B-tree on a 128 GB box. The B-tree alone is 5× the buffer pool, so every WHERE id = X query that doesn’t hit a recently-touched page is a cold disk read. Tail latency is dominated by these misses.

Partition the same table into 48 monthly partitions. A query filtering by created_at >= '2026-04-01' now touches a single ~12 GB partition with a ~12 GB B-tree. The active partition fits in the buffer pool comfortably. The query plan is identical — same scan, same join, same filter — but every page read is now a buffer hit instead of a disk miss. Latency drops by an order of magnitude.

Partitioning’s quiet superpower: it doesn’t make the data smaller, it makes the active working set smaller.

5.2 Retention becomes O(1)

Dropping a partition is a metadata operation. DROP TABLE events_2026_01 on a 200 GB partition takes milliseconds and immediately returns the storage to the OS.

Compare with the equivalent on an unpartitioned table: DELETE FROM events WHERE created_at < '2026-02-01' is a multi-hour scan that rewrites every page touched (Postgres marks deleted rows as dead tuples instead of removing them in place — the consequence of MVCC). After the DELETE finishes, you need autovacuum to reclaim those dead tuples, another multi-hour operation. And even after vacuum, the table’s physical size doesn’t shrink, because Postgres can’t return free space mid-file to the OS without VACUUM FULL, which takes an exclusive lock and blocks reads.

The retention story is the single biggest practical win of partitioning. Append-heavy tables — events, audit logs, activity streams, metrics — all have a natural retention horizon. Partition by time, drop old partitions on a cron, and the table never actually grows unboundedly.

5.3 Operational isolation for DDL

ALTER TABLE on a 4 TB unpartitioned table takes a heavy lock and blocks every other operation on that table. On 48 partitions, you can apply the same schema change one partition at a time, pace the rollout over hours or nights, and the live partitions keep serving queries.

Concrete example. Adding a column with a non-null default to an unpartitioned 4 TB table rewrites every row — hours of downtime. On 48 partitions, you can do it partition by partition over a few off-hours windows. New writes go to the partitions that already have the column; the migration completes lazily without ever blocking traffic for more than a few seconds per partition.

5.4 Autovacuum parallelism

Autovacuum runs a configurable number of workers (autovacuum_max_workers, default 3). Each worker handles one relation at a time. On one giant unpartitioned table, three workers can only contend for the same table — there’s nothing for the other two to do. On 48 partitions, three workers can be vacuuming three partitions concurrently. The total throughput of vacuum work is now bounded by your autovacuum_max_workers and per-worker cost limits, not by the size of one table.

Partitioning is the cheapest way to give autovacuum useful parallelism on a write-heavy workload.

5.5 Per-partition statistics, better query plans

Postgres maintains statistics per table — row counts, value distributions, distinct counts, most-common-values lists — that the planner uses to estimate query costs. On one giant table, these statistics are averages across the entire dataset. On a partitioned table, each partition has its own statistics tuned to that partition’s actual data.

Example. The events table has old, stable partitions (Jan–Feb) and a current churning partition (live data). The query distributions are very different — the live partition has more recent timestamps clustered tightly, while older partitions are evenly distributed. Per-partition stats let the planner pick a different (better) plan for each range. On one giant table, the stats are blurred together and the planner makes one mediocre estimate for everything.

5.6 Index-only scans for old data

Every heap page has a corresponding bit in Postgres’s visibility map — a compact structure that tracks whether all rows on the page are visible to every active transaction. When an index scan touches a page marked all-visible, Postgres can skip the heap fetch entirely and return data directly from the index entries. This is an index-only scan — the fastest possible read path.

For a page to be marked all-visible, it must have had no recent modifications and autovacuum must have run since. On a churning unpartitioned table, most pages are never marked all-visible for long because writes keep touching them. On a partitioned table, old read-mostly partitions stabilize at all-visible permanently. Queries against historical data become index-only scans automatically — Heap Fetches: 0 in EXPLAIN ANALYZE, which is the fastest read path Postgres has.

5.7 Tiered storage via tablespaces

Postgres lets you create tablespaces — named locations on disk — and assign tables to them. Partitions can live in different tablespaces:

CREATE TABLESPACE cold_storage LOCATION '/mnt/slow_ssd';

ALTER TABLE events_2024_01 SET TABLESPACE cold_storage;
ALTER TABLE events_2024_02 SET TABLESPACE cold_storage;

Old, rarely-accessed partitions move to slower (cheaper) storage — slower SSD, spinning rust, even network-attached storage. Hot partitions stay on fast local NVMe. The table is still one logical thing; queries against current data are fast, queries against historical data are slower but possible.

This is the same pattern that data warehouses use natively. Partitioning gives it to you in operational Postgres without leaving the database.

5.8 `CREATE INDEX CONCURRENTLY` scales linearly

Building an index on a large table normally takes an exclusive lock. CREATE INDEX CONCURRENTLY avoids that but does two passes over the table and is several times slower than the locking variant. On a 4 TB unpartitioned table, a concurrent build can take a day.

On 48 partitions, you can build the index on each partition independently, in parallel if you want, and each build is a fraction of the size. The total wall-clock time can be hours instead of days — and if a build fails on one partition, you’ve only lost one partition’s worth of work, not the entire index.

5.9 Capacity planning is visible

A single 4 TB table that’s growing… how fast? You can plot the size over time, but the past-versus-current rate is mushed together.

With monthly partitions, each partition’s size is a clear data point. events_2026_01 is 180 GB; events_2026_02 is 230 GB; events_2026_03 is 290 GB. Now you have a growth curve, per period, that’s trivial to query (SELECT pg_relation_size('events_2026_03')). Capacity planning is a SQL query away.

5.10 Selective backups and archival

pg_dump on a partitioned table can target specific partitions: pg_dump -t events_2024_* dumps only old partitions, perhaps for archival to S3 as Parquet files. You don’t need to dump the whole table to back up the inactive parts.

Many teams build a pipeline: monthly cron job archives the about-to-be-dropped partition to S3 as Parquet, dumps a SQL backup as a sanity copy, then drops the partition. Storage costs go down by an order of magnitude; reads against archived data are still possible (just slower, via S3 Select or a query engine like DuckDB or Athena).

5.11 Lock contention is localized

A long-running VACUUM FULL or REINDEX on one partition does not block queries on the other partitions. A locking schema change on events_2024_01 doesn’t stop the application from inserting into events_2026_05. Operational events that used to take down “the events table” now take down “one slice of the events table” — much smaller blast radius.

6. Internals: What Postgres Does Differently With Partitioned Tables

Behind these advantages are real changes to how Postgres processes queries.

Query planner cost

Planning time grows with the number of partitions. With 12 partitions and good pruning, planning overhead is negligible. With 5,000 partitions and a query that can’t prune, the planner can spend more time deciding than the executor spends running. Postgres 12+ improved per-partition planning cost dramatically (from milliseconds to microseconds), but the rule still holds: prefer hundreds of partitions over thousands. The sweet spot is usually 30–500 partitions per table.

Partition-wise joins and aggregates

Set enable_partitionwise_join = on and enable_partitionwise_aggregate = on (both off by default because they increase planning time) and the planner can join matching partitions of two co-partitioned tables in parallel, and aggregate per-partition before combining. For analytical workloads on partitioned fact tables, this can be the difference between a 30-second query and a 2-second one.

MVCC, dead tuples, and `xid` wraparound

MVCC keeps multiple versions of each row visible to different transactions. Every UPDATE writes a new tuple; the old becomes dead, waiting for vacuum. On a 4 TB unpartitioned table with high churn, the dead-tuple bookkeeping is one giant problem; on 48 partitions it’s 48 smaller problems autovacuum workers handle in parallel.

Postgres’s transaction IDs are 32-bit, and VACUUM FREEZE exists to mark old tuples visible to all future transactions so their xids can be safely reused. On a giant unpartitioned table, freezing is a recurring multi-hour I/O storm. On partitions, freezing is incremental — old read-mostly partitions get frozen once and then never need it again.

What partitioning shares (and doesn’t multiply)

It’s worth being explicit about what partitioning does not duplicate, because that’s what determines whether you need to go further:

One WAL. Every insert across every partition writes to the same Write-Ahead Log. Partitioning doesn’t add write throughput beyond what one machine’s WAL can sustain.
One shared_buffers. All partitions compete for the same buffer pool.
One max_connections. Same connection pool, same pgbouncer, same backend processes.
One replication stream. Logical or physical, the WAL is one stream that one replica replays.
One CPU pool. Partition-wise parallelism helps, but you’re still bounded by the machine’s cores.

Hit any of these ceilings and you’ve left the world of partitioning. That’s the next post in the Distributed Systems series.

7. Pitfalls and Things to Watch For

The advantages above are real, but they assume you set things up correctly. Here’s the failure-mode catalogue.

7.1 Wrong partition key

You hash-partitioned messages by id, but every query filters by channel_id. Every query now touches every partition (no pruning is possible without the key in the predicate), planning time is several times slower than the unpartitioned baseline, and you’ve made things worse.

The fix is a full re-partition — a migration on the same scale as a shard rebalance: double-write to the new partitioning scheme, backfill, verify, cut over. Pick the key like it’s permanent, because in practice it is.

A useful rule: look at the top 20 queries against the table. The column that appears in 90%+ of WHERE clauses is your partition key candidate. If no single column dominates, partitioning by any of them is going to hurt the queries that filter on the others.

7.2 Partition explosion

The daily-partition cron got stuck after a server move; 90 days of rows are stacked in events_default (the catchall partition for rows that don’t match any defined partition). The planner now has to consider hundreds of partition bounds for every query, and the catchall partition has no useful indexes for the workload.

Worse: someone set up sub-partitioning by tenant inside RANGE-by-time, without bounding the tenant count. You now have 200,000 partitions, query planning takes 30 seconds, and autovacuum is permanently rotating through them.

Watch the partition count actively. A monitoring alert at “more than 1,000 partitions on any single relation” catches this class of bug before it becomes an outage.

7.3 The default partition is a trap

CREATE TABLE events_default PARTITION OF events DEFAULT;

The default partition catches rows that don’t match any explicitly-bounded partition. It seems like a safety net, but in practice:

It hides bugs. Your cron didn’t create April’s partition? Rows accumulate in events_default instead of failing loudly. You notice when queries get slow weeks later.
It defeats pruning for every query. As long as events_default exists, the planner can’t be certain a query doesn’t need to scan it, so it scans it for every query.
Removing it later requires moving the rows out — and you can’t add a new bounded partition that overlaps with rows already in the default.

The pattern most teams converge on: don’t create a default partition. Let inserts fail when partitions are missing. The failure is loud, immediate, and trivially fixable; the alternative is silent rot.

7.4 Sub-partitioning is rarely worth it

The temptation: range-partition by month, then hash-partition each month by tenant. Two levels of pruning, smaller chunks. In practice:

The partition count multiplies (12 months × 32 tenants = 384 partitions).
Operational complexity multiplies (creating new top-level partitions now creates 32 sub-partitions).
The pruning benefit is marginal unless both keys are routinely in the query predicate.

If a single level of partitioning has gotten you to chunks of reasonable size (1–50 GB per partition), don’t sub-partition. The exception is genuinely two-dimensional workloads — usually time + tenant — where queries always filter by both.

7.5 `ATTACH` and `DETACH` partition locks

ATTACH PARTITION briefly takes an ACCESS EXCLUSIVE lock on the parent table — usually fast but it blocks every read in the meantime. DETACH PARTITION historically did the same; Postgres 14+ added DETACH PARTITION ... CONCURRENTLY which avoids the heavy lock but takes longer (two phases, validation step in between).

If you’re on PG 14+, always use CONCURRENTLY for detaches. If you’re on an older version, schedule attach/detach operations during low-traffic windows.

7.6 Foreign keys to a partitioned table

Foreign keys on a partitioned table (the partitioned table has the FK) work fine. Foreign keys to a partitioned table (something else FKs into it) were only added in Postgres 12, and even then have some restrictions — for example, the referenced columns must include the partition key.

If you’re targeting cross-version compatibility or you’re on PG 11 or earlier, design around this. A common pattern: store the FK reference but don’t declare it; enforce in application code.

7.7 Hash partitioning hides scatter-gather

Queries that supply the hash partition key prune to one partition. Queries that don’t supply the key — analytics, joins on a different column, full scans — run on every partition. Eight partitions = eight sequential scans, eight index scans, eight result merges.

This is fine for occasional queries. It’s catastrophic if it’s most of your workload. Always check EXPLAIN ANALYZE for Append (Subplans Removed: N) — if N = 0 regularly, you’re hash-partitioning data without using the key, and the partitioning is hurting you.

7.8 Function-on-column defeats pruning

The single most common partitioning bug. The partition key column has to appear plainly in the WHERE clause for the planner to use the partition bounds.

-- ❌ no pruning, scans every partition
WHERE date_trunc('month', created_at) = '2026-02-01'
WHERE created_at::date = '2026-02-15'

-- ✅ pruning works
WHERE created_at >= '2026-02-01' AND created_at < '2026-03-01'
WHERE created_at >= '2026-02-15' AND created_at < '2026-02-16'

Build a habit of writing predicates against the raw column, and use EXPLAIN ANALYZE to verify partitions are being skipped on any new query against a partitioned table.

7.9 Execution-time pruning is fragile

Plan-time pruning handles literal predicates. Execution-time pruning handles values that arrive at runtime (a parameterized query, a JOIN key, a subquery result). It works — sometimes. Some query shapes never trigger runtime pruning even when the values are clearly available. The query then scans every partition with the per-partition index, which is less efficient than the unpartitioned baseline.

Look for Subplans Removed: N in EXPLAIN ANALYZE. If N = 0 for a query where the partition key is determined at runtime, you’ve hit a planner limitation. Workarounds include rewriting as a CTE that fixes the value first, or splitting into two queries.

7.10 Sequences and uniqueness

bigserial columns work fine across partitions — the sequence is global to the cluster, so IDs don’t collide. But: the uniqueness check is local to each partition. If you somehow inserted a row with id = 42 on events_2026_01 and id = 42 on events_2026_02, Postgres would not catch it, because the unique index that enforces the PK only exists per-partition.

In practice this is fine — bigserial never reuses values — but it’s worth knowing. If you have any path that allows specifying the id directly (a backfill, a restore, an INSERT ... SELECT), you can produce duplicate IDs across partitions and not get an error.

7.11 Statistics aren’t always up to date

ANALYZE collects statistics. On a partitioned table, you can analyze the parent (which combines stats from all partitions) or the individual partitions. The default autovacuum_analyze_* thresholds are evaluated per-partition, which is good — but the parent table’s stats only update when you analyze it explicitly.

If you run queries that aggregate across partitions, the planner uses parent-level stats. Schedule a periodic ANALYZE events on the parent — once a day is usually fine.

7.12 Forgetting to create future partitions

You partition by month. You manually create partitions through December. December rolls around. January arrives. Inserts to January start failing because no partition covers 2027-01-01.

Either use pg_partman to create partitions automatically on a schedule, or wire up your own cron job that creates the next several months ahead. Set up an alert that fires when the latest partition’s upper bound is less than 30 days in the future.

7.13 EXPLAIN output is much larger

EXPLAIN ANALYZE on a partitioned table with 48 partitions and no pruning produces a plan with 48 Seq Scan or Index Scan nodes — pages of output to scroll through. This is normal but disorienting at first. Get used to looking for Append (Subplans Removed: N) at the top and reading the surviving sub-plans.

7.14 Version dependencies

Partitioning capabilities have evolved fast. A short table:

PG 10 — declarative partitioning introduced, RANGE and LIST only.
PG 11 — HASH partitioning, FK to partitioned tables, partition-wise joins (off by default).
PG 12 — partition pruning improvements, FK to partitioned tables fully supported.
PG 13 — logical replication for partitioned tables.
PG 14 — DETACH PARTITION ... CONCURRENTLY, partition-wise stats improvements.
PG 15 — MERGE works on partitioned tables.

If you’re on an older version, some of the advantages above don’t apply yet. Check your PG version before promising any specific behaviour.

7.15 `pg_dump` dumps each partition separately

pg_dump of a partitioned table emits separate CREATE TABLE and COPY statements per partition. Restore order matters: parents before children, or use --section=pre-data and --section=data separately. Most of the time this Just Works, but it can surprise teams running custom backup/restore scripts.

8. The Ecosystem

Two tools come up everywhere in production.

pg_partman — the de-facto partition-management extension. Creates partitions ahead of time, drops old ones, optionally indexes new ones. If you’re range-partitioning by time, you almost certainly want pg_partman. The alternative is a homegrown cron job that gets the timezone wrong on a Sunday at 3 AM.

TimescaleDB — a Postgres extension purpose-built for time-series. Hypertables auto-partition by time (and optionally by space); also gives you continuous aggregates, compression, retention policies, and time-bucket query helpers. For unambiguously time-series workloads (metrics, IoT sensor data, financial ticks), it beats hand-rolled partitioning on ergonomics. For mixed OLTP (transactional workloads like e-commerce or SaaS apps) where time is one dimension among many, plain partitioning + pg_partman is usually better.

9. When Partitioning Isn’t Enough

Partitioning solves problems about size per relation. The dataset is bigger than any single relation can comfortably hold; partitioning splits it into chunks that the planner, the buffer pool, autovacuum, and operational tooling can all handle.

It does not solve problems about per-cluster physical resources. When you’ve maxed out:

One machine’s CPU cores
One machine’s RAM
One disk’s IOPS
One WAL’s write throughput
One replication stream’s apply rate
One max_connections ceiling

…partitioning won’t help. Every partition still shares those resources. That’s when you start thinking about sharding — splitting the dataset across multiple Postgres instances. Sharding is a completely different category of operation, with its own architectural questions and its own catalogue of failure modes. It’s covered in the next post in this series’ sibling: Sharding Postgres, the first post in the Distributed Systems series.

The honest framing: most teams who think they need sharding actually need partitioning, more RAM, better query plans, or a vertical decomposition of their database (splitting billing, analytics, notifications into their own Postgres instances). Real horizontal sharding is for the small number of systems that genuinely outgrow one machine in write throughput or total data size — and even then, often after a year or two of partitioning and vertical decomposition have bought them runway.

10. The Takeaway

The single highest-leverage operation a Postgres team can perform on a large table is range-partitioning by time, if the workload has any natural retention horizon. The benefits compound: smaller working set, cheaper retention, autovacuum parallelism, index-only scans on old data, operational isolation. The cost — partition creation cron jobs, slightly more complex EXPLAIN output, a few rules to follow when writing queries — is a small fraction of what you get back.

The pitfalls above are real, but they’re all knowable up front. Pick the partition key like it’s permanent, don’t create a default partition, watch the partition count, and write your WHERE clauses without functions on the partition column. Get those right and partitioning is one of the few Postgres features where the upside is huge and the downside is small.

The architecture that survives is the one you can still understand at 3 AM with a single psql window open. Partitioning keeps you in that world for much longer than people expect.

How Databases Actually Store and Find Your Data — pages, heaps, and TIDs that partitioning splits at the physical level.
PostgreSQL Indexing Deep Dive — local indexes per partition, and how partition pruning interacts with index choice.
Reading PostgreSQL EXPLAIN Output — recognising Append, Subplans Removed, and partition-wise joins in real plans.
Sharding Postgres: When One Database Stops Being Enough — the next step when partitioning isn’t enough.
Database Replication: What to Use When — how partitions interact with WAL streaming and logical replication.

Sharding Postgres: When One Database Stops Being Enough

Wed, 13 May 2026 00:00:00 GMT

The hot tables are partitioned. The old data is on cheap storage. billing, analytics, and notifications each live on their own Postgres instances now. Read replicas absorb the analytical load. Autovacuum is keeping up. Every single-node tuning lever has been pulled.

And still — the central tenant data is 12 TB, the write rate exceeds what one disk’s Write-Ahead Log (WAL) can sustain, one machine’s max_connections ceiling caps how many application servers can talk to the database, and the replica is permanently a few minutes behind during peak traffic. You’ve genuinely outgrown a single Postgres cluster.

This is when sharding stops being theoretical and starts being the next thing on the roadmap. Sharding is the horizontal distribution of one dataset across multiple Postgres servers. Each shard is a complete Postgres instance, with its own WAL, its own shared_buffers (Postgres’s in-memory page cache), its own backends, its own backups. They don’t know about each other. A routing layer in the application or in a proxy decides which shard owns which row.

The honest framing — and the thing every public engineering post about sharding eventually says: you are trading one big problem (a database that can’t keep up) for a dozen smaller problems (routing, fan-out, hot shards, cross-shard joins, rebalancing, schema migrations, distributed transactions). The smaller problems are tractable. They don’t disappear because you picked the right tool.

This post is the first in the Distributed Systems series. It assumes you’ve already exhausted what one Postgres can do — partitioned the hot tables, decomposed independent domains into their own instances, added read replicas. (If you haven’t, start with the sibling post: Postgres Partitioning.) From here on, we’re talking about the operational regime that begins when one database stops being enough.

1. The Three Architectural Questions

Every sharded system answers these. The answers determine everything else.

What’s the partition key? The column you hash or range-distribute on. It separates the single-shard queries (cheap, scale linearly with shards added) from the scatter-gather queries (touch every shard, don’t scale). Pick badly and the design fights you for years.
How do clients find the right shard? Three approaches, often combined: a directory service (a metadata DB that maps tenants/users to shards), deterministic hashing (compute the shard from the key with no lookup), or a SQL-aware routing proxy.
How do you reshard? Eventually a shard gets too hot, a region is added, or the original key was wrong. Online rebalancing is the hardest engineering problem in this space.

2. Shard Keys — The Hardest Decision

For a multi-tenant B2B SaaS, the answer is almost always tenant_id. Every tenant’s data lives on one shard. Queries already filter by tenant. No single tenant is large enough to need their data spread across machines. This is the architecture Notion, Figma, Slack (on Vitess), and most B2B systems converge on.

For a B2C product, user_id works for user-centric reads (profile, feed, messages) but breaks down for global features (timelines of accounts you follow, group chats, marketplace browses). Large B2C systems usually shard by user and maintain denormalized cross-cutting structures — fanout tables, materialized timelines — that absorb the scatter-gather cost up front.

For geo-distributed products, region works as a coarse shard key (data sovereignty, latency), with finer-grained tenant or user sharding nested inside each region.

The wrong shard keys are obvious in retrospect:

Timestamps for write-heavy workloads. Every write goes to the newest shard. The hot-shard problem in its purest form.
Auto-incrementing IDs. Same issue when range-sharded; hashing fixes the imbalance but you still scatter-gather for almost everything.
Low-cardinality columns (country code, plan tier). Some shards end up 10× the size of others, and you can’t rebalance.

A useful rule: look at your top 20 query predicates. If 90%+ filter by a particular column, that’s your shard key. If no single column dominates, you don’t have a sharding problem — you have a data-model problem.

3. Consistent Hashing and the Logical/Physical Split

A naive hash(key) % N design hits a wall when N changes. Suppose today N = 3, so tenant 42 resolves to hash(42) % 3 = 0 — shard 0. Tomorrow you add a fourth machine: hash(42) % 4 = 2 — shard 2. The same tenant ID now points to a different shard. Almost every tenant gets a new assignment, and every row in the system has to be physically copied to its new home. That’s called re-keying: the routing function changed, so the answer to “which shard does this row belong to?” changed for nearly everyone.

Consistent hashing fixes this by mapping both keys and nodes onto a circular hash space. Each key is owned by the next node clockwise from it. When you add a node, only the keys in the arc between the new node and its clockwise neighbour need to move — the rest stay where they were.

In a 3-node ring, adding a 4th node moves roughly 1/4 of the keys. With hash(key) % N, adding a 4th node would have moved almost all of them. That’s the whole point.

But most production systems use a simpler trick on top: decouple logical and physical shards.

Pick a logical shard count larger than you’ll ever need (256, 1024, 4096).
Distribute logical shards across a smaller number of physical instances.
Keep a directory tenant_id → logical_shard → physical_instance.
When a machine gets hot, move a few logical shards. No re-keying.

To be precise about what does and doesn’t move: the data still gets copied. When LS-3 migrates from pg-02 to pg-03, every row belonging to LS-3 is physically transferred across — there’s no magic that avoids the bytes-on-disk part. That copy uses the online-rebalancing playbook later in this post (double-write → backfill → verify → cut over), and at real scale it can take hours.

What the indirection avoids is re-keying. With naive hash(tenant_id) % N, adding one machine to a 3-node cluster changes N from 3 to 4, which changes the hash result for roughly 3/4 of all tenants, which means 3/4 of the entire dataset has to move. With the logical/physical split, the routing function (hash(tenant_id) % 256) is permanent — every tenant always resolves to the same logical shard, forever. To rebalance, you pick one logical shard, update its directory row, and copy that shard’s data. The difference is “copy 1/256 of the dataset” versus “copy 3/4 of the dataset” — for the same operational outcome of adding one machine.

This is the architecture published by Notion (480 logical shards across 32 instances), Instagram (logical shards as Postgres schemas across a few dozen machines), and most large Postgres shops.

4. Routing Layers

Three styles:

Application-level. The app computes the shard and opens a connection to the right Postgres. Maximum control, maximum responsibility. Each shard is vanilla Postgres — same monitoring, same backups, same psql. You write the fan-out logic, the directory cache, the rebalancing tools. Most large Postgres shops do this.
SQL-aware proxy (PgCat, Citus’s coordinator, vendor proxies). One endpoint the app talks to; it parses queries, computes shards, fans out reads, merges results. Strengths: app stays simple. Weakness: the proxy has a worldview, and your queries have to fit it.
Distributed Postgres-compatible databases (CockroachDB, YugabyteDB). A different system that speaks the Postgres wire protocol. Covered in the Ecosystem section.

5. The Hard Parts (And There Are a Lot of Them)

When teams build sharded Postgres for the first time, the first six months are mostly about the things they didn’t plan for. Here’s the catalogue. Each one is a project, not a paragraph — but you need to know all of them exist before you start.

5.1 Cross-shard `JOIN`s

A single Postgres JOIN runs in microseconds because both sides of the join are on the same machine, in the same buffer cache. The moment you shard, a JOIN between two tables that don’t share the same shard key becomes a fan-out query: the app or proxy asks every shard for matching rows, ships them over the network, and merges the results.

Concrete example. You have:

users sharded by tenant_id
audit_logs sharded by tenant_id

A query like this stays on a single shard — Postgres handles it the same way it always did:

SELECT u.email, l.action
FROM users u
JOIN audit_logs l ON l.user_id = u.id
WHERE u.tenant_id = 42;

But add a global feature: “show every audit log entry across all tenants for users who signed up in the last 24 hours.” Now the routing layer has to:

Query every shard for users who signed up in the last 24 hours.
Group those user IDs by which shard they came from.
Query every shard again for matching audit log entries.
Merge everything into one result set.

With 32 shards that’s 64 round-trips, network latency added per call, partial-failure modes when one shard times out, and merge logic somewhere in the app. The fix is co-location — design the schema so tables you’d join always share the shard key. The cost of co-location is that occasionally you’ll want a cross-tenant join and simply won’t have it.

5.2 Cross-shard aggregations and reporting

Single-shard reads scale linearly with shard count. Cross-shard aggregations don’t.

“How many active users do we have?” is now SELECT count(*) FROM users WHERE last_active > now() - interval '30 days' on every shard, summed in the app. “Top 10 products globally by revenue this month” is a partial sort on every shard followed by a global merge-sort.

This is why every sharded company eventually builds a parallel analytics path — a data warehouse like Snowflake, BigQuery, Redshift, or ClickHouse, fed by Change Data Capture (CDC) streams from each shard. Operational queries hit Postgres; analytics queries hit the warehouse. You absorb the latency cost (warehouse data is minutes behind real-time) in exchange for queries that can scan globally.

If you don’t plan for the warehouse before sharding, every reporting feature becomes a scatter-gather of doom and the analytics team starts asking why the dashboard takes 90 seconds to load.

5.3 Foreign keys disappear

A FOREIGN KEY in Postgres is the database telling you “this user_id must reference a real row in users.id.” It’s enforced at write time — try to insert an order for a user who doesn’t exist and Postgres refuses.

Across shards, this guarantee doesn’t exist. If users and orders live on different shards, Postgres has no way to check. Two ways to handle it:

Co-locate the related tables. If orders and users are both sharded by tenant_id, the FK can be enforced normally within each shard’s vanilla Postgres. The constraint is local — and local is fine because all the rows you care about for that tenant are on one shard.
Enforce in application code. Check that the parent exists before the insert. Accept the race condition (the parent could be deleted between check and insert, leaving an orphan) or guard it with application-level locking. Most teams accept the race because the failure mode — an orphan row — is recoverable, and the alternative is much more expensive.

5.4 Globally unique IDs

bigserial and SERIAL columns are backed by a sequence — a per-database counter that increments on every insert. Sequences are local to each Postgres instance. Once you have N shards, two different shards will both happily emit id = 1 for the first inserted row, and now you have collisions everywhere.

Three common fixes:

UUIDs. Universally Unique Identifiers — 128-bit random values, unique by construction across any system. 16 bytes instead of 8, and the randomness defeats the locality optimizations that B-tree indexes use for inserts. v7 UUIDs (time-prefixed) are a good middle ground; standard v4 random UUIDs hurt insert performance because each new row lands on a random index page.
Snowflake-style IDs. A 64-bit integer composed of (timestamp_ms | shard_id | sequence_within_ms). Fits in a bigint, sorts roughly by time (good for indexes), unique by construction (different shards have different shard_id bits). Originally Twitter’s design, now copied widely.
Centralised ID service. A single coordinator hands out ID ranges to each shard. Simple to reason about, but it’s now a hop on every insert and a single point of failure.

The decision matters far beyond ID generation. IDs are everywhere — in URLs, in logs, in cross-system traces. Choosing badly means migrating IDs later, which is much worse than migrating data.

5.5 Distributed transactions

A single Postgres transaction is atomic. This works fine on one machine:

BEGIN;
UPDATE accounts SET balance = balance - 10 WHERE id = 1;
UPDATE accounts SET balance = balance + 10 WHERE id = 2;
COMMIT;

Either both updates apply or neither does. Across shards (say id = 1 is on shard A and id = 2 is on shard B), you have two bad options.

Two-Phase Commit (2PC). A coordinator asks every participating shard “can you commit?” Each shard prepares the transaction (writes the changes durably but doesn’t make them visible) and replies. Once all shards say yes, the coordinator broadcasts “commit.” Sounds clean. In practice, the prepare phase locks rows on every shard for the round-trip duration; if the coordinator crashes between phases, you have prepared transactions that can’t be resolved without manual intervention. Production sharded systems almost universally avoid 2PC.

Sagas. Break the cross-shard operation into a sequence of single-shard transactions, each with a compensating undo step. If a later step fails, run the inverse of the earlier steps.

Concrete example — transferring $10 between two users on different shards:

Step 1:  shard A → debit user-1 by $10      (commit on shard A)
Step 2:  shard B → credit user-2 by $10     (commit on shard B)

If Step 2 fails:
   Compensation → shard A → credit user-1 by $10

The sequence has to be idempotent (running step 2 twice must be the same as running it once), because retries are unavoidable in distributed systems. It has to handle the case where the compensation itself fails (you ledger the failure and route it to a human-review queue). And it has to be observable, so you can answer “where is transfer #1234 stuck?” three weeks later when a customer complains.

Stripe’s Ledger system is essentially a giant saga orchestrator on top of sharded storage. Done well, sagas are robust. Done badly, they’re a permanent source of data inconsistency.

5.6 Online rebalancing

Sooner or later one shard is too hot, a region is being added, or the original key choice was wrong for a workload that has since changed shape. You need to move data while the application keeps reading and writing. There’s no maintenance window — the system is live.

The pattern that has emerged across published architectures (Notion, Figma, Slack on Vitess, Stripe):

Double-write. Application writes go to both the old shard and the new shard for every row in the migrating range. Reads still come from the old shard.
Backfill. Stream historical data from old to new using logical replication (Postgres’s built-in publication/subscription) or a CDC tool like Debezium.
Verify. Compare random samples between old and new. Run the application against the new shard in shadow mode — every read also queries the new shard, results are compared but the old shard’s response is what’s returned to the user.
Cutover. Flip reads to the new shard. Keep double-writing for a safety window — usually days — in case you need to roll back.
Tear down. Stop writing to the old shard. Eventually drop the data.

Each step is hours-to-days at real scale. Each has subtle failure modes:

Replication lag — the new shard is minutes behind because the backfill is competing with live writes.
Diverging writes — the double-write got applied to the new shard but not the old (network blip), or vice versa, and a row’s value differs between the two.
Idempotency edge cases — the same write got applied twice on one side because of a retry.

Teams that get this right build it once as a generic framework and reuse it for every subsequent rebalance. Teams that don’t end up doing the migration by hand the first three times, with a different bug each time.

5.7 Hot shards and tenant skew

Eventually one shard becomes a latency outlier. The usual causes:

One tenant grew 40× the average (Notion’s published numbers describe this happening repeatedly).
A region with disproportionate traffic concentration (us-east always lighting up).
A “celebrity user” pattern — one row receiving thousands of reads per second because some piece of content is going viral.

Mitigations are uniformly painful:

Split the giant tenant across multiple shards. Breaks co-location, requires application changes, makes their data harder to back up coherently.
Isolate them on a dedicated shard so their problems don’t affect other tenants. Operationally fine. But you’ve now admitted the architecture isn’t really “shard by tenant” — it’s “shard by tenant, except for the special ones.”
Application-level rate limiting for the hot row or hot tenant. Buys time, doesn’t fix the underlying skew.
Caching layer in front of the hot reads. Helps for read-heavy hot rows, doesn’t help if writes are the problem.

The lesson Notion and others have written about publicly: you can’t predict which tenant will go viral. Build the shard-migration tooling early, because you’ll need it.

5.8 Schema migrations across N shards

ALTER TABLE on one Postgres is operationally simple. On 32 shards it’s a different problem.

The migration has to run on every shard. If it fails halfway through, some shards have the new schema and some don’t.
The application code that depends on the new schema can’t roll out before the schema. So the schema change must be deployed first, and it must be backward-compatible — the application still works against shards that haven’t migrated yet.
The rollout takes minutes-to-hours, and during that window the application is running against shards in mixed states.

The pattern: every schema change is two PRs.

PR 1 — make the schema change in a way the old code still works against. Add a new nullable column with a default; don’t drop the old column. Deploy. Run the migration on every shard.
PR 2 — once every shard has migrated and you’ve waited a deploy cycle, remove the old column and the now-unused code. Deploy that.

Expensive migrations (rewriting a 500 GB table) need pg_repack or batched UPDATE loops on each shard, with monitoring to catch the one shard where it fails partway through and resume from where it stopped. The thing you might be slightly lazy about on one Postgres becomes religion on N.

5.9 Backups and point-in-time recovery

A single Postgres backup is pg_basebackup plus an archived WAL stream. Point-in-time recovery (PITR) means “restore to the state the database was in at 14:32:11 last Tuesday.” You replay WAL forward from the base backup until you reach that timestamp.

On 32 shards, you have 32 base backups taken at 32 slightly different points in time, with 32 separate WAL streams. If you need to restore the whole system to a globally consistent moment — say, before a bug corrupted data across multiple tenants — you need:

Each shard’s base backup, taken near the same wall-clock time.
Each shard’s archived WAL.
Tooling that orchestrates a parallel restore across all 32 shards to the same target timestamp.

Most teams accept the approximation: shards are restored to near the same point in time, and any cross-shard inconsistencies introduced by the restore are reconciled manually after. True globally-consistent PITR on a sharded Postgres is hard enough that most teams don’t attempt it. They invest instead in audit logs and replay tools that can selectively undo a bad transaction.

5.10 Connection pooling explosion

Your application talks to one Postgres? It opens N database connections. Talks to 32 shards? It potentially needs N × 32 connections, one pool per shard. With pgbouncer (a connection pooler) in front of each shard, the pgbouncer fleet is now managing 32× more connections too.

Concrete numbers. Suppose your application server keeps 50 connections per Postgres. You have 100 application servers. With one database: 5,000 connections — already close to Postgres’s practical ceiling (~10,000) and a clear case for pgbouncer. With 32 shards: potentially 160,000 connections in the worst case. You’ll exhaust file descriptors before you exhaust the database.

Solutions:

Lazy pools. Only open a connection to a shard when a request actually needs it. Most requests touch one shard, so most pools stay small.
Reverse routing. Application servers are assigned to subsets of shards, so each server only pools connections to its assigned shards.
Shared connection pool layer. A central pgbouncer cluster fronting all shards, sized for the global connection count rather than per-app.

You’ll discover this in production when traffic spikes against one shard exhaust its max_connections and the application starts failing every request that needs that shard, regardless of which app server is doing the asking.

5.11 Failover and disaster recovery

One Postgres going down = your app is down. 32 shards = one shard going down means 1/32 of users see errors. Better in some ways, worse in others.

Better: the blast radius is smaller. Most users don’t notice. You have time to respond.
Worse: failover has to be automated per shard because manual intervention doesn’t scale across 32 instances. Each shard needs a hot standby (a streaming replica ready to take over) and an automated promotion script. Patroni or Stolon are the two common tools for managing this.
Worse: “is the database up?” is no longer a single health check. It’s 32. Your dashboards have to handle “27/32 healthy, 5 degraded” — and the routing layer has to route around the failing ones gracefully.

You also need to think about correlated failures. A bad query pattern or a bug in a schema migration can take down many shards simultaneously. “One shard at a time is fine” is a comforting story that doesn’t survive contact with reality.

5.12 Observability becomes 32× harder

A slow query on one shard is invisible on the dashboards for the other 31. The questions you ask change:

“What’s the p99 query latency?” (the latency the slowest 1% of queries experience) → “What’s the p99 latency per shard, and which shard is slowest?”
“Which queries are the worst offenders?” → “Which queries, on which shards, and is it the same workload everywhere?”
“Disk usage?” → 32 numbers, hopefully one dashboard.

You need per-shard tagging in your metrics, log aggregation that can group by shard, and dashboards that surface the outlier shard rather than averaging across all 32 (averages hide the one shard that’s on fire). The “show me the worst shard right now” view is the single most valuable dashboard a sharded system has.

5.13 Testing and dev environments

Production has 32 shards. Your laptop has one Postgres. The bugs that only appear in production are exactly the ones related to the sharding boundary — cross-shard joins, routing logic, the saga compensation step you never exercised.

Most teams settle on one of:

Run a small number of shards locally (2 or 4) to exercise the routing logic. Not a faithful reproduction, but catches most routing bugs.
Maintain a staging environment that mirrors production at smaller scale and is the only place where the sharded code path is genuinely tested before production.
Integration tests against a small sharded cluster in CI. Slow but catches real bugs.

Neither is great. Sharded systems have a long tail of “only-fails-in-production” bugs that you pay for through monitoring discipline and quick rollback rather than test coverage.

5.14 Multi-tenant security boundaries

When tenant_id is the shard key, the database physically separates tenants. A query that forgets to filter by tenant_id returns nothing useful, because that tenant’s data isn’t on the connected shard. This is accidentally good for security.

But the inverse: if the routing layer has a bug that sends a query to the wrong shard, you’ve just leaked one tenant’s data to another. Routing-layer bugs become security incidents in sharded systems in a way they aren’t in single-database systems. Teams that take this seriously add:

Pre-flight assertions. The application checks that the connection it’s about to use is the right shard for the tenant in the request context. Cheap and catches most bugs immediately.
Result validation. Every query result is filtered through a check that all returned rows match the expected tenant_id. A row that doesn’t match means routing failed; the request is rejected and an alert fires.
Periodic audits. Random sampling of query results to confirm no cross-tenant leakage in production.

5.15 The cost — in dollars and headcount

Sharding multiplies your infrastructure cost roughly linearly with shard count. 32 shards is 32 Postgres instances, each with its own storage, backups, monitoring, hot standby. The savings come from using smaller machines per shard, but rarely fully offset the multiplier — you usually end up paying 1.5–3× what one giant machine would have cost.

The bigger cost is headcount. A team running one Postgres can treat it as part-time work for a thoughtful backend engineer. A team running 32 shards needs at least one full-time database engineer thinking about capacity, rebalancing, failover, and the migration framework. The platform is now a product — with an on-call rotation, a roadmap, and an expectation that someone owns it 24/7.

This is the hidden tax of premature sharding: you’re permanently adding a person (or three) to your headcount in exchange for a problem you might not have actually had yet.

6. What Sharding Actually Does to Postgres

The two strategies (partitioning and sharding) route around fundamentally different bottlenecks. Partitioning shrinks the active working set and gives operations a smaller object to grip. Sharding adds physical resources — more disks, more cores, more RAM, more max_connections — at the cost of coordination.

WAL — the resource sharding actually adds

Postgres writes every change to the Write-Ahead Log before modifying the heap. All writes across all partitions go to the same WAL on the same physical disk, serialized through XLogInsert and ultimately one fsync per group commit. There is one WAL per Postgres cluster.

This is the throughput ceiling partitioning can’t break. A modern NVMe disk fsyncs in tens of microseconds; with synchronous commit and a busy backend you can push tens of thousands of WAL records per second per instance, but you don’t get to add more WAL by adding partitions. Sharding spreads WAL across N machines and N disks — this is the actual write-scaling mechanism.

Replication

A streaming replica replays WAL serially. When the primary writes WAL faster than the replica can apply, lag expands under load. Postgres 16+ has parallel apply for some WAL record types, but the constraint stands: one replica can’t keep up with a primary running on hardware many times its size. Sharding turns “one fast primary, one struggling replica” into “N modest primaries, each with a comfortable replica.”

Connections

max_connections is per-cluster, and each backend is a heavyweight process. The practical ceiling is a few thousand — beyond which you’re using pgbouncer in transaction-pooling mode and accepting its tradeoffs (no session state, fewer prepared statements, no LISTEN/NOTIFY). Sharding multiplies the ceiling by shard count.

Checkpoints

The checkpointer flushes dirty buffers and trims the WAL every 5 minutes or 1 GB, whichever first. On a single instance with TB of data and a hot working set, every checkpoint is an I/O spike that shows up in tail latency. Sharding spreads checkpoints across machines so they’re no longer correlated.

The shape of this: sharding solves problems that are about per-cluster physical resources. If your problem isn’t physical resources, sharding adds a lot of complexity without buying you anything. Partition first; vertical-decompose second; shard only when nothing cheaper is left.

7. The Postgres Ecosystem

The tools that show up everywhere in production sharded Postgres:

Citus — turns one coordinator and N worker nodes into a distributed Postgres. Define a distribution column; Citus rewrites queries to fan out and merge results. Co-located tables join efficiently. Owned by Microsoft now, powers Azure Cosmos DB for PostgreSQL. Best fit when the workload has a clear distribution column and most queries already filter by it.

Logical replication — Postgres’s publication/subscription system streams row changes (not WAL bytes) to subscribers. The mechanism behind every serious online shard migration, every Debezium-based CDC pipeline, and tools like pg_easy_replicate for upgrades.

Distributed Postgres-compatible databases — CockroachDB, YugabyteDB. Not Postgres, but speak its wire protocol. Strong consistency via Raft, automatic resharding, multi-region support. The catch: write latency is higher (consensus isn’t free), some PG features are missing or partial (certain extensions, advisory locks, custom types, some JSONB operators). Right answer for new systems needing multi-region transactional writes from day one; harder sell as a migration target for a mature Postgres workload.

Patroni / Stolon — automated failover managers for Postgres clusters. Essential for running 32 shards each with a hot standby — manual failover does not scale.

8. Failure Modes Specific to Sharding

Beyond the hard parts above, here are the incident-report patterns you’ll see most often in mature sharded systems.

Uneven shard distribution. You sharded by tenant_id and one tenant is 40× the average. That shard is the bottleneck; everyone else is bored. Splitting the giant tenant breaks co-location; isolating it on a dedicated shard works but admits the architecture isn’t really “shard by tenant” anymore.

Painful rebalancing. Shard 7 moving to a new machine. Double-write on, backfill at 80%, a bug in the comparison job silently passing diverging writes through. You discover it after cutover. Two sources of truth and a Saturday.

Cross-shard transactions, snuck in. A feature needed atomic updates across two tenants. You don’t have 2PC. The code stitched together unrelated commits, and now an orphan in shard B points to a missing parent in shard A. The second-worst class of bug in sharded systems. (The worst: silent divergence during a rebalance.)

Schema migrations that aren’t backward-compatible. You dropped a column and the code that read it in one PR — but the rollout is shard-by-shard, and now some shards have the new schema and some don’t. Always two-step: schema-compatible change first, code change second, after every shard has migrated.

The slow shard nobody noticed. Per-shard metrics exist but the dashboard averages across all of them. One shard is at p99 = 2 seconds; the global average is fine; nobody notices until a customer complains. The single most valuable sharded-system dashboard is the outlier view, not the average.

9. The Full Scaling Journey

What actually happens, in order, in systems that reach interesting scale:

Stage 0 — One Postgres. Tune shared_buffers (~25% of RAM), work_mem, effective_io_concurrency, autovacuum cost limits. Add the right indexes. Use EXPLAIN ANALYZE. Orders of magnitude on the table.

Stage 1 — Read replicas. Send analytical and reporting traffic to followers. Buys read scale, not writes. Replicas aren’t transactionally consistent with the primary; design around it or pay write latency for synchronous replication.

Stage 2 — Partition the hot tables. Range-partition the 4 TB events table by day or month. Drop old partitions on a cron. Same for audit logs, activity streams, anything append-mostly. Often buys 2–5 years of runway. (Covered in detail in the sibling post: Postgres Partitioning.)

Stage 3 — Vertical decomposition. Pull large independent domains into their own Postgres instances: billing, analytics, notifications, search. Multiple connection pools, but each database is still single-node. This is where most companies stop.

Stage 4 — Shard the core. When central tenant/user data still won’t fit, pick a partition key (almost always tenant_id), pick a logical shard count, build the routing layer, migrate over a quarter using double-write. Build the runbooks. Hire a database team.

Stage 5 — Operate the platform. Sharding isn’t a project, it’s a regime. Per-shard observability, capacity planning, rebalancing tooling, migration automation. The sharding system is a product.

The mistake is jumping from Stage 1 to Stage 4. Avoid premature sharding the same way you’d avoid premature microservices: the costs are paid forever, the benefits arrive later than expected, and most wins were available cheaper one layer up.

10. The Insight

Partitioning is usually about making one database manageable. Sharding is about admitting one database is no longer enough. They’re not points on the same continuum — they’re answers to different questions, taken from different toolkits, costing different things, paid forever.

The teams that scale Postgres well are not the teams who shard early. They’re the teams who partition aggressively, decompose their data domains, build serious observability around their one cluster, and only reach for sharding when they’ve genuinely exhausted what one big machine can do. By the time they do shard, the architecture has been forced into a shape that can be sharded — a clear partition key emerging from the query patterns, related data co-located by accident of the data model, transactional boundaries that already respect tenant edges.

The architecture that survives is the one you can still understand at 3 AM with a single psql window open. Postgres makes that achievable longer than most people think, if you use partitioning well, vertical decomposition deliberately, and sharding only when nothing cheaper is left.

This is the first post in the Distributed Systems series. Next up: replication strategies — synchronous vs asynchronous, physical vs logical, and the consistency tradeoffs each forces on the application above it. After that: consensus protocols (Raft and friends), distributed transactions (2PC, sagas, the outbox pattern), CDC pipelines, and distributed locks. Sharding was the easy one; the rest of this is about coordinating machines that don’t share memory and can’t fully trust each other.

Postgres Partitioning: Strategies, Advantages, and Pitfalls — the cheaper step you should exhaust before sharding.
Database Replication: What to Use When — Part 2 of this series; covers single-leader, multi-leader, leaderless, and the consistency tradeoffs.
ACID, Read Phenomena, and Isolation Levels — what cross-shard transactions cost you compared to single-node.
Reading PostgreSQL EXPLAIN Output — useful for the partitioning step and for diagnosing slow per-shard queries.

The Crypto Module: Hashing, Encryption, and What to Use When

Mon, 11 May 2026 00:00:00 GMT

Every serious programming language ships a crypto module in its standard library. Go has crypto/*. Node.js has node:crypto. Python has hashlib and secrets. They all expose the same underlying primitives — but using the wrong one for the job can silently destroy your security.

This post maps the terminology to the concepts, shows you what each primitive actually does, and gives you a clear decision tree for common scenarios.

The Vocabulary (Defined Once, Used Everywhere)

Before touching code, let’s fix the words. A lot of crypto confusion comes from these terms being used interchangeably when they shouldn’t be.

Encoding — transforms data into a different representation. Base64, hex, URL-encoding. Reversible. Provides zero security — it’s not encryption.

Hashing — a one-way transformation. Input of any size → fixed-size output (the digest). Practically impossible to reverse. The same input always produces the same digest.

Encryption — a two-way transformation. Requires a key. You can decrypt back to the original if you have the key.

Digest — the output of a hash function. SHA-256("hello") → a 32-byte digest, usually shown as 64 hex characters.

Salt — random data mixed with a password before hashing it, so identical passwords produce different digests. Defeats precomputed rainbow-table attacks.

HMAC — Hash-based Message Authentication Code. A hash that requires a secret key to produce and verify. Proves both integrity and authenticity.

Key Derivation Function (KDF) — a deliberately slow hash designed specifically for passwords. Bcrypt, Argon2, PBKDF2 are KDFs. Slowness is the feature — it makes brute-force expensive.

Pepper — a server-side secret (e.g. in an environment variable) mixed into every password hash. Stored separately from the database, so a DB leak alone isn’t enough to crack hashes.

IV / Nonce — Initialisation Vector / Number Used Once. Random bytes fed into an encryption operation so that encrypting the same plaintext twice produces different ciphertexts.

Hashing: Integrity, Not Secrecy

Use a hash when you need to verify that data hasn’t changed, or to produce a fixed-size fingerprint of something.

What it’s good for:

Checksums on file downloads
Deduplication (content-addressable storage)
Building blocks for HMAC and KDFs

What it is NOT for: storing passwords, or anything where the data needs to be kept secret.

// Go
import (
    "crypto/sha256"
    "fmt"
)

data := []byte("important document")
digest := sha256.Sum256(data)
fmt.Printf("%x\n", digest) // 64 hex chars

// Node.js
import { createHash } from 'node:crypto';

const digest = createHash('sha256')
  .update('important document')
  .digest('hex');
console.log(digest); // 64 hex chars

Which algorithm to use:

Algorithm	Status	Use for
SHA-256	✅ Safe	General-purpose checksums, HMAC, signatures
SHA-512	✅ Safe	Same as SHA-256, larger output
SHA-3	✅ Safe	Rarely needed; useful when SHA-2 is disallowed by policy
MD5	❌ Broken	Nothing security-related. Legacy file checksums only
SHA-1	❌ Broken	Nothing security-related

MD5 and SHA-1 are collision-vulnerable — two different inputs can produce the same digest. Never use them where security matters. Legacy systems still use them for non-security checksums (git uses SHA-1 for object IDs, for instance), but that doesn’t make them acceptable for new code dealing with passwords or signatures.

Practical example: package and image integrity. When you npm install, npm verifies every downloaded tarball against a SHA-256 hash recorded in package-lock.json. If a byte differs — because the file was corrupted mid-download or because a malicious registry served a swapped tarball — the install fails before any code runs. The same pattern is everywhere: Docker pulls verify the sha256:abc123… digest of each image layer, apt checks SHA-256 sums in its Packages.gz index, and download pages publish hashes so you can run sha256sum file.iso locally and compare. The hash isn’t secret; it’s a public fingerprint that lets anyone detect tampering.

Salting and Password Hashing

The single most common crypto mistake engineers make: hashing a password with SHA-256 directly.

// ❌ WRONG — never do this
hash := sha256.Sum256([]byte("mypassword"))

The problem: SHA-256("password123") is the same hash for every user with that password. An attacker with a precomputed table of common password hashes (a rainbow table) can reverse millions of hashes instantly.

The fix: use a purpose-built key derivation function (KDF).

A salt is a random value mixed with the password before hashing, so identical passwords produce different digests.

A KDF is a hash function deliberately designed to be slow — it makes brute-force attacks expensive. Good KDFs (bcrypt, Argon2id) generate and manage the salt for you automatically; you don’t store the salt separately.

// Go — bcrypt handles salt generation internally
import "golang.org/x/crypto/bcrypt"

// when user signs up:
hash, err := bcrypt.GenerateFromPassword([]byte("mypassword"), bcrypt.DefaultCost)

// when user logs in:
err = bcrypt.CompareHashAndPassword(hash, []byte("mypassword"))
if err == nil {
    // password matches
}

// Node.js — use the bcryptjs or @node-rs/bcrypt package
import bcrypt from 'bcryptjs';

// signup
const hash = await bcrypt.hash('mypassword', 12); // 12 = cost factor

// login
const ok = await bcrypt.compare('mypassword', hash);

The output looks like $2a$12$xyz... — the format embeds the algorithm, cost factor, salt, and hash all together. You store this single string and never need to manage the salt separately.

What is the cost factor?

The cost factor (sometimes called work factor) controls how slow the KDF is. It’s not a linear multiplier — in bcrypt it’s an exponent: cost 12 means 2¹² = 4,096 internal iterations; cost 13 means 8,192. Each increment roughly doubles the time.

cost 10 → ~80ms   per hash  (too fast for high-security systems today)
cost 12 → ~300ms  per hash  (good default)
cost 13 → ~600ms  per hash
cost 14 → ~1200ms per hash

The right cost factor is the highest one your system can afford without making login feel sluggish. A rule of thumb: aim for 200–500ms on your production hardware, and recalibrate every few years as hardware gets faster.

For PBKDF2 (Password-Based Key Derivation Function 2), the equivalent knob is the iteration count — the same principle applies: more iterations = more time = harder to brute-force.

For Argon2id, you tune time cost (iterations), memory cost (RAM), and parallelism. OWASP recommends a starting point of m=64MB, t=1, p=1 and increasing memory until you hit your latency budget.

KDF comparison:

KDF	Recommended cost	Memory-hard?	Notes
Argon2id	default params	✅ Yes	Best choice for new systems
bcrypt	cost ≥ 12	No	Widely supported, still fine
PBKDF2-SHA256	≥ 600,000 iterations	No	FIPS-compliant environments
SHA-256 alone	—	—	❌ Never for passwords

Memory-hard means the algorithm requires a lot of RAM, not just CPU time. This makes GPU-based brute-force attacks much more expensive — a GPU can run millions of SHA-256 operations per second in parallel, but struggles when each operation needs gigabytes of memory.

Bcrypt gotcha: bcrypt silently truncates passwords longer than 72 bytes. A user with a 100-character passphrase only gets the first 72 hashed. If you allow long passwords, either pre-hash with SHA-256 before passing to bcrypt, or use Argon2id which has no length limit.

When NOT to salt

Salt is valuable for low-entropy inputs like passwords where precomputed tables are a realistic attack. Skip it when:

Input is already high-entropy — API keys, UUIDs, cryptographic tokens (128-bit random values). Precomputing 2¹²⁸ hashes is impossible; salt adds nothing.
File integrity checksums — SHA-256 of a file download exists to be verified against a published hash. A salt would change the digest and break the comparison.
Content-addressable storage / deduplication — git object IDs, IPFS CIDs. The entire point is that identical content produces identical hashes. Salt breaks this by design.
HMAC — the secret key already plays the role of salt (and provides authentication on top). A separate salt is redundant.
Deterministic lookup — if you need to query “does this value exist?” against stored hashes (e.g. checking whether an email is already registered), per-record salts prevent direct comparison. You’d need a server-side secret applied consistently to all records (a pepper) rather than a random per-record salt.

The one-line rule: salt when the input space is small enough for an attacker to precompute hashes for; skip salt when the input is already unpredictable, or when identical-input → identical-hash is a requirement.

HMAC: Signed Hashes

A plain hash proves the content hasn’t been corrupted. But it doesn’t prove who produced it. An attacker who intercepts a payload can recompute the hash after modifying the data.

HMAC adds a secret key to the mix. Only someone who knows the key can produce or verify the MAC (Message Authentication Code).

Both sides share the secret key. The sender signs, the receiver re-signs the same message and compares.

// Go
import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
)

var sharedKey = []byte("my-secret-key")

// SENDER: sign and transmit (message, signature)
func sign(message []byte) string {
    mac := hmac.New(sha256.New, sharedKey)
    mac.Write(message)
    return hex.EncodeToString(mac.Sum(nil))
}

// RECEIVER: recompute and compare in constant time
func verify(message []byte, receivedSig string) bool {
    mac := hmac.New(sha256.New, sharedKey)
    mac.Write(message)
    expected := mac.Sum(nil)

    actual, err := hex.DecodeString(receivedSig)
    if err != nil {
        return false
    }
    return hmac.Equal(expected, actual) // constant-time
}

// Node.js
import { createHmac, timingSafeEqual } from 'node:crypto';

const SECRET = 'my-secret-key';

function sign(message) {
  return createHmac('sha256', SECRET).update(message).digest('hex');
}

function verify(message, receivedSig) {
  const expected = createHmac('sha256', SECRET).update(message).digest();
  const actual   = Buffer.from(receivedSig, 'hex');
  if (actual.length !== expected.length) return false;
  return timingSafeEqual(expected, actual);
}

Use HMAC for:

API request signing (webhook signatures from Stripe, GitHub, etc.)
JWT (JSON Web Token) secrets — the HS256 algorithm is HMAC-SHA256
Signed cookies or session tokens

Practical example: verifying a Stripe webhook. When a payment succeeds, Stripe POSTs a JSON event to your endpoint and includes a Stripe-Signature HTTP header containing an HMAC-SHA256 of the raw request body, computed with a webhook secret that you and Stripe both hold. Your handler reads the secret from your environment, recomputes the HMAC over the exact bytes of the request body, and compares it to the header value using a constant-time check. If they match, you know two things: the request actually came from Stripe (nobody else has the secret), and the body wasn’t altered in transit. Without this verification, anyone who learns your webhook URL could send fake payment.succeeded events and trick your app into shipping products for free.

Why constant-time comparison matters

When you compare two strings with ===, most implementations short-circuit — they stop at the first byte that differs and return false immediately. This creates a tiny but measurable timing difference: a signature that matches the first 20 bytes takes slightly longer to reject than one that matches 0 bytes.

An attacker who can send millions of requests and measure response times can exploit this. They try signatures that differ one byte at a time, watching for the requests that take slightly longer. Eventually they can reconstruct the correct signature one byte at a time — without ever knowing the secret key. This is called a timing attack.

hmac.Equal in Go and timingSafeEqual in Node always take the same amount of time regardless of how many bytes match. The comparison never short-circuits. That removes the timing signal entirely.

When NOT to use HMAC

For data integrity alone over HTTPS, between services you control — if you’re calling your own internal service over TLS (Transport Layer Security, the protocol behind HTTPS), the TLS layer already provides integrity. Adding HMAC is redundant. (But for webhooks like Stripe → your endpoint, HMAC is still needed — HTTPS only proves the data wasn’t tampered with in transit, not that the sender is Stripe specifically.)
When the other party doesn’t share your key — HMAC requires both sides to know the secret. If you need to prove identity to a third party who doesn’t have your key (e.g. a public API client), use asymmetric signatures (RS256, ES256) instead.
For password storage — HMAC is not a key derivation function. Don’t use it to hash passwords; use bcrypt or Argon2id.

Encryption: AES-256-GCM

When you need to encrypt data and decrypt it later (unlike hashing, which is one-way), use AES-256-GCM. This is the current industry standard for symmetric encryption.

AES-256: Advanced Encryption Standard with a 256-bit key. Extremely fast on modern hardware (dedicated AES instructions exist in every modern CPU).
GCM mode: Galois/Counter Mode. It gives you both confidentiality (nobody can read the data) and integrity (nobody can tamper with it without detection). This combination is called AEAD (Authenticated Encryption with Associated Data) — “Associated Data” is optional unencrypted metadata, like a record ID, that you want authenticated alongside the ciphertext but not hidden.

// Go
import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
    "io"
)

func encrypt(key, plaintext []byte) ([]byte, error) {
    block, _ := aes.NewCipher(key)
    gcm, _ := cipher.NewGCM(block)

    nonce := make([]byte, gcm.NonceSize()) // 12 bytes
    io.ReadFull(rand.Reader, nonce)        // cryptographically random

    // nonce is prepended to ciphertext so decryption can extract it
    return gcm.Seal(nonce, nonce, plaintext, nil), nil
}

func decrypt(key, ciphertext []byte) ([]byte, error) {
    block, _ := aes.NewCipher(key)
    gcm, _ := cipher.NewGCM(block)

    nonce := ciphertext[:gcm.NonceSize()]
    data  := ciphertext[gcm.NonceSize():]
    return gcm.Open(nil, nonce, data, nil)
}

// Node.js
import { randomBytes, createCipheriv, createDecipheriv } from 'node:crypto';

function encrypt(key, plaintext) {
  const nonce = randomBytes(12);
  const cipher = createCipheriv('aes-256-gcm', key, nonce);
  const encrypted = Buffer.concat([cipher.update(plaintext), cipher.final()]);
  const tag = cipher.getAuthTag(); // 16-byte auth tag
  return Buffer.concat([nonce, tag, encrypted]);
}

function decrypt(key, data) {
  const nonce      = data.subarray(0, 12);
  const tag        = data.subarray(12, 28);
  const ciphertext = data.subarray(28);
  const decipher   = createDecipheriv('aes-256-gcm', key, nonce);
  decipher.setAuthTag(tag);
  return Buffer.concat([decipher.update(ciphertext), decipher.final()]);
}

What is a nonce?

A nonce (Number Used Once) is a random value you generate fresh for every single encryption. Think of it like a unique order number on an invoice — every invoice gets a different number, even if the contents are identical.

AES-GCM uses a 12-byte (96-bit) nonce. It is not secret — the decryptor needs it to decrypt, so it’s stored alongside the ciphertext. The only requirement is that it must be unique per encryption with a given key. That’s why you generate it with a CSPRNG rather than counting up from 0 (a counter can work but is error-prone to manage).

What happens if you reuse a nonce?

AES-GCM works by generating a keystream from the key and nonce, then XOR-ing your plaintext with it. If you encrypt two different messages with the same key and nonce, an attacker can XOR the two ciphertexts together — the keystream cancels out and they’re left with the XOR of the two plaintexts. Recovering readable content from that is often straightforward. Worse, the authentication tag (see below) becomes forgeable. This is called a two-time pad attack and it’s completely catastrophic.

The fix is trivial: always call randomBytes(12) or rand.Reader fresh for each encryption. Don’t save and reuse a nonce from a previous session.

What is the auth tag?

GCM produces a 16-byte authentication tag alongside the ciphertext. It’s a signature over the ciphertext, computed using the key and nonce. When you decrypt, GCM recomputes the tag and compares it to the stored one. If any byte of the ciphertext was modified — by an attacker, by corruption, by anything — the tags won’t match and decryption fails with an error.

This is why AES-GCM gives you both confidentiality and integrity in one operation. You don’t need a separate HMAC on top of an AES-GCM ciphertext.

Rules:

Generate a new random nonce for every encryption. Reusing a nonce with the same key in GCM mode is catastrophic — it breaks both confidentiality and integrity.
Never use ECB mode. ECB (Electronic Code Book) encrypts each 16-byte block independently, so identical blocks produce identical ciphertext. It famously leaks patterns in data (look up the “ECB penguin” to see why this matters).
Key size: AES-128 (16 bytes) is still considered secure, but AES-256 (32 bytes) is the standard recommendation. Generate keys with a CSPRNG (see below), never derive them from a password with SHA-256.

Practical example: encrypting sensitive fields in a database. Say your app stores health records and the diagnosis column must be encrypted at rest so that even a stolen database dump leaks nothing useful. On write, you generate a fresh 12-byte nonce with randomBytes(12), encrypt the plaintext diagnosis with aes-256-gcm using your data key, and store nonce || auth_tag || ciphertext together as a single blob in the column. On read, you split the blob back apart and call decipher.setAuthTag(tag) before decrypting — if anyone has flipped a single byte in the ciphertext (or tried to swap one patient’s record into another), decipher.final() throws and you reject the read. The encryption key itself lives in your KMS, not in the database, so leaking the database alone is useless to the attacker.

Where Do Encryption Keys Live?

Encryption is only as strong as the secrecy of the key. The most common real-world failure isn’t broken crypto — it’s a key checked into a git repo or hardcoded in source code.

Never commit keys to source code, .env files in the repo, or container images.
Development: environment variables loaded from a local .env that’s git-ignored.
Production: a dedicated secrets manager or Key Management Service (KMS) — AWS KMS, GCP Cloud KMS, Azure Key Vault, HashiCorp Vault, Doppler. These services hold the key, enforce access control, and log every use. Your application asks the KMS to encrypt or decrypt on its behalf; the raw key material never lives in your application memory.

Rotating Keys

Key rotation means replacing an old key with a new one periodically or immediately after a suspected leak. The goal: if a key is ever exposed, the window of damage is bounded.

The challenge: data encrypted with the old key can’t be decrypted with the new one. The solution is to version your keys and tag your ciphertext with which version encrypted it:

# store with a version prefix
v1:3f8a12...:aabbcc...   ← encrypted with key version 1
v2:9d1f44...:112233...   ← encrypted with key version 2

During a rotation:

Generate key v2. Start encrypting all new data with it.
On reads, check the version prefix and decrypt with the matching key.
Lazily re-encrypt old records as they’re accessed — write them back encrypted with v2.
Once no v1-tagged records remain, retire key v1.

Nonces don’t rotate. A nonce is generated fresh for every single encryption operation and stored alongside that one ciphertext forever. You never reuse or “update” a nonce. If you’re thinking about nonce management — don’t. Just always call randomBytes(12) and you’re done.

Envelope Encryption

If you encrypted all your records directly with a single master key, rotating that key would mean re-encrypting every record through KMS — slow, expensive, and risky at scale.

Envelope encryption solves this with two layers:

Master Key  ← lives in KMS, you never see the raw bytes
     ↓  encrypts
 Data Key   ← one per record (or batch), stored alongside the ciphertext
     ↓  encrypts
 Your data

The workflow:

Ask KMS to generate a data key. KMS returns two things: the plaintext data key, and an encrypted copy of it (the “wrapped key”).
Encrypt your data locally with the plaintext data key using AES-256-GCM (fast, no KMS call per byte).
Discard the plaintext data key from memory immediately.
Store the wrapped key + nonce + ciphertext together.

To decrypt:

Send the wrapped key to KMS. KMS decrypts it with the master key and returns the plaintext data key.
Decrypt your data locally.
Discard the plaintext data key again.

Why this is better than encrypting everything with one key:

The master key never leaves KMS, so it’s never exposed in your app’s memory.
To rotate the master key, you only re-wrap the small data keys — not the entire dataset. Re-keying a billion records is a billion tiny KMS calls on kilobyte-sized keys, not re-encrypting terabytes of data.
Different records can have different data keys, so a single compromised key doesn’t expose everything.

AWS S3 server-side encryption, Google Cloud Storage, and most database encryption-at-rest systems all use this pattern under the hood.

Asymmetric Encryption: RSA and Elliptic Curves

So far everything has been symmetric — the same key encrypts and decrypts. Asymmetric cryptography uses a key pair: a public key and a private key.

Public key — safe to share with anyone. Used to encrypt data or verify a signature.
Private key — never shared. Used to decrypt data or create a signature.

This solves the key distribution problem: how do you securely send a key to someone you’ve never met? You publish your public key, they encrypt with it, only you can decrypt.

Asymmetric algorithms are roughly 1000× slower than AES, so in practice you never encrypt bulk data with them directly. The standard pattern (hybrid encryption, which is essentially what TLS does on every HTTPS request):

Generate a random AES key (the “session key”).
Encrypt the data with AES.
Encrypt the AES key with the recipient’s public RSA (Rivest–Shamir–Adleman) key.
Send both the encrypted key and the encrypted data.

ECDSA (Elliptic Curve Digital Signature Algorithm) and Ed25519, despite using elliptic curves, are signature algorithms — they cannot encrypt. For elliptic-curve encryption you use ECDH (Elliptic Curve Diffie–Hellman) to derive a shared AES key, then encrypt with AES.

Where asymmetric crypto shows up for engineers:

JWT with RS256/ES256: signing tokens with a private key, verifying with a public key.
SSH keys: your ~/.ssh/id_ed25519 is a private key; the .pub file is the public key added to servers.
HTTPS/TLS: the certificate is a public key. Your browser uses it during the handshake.

Practical example: GitHub SSH authentication

When you set up git push over SSH, you run ssh-keygen -t ed25519 once. It produces two files in ~/.ssh/:

id_ed25519 — your private key. Stays on your laptop. Never shared.
id_ed25519.pub — your public key. A single line of text starting with ssh-ed25519 ….

You open GitHub → Settings → SSH and GPG keys → New SSH key, and paste the contents of the .pub file. From now on, every time you git push:

GitHub’s SSH server sends your machine a random challenge string.
Your local ssh-agent signs the challenge with your private key.
Your machine sends the signature back.
GitHub verifies the signature using the public key you uploaded.

If the signature checks out, GitHub knows the connection is from someone who holds the matching private key — i.e. you. The private key never leaves your laptop, not even during the handshake. And here’s the magic: if GitHub’s database of public keys leaked tomorrow, the attacker still couldn’t impersonate you, because public keys are useless for creating signatures. They can only verify them.

The same pattern powers gh auth with GPG signing, signed git commits (the green “Verified” badge on github.com), git push over HTTPS with a personal access token wrapped by a passkey, deploy keys, and CI/CD runners pulling private repos.

Choosing a curve

For new asymmetric work, prefer Ed25519 (EdDSA) or ECDSA with P-256 over RSA. They’re smaller, faster, and more resistant to implementation mistakes. A 256-bit Ed25519 key offers roughly the same security as a 3072-bit RSA key, with much smaller signatures and faster verification.

Secure Random: The Foundation Everything Else Needs

Keys, salts, nonces, session tokens, CSRF (Cross-Site Request Forgery) tokens — all require cryptographically secure pseudorandom numbers (CSPRNG). A CSPRNG draws entropy from the operating system (hardware events, timing jitter, CPU noise) and produces output that is computationally indistinguishable from true randomness.

// Go — crypto/rand, not math/rand
import "crypto/rand"

token := make([]byte, 32)
if _, err := rand.Read(token); err != nil {
    panic(err) // entropy source unavailable — refuse to continue
}

// Node.js — node:crypto, not Math.random()
import { randomBytes } from 'node:crypto';

const token = randomBytes(32); // Buffer of 32 random bytes
const tokenHex = token.toString('hex'); // 64-char hex string

Practical example: password reset links. When a user clicks “forgot password,” your server generates 32 bytes from randomBytes(32), stores its SHA-256 in the database alongside the user ID and a 15-minute expiry, and emails the user a link like https://app.com/reset?token=<hex>. When the link is clicked, you SHA-256 the supplied token, look up the record, check the expiry, and let the user choose a new password. Because the token has 256 bits of entropy, an attacker can’t guess valid reset tokens by trying random URLs. The same primitive powers session cookies, API keys returned on signup, OAuth state parameters, and CSRF form tokens — anywhere you need an unguessable identifier.

What is entropy?

Entropy is a measure of unpredictability. A coin flip has 1 bit of entropy — two possibilities. A 32-byte random token has 256 bits of entropy — there are 2²⁵⁶ possible values. At a billion guesses per second, exhausting that space would take longer than the age of the universe.

Math.random() and rand.Intn() typically have 32–64 bits of internal state, seeded from a predictable source like the system clock or process ID. An attacker who knows roughly when your server started can narrow the seed space to a few million values and brute-force a session token in seconds.

A CSPRNG seeds itself from sources that are genuinely unpredictable to an outside observer — hardware interrupts, CPU timing jitter, OS entropy pools. This is what crypto/rand and node:crypto use under the hood.

How many bytes do you need?

Use case	Bytes	Why
Session token / API key	32	256-bit security margin
CSRF token	32	Same
AES-256 encryption key	32	Exactly fills the key size
AES-GCM nonce	12	Standard for GCM mode
KDF salt	16	128 bits is more than sufficient
Password reset token	32	Short-lived, but still needs full entropy

More bytes is rarely better — 32 bytes is already far beyond what anyone can brute-force. The risk is always the generator, not the length.

When You’d Rather Not Choose: High-Level Libraries

The standard crypto modules expose primitives — building blocks. Picking and combining them correctly is where most bugs live. If you’d rather not, a few well-regarded “use this and you’ll probably be fine” options:

libsodium (crypto_secretbox, crypto_box) — bindings exist for every language. Picks safe defaults: XChaCha20-Poly1305 for symmetric, X25519 for key exchange, Ed25519 for signing. You don’t choose the nonce size or the cipher mode — there’s only one of each, and it’s a good one.
age (age-encryption.org) — file-encryption tool and library. Simple key format, small surface area, hard to misuse.
Web Crypto API (crypto.subtle in browsers and Node.js) — the cross-platform standard for client-side JavaScript. Async by design.

Reach for these when you don’t have a specific reason to use the lower-level primitives.

Decision Tree: What to Use When

Scenario	Primitive	Algorithm
Store a user password	KDF	Argon2id or bcrypt (cost ≥ 12)
Verify a file download	Hash	SHA-256
Sign an API webhook payload	HMAC	HMAC-SHA256
Encrypt data at rest	Symmetric	AES-256-GCM
Sign a JWT	HMAC or asymmetric	HS256 (HMAC) or ES256 (ECDSA)
Generate a session token	Secure random	`crypto/rand`, `randomBytes`
Derive a key from a password	KDF	Argon2id or PBKDF2
Exchange keys with a stranger	Asymmetric	ECDH (key exchange)

Common Mistakes Checklist

The last point is worth emphasising: don’t invent new algorithms. Every primitive described in this post has been analysed by thousands of cryptographers for decades. The crypto module exists so you never have to think about the mathematics — your only job is to pick the right primitive and use it correctly.

Vectors and Matrices: The Language Neural Networks Speak

Sun, 10 May 2026 00:00:00 GMT

Every neural network, no matter how large, is doing one thing repeatedly: multiplying matrices and adding vectors. GPT-4 with its hundreds of billions of parameters is, at its core, a long chain of these operations. Before you can understand attention, backpropagation, or transformers, you need to be fluent in the arithmetic they’re written in.

This post builds that fluency from scratch, with worked examples at every step.

Scalars, Vectors, and Matrices

A scalar is a single number. Temperature (22°C), price ($9.99), age (31) — all scalars.

A vector is an ordered list of numbers. It has both a magnitude (how big) and a direction (which way).

v = [3, 2]       (a 2D vector)
w = [1, 0, -4]   (a 3D vector)

Geometrically, v = [3, 2] means “go 3 units right and 2 units up.” Vectors live in space.

A matrix is a rectangular grid of numbers, arranged in rows and columns.

A = [[2, 0],
     [1, 3]]    (a 2×2 matrix — 2 rows, 2 columns)

We say matrix A has shape (2, 2). A matrix with m rows and n columns has shape (m, n).

Vector Operations

Addition

Add vectors component by component. Both vectors must have the same length.

a = [1, 2, 3]
b = [4, 5, 6]

a + b = [1+4, 2+5, 3+6] = [5, 7, 9]

Geometrically: place vector b at the tip of vector a. The result points from the origin to where b ends.

Scalar Multiplication

Multiply every component by the same number.

a = [1, 2, 3]
3 * a = [3, 6, 9]

This stretches (or shrinks, or flips) the vector without changing its direction.

The Dot Product

This one is crucial. The dot product of two vectors is a single number (a scalar):

a · b = a[0]*b[0] + a[1]*b[1] + ... + a[n]*b[n]

Worked example:

a = [1, 2, 3]
b = [4, 5, 6]

a · b = (1×4) + (2×5) + (3×6)
      = 4 + 10 + 18
      = 32

The dot product measures how much two vectors point in the same direction. If two vectors are perpendicular, their dot product is 0. If they point in the same direction, it’s large and positive. This property becomes the foundation of the attention mechanism in transformers.

Geometric interpretation:

a · b = |a| × |b| × cos(θ)

Where |a| is the length (or magnitude) of vector a — computed as sqrt(a[0]² + a[1]² + ...). θ is the angle between the two vectors. cos(0°) = 1 (same direction), cos(90°) = 0 (perpendicular), cos(180°) = -1 (opposite).

Matrices

What a Matrix Represents

A matrix can represent many things: a dataset (rows = examples, columns = features), a transformation of space, or — most importantly for neural networks — the weights connecting one layer to the next.

Matrix Multiplication

This is the core operation. To multiply matrix A (shape m×n) by matrix B (shape n×p), the output C has shape m×p.

The rule: entry C[i][j] is the dot product of row i of A with column j of B.

Worked example (2×2 × 2×2):

A = [[1, 2],    B = [[5, 6],
     [3, 4]]         [7, 8]]

C[0][0] = row 0 of A · col 0 of B = [1,2]·[5,7] = (1×5)+(2×7) = 19
C[0][1] = row 0 of A · col 1 of B = [1,2]·[6,8] = (1×6)+(2×8) = 22
C[1][0] = row 1 of A · col 0 of B = [3,4]·[5,7] = (3×5)+(4×7) = 43
C[1][1] = row 1 of A · col 1 of B = [3,4]·[6,8] = (3×6)+(4×8) = 50

C = [[19, 22],
     [43, 50]]

Worked example (2×3 × 3×2 → 2×2):

A = [[1, 0, 2],     B = [[1, 2],
     [3, 1, 0]]          [0, 1],
                          [4, 0]]

C[0][0] = [1,0,2]·[1,0,4] = 1+0+8 = 9
C[0][1] = [1,0,2]·[2,1,0] = 2+0+0 = 2
C[1][0] = [3,1,0]·[1,0,4] = 3+0+0 = 3
C[1][1] = [3,1,0]·[2,1,0] = 6+1+0 = 7

C = [[9, 2],
     [3, 7]]

Important: matrix multiplication is not commutative. A × B ≠ B × A in general. The inner dimensions must match: (m×n) × (n×p) works; (m×n) × (p×n) does not (unless p = m).

The Transpose

The transpose of a matrix, written Aᵀ, flips rows and columns.

A = [[1, 2, 3],       Aᵀ = [[1, 4],
     [4, 5, 6]]              [2, 5],
                              [3, 6]]

If A has shape (m, n), then Aᵀ has shape (n, m).

The transpose appears constantly in neural networks, particularly in the attention formula QKᵀ (which you’ll see in post 7).

Why Shape Tracking Matters

In code, shape errors are the most common bug when building neural networks. Getting comfortable with shapes now prevents a lot of pain later.

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])   # shape (2, 3)

B = np.array([[7, 8],
              [9, 10],
              [11, 12]])    # shape (3, 2)

C = A @ B                   # shape (2, 2) — inner dims match (3 == 3)
print(C)
# [[58  64]
#  [139 154]]

The @ operator in Python is matrix multiplication (same as np.matmul). If you try to multiply shapes that don’t align, NumPy will raise an error immediately.

A Neural Network Layer in One Line

Here’s the payoff. A single layer in a neural network computes:

output = activation(W @ input + b)

Where:

W is the weight matrix — shape (output_size, input_size)
input is the input vector — shape (input_size,)
b is the bias vector — shape (output_size,)
activation is a nonlinear function (ReLU, sigmoid, etc.)

Worked example:

W = [[0.5, -0.2],    # shape (3, 2)
     [0.1,  0.8],
     [-0.3, 0.4]]

input = [2.0, 1.0]   # shape (2,)
b     = [0.1, -0.1, 0.2]

z = W @ input + b
  = [0.5*2 + (-0.2)*1 + 0.1,
     0.1*2 +   0.8*1 + (-0.1),
     -0.3*2 + 0.4*1 + 0.2]
  = [0.9, 0.9, 0.0]

That’s a 3-neuron layer processing a 2-dimensional input. The matrix W has 3×2 = 6 learnable weights — the numbers that training will adjust to minimise the loss.

Key Takeaways

A vector is an ordered list of numbers with magnitude and direction.
The dot product a · b measures similarity between two vectors — large when they point the same way, zero when perpendicular.
A matrix is a grid of numbers that can represent a transformation, a dataset, or a set of weights.
Matrix multiplication chains transformations together. Shape (m×n) × (n×p) → (m×p).
One neural network layer = W @ x + b followed by an activation function.

Everything in the rest of this series builds directly on these operations. Next: the calculus that teaches a network to improve.

Derivatives and Gradients: Teaching Machines to Improve

Sun, 10 May 2026 00:00:00 GMT

A neural network learns by making a mistake and then figuring out which weights caused it — and by how much. The tool that makes this possible is calculus: specifically, derivatives and gradients.

If you learned derivatives in school and then forgot them, this post will rebuild exactly what you need. If you’ve never seen them, this covers everything from the ground up.

What Is a Derivative?

A derivative measures how much a function’s output changes when you nudge its input.

Formally, the derivative of f(x) at a point x₀ is:

f'(x₀) = lim(h→0) [f(x₀ + h) - f(x₀)] / h

The notation lim(h→0) means “the value this expression approaches as h gets vanishingly small.” The whole formula is asking: if I move x by a tiny amount h, how much does f(x) change per unit of that movement? The answer is the slope of the function at that point.

Example: f(x) = x²

f(x₀ + h) - f(x₀)   (x₀+h)² - x₀²   x₀² + 2x₀h + h² - x₀²
——————————————————— = ——————————————— = ——————————————————————
         h                  h                      h
                                       = 2x₀ + h

As h → 0, this becomes 2x₀. So the derivative of x² is 2x. At x = 3, the slope is 6.

Derivative Rules

You rarely need the limit definition in practice. These rules cover almost everything:

Function	Derivative
`xⁿ`	`n·xⁿ⁻¹` (power rule)
`eˣ`	`eˣ`
`ln(x)`	`1/x`
`c` (constant)	`0`
`c·f(x)`	`c·f'(x)`
`f(x) + g(x)`	`f'(x) + g'(x)`

The chain rule is the most important one for neural networks:

If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x)

In plain words: the derivative of a composition equals the outer derivative times the inner derivative.

Example:

y = (3x + 1)²

Let g(x) = 3x + 1   →  g'(x) = 3
Let f(u) = u²        →  f'(u) = 2u

dy/dx = f'(g(x)) · g'(x) = 2(3x+1) · 3 = 6(3x+1)

At x = 1: dy/dx = 6(3+1) = 24.

A neural network is a long composition of functions. The chain rule is how gradients flow backward through all of them — which is exactly what backpropagation does (covered in post 5).

Partial Derivatives

Neural networks have millions of parameters, not one. A function of multiple variables uses partial derivatives: the derivative with respect to one variable, treating all others as constants.

Notation: ∂f/∂x means “the partial derivative of f with respect to x.”

Example: f(x, y) = x² + 3xy + y²

∂f/∂x = 2x + 3y      (treat y as a constant, differentiate by x)
∂f/∂y = 3x + 2y      (treat x as a constant, differentiate by y)

At (x=2, y=1):

∂f/∂x = 2(2) + 3(1) = 7
∂f/∂y = 3(2) + 2(1) = 8

This tells us: at the point (2, 1), moving slightly in the x direction increases f by about 7× that movement; moving slightly in the y direction increases f by about 8×.

The Gradient

The gradient of a function is the vector of all its partial derivatives:

∇f(x, y) = [∂f/∂x, ∂f/∂y]

For a function of n variables, the gradient is an n-dimensional vector.

The gradient always points in the direction of steepest increase. If you’re standing on a hillside, the gradient points uphill. Its negative, -∇f, points downhill — toward the minimum.

For the example above at (2, 1):

∇f(2, 1) = [7, 8]

The function increases fastest in the direction [7, 8]. To find a minimum, we move in the direction [-7, -8].

Gradient Descent

Training a neural network means minimising a loss function L(w), where w represents all the weights. The loss measures how wrong the network’s predictions are.

Gradient descent does this iteratively:

Compute the gradient ∇L(w) — which direction does the loss increase fastest?
Take a small step in the opposite direction (downhill).
Repeat.

The update rule is:

w ← w - η · ∇L(w)

Where η (eta) is the learning rate — a small positive number like 0.001 or 0.01 that controls how large each step is.

Why not take huge steps? If η is too large, you overshoot the minimum and the loss oscillates or diverges. Too small, and training takes forever. Finding a good learning rate is one of the main practical challenges in training neural networks.

Worked example:

Suppose the loss is L(w) = (w - 3)² (a simple parabola with minimum at w = 3). Start at w = 0.

L'(w) = 2(w - 3)     (derivative of the loss)
η = 0.3

Step 1: w = 0   L'(0) = 2(0-3) = -6   w ← 0 - 0.3×(-6) = 1.8
Step 2: w = 1.8 L'(1.8) = 2(1.8-3) = -2.4  w ← 1.8 - 0.3×(-2.4) = 2.52
Step 3: w = 2.52  L'(2.52) = -0.96   w ← 2.52 + 0.288 = 2.808
Step 4: w = 2.808 → ...

Each step gets closer to w = 3. Gradient descent is working.

The Loss Surface

In a real neural network, L(w) is a function of millions of weights — the “loss surface” is a high-dimensional landscape. We can’t visualise it, but the mathematics is identical: compute the gradient, step downhill, repeat.

import numpy as np

# simple quadratic loss: L(w) = (w - 3)^2
def loss(w):
    return (w - 3) ** 2

def grad_loss(w):
    return 2 * (w - 3)

w = 0.0
lr = 0.3

for step in range(20):
    g = grad_loss(w)
    w = w - lr * g
    print(f"step {step+1:2d}: w={w:.4f}, L={loss(w):.4f}")

Output converges to w ≈ 3.0 within a handful of steps.

Why the Chain Rule Is Central

In a neural network, the weights w produce hidden activations, which produce the output ŷ, which determines the loss L. This is a chain of compositions:

w  →  hidden  →  ŷ  →  L

To compute ∂L/∂w, you apply the chain rule repeatedly:

∂L/∂w = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂w

This is the mathematical heart of backpropagation. Post 5 works through this in full detail.

Key Takeaways

A derivative f'(x) measures the slope of f at x — how much the output changes per unit change in input.
The chain rule handles compositions: d/dx f(g(x)) = f'(g(x)) · g'(x).
A partial derivative ∂f/∂x treats all other variables as constants.
The gradient ∇f is the vector of all partial derivatives. It points in the direction of steepest increase.
Gradient descent moves weights in the direction -∇L, reducing the loss one step at a time.

Next up: probability and the Gaussian — the mathematical language that neural networks use to express uncertainty and produce predictions.

Probability and the Gaussian: How Neural Networks Express Uncertainty

Sun, 10 May 2026 00:00:00 GMT

When a language model says “the next word is probably ‘cat’”, it is producing a probability distribution over its entire vocabulary. When a classifier is 94% confident an image is a dog, that confidence is a number from probability theory. Understanding how neural networks reason about uncertainty requires understanding a handful of core probability concepts.

Probability Basics

A probability is a number between 0 and 1 that expresses how likely an event is.

P(event) = 0 means impossible.
P(event) = 1 means certain.
P(heads on a fair coin) = 0.5.

For a set of mutually exclusive outcomes that cover all possibilities, probabilities must sum to 1:

P(heads) + P(tails) = 0.5 + 0.5 = 1.0

This constraint — probabilities summing to 1 — is something we’ll enforce explicitly with softmax.

Expected Value

The expected value E[X] is the probability-weighted average of all possible outcomes:

E[X] = Σ x · P(X = x)

Example: Roll a fair six-sided die. What’s the expected value?

E[X] = 1×(1/6) + 2×(1/6) + 3×(1/6) + 4×(1/6) + 5×(1/6) + 6×(1/6)
     = (1+2+3+4+5+6)/6
     = 21/6 = 3.5

You’ll never actually roll 3.5, but on average over many rolls, this is what you expect.

Probability Distributions

A probability distribution describes how probability is spread across possible values of a random variable.

For a discrete distribution (finite outcomes), we list each outcome and its probability.

For a continuous distribution (infinite outcomes), we use a probability density function (PDF): a curve where the area under the curve equals 1, and the area between two values a and b gives P(a ≤ X ≤ b).

The Gaussian Distribution

The most important distribution in mathematics and statistics. Also called the normal distribution.

Its PDF is:

f(x) = (1 / σ√(2π)) · exp(-(x - μ)² / (2σ²))

Two parameters control it completely:

μ (mu) — the mean, where the peak sits.
σ (sigma) — the standard deviation, how spread out the distribution is. σ² is the variance.

The bell curve is symmetric around μ. About 68% of the probability mass falls within one standard deviation of the mean (μ ± σ). About 95% falls within μ ± 2σ.

Why the Gaussian Appears Everywhere

The Central Limit Theorem says: the sum of many independent random variables, regardless of their individual distributions, tends toward a Gaussian. Measurement errors, natural variation in populations, noise in signals — all of these arise from the sum of many small independent effects, so they all look Gaussian.

In neural networks, the Gaussian appears in:

Weight initialisation — weights are often initialised by sampling from N(0, σ²), which is shorthand for “a normal distribution with mean 0 and variance σ²”.
Noise — dropout and other regularisation techniques add controlled randomness.
Gaussian processes — probabilistic models closely related to neural networks.

Softmax: Turning Numbers Into Probabilities

A neural network’s output layer for classification typically produces raw scores called logits — one per class, any real value, not constrained to sum to 1. Softmax converts these into a proper probability distribution.

For a vector of logits z = [z₁, z₂, ..., zₙ]:

softmax(z)ᵢ = e^zᵢ / Σⱼ e^zⱼ

Each output is always between 0 and 1, and they sum to exactly 1.

Worked example:

z = [2.0, 1.0, 0.1]

e^2.0 = 7.389
e^1.0 = 2.718
e^0.1 = 1.105
sum   = 11.212

softmax = [7.389/11.212, 2.718/11.212, 1.105/11.212]
        = [0.659, 0.242, 0.099]

Check: 0.659 + 0.242 + 0.099 = 1.0  ✓

The class with logit 2.0 gets about 66% of the probability. The exponential function amplifies differences: adding 1 to a logit multiplies its (unnormalised) weight by e ≈ 2.72, so small changes in logits become large probability gaps.

The Temperature Parameter

In language models, you’ll see softmax with a temperature T:

softmax(z/T)ᵢ = e^(zᵢ/T) / Σⱼ e^(zⱼ/T)

T = 1.0 — standard softmax.
T < 1.0 — sharper distribution (more confident, more predictable outputs).
T > 1.0 — flatter distribution (more random, more creative outputs).

When you use “temperature” in a language model API, this is the parameter you’re controlling.

Cross-Entropy Loss

How do you measure how wrong a predicted probability distribution is? With cross-entropy.

For a single example with true label y (one-hot encoded — a vector of zeros with a single 1 at the correct class) and predicted probabilities ŷ:

L = -Σᵢ yᵢ · log(ŷᵢ)

Because y is one-hot (only one class is correct), this simplifies to:

L = -log(ŷ_correct)

Where ŷ_correct is the predicted probability assigned to the true class.

Worked example:

Throughout this series, log means the natural logarithm (base e), which is the standard in machine learning.

True class: cat (index 1)
Predictions: [0.10, 0.70, 0.20]

L = -log(0.70) ≈ 0.357

If the model had been confident and wrong:

Predictions: [0.05, 0.10, 0.85]   (cat is index 1, model predicts index 2)

L = -log(0.10) = 2.303   ← much higher loss

Cross-entropy loss grows sharply as the predicted probability of the correct class approaches zero. This is what motivates the model to assign high probability to correct classes.

Why log?

The logarithm has a key property: log(1) = 0 (perfect prediction → zero loss), and log(x) → -∞ as x → 0 (confident wrong prediction → infinite loss). It also has nice mathematical properties that make the gradients clean.

Putting It Together: The Classification Pipeline

Input → Neural Network → Logits → Softmax → Probabilities → Cross-Entropy Loss

Step by step for a 3-class classifier:

input = [0.5, 1.2]              # some input features
W = [[0.3, -0.1],               # weight matrix (3×2)
     [0.2,  0.4],
     [-0.5, 0.7]]
b = [0.1, 0.0, -0.2]           # biases

logits = W @ input + b
       = [0.3*0.5 + (-0.1)*1.2 + 0.1,
          0.2*0.5 + 0.4*1.2 + 0.0,
          -0.5*0.5 + 0.7*1.2 + (-0.2)]
       = [0.13, 0.58, 0.39]

# softmax:
e^0.13 ≈ 1.139,  e^0.58 ≈ 1.786,  e^0.39 ≈ 1.477
sum ≈ 4.402

probs = [1.139/4.402, 1.786/4.402, 1.477/4.402]
      ≈ [0.259, 0.406, 0.336]

true class: index 1 (probability = 0.406)
loss = -log(0.406) ≈ 0.902

Training reduces this loss by adjusting W and b using gradient descent.

Key Takeaways

A probability distribution describes how likely each possible outcome is. Probabilities sum to 1.
The Gaussian (normal) distribution is defined by mean μ and standard deviation σ. It appears everywhere because of the Central Limit Theorem.
Softmax converts raw logits to probabilities: e^zᵢ / Σ e^zⱼ. Always positive, always sums to 1.
Temperature controls how sharp or flat the softmax output is.
Cross-entropy loss = -log(ŷ_correct) measures how wrong a prediction is. High when the model assigns low probability to the true class.

With vectors, derivatives, and probability in hand, you’re ready to build an actual neural network from scratch. That’s next.

A Neural Network from Scratch: Perceptrons, Layers, and Forward Pass

Sun, 10 May 2026 00:00:00 GMT

The previous three posts gave you vectors, matrices, derivatives, and probability. Now you have everything needed to build and run a neural network from scratch — no libraries, just arithmetic.

The Perceptron: One Neuron

A single neuron takes a vector of inputs, multiplies each by a weight, adds a bias, and passes the result through an activation function.

z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
a = activation(z)

In matrix form (using the dot product):

z = w · x + b
a = activation(z)

Where w and x are vectors of the same length, and b is a scalar.

Worked example:

x = [0.5, 1.2, -0.3]       # input (3 features)
w = [0.4, -0.1, 0.8]       # weights
b = 0.2                      # bias

z = (0.4×0.5) + (-0.1×1.2) + (0.8×-0.3) + 0.2
  = 0.20 - 0.12 - 0.24 + 0.2
  = 0.04

a = activation(0.04)        # apply activation function

The weights w and bias b are the parameters that training will learn. A single neuron can only learn a linear decision boundary — it separates space with a line (or hyperplane). That’s why we stack neurons into layers and add nonlinear activations.

Activation Functions

Without activation functions, stacking layers is pointless: W₂(W₁x + b₁) + b₂ is still just a linear transformation, equivalent to a single layer. Activations introduce nonlinearity, enabling networks to learn complex patterns.

ReLU (Rectified Linear Unit)

ReLU(z) = max(0, z)

If the input is positive, pass it through unchanged. If negative, output 0.

ReLU(-2.5) = 0
ReLU(0)    = 0
ReLU(3.1)  = 3.1

ReLU is the default choice for hidden layers in modern networks. It’s fast to compute and its gradient is either 0 or 1, which makes backpropagation simple.

Derivative of ReLU:

d/dz ReLU(z) = 1  if z > 0
             = 0  if z < 0

(Undefined at z = 0, but we set it to 0 by convention — rarely matters in practice.)

Sigmoid

sigmoid(z) = 1 / (1 + e^(-z))

Squashes any input to the range (0, 1). Historically used everywhere, now mostly used in output layers for binary classification.

sigmoid(-3) ≈ 0.047
sigmoid(0)  = 0.5
sigmoid(3)  ≈ 0.953

Derivative of sigmoid (a neat identity):

d/dz sigmoid(z) = sigmoid(z) · (1 - sigmoid(z))

tanh

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Squashes to (-1, 1). Outputs are centred around 0 (unlike sigmoid, which is centred around 0.5), which makes training easier in some cases.

tanh(-2) ≈ -0.964
tanh(0)  = 0.0
tanh(2)  ≈  0.964

From One Neuron to a Layer

A layer applies a weight matrix W to the input vector x, adds a bias vector b, and then applies an activation elementwise.

Z = W @ x + b        # linear transformation
A = activation(Z)    # nonlinear activation

If the layer takes n inputs and produces m outputs:

W has shape (m, n)
x has shape (n,)
b has shape (m,)
Z and A both have shape (m,)

Worked example — a 2-neuron layer with 3 inputs:

W = [[ 0.5, -0.2,  0.1],   # shape (2, 3)
     [-0.3,  0.4,  0.7]]

x = [1.0, 2.0, -1.0]       # shape (3,)
b = [0.1, -0.2]             # shape (2,)

Z = W @ x + b
  = [(0.5×1 + -0.2×2 + 0.1×-1 + 0.1),
     (-0.3×1 + 0.4×2 + 0.7×-1 + -0.2)]
  = [(0.5 - 0.4 - 0.1 + 0.1),
     (-0.3 + 0.8 - 0.7 - 0.2)]
  = [0.1, -0.4]

A = ReLU(Z) = ReLU([0.1, -0.4]) = [0.1, 0.0]

The second neuron got a negative pre-activation, so ReLU clamps it to 0. That neuron is “off” for this input.

A Complete 2-Layer Network

Now let’s stack two layers and run a full forward pass. The network will classify 2D points into 2 categories.

Architecture:

Input: 2 features
Hidden layer: 3 neurons, ReLU activation
Output layer: 2 neurons, softmax (class probabilities)

Layer 1 weights and biases:
W1 = [[ 0.4, -0.3],     # shape (3, 2)
      [-0.2,  0.5],
      [ 0.8,  0.1]]
b1 = [0.1, -0.1, 0.0]

Layer 2 weights and biases:
W2 = [[ 0.6, -0.4,  0.2],   # shape (2, 3)
      [-0.1,  0.3,  0.5]]
b2 = [0.0, 0.1]

Input:
x = [1.5, -0.5]

Forward pass — Layer 1:

Z1 = W1 @ x + b1

Z1[0] = 0.4×1.5 + (-0.3)×(-0.5) + 0.1 = 0.6 + 0.15 + 0.1 = 0.85
Z1[1] = (-0.2)×1.5 + 0.5×(-0.5) + (-0.1) = -0.3 - 0.25 - 0.1 = -0.65
Z1[2] = 0.8×1.5 + 0.1×(-0.5) + 0.0 = 1.2 - 0.05 = 1.15

Z1 = [0.85, -0.65, 1.15]

A1 = ReLU(Z1) = [0.85, 0.0, 1.15]

The second neuron fired negatively and was clamped to 0 by ReLU.

Forward pass — Layer 2:

Z2 = W2 @ A1 + b2

Z2[0] = 0.6×0.85 + (-0.4)×0.0 + 0.2×1.15 + 0.0 = 0.51 + 0 + 0.23 = 0.74
Z2[1] = (-0.1)×0.85 + 0.3×0.0 + 0.5×1.15 + 0.1 = -0.085 + 0 + 0.575 + 0.1 = 0.59

Z2 = [0.74, 0.59]

# softmax
e^0.74 = 2.096
e^0.59 = 1.804
sum    = 3.900

ŷ = [2.096/3.900, 1.804/3.900] = [0.538, 0.463]

The network assigns 53.8% probability to class 0 and 46.3% to class 1.

Computing the loss (assuming class 0 is correct):

L = -log(ŷ₀) = -log(0.538) = 0.620

Training will adjust all 14 parameters (W1, b1, W2, b2) to reduce this loss.

Batched Computation

In practice, you don’t process one example at a time — you process a batch of many examples simultaneously using matrix operations, which is far more efficient on GPU hardware.

If your batch has B examples and each example has n features, the input X has shape (B, n). The layer computes:

Z = X @ W.T + b     # shape (B, m)
A = activation(Z)

(Note: X @ W.T rather than W @ x because X has examples as rows, not columns. Both are equivalent for a single example.)

PyTorch and NumPy handle this transparently:

import numpy as np

# batch of 4 examples, 2 features each
X = np.array([[1.5, -0.5],
              [0.0,  1.0],
              [-1.0, 2.0],
              [0.5,  0.5]])  # shape (4, 2)

W1 = np.array([[ 0.4, -0.3],
               [-0.2,  0.5],
               [ 0.8,  0.1]])  # shape (3, 2)
b1 = np.array([0.1, -0.1, 0.0])

Z1 = X @ W1.T + b1   # shape (4, 3)
A1 = np.maximum(0, Z1)  # ReLU, shape (4, 3)
print(A1)

All 4 examples are computed simultaneously, in parallel.

Key Takeaways

A perceptron computes z = w·x + b, then a = activation(z).
Without activations, stacking layers is equivalent to one linear layer.
ReLU max(0,z) is the default hidden-layer activation — simple, fast, and effective.
Sigmoid squashes to (0,1) for binary outputs; tanh squashes to (-1,1).
A layer is A = activation(W @ x + b).
The forward pass chains these layer computations to produce a prediction.
Batching processes many examples at once via matrix multiply — essential for GPU efficiency.

The network can now produce predictions. But how does it learn? Next post: backpropagation — where the chain rule does the heavy lifting.

Backpropagation: How Neural Networks Learn from Mistakes

Sun, 10 May 2026 00:00:00 GMT

The forward pass computes a prediction. The loss measures how wrong it is. Backpropagation answers the question: which weights should change, and in which direction, to reduce the loss?

The answer comes from the chain rule, applied systematically from the output backward through every layer to every weight. This post walks through that calculation in full.

The Computation Graph

A neural network is a composition of functions. The easiest way to reason about it is as a computation graph: a directed acyclic graph where each node is an operation and each edge carries a value.

For a minimal network output = W·x + b → sigmoid → loss:

Forward pass: values flow left to right; we compute and store each intermediate result.

Backward pass: gradients flow right to left; at each node we apply the chain rule and multiply the incoming gradient by the local derivative.

The Key Idea: Local Gradients

At each node in the graph, we only need to know:

The local gradient — how the node’s output changes with respect to its input.
The upstream gradient — how the loss changes with respect to this node’s output (arriving from the right).

Then, by the chain rule:

gradient flowing backward = upstream gradient × local gradient

This is the entire algorithm. Let’s work through it.

Worked Example: A Single Neuron

Setup:

x = 2.0, w = -3.0, b = -3.0
z = w·x + b
ŷ = sigmoid(z)
L = (ŷ - y)² / 2    (MSE loss, true label y = 0)

Forward pass:

z  = (-3.0)(2.0) + (-3.0) = -6.0 - 3.0 = -9.0
ŷ  = sigmoid(-9.0) = 1/(1+e^9.0) = 1/8104.1 ≈ 0.000123
L  = (0.000123 - 0)² / 2 ≈ 0.0000000076   (very small — already close)

Let’s use a worse weight so the gradients are more instructive. Try w = 1.0, b = 0.5:

x = 2.0, w = 1.0, b = 0.5, y = 0

z  = 1.0×2.0 + 0.5 = 2.5
ŷ  = sigmoid(2.5) = 1/(1+e^-2.5) = 1/1.082 ≈ 0.924
L  = (0.924 - 0)² / 2 = 0.427

Loss is 0.427. We want to reduce it by adjusting w and b.

Backward pass — computing ∂L/∂w and ∂L/∂b:

Step 1: gradient of loss with respect to ŷ

∂L/∂ŷ = ŷ - y = 0.924 - 0 = 0.924

Step 2: gradient through sigmoid. Using the identity sigmoid'(z) = sigmoid(z)·(1-sigmoid(z)):

∂ŷ/∂z = sigmoid(2.5)·(1 - sigmoid(2.5)) = 0.924 × 0.076 = 0.0702

∂L/∂z = ∂L/∂ŷ · ∂ŷ/∂z = 0.924 × 0.0702 = 0.0649

Step 3: gradient through the linear layer z = wx + b:

∂z/∂w = x = 2.0
∂z/∂b = 1

∂L/∂w = ∂L/∂z · ∂z/∂w = 0.0649 × 2.0 = 0.1298
∂L/∂b = ∂L/∂z · ∂z/∂b = 0.0649 × 1.0 = 0.0649

Gradient descent update (with η = 0.5):

w ← 1.0 - 0.5 × 0.1298 = 1.0 - 0.0649 = 0.9351
b ← 0.5 - 0.5 × 0.0649 = 0.5 - 0.0325 = 0.4675

After one update, let’s check the new loss:

z  = 0.9351×2.0 + 0.4675 = 2.337
ŷ  = sigmoid(2.337) ≈ 0.912
L  = 0.912² / 2 = 0.416    (was 0.427 — improved slightly)

After thousands of such steps, the loss converges toward 0.

Backprop Through a 2-Layer Network

Now let’s extend this to the 2-layer network from the previous post. The structure:

x → [W1, b1, ReLU] → A1 → [W2, b2, Softmax] → ŷ → Cross-Entropy → L

Using the values from post 4’s forward pass:

x  = [1.5, -0.5]
Z1 = [0.85, -0.65, 1.15]
A1 = [0.85, 0.0, 1.15]      (after ReLU)
Z2 = [0.74, 0.59]
ŷ  = [0.538, 0.463]
y  = [1, 0]                  (class 0 is correct)
L  = -log(0.538) = 0.620

Backward pass:

Step 1: gradient of cross-entropy + softmax (combined)

The combined gradient of cross-entropy loss + softmax has a beautifully clean form:

∂L/∂Z2 = ŷ - y = [0.538-1, 0.463-0] = [-0.462, 0.463]

This is one of the most elegant results in neural network math — the gradient is just the prediction error.

Step 2: gradient w.r.t. W2 and b2

For each weight W2[i,j], the gradient is the product of the upstream gradient at output i and the input activation at position j:

∂L/∂W2[i, j] = (∂L/∂Z2)[i] × A1[j]

# Row 0 (output neuron 0):
∂L/∂W2[0, :] = -0.462 × [0.85, 0.0, 1.15] = [-0.393, 0.0, -0.531]

# Row 1 (output neuron 1):
∂L/∂W2[1, :] =  0.463 × [0.85, 0.0, 1.15] = [ 0.394, 0.0,  0.532]

∂L/∂b2 = ∂L/∂Z2 = [-0.462, 0.463]

Step 3: gradient flowing back to A1

∂L/∂A1 = W2ᵀ @ (∂L/∂Z2)

W2ᵀ has shape (3, 2), ∂L/∂Z2 has shape (2,)

∂L/∂A1[0] = 0.6×(-0.462) + (-0.1)×0.463 = -0.277 - 0.046 = -0.323
∂L/∂A1[1] = (-0.4)×(-0.462) + 0.3×0.463 = 0.185 + 0.139 = 0.324
∂L/∂A1[2] = 0.2×(-0.462) + 0.5×0.463  = -0.092 + 0.232 = 0.140

Step 4: gradient through ReLU

ReLU’s derivative is 1 where Z1 > 0 and 0 where Z1 ≤ 0. This is the ReLU mask:

Z1 = [0.85, -0.65, 1.15]
mask = [1, 0, 1]              (1 where Z1 > 0, 0 elsewhere)

∂L/∂Z1 = ∂L/∂A1 × mask = [-0.323, 0.0, 0.140]

The second neuron was inactive (ReLU output was 0), so no gradient flows through it. This is correct — if a neuron doesn’t contribute to the output, adjusting its weights has no effect on the loss.

Step 5: gradient w.r.t. W1 and b1

Same pattern as Step 2 — multiply the upstream gradient by each input feature:

∂L/∂W1[i, j] = (∂L/∂Z1)[i] × x[j]

∂L/∂W1[0, :] = -0.323 × [1.5, -0.5] = [-0.485,  0.162]
∂L/∂W1[1, :] =  0.0   × [1.5, -0.5] = [ 0.0,    0.0  ]   (zeroed by ReLU mask)
∂L/∂W1[2, :] =  0.140 × [1.5, -0.5] = [ 0.210, -0.070]

∂L/∂b1 = ∂L/∂Z1 = [-0.323, 0.0, 0.140]

The Pattern

Every layer in a neural network has the same two ingredients:

Forward: Z = W @ A_prev + b, A = activation(Z).
Backward: receive ∂L/∂A from the next layer, compute ∂L/∂W, ∂L/∂b (for gradient updates), and ∂L/∂A_prev (to pass backward further).

The backward pass is just the chain rule applied layer by layer, starting from the loss and working back to every weight.

Why “Backpropagation”?

The word comes from “backward propagation of errors.” Information about the loss is propagated backward through the network to every weight. Each weight receives a gradient proportional to how much it contributed to the error.

Weights that contributed a lot to a large error get a large gradient and are adjusted significantly. Weights that barely affected the output get nearly zero gradient and change little. The network naturally focuses learning on the parameters that matter most.

Practical Implementation

In PyTorch, this is handled automatically:

import torch

x  = torch.tensor([1.5, -0.5])
y  = torch.tensor([1.0, 0.0])

# Define weights with gradient tracking
W1 = torch.randn(3, 2, requires_grad=True)
b1 = torch.zeros(3, requires_grad=True)
W2 = torch.randn(2, 3, requires_grad=True)
b2 = torch.zeros(2, requires_grad=True)

# Forward pass
A1 = torch.relu(W1 @ x + b1)
Z2 = W2 @ A1 + b2
probs = torch.softmax(Z2, dim=0)
loss = -torch.log(probs[0])   # true class is index 0

# Backward pass — PyTorch computes all gradients automatically
loss.backward()

# Gradients are now in W1.grad, b1.grad, W2.grad, b2.grad
print(W1.grad)

loss.backward() builds and traverses the computation graph, computing every gradient via the chain rule. The code above does exactly what we computed by hand.

Key Takeaways

Backpropagation is the chain rule applied to a computation graph, starting from the loss and moving backward through each layer.
At each node, the local gradient × upstream gradient gives the gradient to pass further back.
The gradient of cross-entropy + softmax is simply ŷ - y — the prediction error.
The gradient through ReLU is either 1 (pass through) or 0 (blocked). Neurons with negative pre-activations don’t update their weights.
Every weight receives a gradient proportional to its contribution to the error.

With backpropagation understood, you know how neural networks learn. The next two posts cover the specific techniques that make large language models work: embeddings and attention.

Embeddings and Similarity: Turning Words into Vectors

Sun, 10 May 2026 00:00:00 GMT

A neural network can only process numbers. Before a language model can reason about the word “cat”, it must be converted into a vector of numbers. This conversion is called an embedding, and the mathematical properties of those vectors are what give language models their apparent ability to understand meaning.

Why Not Just Use Integers?

The simplest approach: assign each word an integer. “cat” = 1, “dog” = 2, “car” = 3.

This has a fatal problem. The number 3 is larger than 2, which is larger than 1 — but “car” is not more than “dog” in any meaningful sense. Neural networks will try to learn from these numerical relationships and get confused.

One-Hot Encoding

A better idea: give each word a vector of zeros with a single 1 at the word’s index.

vocabulary = ["cat", "dog", "car", "runs", "sleeps"]  # size = 5

"cat"    → [1, 0, 0, 0, 0]
"dog"    → [0, 1, 0, 0, 0]
"car"    → [0, 0, 1, 0, 0]

This fixes the ordering problem — no word is numerically larger than another. But it has two major flaws:

Dimensionality: Real vocabularies have 50,000–100,000+ words. Each vector would be 50,000+ dimensions, mostly zeros. Storing and multiplying with these is extremely wasteful.
No notion of similarity: The dot product of any two different one-hot vectors is 0. The model has no way to know that “cat” and “dog” are more related to each other than “cat” and “car”.

Dense Embeddings

The solution: represent each word as a dense vector in a much smaller space, typically 64 to 1024 dimensions. These vectors are learned during training.

"cat"    → [0.2, -0.4,  0.7,  0.1, ...]   # 768-dimensional vector
"dog"    → [0.3, -0.3,  0.8,  0.0, ...]   # similar to "cat"
"car"    → [-0.5, 0.6, -0.2,  0.9, ...]   # different direction

Words with similar meanings end up with similar vectors. Words that appear in similar contexts during training pull their vectors together. After training, the geometry of the embedding space encodes semantic relationships.

The Embedding Matrix

In practice, embeddings are stored in an embedding matrix E of shape (vocab_size, embedding_dim). Row i of E is the embedding vector for word with index i.

To look up the embedding for word i, multiply the one-hot vector eᵢ by E:

embedding = eᵢ @ E

But multiplying a one-hot vector by a matrix just selects a row — it’s equivalent to a table lookup:

embedding = E[i, :]

This is all an embedding layer does: look up a row in a learned table. The parameters E are learned through backpropagation, just like any other weights.

In PyTorch:

import torch
import torch.nn as nn

vocab_size = 10000
embed_dim  = 256

embedding_layer = nn.Embedding(vocab_size, embed_dim)

# Look up embeddings for a batch of token indices
token_ids = torch.tensor([4, 17, 3, 42])  # 4 tokens
embeddings = embedding_layer(token_ids)    # shape (4, 256)

Dot Product as Similarity

From post 1, the dot product of two vectors a · b = |a||b|cos(θ). The dot product is large when vectors point in the same direction and small (or negative) when they don’t.

This makes it a natural similarity measure:

sim("cat", "dog")  = cat_vec · dog_vec  ≈ large positive
sim("cat", "car")  = cat_vec · car_vec  ≈ small
sim("cat", "hate") = cat_vec · hate_vec ≈ small or negative

The dot product is used directly in the attention mechanism (post 7).

Cosine Similarity

The dot product depends on the magnitude of both vectors. A long vector will have large dot products with everything. Cosine similarity normalises for this:

cosine_sim(a, b) = (a · b) / (|a| × |b|)

Where |a| is the length (L2 norm) of vector a: |a| = sqrt(a[0]² + a[1]² + ... + a[n]²).

The result is always between -1 and +1:

+1 — same direction (identical meaning).
0 — perpendicular (unrelated).
-1 — opposite directions (opposite meaning).

Worked example:

a = [0.6, 0.8]    (embedding for "king")
b = [0.5, 0.9]    (embedding for "queen")
c = [-0.9, 0.2]   (embedding for "terrible")

a · b = 0.6×0.5 + 0.8×0.9 = 0.30 + 0.72 = 1.02
|a| = sqrt(0.6² + 0.8²) = sqrt(0.36 + 0.64) = 1.0
|b| = sqrt(0.5² + 0.9²) = sqrt(0.25 + 0.81) = sqrt(1.06) ≈ 1.03

cosine_sim(a, b) = 1.02 / (1.0 × 1.03) ≈ 0.99   ← very similar

a · c = 0.6×(-0.9) + 0.8×0.2 = -0.54 + 0.16 = -0.38
|c| = sqrt(0.81 + 0.04) = sqrt(0.85) ≈ 0.92

cosine_sim(a, c) = -0.38 / (1.0 × 0.92) ≈ -0.41   ← dissimilar

Semantic Arithmetic

One of the most surprising properties of trained embeddings is that arithmetic on vectors corresponds to semantic relationships.

The classic example:

embedding("king") - embedding("man") + embedding("woman") ≈ embedding("queen")

The embedding space has learned to encode the concept of “royalty” and “gender” as directions in space. Subtracting the “man” direction and adding the “woman” direction finds the nearby word “queen”.

Other examples:

Paris - France + Germany ≈ Berlin
walking - walk + swim ≈ swimming
puppy - dog + cat ≈ kitten

These relationships are not explicitly programmed. They emerge from training on large corpora, because words that appear in similar contexts end up with similar vector directions.

A caveat: because embeddings are shaped entirely by the text they were trained on, they also absorb biases present in that text. Early word embeddings notoriously encoded sexist and racist stereotypes (e.g. doctor - man + woman ≈ nurse). Modern models partially mitigate this with curated training data and additional fine-tuning, but it remains an active area of research.

Subword Tokenisation

Modern language models don’t embed whole words — they embed subword tokens. The word “unbelievable” might be tokenised as ["un", "believ", "able"], and each subword gets its own embedding. This handles rare words and morphology automatically.

GPT-2 uses Byte Pair Encoding (BPE), which builds a vocabulary of common character sequences. GPT-4 uses a similar approach. The vocabulary size is typically 50,000–100,000 tokens.

Positional Embeddings

Embeddings give each word an identity, but they carry no positional information — the embedding of “cat” is the same whether it appears first or last in a sentence. Transformers solve this by adding a positional embedding to each token’s embedding:

final_embedding[i] = token_embedding[i] + positional_embedding[i]

The positional embeddings encode the position of each token in the sequence. Post 8 covers how these are constructed.

Key Takeaways

One-hot encoding is sparse and carries no semantic information. Dot products between one-hot vectors are always 0 or 1.
Dense embeddings are learned vectors of typically 64–1024 dimensions. Similar words get similar vectors.
An embedding matrix E stores one row per vocabulary word. Looking up an embedding is just indexing a row.
The dot product a · b measures how much two vectors point in the same direction — the foundation of attention.
Cosine similarity normalises for vector length, giving a similarity score in [-1, +1].
Trained embeddings exhibit semantic arithmetic: king - man + woman ≈ queen.

Next: the attention mechanism — where embeddings meet dot products to build context-aware representations.

The Attention Mechanism: How Transformers Focus

Sun, 10 May 2026 00:00:00 GMT

Before transformers, recurrent neural networks processed sequences one token at a time. To remember something from 100 words ago, the information had to survive through 100 sequential updates — often fading or being overwritten. The attention mechanism solves this by allowing every position in the sequence to directly attend to every other position, in a single step.

The math behind this is elegant and entirely built from tools you already have: matrix multiplication, dot products, and softmax.

The Problem: Context-Dependent Meaning

The word “bank” means different things in:

"I deposited money in the bank"     → financial institution
"I sat by the river bank"           → side of a river

For a language model to process “bank” correctly, it needs to know which other words in the sentence are relevant. That’s what attention computes.

The Library Analogy

Imagine a library system:

You arrive with a query — what you’re looking for.
Every book has a key — a label describing its contents.
Each book also has a value — the actual content.

The system compares your query to every key, figures out which books are most relevant, and returns a weighted blend of those books’ values.

Attention works exactly the same way:

Q (Query): what this position is looking for.
K (Key): what each position offers.
V (Value): the actual content each position contributes.

Scaled Dot-Product Attention

The full formula:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

Where dₖ is the dimension of the key vectors.

This looks concise but unpacks into four meaningful steps:

Step 1: Compute attention scores

scores = Q @ Kᵀ

Each entry scores[i, j] is the dot product of query i with key j — how relevant position j is to position i. Shape: (seq_len, seq_len).

Step 2: Scale

scores = scores / √dₖ

Why scale? The dot product of two dₖ-dimensional vectors has variance that grows with dₖ. Without scaling, large dₖ values push the dot products into regions where softmax produces near-zero gradients (the “saturated” region). Dividing by √dₖ keeps the variance stable regardless of dimension.

Step 3: Softmax to get weights

weights = softmax(scores, applied row by row)

Each row of weights is a probability distribution over all positions. weights[i, j] is how much position i should attend to position j. Each row sums to 1.

Step 4: Weighted sum of values

output = weights @ V

Position i’s output is a weighted average of all value vectors, weighted by how much i should attend to each position.

A Worked Numerical Example

Sequence: “the cat sat” — 3 tokens. Embedding dimension dₖ = 4.

Input embeddings (3 × 4):

X = [[0.9, 0.3, 0.1, 0.5],    # "the"
     [0.1, 0.8, 0.4, 0.2],    # "cat"
     [0.6, 0.1, 0.9, 0.3]]    # "sat"

In practice, Q, K, V are computed by multiplying X by learned weight matrices Wq, Wk, Wv. For simplicity, let’s use Q = K = V = X (which is what happens when those weight matrices are identity matrices).

Step 1: Scores = Q @ Kᵀ

Kᵀ (transposed, shape 4×3):
[[0.9, 0.1, 0.6],
 [0.3, 0.8, 0.1],
 [0.1, 0.4, 0.9],
 [0.5, 0.2, 0.3]]

Scores = X @ Kᵀ (shape 3×3):

scores[0,0] = 0.9×0.9 + 0.3×0.3 + 0.1×0.1 + 0.5×0.5 = 0.81+0.09+0.01+0.25 = 1.16
scores[0,1] = 0.9×0.1 + 0.3×0.8 + 0.1×0.4 + 0.5×0.2 = 0.09+0.24+0.04+0.10 = 0.47
scores[0,2] = 0.9×0.6 + 0.3×0.1 + 0.1×0.9 + 0.5×0.3 = 0.54+0.03+0.09+0.15 = 0.81

scores[1,0] = 0.1×0.9 + 0.8×0.3 + 0.4×0.1 + 0.2×0.5 = 0.09+0.24+0.04+0.10 = 0.47
scores[1,1] = 0.1×0.1 + 0.8×0.8 + 0.4×0.4 + 0.2×0.2 = 0.01+0.64+0.16+0.04 = 0.85
scores[1,2] = 0.1×0.6 + 0.8×0.1 + 0.4×0.9 + 0.2×0.3 = 0.06+0.08+0.36+0.06 = 0.56

scores[2,0] = 0.6×0.9 + 0.1×0.3 + 0.9×0.1 + 0.3×0.5 = 0.54+0.03+0.09+0.15 = 0.81
scores[2,1] = 0.6×0.1 + 0.1×0.8 + 0.9×0.4 + 0.3×0.2 = 0.06+0.08+0.36+0.06 = 0.56
scores[2,2] = 0.6×0.6 + 0.1×0.1 + 0.9×0.9 + 0.3×0.3 = 0.36+0.01+0.81+0.09 = 1.27

Scores:
[[1.16, 0.47, 0.81],
 [0.47, 0.85, 0.56],
 [0.81, 0.56, 1.27]]

Step 2: Scale by √dₖ = √4 = 2

Scaled scores:
[[0.58, 0.235, 0.405],
 [0.235, 0.425, 0.28],
 [0.405, 0.28, 0.635]]

Step 3: Softmax (row by row)

For row 0: [0.58, 0.235, 0.405]

e^0.58 = 1.786, e^0.235 = 1.265, e^0.405 = 1.500
sum = 4.551

weights[0] = [1.786/4.551, 1.265/4.551, 1.500/4.551]
           = [0.393, 0.278, 0.330]

For row 1: [0.235, 0.425, 0.28]

e^0.235 = 1.265, e^0.425 = 1.530, e^0.28 = 1.323
sum = 4.118

weights[1] = [0.307, 0.371, 0.321]

For row 2: [0.405, 0.28, 0.635]

e^0.405 = 1.500, e^0.28 = 1.323, e^0.635 = 1.887
sum = 4.710

weights[2] = [0.318, 0.281, 0.401]

Attention weights:
[[0.393, 0.278, 0.330],   "the" attends to: the(39%), cat(28%), sat(33%)
 [0.307, 0.371, 0.321],   "cat" attends to: the(31%), cat(37%), sat(32%)
 [0.318, 0.281, 0.401]]   "sat" attends to: the(32%), cat(28%), sat(40%)

Each position attends most strongly to itself (the diagonal), but also gathers information from other positions.

Step 4: Output = weights @ V

Using V = X:

output[0] = 0.393×[0.9,0.3,0.1,0.5] + 0.278×[0.1,0.8,0.4,0.2] + 0.330×[0.6,0.1,0.9,0.3]
          = [0.354,0.118,0.039,0.197] + [0.028,0.222,0.111,0.056] + [0.198,0.033,0.297,0.099]
          = [0.580, 0.373, 0.447, 0.352]

output[0] is now a blend of all three value vectors, weighted by how relevant each token is to “the”. The embedding for “the” has been enriched with contextual information from the whole sentence.

Causal Masking (GPT-style)

GPT generates text left to right. When predicting the next token, it must not be able to see future tokens. This is enforced with a causal mask: before the softmax, positions are blocked from attending to future positions by adding -∞ to those scores.

Masked scores for sequence "the cat sat":

[[0.58,  -inf, -inf],   "the" can only see itself
 [0.235, 0.425, -inf],  "cat" can see "the" and itself
 [0.405, 0.28,  0.635]] "sat" can see all three

softmax(-inf) = 0, so those positions get zero weight. The model genuinely cannot leak information from the future.

Multi-Head Attention

Running attention once captures one type of relationship. Multi-head attention runs h parallel attention computations (“heads”), each with its own learned Wq, Wk, Wv matrices, then concatenates and projects the results.

head_i = Attention(X·Wqᵢ, X·Wkᵢ, X·Wvᵢ)

MultiHead(X) = Concat(head_1, ..., head_h) · Wo

Different heads learn to attend to different types of relationships simultaneously:

One head might focus on syntactic dependencies.
Another on semantic similarity.
Another on positional proximity.

GPT-2 (small) uses 12 heads each of dimension 64, for a total dimension of 768.

Key Takeaways

Attention computes a context-aware representation of each position by attending to all other positions.
Q (query), K (key), V (value) are linear projections of the input embeddings.
Scaled dot-product attention: softmax(QKᵀ / √dₖ) · V.
Scaling by √dₖ prevents the dot products from growing too large and saturating softmax.
Causal masking prevents future positions from influencing past predictions — essential for autoregressive text generation.
Multi-head attention runs h attention heads in parallel, each learning different relationships.

The final post puts everything together into the full transformer architecture — and walks through exactly what GPT does on every forward pass.

The Transformer and GPT: Putting It All Together

Sun, 10 May 2026 00:00:00 GMT

You now have every mathematical tool needed to understand GPT. Vectors and matrices (post 1) move data. Derivatives and gradients (post 2) drive learning. Probability and softmax (post 3) produce predictions. Layers with activations (post 4) learn nonlinear functions. Backpropagation (post 5) trains all the weights. Embeddings (post 6) represent words. Attention (post 7) builds contextual representations.

This post assembles these pieces into the transformer architecture and walks through a complete forward pass.

The Transformer Block

The core unit of a transformer is the transformer block (also called a transformer layer). GPT stacks many of these in sequence.

Each block has two sublayers:

Multi-Head Self-Attention — lets positions attend to each other.
Feed-Forward Network — processes each position independently with a two-layer MLP.

Each sublayer is wrapped with two important additions:

A residual connection (add the input to the output).
Layer normalisation (applied before each sublayer).

The full computation for one block:

# Sub-layer 1: Multi-Head Attention
X' = X + MultiHeadAttention(LayerNorm(X))

# Sub-layer 2: Feed-Forward Network
X'' = X' + FFN(LayerNorm(X'))

Layer Normalisation

Layer normalisation stabilises training by ensuring the activations at each layer have a consistent distribution, regardless of the scale of the previous layer’s outputs.

For a vector x:

μ = mean(x)           # mean of all elements
σ = std(x)            # standard deviation

x_norm = (x - μ) / σ  # normalise

output = γ · x_norm + β   # learnable scale γ and shift β

γ and β are learned parameters (each a vector of length d_model). After normalising, the network can still learn to rescale and shift the distribution as needed.

Worked example (4-dimensional vector):

x = [0.5, -0.3, 1.2, 0.8]

μ = (0.5 - 0.3 + 1.2 + 0.8) / 4 = 2.2 / 4 = 0.55
σ = sqrt(((0.5-0.55)² + (-0.3-0.55)² + (1.2-0.55)² + (0.8-0.55)²) / 4)
  = sqrt((0.0025 + 0.7225 + 0.4225 + 0.0625) / 4)
  = sqrt(0.3025) = 0.55

x_norm = [(0.5-0.55)/0.55, (-0.3-0.55)/0.55, (1.2-0.55)/0.55, (0.8-0.55)/0.55]
       = [-0.091, -1.545, 1.182, 0.455]

# With γ = [1,1,1,1] and β = [0,0,0,0], output = x_norm

Without LayerNorm, deep networks become difficult to train because small changes in early layers compound through many subsequent layers, causing exploding or vanishing gradients.

Residual Connections

A residual connection adds the sublayer’s input directly to its output:

output = x + Sublayer(x)

This looks simple but has a profound effect: during backpropagation, gradients can flow directly from the output back to any earlier layer without being diminished by each intermediate layer. This is why transformers with dozens of layers can be trained at all.

Without residuals, training a 96-layer transformer (like GPT-3) would be practically impossible — gradients would vanish long before reaching the early layers.

The Feed-Forward Network

The FFN sublayer is a standard two-layer MLP applied independently to each position’s vector:

FFN(x) = ReLU(x · W₁ + b₁) · W₂ + b₂

(This post uses the row-vector convention x · W, common in transformer papers. It’s equivalent to Wᵀ @ x from the earlier posts — just with the matrix transposed.)

W₁ has shape (d_model, d_ff) — expands to a larger dimension.
W₂ has shape (d_ff, d_model) — projects back.
d_ff is typically 4 × d_model. In GPT-2 small: d_model = 768, d_ff = 3072.

The FFN is applied position-wise: each token’s vector goes through the same MLP independently. It does not mix information across positions — that’s attention’s job.

The expansion to 4×d_model gives the network capacity to store complex transformations. Research suggests the FFN layers act as associative memories, storing factual knowledge in their weights.

Positional Encoding

Attention has no built-in sense of position — the operation softmax(QKᵀ/√dₖ)V is identical regardless of token order. Position is injected by adding a positional encoding to each token’s embedding before it enters the transformer stack.

The original transformer paper (Vaswani et al. 2017) uses sinusoidal functions:

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Where pos is the token position and i is the dimension index.

Why sinusoids?

Each position gets a unique pattern of sine/cosine values.
The model can generalise to sequence lengths not seen during training.
Relative positions can be represented as linear transformations of absolute positions.

GPT-2 uses learned positional embeddings instead: a separate embedding matrix of shape (max_seq_len, d_model) where each row is a trainable vector. Both approaches work; learned embeddings are simpler.

The Full GPT Forward Pass

GPT is a decoder-only transformer. It takes a sequence of tokens and predicts the next token.

Let’s trace through GPT-2 Small (12 layers, 12 heads, d_model = 768):

Input: the sequence “The cat sat on the”

Step 1: Tokenise and embed

tokens = [464, 3797, 3332, 319, 262]   # BPE token ids
# 464="The", 3797="cat", 3332="sat", 319="on", 262="the"

# Look up token embeddings:
E = embedding_matrix[tokens]           # shape (5, 768)

# Add positional embeddings:
P = positional_matrix[[0,1,2,3,4]]     # shape (5, 768)

X = E + P                              # shape (5, 768)

Step 2: Pass through 12 transformer blocks

For each block i in [0..11]:

# Attention sublayer
X_norm = LayerNorm(X)                   # shape (5, 768)
attn_out = MultiHeadAttention(X_norm)   # shape (5, 768)
X = X + attn_out                        # residual

# FFN sublayer
X_norm = LayerNorm(X)
ffn_out = FFN(X_norm)                   # shape (5, 768)
X = X + ffn_out                         # residual

After 12 blocks, X still has shape (5, 768), but each token’s vector is now a rich contextual representation incorporating information from all other tokens.

Step 3: Final layer norm + linear projection

X_final = LayerNorm(X)                  # shape (5, 768)
logits = X_final @ W_unembed            # shape (5, 50257)

W_unembed has shape (768, 50257) — one column per vocabulary token. (In practice, GPT ties this to the transposed embedding matrix to save parameters.)

Step 4: Sample the next token

We only need the last position’s logits (predicting what comes after “the”):

last_logits = logits[-1]                # shape (50257,)
probs = softmax(last_logits / T)        # T = temperature (post 3)

next_token = sample(probs)              # e.g., "mat" (token 17680)

The model outputs “mat” — completing “The cat sat on the mat.”

Parameter Count

Let’s understand where GPT-2 Small’s 117M parameters come from:

Component	Parameters
Token embeddings	50,257 × 768 = 38.6M
Positional embeddings	1,024 × 768 = 0.8M
Per block: Attention (Q,K,V,O matrices)	4 × (768 × 768) = 2.4M
Per block: FFN (W₁, W₂)	(768×3072) + (3072×768) = 4.7M
Per block: LayerNorms	≈ 0.003M
12 blocks × (2.4 + 4.7)M	85.2M
Final LayerNorm + unembedding	≈ 0.6M
Total	≈ 117M

GPT-3 uses 175 billion parameters — the same architecture, scaled up: 96 layers, 96 heads, d_model = 12,288. Every forward pass still executes the same sequence of operations you now understand.

What Training Actually Does

During training on a large text corpus, the loss is the average cross-entropy across all positions (here N is sequence length):

L = -(1/N) Σₜ log P(tokenₜ | token₁, ..., tokenₜ₋₁)

For each training step:

Run a forward pass on a batch of sequences.
Compute the cross-entropy loss.
Run backpropagation to compute gradients for all 117M parameters.
Update all parameters with gradient descent (usually Adam optimiser).

After training on hundreds of billions of tokens, the weights encode enough statistical structure about language that the model can answer questions, write code, translate, summarise, and reason — all from next-token prediction.

Key Takeaways

A transformer block = MultiHeadAttention + FFN, each wrapped with LayerNorm and a residual connection.
Layer normalisation stabilises training by keeping activations at a consistent scale.
Residual connections allow gradients to flow directly to early layers, making deep networks trainable.
The FFN sublayer applies a two-layer MLP at each position independently, with hidden dimension 4×d_model.
Positional encoding injects position information by adding sinusoidal or learned vectors to token embeddings.
GPT is decoder-only: it predicts the next token autoregressively using a causally masked transformer stack.
One forward pass = embed → position encode → N×(LayerNorm+Attention+Residual+LayerNorm+FFN+Residual) → linear → softmax.

You now have the complete mathematical picture of how GPT works, built from nothing but addition, multiplication, and the chain rule. Every “emergent” capability in a language model traces back to these operations, repeated billions of times on billions of tokens.

People, Character, and the Foundation of Learning

Thu, 07 May 2026 00:00:00 GMT

Disclaimer: This post is part of a personal learning series based on Naval Ravikant’s “How to Get Rich” transcript, available at nav.al/rich. The ideas and frameworks here belong to Naval — I’m working through them for my own understanding. This is not financial advice.

Strategy, leverage, and specific knowledge are built individually. But wealth at any meaningful scale is built with people, over time. The final layer of Naval’s framework is about who to build with, how trust compounds when you play long games, and how to develop the intellectual foundation that makes everything else possible.

Play Long-Term Games With Long-Term People

Naval’s observation about compound interest extends far beyond money: all the benefits in life come from compound interest — in relationships, in knowledge, and in reputation, not just in investments.

The reason long-term games compound is game theory. In a one-shot interaction, the rational move is often to defect — extract maximum value and move on, because there are no future rounds to lose. In a repeated game with the same people, the rational move is to cooperate, because both sides know the game continues. Trust develops precisely because both parties have skin in future iterations.

This dynamic is what Naval points to when describing Silicon Valley’s productivity. Long-term geographic proximity — encountering the same people across companies, across deals, across decades — changes how people behave toward each other. Reputation follows you. Defection in round one affects every subsequent round. The result is a tit-for-tat equilibrium where cooperation is the dominant strategy, not because of idealism but because of rational self-interest across a long time horizon.

In long-term games, everyone makes each other wealthier — positive sum. In short-term games, people divide what already exists — zero sum. The type of game you’re playing determines what strategy is rational.

The corollary is the cost of switching too frequently: every time you reset your context — new industry, new city, new network — you start compounding from zero. The trust you’ve built, the reputation you’ve earned, the relationships that took years to develop don’t transfer. They compound in place. People who switch industries and geographies repeatedly never benefit from the long tail of compounding that comes from sustained presence in one domain and community.

Pick Partners With Intelligence, Energy, and Integrity

Naval’s partner selection framework is three non-negotiable traits:

Intelligence — seeing problems clearly, acting in the right direction, making sound decisions under uncertainty.

Energy — actually executing. Showing up, doing the work, sustaining effort over the duration needed for things to compound.

Integrity — not cheating you when it becomes advantageous to do so.

All three are required. Any two without the third is dangerous. High intelligence and high energy without integrity produces “a smart and hardworking crook who’s eventually going to cheat you.” The smarter and more energetic they are, the more thoroughly they can cause damage when their interests diverge from yours. Integrity is the constraint that makes the other two safe to rely on.

The difficulty is that integrity is the hardest trait to verify upfront. People present well in early interactions. The tell is how they behave when they think no one is watching: how they treat people with less power than them, how they handle situations where cutting corners would benefit them, whether they hold grudges out of proportion to the offence, whether they speak badly about people who aren’t in the room.

These patterns are consistent. The person who treats a waiter dismissively will eventually treat you dismissively. The person who rationalises cheating a previous employer will find a rationalisation for cheating you when the circumstances align. You cannot manufacture high-integrity behaviour through incentives. It either exists in someone’s character or it doesn’t.

Intrinsic Motivation Cannot Be Manufactured

A related principle: you cannot sustainably motivate another person through external pressure. Motivation has to come from within. This matters for choosing collaborators because people who are genuinely excited by what they’re building — who would work on it regardless of the specific compensation — sustain output over the long run in a way that externally motivated people don’t.

External incentives produce performance while the incentive is present. Remove it and performance returns to the intrinsic baseline. If that baseline is low, you’ve built a dependency rather than a partnership.

The practical consequence: look for people who find the problems genuinely interesting, not primarily those drawn by the package. Their motivation compounds over time. Everyone else’s degrades.

Partner With Rational Optimists

Pessimism is the comfortable position. Pointing out why something won’t work requires no execution and insulates you from the embarrassment of trying and failing publicly. Naval identifies the specific failure mode: the cynic who won’t lead, won’t follow, and won’t get out of the way — blocking progress without contributing any direction.

The alternative isn’t naive optimism — ignoring real risks and charging forward on enthusiasm. It’s rational optimism: seeing reality clearly while maintaining confidence in your ability to navigate it.

“Know all the pitfalls. Know the downsides, but still keep your chin up.”

Naval grounds this in evolutionary reasoning. Humans evolved as pessimists — acute threat sensitivity kept our ancestors alive. But the risk structure of the modern world is fundamentally different: limited downside, potentially unlimited upside. Taking a bet that costs you a year of effort but could generate decade-long returns is rational. The evolved pessimism response — avoid the bet — is no longer calibrated to the actual risk environment.

The only thing genuinely worth avoiding is ruin: paths that lead to catastrophic, irreversible loss — bankruptcy, jail, permanent reputation damage. Everything short of that is recoverable. Rational optimists understand this distinction and take the bets. Pessimists avoid all of them to preserve the feeling of safety.

One more observation: successful people “just do things.” The fastest way to determine whether something is viable is to try it. Doing is faster than planning, and far faster than watching.

Read What You Love Until You Love to Read

The intellectual foundation of the entire framework is reading — not as a performance but as a compounding habit that changes how you think over years and decades.

The barrier most people face isn’t access to books. It’s that they haven’t built the habit itself. Naval’s solution to the catch-22: read what you love until you love to read. Start with fiction if that’s what holds attention. Move to science fiction, then narrative non-fiction, then denser material, then primary scientific and philosophical texts. Follow genuine interest rather than a curated syllabus.

What to read once the habit exists matters. Naval is specific: read foundational texts over modern commentary. Adam Smith’s Wealth of Nations rather than business school summaries. Darwin’s Origin of Species rather than popular science books about evolution. Feynman’s Six Easy Pieces rather than cosmology written for a general audience.

The reason is fidelity. Every layer of commentary filters, selects, and interprets what came before. Reading the original means encountering the argument as its author constructed it — not as someone else decided to summarise it. You build mental models from primary sources, which are defensible from first principles. Memorised conclusions from secondary sources are not.

Depth over volume. It is better to read one book slowly, struggling with difficult passages, sitting with ideas until they become part of how you think, than to skim fifty books and accumulate the count. The goal is not to have read things — it’s to understand them well enough that the insight is available when you need it. That only happens through genuine engagement.

Mathematics and Logic Are the Bedrock

Underlying everything else: mathematics and logic. Once you’ve internalised mathematical thinking, you can evaluate any field’s claims from first principles rather than memorising expert opinions. You can follow a statistical argument to its assumptions, identify what’s being asserted versus what’s been demonstrated, and separate truth from opinion dressed as fact.

This is the difference between genuine understanding and accumulated information. Someone with mathematical fluency reads a study and evaluates the methodology. Someone without it has to trust the presenter’s summary — which may or may not be accurate.

Five skills Naval considers foundational for operating effectively in the current world: reading, writing, arithmetic, persuasion, and computer programming. Not a specific domain. Not a specific credential. Five general capabilities that make every other form of learning faster and every other skill more deployable. Master these and the specific content of your career remains flexible across decades.

The full framework, taken together: define wealth correctly, engineer the conditions for luck, build ownership not just income, develop specific knowledge through genuine curiosity, deploy it through permissionless leverage, play long games with high-integrity people, and compound your understanding through deep reading. None of it is a shortcut. All of it compounds.

Part 5 of 5. Start from the beginning with Part 1: Wealth, Money, and Status.

Full original transcript: nav.al/rich

Leverage, Scale, and What the Internet Changed

Wed, 06 May 2026 00:00:00 GMT

Disclaimer: This post is part of a personal learning series based on Naval Ravikant’s “How to Get Rich” transcript, available at nav.al/rich. The ideas and frameworks here belong to Naval — I’m working through them for my own understanding. This is not financial advice.

Specific knowledge tells you what to build. Leverage tells you how that knowledge can generate returns that dwarf the time you put in. Without leverage, even exceptional skill produces linear returns — more hours, proportionally more output. With it, one unit of effort can multiply into thousands. This is the mechanism that separates income from wealth.

Give Society What It Doesn’t Know How to Get

Before getting to leverage itself, it helps to understand what you’re deploying it toward. Naval’s framing for identifying opportunity: find things society wants but doesn’t yet have a reliable way to obtain.

If society already knew how to produce something reliably, it would already exist at scale. The gap between what people want and what they can currently get is where value is created. Oil made Rockefeller wealthy because it solved lighting and energy problems that no existing infrastructure addressed. Cars made Ford wealthy because affordable personal transport didn’t exist. Each era has its equivalent gaps.

Creation alone isn’t enough though — distribution matters equally. Uber didn’t invent private drivers. It removed the friction in accessing them and made that access affordable at scale. The wealth came from solving both the product and the distribution problem. When you find your gap, the question isn’t just “can I build this?” but “can I scale it to the people who need it?”

The Four Types of Leverage

Leverage is getting outputs disproportionate to your inputs. Naval identifies four types, and understanding the differences between them matters practically.

Labour leverage — other people working on your behalf. The oldest form of leverage. A general commanding an army, a founder building a team. It scales, but slowly and expensively. Every person added requires hiring, onboarding, motivating, and coordinating. Managing people is a skill in itself, and it has hard limits. At some point, communication overhead consumes the gains.

Capital leverage — money working on your behalf through investments. More scalable than labour: once deployed, capital doesn’t need management in the same way. But it requires capital to start with, and the returns are heavily competed over by people with access to the same markets and information. Capital leverage alone tends to compress toward market returns over time. It’s powerful, but not distinctive.

Code leverage — software running on your behalf at essentially zero marginal cost per additional user. Write a program once; it runs for millions of people simultaneously. This is qualitatively different from labour and capital. The millionth user costs nothing more than the first. A single engineer’s output can serve a global user base without any proportional increase in effort. The upside is uncapped in a way that neither labour nor capital leverage achieves.

Media leverage — content working on your behalf. A blog post, a book, a podcast, a video — created once, distributable indefinitely. Joe Rogan records one conversation; it reaches tens of millions of people. The creator is asleep while the content is being consumed. Like code, the marginal cost of an additional listener or reader is zero.

Permissionless Leverage Is the Real Unlock

Naval draws a distinction that matters more than the specific type: permissionless leverage vs. leverage that requires permission.

Labour and capital leverage require permission. To hire, you need resources and someone willing to work for you. To deploy capital, you need capital — which means either earning it first or convincing someone else to give it to you. These require gatekeepers to say yes.

Code and media are permissionless. You don’t need someone’s approval to write software. You don’t need a publishing contract to write content. You don’t need an investor to start a YouTube channel or a blog. The barrier to accessing these forms of leverage is effort and skill, not permission.

This is new. For most of human history, meaningful leverage required institutional access — a factory, a printing press, a radio station, a bank. Those were expensive and controlled. The internet made code and media leverage available to anyone with a laptop and skill. This is the structural change that makes individual wealth creation more achievable now than at any point before.

The practical consequence: if you’re choosing where to develop skills, weight heavily toward domains where your output can be distributed digitally at zero marginal cost. Writing, software, video, audio. These are the leverage vehicles that don’t require permission to access.

What the Internet Did to Career Possibilities

Before the internet, your potential market was bounded by geography. Specialised expertise could only reach the people physically near you — your city, your industry, your local network. Most niche knowledge generated limited returns because the audience for it was too small in any one place to sustain a career.

The internet removes that constraint entirely. Any human is now reachable by any other human. Any niche obsession — obscure historical research, hardware hacking, competitive chess strategy, miniature cooking — can reach its global audience. The passionate 50,000 people worldwide who would pay for what you know are now findable. That number is often more than enough to build a sustainable livelihood around.

This has permanently expanded what a viable career looks like. E-sports players earn competitive salaries. Independent creators build audiences larger than cable news networks. Niche newsletter writers support themselves entirely through subscribers they’ve never met. None of these were structurally possible before the internet dissolved the geographic constraint on distribution.

The other change: the economics of niche are now better than the economics of generic. Serving a small passionate audience is more sustainable than competing for attention in a crowded mainstream category. Generic content fights for attention against everyone. Specific content finds the people for whom it is exactly right.

Authenticity Eliminates Competition

The usual advice for building an audience or a business is to find an underserved market and position competitively. Naval’s observation cuts past this: escape competition through authenticity.

Every human is a unique intersection of knowledge, experience, curiosity, and personality. No one else has yours. If you build around what you genuinely are — rather than optimising for what seems most commercially validated — you become irreplaceable by definition. The specific combination of you cannot be replicated, so competition converges to zero.

The trap is the opposite instinct: looking at what’s working for someone else and copying the format, the style, the positioning. That path leads directly into competition with the original, on their terms, where they have the advantage of being first and authentic to the approach. The thing that made them successful is that it’s genuinely theirs. Copying it makes you the imitation.

This is harder in practice than in principle because authenticity requires resisting a strong social pull. When something works for others, the obvious move is to do the same thing. Resisting that and building from your actual interests and strengths is the thing that generates genuine differentiation.

Taken together: find the gap society needs filled, build specific knowledge around it, deploy it through permissionless leverage (code or media), reach a global niche through the internet, and let your authentic combination of skills and perspective eliminate direct competition. Each element enables the others.

Part 4 of 5. Read Part 3: Specific Knowledge Is Your Moat or continue to Part 5: People, Character, and the Foundation of Learning.

Specific Knowledge Is Your Moat

Tue, 05 May 2026 00:00:00 GMT

Disclaimer: This post is part of a personal learning series based on Naval Ravikant’s “How to Get Rich” transcript, available at nav.al/rich. The ideas and frameworks here belong to Naval — I’m working through them for my own understanding. This is not financial advice.

Once you understand what wealth is and why ownership matters, the next question is: what do you actually build expertise in? Naval’s answer is specific knowledge — and the definition is precise enough to be actionable.

What Specific Knowledge Is (And Isn’t)

Specific knowledge is expertise that cannot be formally taught, and therefore cannot be mass-produced, commoditised, or automated away.

The distinction Naval draws: everything can be learned, but not everything can be taught. Specific knowledge is developed through lived experience, genuine obsession, and working at the edge of a domain — not through standardised curricula. If a skill can be taught in a classroom, it can be certified, systematised, scaled, and eventually wage-compressed. The moment a skill becomes teachable at scale, it loses its economic scarcity.

This is why conventional career advice — study the high-paying field, get the credential, join the sector — often leads to exactly the outcome it promises initially but no more. You end up in a competitive pool of similarly credentialled people, competing on marginal differences.

Specific knowledge sits outside that pool because it can’t be replicated through the same pathway that created it in you.

How to Find Yours

The path Naval describes isn’t strategic — it’s introspective.

“Specific knowledge is found much more by pursuing your innate talents, your genuine curiosity, and your passion.”

The signal isn’t what sounds impressive or pays well. It’s what feels like play to you while appearing as work to others. If a domain pulls your attention regardless of external reward — if you’d explore it whether or not anyone paid you — that’s where specific knowledge forms.

There’s also an observational pattern: specific knowledge is often noticed by people who know you well before you’ve identified it yourself. Naval mentions his mother observing his entrepreneurial nature as a child — critiquing pizza parlour operations, noticing inefficiencies. He hadn’t framed it that way. She had. The people closest to you often see your specific knowledge before you do.

The practical test: find the intersection of what you’re obsessively curious about and what you can get better at faster than most people around you. That’s the seed.

Skill Stacking

You don’t have to be the world’s best at one thing. Naval references Scott Adams’ framework: being in the top 25% at three different skills creates a competitive advantage that doesn’t exist if you’re pursuing any one of them in isolation.

The combination matters more than the individual rank. A software engineer who is also an excellent communicator and has deep domain knowledge in a specific industry is rarer than three separate people occupying those positions. When the need for that intersection arrives, you’re the only candidate.

This also means specific knowledge can be assembled over time, across roles and domains, through genuine curiosity rather than linear credential accumulation. The unconventional path creates the unconventional combination.

Build and Sell — The Most Powerful Combination

Naval identifies the two fundamental skills of wealth creation:

Building — creating products, writing code, designing systems, making things that work
Selling — distributing, communicating, recruiting, fundraising, persuading

The most successful companies pair these in complementary founders. Jobs and Wozniak. Gates and Allen. One who builds deeply, one who sells effectively. The pairing works because each skill is genuinely hard, and most people are strong in at most one.

But one person who masters both becomes, in Naval’s word, “unstoppable.” Elon Musk understands rockets well enough to make credible commitments. Steve Jobs developed product intuition alongside distribution genius. Marc Andreessen programmed Netscape, then built the firm that funds the next generation.

The sequence Naval recommends: learn to build first, then learn to sell. Building requires sustained focus and is harder to acquire late. Selling — defined broadly as writing, marketing, communication, and persuasion — can be layered on later, particularly for people who are naturally communicative.

He goes further: “I’d rather teach an engineer marketing than a marketer engineering.” Technical depth provides better initial differentiation. Engineers who develop sales skills scale better over time than non-technical salespeople who attempt to understand what they’re selling.

Selling also doesn’t require hand-to-hand interaction. Writing is a form of sales. A clear technical blog post, a well-argued README, a compelling product page — these are all distribution. Engineers uncomfortable with traditional sales often find leverage through writing, which scales without continuous personal effort.

There Is No Skill Called “Business”

Naval is direct about this: “business” is too broad to mean anything, in the same way that “relating to humans” is too broad to be a skill.

Business schools teach case studies — anecdotes presented as analysis. The problem with anecdotes is that they can be assembled to justify almost any conclusion. Case study education produces pattern-matched intuition that breaks down in novel situations, which is precisely where most interesting business challenges arise.

The fields worth studying instead:

Microeconomics — how individuals and firms make decisions under constraints
Game theory — how strategic actors behave when outcomes depend on others’ choices
Psychology — how people actually think and make decisions (rather than how they should)
Persuasion — the mechanics of changing minds
Ethics — not as philosophy but as a guide to sustainable decision-making
Mathematics and logic — the foundation that lets you evaluate any domain’s truth claims
Computing — the leverage tool of the current era

These are generative rather than descriptive. Understanding microeconomics lets you reason about any market. Understanding game theory lets you think through any negotiation. Understanding psychology applies everywhere. Knowing a specific case study from 1995 applies almost nowhere.

The deeper principle: doing is faster than watching. Building something, even something small, teaches more than an equivalent amount of time spent reading about what others built. First-principles understanding — built through doing — is defensible from any angle. Memorised facts from case studies are not.

Part 3 of 5. Read Part 2: The Luck You Can Engineer or continue to Part 4: Leverage, Scale, and the Internet.

The Luck You Can Engineer

Mon, 04 May 2026 00:00:00 GMT

Disclaimer: This post is part of a personal learning series based on Naval Ravikant’s “How to Get Rich” transcript, available at nav.al/rich. The ideas and frameworks here belong to Naval — I’m working through them for my own understanding. This is not financial advice.

The phrase “how to get rich without getting lucky” sounds like a contradiction. But Naval’s argument is more precise than it first appears — not that luck doesn’t exist, but that there are four distinct types, and most of the ones that matter are ones you can engineer through deliberate effort.

Four Types of Luck

1. Blind luck. Pure chance. Lightning strikes, you happen to be in the right place. You have no influence over it, and it’s the least interesting type for building wealth — you can’t optimise for it.

2. Luck from hustling. You increase your exposure to opportunity by staying in motion — shipping projects, taking meetings, reaching out, trying things. Naval uses the image of stirring a petri dish: the more you stir, the more reactions you create. More activity expands your surface area for things to happen. This type is fully controllable.

3. Luck from preparation. You develop expertise deep enough that you recognise opportunities others miss. Two people encounter the same situation; one sees nothing, the other sees a breakthrough. The difference is pattern recognition built from sustained, focused work. Preparation doesn’t create luck — it lets you catch it when it passes.

4. Luck from character. You build a specific reputation and identity over time until opportunities seek you out. If you become known as the world’s best expert in a narrow domain, people with relevant problems will find you. The opportunity arrives because of who you are, not what you happened to be doing that day.

The progression is the point. Each type is more controllable than the last. The fourth type becomes so reliable it stops functioning like luck at all.

The 999 Out of 1,000 Framework

Naval frames the goal as: in 1,000 parallel universes, you want to be wealthy in 999 of them. Not all of them — some unlucky outcomes are genuinely unavoidable. But 999 out of 1,000.

That target requires building yourself so thoroughly that luck becomes marginal. The fourth type — character-driven luck — starts becoming deterministic. “Your character becomes your destiny.” When an opportunity arrives that requires your specific skills and reputation, you’re the only plausible candidate. The randomness is gone.

This connects character to outcome in a way most frameworks miss. Getting rich without getting lucky means building the version of yourself for whom wealth is a natural consequence, not a fortunate accident.

You Won’t Get Rich Renting Your Time

Even with the right kind of luck, you need the right vehicle. Salaried employment is the wrong one.

The structural problem is simple: inputs match outputs directly. Sleep, and you earn nothing. Vacation, and you earn nothing. Your output is capped by your hours, your hours are capped by your lifespan, and there is no mechanism for non-linear returns.

The deeper problem is replaceability. When you take a salaried role, you agree to perform a defined set of functions. Defined functions can be systematised, taught, and eventually automated. You’re filling a role, not expressing a unique capability. Your replacement is always findable.

Contrast this with leverage: one engineer creates Bitcoin and generates billions in value. Another engineer writes code that goes unused despite identical hours invested. The inputs are the same. The outputs differ by orders of magnitude. Salary compresses this variance deliberately — your pay doesn’t fluctuate with actual impact. That’s comfortable, and it’s also the mechanism that makes wealth creation through employment alone structurally impossible.

Ownership Is the Vehicle

The alternative is ownership — equity in something with productive potential. Stock options. Founding a company. Building something that generates returns without your continuous presence. Investing in assets.

Naval is unambiguous: “You must own equity, a piece of a business, to gain your financial freedom.” Not a higher salary. Not a better title. Equity — because equity is where upside is uncapped.

A salary has a ceiling negotiated in advance. Equity reflects what the business actually creates. If the business generates disproportionate value, your ownership stake reflects that. The ceiling doesn’t exist.

This doesn’t mean abandoning employment immediately. It means orienting every career decision toward increasing ownership of something — joining early-stage companies for equity rather than established ones for salary, building side projects with compounding potential, making investments. The vehicle matters as much as the effort.

Live Below Your Means — Preserving Optionality

There’s a companion principle: don’t inflate your lifestyle as your income rises.

Naval calls a monthly salary “one of the most dangerous drugs” — not because income is bad, but because predictable income trains you to spend up to it. As income rises, lifestyle expectations rise proportionally. The result is higher income with the same financial freedom — which is none. You’re running faster on a treadmill that also gets faster.

The alternative pattern is making money in discrete lumps separated over long periods, so your lifestyle baseline doesn’t have the chance to adapt. A founder who earns nothing for several years and then has a large exit hasn’t built the spending habits that a steadily rising salary would have created. They have optionality that a well-compensated employee — who spent every dollar as it arrived — doesn’t.

Freedom comes from the gap between what you earn and what you spend. Closing that gap through consumption eliminates the resource that funds the ownership path in the first place.

Accountability Is the Other Side of Ownership

Ownership requires accountability — your name attached to outcomes, good or bad. This discomfort is why most people avoid it. The downside is visible and real.

But accountability is also what makes ownership effective. When outcomes are tied to you specifically, you take the kind of care over your work that generates the best results. And it’s a market signal: founders with skin in the game attract better collaborators than those executing for a salary with no exposure to downside.

The combination — equity ownership, accountability, specific skills, lifestyle below means — is what separates the wealth track from the income track. Each element enables the others.

Part 2 of 5. Read Part 1: Wealth, Money, and Status or continue to Part 3: Specific Knowledge Is Your Moat.

Wealth, Money, and Status: Getting the Definitions Right

Sun, 03 May 2026 00:00:00 GMT

Disclaimer: This post is part of a personal learning series based on Naval Ravikant’s “How to Get Rich” transcript, available at nav.al/rich. The ideas and frameworks here belong to Naval — I’m working through them for my own understanding. This is not financial advice.

Most conversations about money collapse three distinct things into one word. Naval Ravikant’s framework starts by separating them cleanly, because until you define what you’re actually chasing, you can’t build a coherent strategy to get there.

Wealth Is Assets That Earn While You Sleep

Naval’s definition is precise: wealth is assets that earn while you sleep — businesses, investments, intellectual property. Not your salary. Not your net worth as a number on a screen. Productive assets that generate returns independently of your time.

This definition rules out money earned by showing up. It rules out status. It points exclusively at things that compound without your direct presence. A business you own that generates profit while you’re on holiday is wealth. A bonus you earned by working 80-hour weeks is not — you traded time for it once and it won’t repeat without more time.

The goal behind wealth, in his framing, isn’t luxury. It’s freedom:

“So you don’t have to wear a tie like a collar around your neck. So you don’t have to wake up at 7:00 a.m. to rush to work.”

Sovereignty over your own time. That is the operative target.

Money is the transfer mechanism for wealth — a claim on human effort that can be redeemed for goods, services, or productive assets. When you earn a salary, you accumulate transferable claims on society. Spend them on consumption and they’re gone. Convert them into assets and they compound.

The critical insight is that money is a means, not an end. People who confuse accumulating money with building wealth often end up with large account balances and no passive income. They have stored claims but nothing producing new ones.

Status Is Zero-Sum

Status is your rank within a social hierarchy. Unlike wealth, it is strictly zero-sum. For you to rise in status, someone else must fall. Every status point gained is, by definition, a status point lost by someone else.

Wealth creation is the opposite — positive-sum. A software product built by one engineer can serve millions of people simultaneously. The engineer becomes wealthier, users get something useful, and no one loses anything in the exchange. Value was created, not redistributed.

This distinction explains a real pattern: people who attack successful individuals are almost always playing status games, not engaging with economic reality. Dragging down someone wealthy does not make you wealthier. It moves someone lower in the hierarchy, which moves you up relatively. Naval is direct about this — it’s status-seeking disguised as moral positioning.

Recognising the pattern means you can ignore the attacks. They’re irrelevant to building actual assets.

Ethical Wealth Creation Makes the World Better

Naval rejects the idea that wealth is extracted from others. Historical wealth creation — antibiotics, electricity, cars, smartphones — generated genuine abundance. The people who built those things got rich, and the world got better. Neither outcome was at the expense of the other.

His thought experiment: if everyone had the engineering skills to build software and hardware, society would achieve enormous abundance with minimal required labour. Wealth creation at its core is building things that solve problems at scale. The builder captures a fraction of the value generated for others; the rest belongs to the world.

The corollary is what happens when the balance tips. When takers outnumber makers, productive capacity shrinks and systems deteriorate — not because wealth is unfair, but because the incentive to create disappears.

Free Markets Emerge From Human Nature

Naval argues that voluntary exchange and credit-debit tracking are evolutionary traits, not political inventions. Humans uniquely cooperate across genetic lines — we form relationships, track obligations, and trade with strangers because we developed the capacity to maintain mental ledgers of who owes what to whom.

Markets formalise this instinct. The mechanism by which billions of people coordinate effort and allocate resources without central direction is not a social construct imposed from above — it’s the formalisation of something humans already do naturally.

What This Means in Practice

Before building any strategy, it helps to be clear about what you’re actually trying to build:

Wealth — assets earning independently of your time. The actual goal.
Money — the transfer medium. Useful, but transient unless converted into assets.
Status — social hierarchy rank. Zero-sum, mostly irrelevant to financial freedom.

The game worth playing is wealth creation. Misaligning on these definitions leads to building the wrong things — engineers optimising for prestigious titles instead of equity, founders chasing recognition instead of revenue, professionals maximising salary instead of ownership. All of them are playing a version of the status game while believing they’re on the wealth track.

The framework starts here because everything else — leverage, specific knowledge, long-term games — only works if the target is right.

This is Part 1 of 5. Continue with Part 2: The Luck You Can Engineer.

B-Trees and B+ Trees: From Binary Trees to Database Indexes

Sat, 02 May 2026 00:00:00 GMT

You already understand binary search trees. Each node has one key, two children, the left subtree is always smaller, the right always larger. For a tree with a million nodes, you find any key in about 20 comparisons.

That sounds fast. At in-memory speeds it is. At disk speeds, it kills you.

Why BSTs Break at Disk Scale

Every time you descend a level in a BST, you are potentially reading from a different location on disk. In RAM, that is a cache miss — nanoseconds. On a spinning disk, that is a random seek (a seek is the disk arm physically moving to a new track on a hard drive) — milliseconds. A million-node BST has ~20 levels. Twenty disk reads per query. At 5ms per seek, that is 100ms before you return a single row.

A page is the fixed-size block (typically 4–16 KB) that databases read from disk in a single I/O — we’ll see why this matters in a moment.

The fix is not to speed up disk. The fix is to read fewer levels.

If each node could hold 1,000 keys instead of 1, the tree would be 1,000x shallower. A million-row table needs only 2 levels. A billion-row table needs 3.

That is the entire idea behind B-trees.

B-Trees: BSTs Gone Wide

A B-tree generalizes the BST by allowing each node to hold many keys and have many children. The sorted invariant is identical — it just applies across more keys per node.

Formal rules for a B-tree of order t:

Each node holds between t-1 and 2t-1 keys (the root can have as few as 1)
Each internal node has exactly one more child than it has keys
All leaves sit at the same depth — always
Keys within a node are sorted ascending

Here t is the tree’s minimum degree — the lower bound on keys per node. The branching factor (children per internal node) is up to 2t.

For t=2 (the smallest B-tree allowed by the definition, also called a 2-3-4 tree), each node holds 1 to 3 keys and has 2 to 4 children.

Here is what a B-tree of order t=2 looks like:

The same BST invariant holds: every key in n1 is less than 30, every key in n2 is between 30 and 70, every key in n3 is greater than 70. It is just a wider tree.

In a B-tree, internal nodes store both keys and the actual data values. You can hit your target at any level of the tree.

Search: Same Idea, Wider Nodes

Search works identically to a BST. At each node, scan the keys left-to-right to find which child to follow (or return if you find the key directly).

Find key 50:

At root [30 | 70]: 50 > 30 and 50 < 70 → take middle child
At [40 | 50 | 60]: scan left to right, find 50 → return value

Two disk reads. In a BST of 9 nodes that is also two reads — but the B-tree scales to millions of nodes while still needing only 2–3 reads. The branching factor is the lever.

The per-node scan is O(t) (or O(log t) with binary search), and tree height is O(log of n in base t) — so total work simplifies to O(log n) since t is a constant for any given database. The win is entirely in the constant — fewer levels means fewer disk reads.

Insertion: Splits Keep the Tree Balanced

You always insert at a leaf. Walk down using the same search logic, land at the right leaf, insert the key in sorted position.

The problem: a leaf can overflow — it may already have 2t-1 keys (the maximum). The fix is to split the overfull node before inserting into it.

Splitting takes a full node with 2t-1 keys and produces:

A left child with the first t-1 keys
A right child with the last t-1 keys
The middle key, which is promoted up to the parent

Example: insert 55. The right child is full and must be split first.

Before:

Split [40 | 50 | 60]. Middle key 50 is promoted to the parent:

Now insert 55 into right child [60]:

The critical detail: split on the way down, not on the way up. Pre-splitting full children means we never need to walk back up to handle parent overflow — a one-pass algorithm that also simplifies concurrent locking.

If the root itself fills and splits, a new root is created holding only the promoted middle key. This is the only way a B-tree grows taller — always from the top.

Deletion: Three Cases

Deletion mirrors insertion. You always want to delete from a leaf, and no node can drop below t-1 keys (the minimum).

Case 1 — Leaf has more than the minimum: just remove the key.

Case 2 — Key is in an internal node: replace it with its in-order predecessor (the largest key in the left subtree) or in-order successor (the smallest key in the right subtree), then delete that key from the leaf. Same trick as BST deletion.

Case 3 — The leaf is at the minimum (t-1 keys): fix the underflow before deleting. Two options:

Borrow from a sibling that has a spare key. The sibling donates a key up to the parent; the parent sends its separator key down to the underflowing node. This is sometimes called a rotation — distinct from the AVL/red-black rotations used to rebalance binary search trees.
Merge with a sibling when no sibling has a spare. Pull the parent’s separator key down and combine two underfull siblings into one valid node. The parent loses a key — which might trigger a cascade upward.

If the root loses its last key during a merge, the root is removed and the merged child becomes the new root. This is the only way a B-tree shrinks — always from the top.

B+ Trees: The Database Standard

B+ trees make one structural change to the B-tree that turns out to be decisive: internal nodes store only keys, not values. All actual data lives exclusively at the leaf level.

Two things stand out immediately.

Internal nodes are denser. Without values, they hold far more keys per disk page. A 16KB page with 8-byte keys and 8-byte child pointers fits roughly several hundred to ~1,000 keys after accounting for page header and metadata. That is the branching factor. A 3-level tree now covers 1,000³ = one billion rows. The same tree in a standard B-tree, carrying 100-byte records in every node, might only branch by 100 — covering one million rows.

Leaf nodes form a sorted linked list. Every leaf has a pointer to the next leaf in key order. This is the feature that makes databases fast for the queries they actually run.

Range Queries: The Decisive Advantage

A point lookup (WHERE id = 42) is equally fast on a B-tree and a B+ tree — navigate to the leaf, return the value.

A range query (WHERE age BETWEEN 25 AND 55) is where B+ trees dominate.

In a B-tree, matching keys can be scattered across internal and leaf nodes. You must traverse the tree for each one.

In a B+ tree:

Navigate to the first leaf containing 25 — O(log n), one tree traversal
Follow the linked list rightward, collecting every key until you pass 55

No backtracking. No re-traversing the tree. The linked list turns a range query into a sequential scan of exactly the relevant pages. Sequential reads are 10–100x faster than random reads on spinning disks; even on SSDs the gap is 2–10x and prefetching makes sequential scans dramatically cheaper.

This is why every major relational database — PostgreSQL, MySQL InnoDB, SQLite, SQL Server — uses B+ trees as the primary index structure.

One Structural Difference: Promoted vs. Copied Keys

When a node splits in a B-tree, the middle key moves up — it leaves the child and lives only in the parent.

When a node splits in a B+ tree, the middle key copies up — it stays in the right leaf and also appears in the parent as a routing guide.

The leaf [20 | 30] in the B+ tree still contains 20, even though 20 also appears in the parent. Internal nodes are routing indexes, not the authoritative location of data. All data lives in the leaves.

Numbers That Put It in Perspective

Assume a 16KB page, 8-byte keys, 8-byte pointers, 100-byte records:

Structure	Keys per internal node	Rows covered at 3 levels
BST	1	7
B-tree (t=50)	~99	~970,000
B+ tree	~1,000	~1,000,000,000

B+ trees get that extra order of magnitude because internal nodes never carry record data — every byte goes toward branching. At a billion rows and 3 levels, you need exactly 3 disk reads per point lookup. In practice the upper levels of the tree are kept hot in the buffer cache, so the amortized cost is closer to one read per query.

Summary

	B-Tree	B+ Tree
Data stored at	Any node	Leaves only
Internal nodes contain	Keys + values	Keys only (denser)
Leaf linked list	No	Yes
Point query	O(log n)	O(log n)
Range query	Tree traversal per key	Follow linked list — O(log n + k)
Typical height (1B rows)	~5 levels	~3 levels
Used by	Many filesystems (HFS+, ext4, NTFS, APFS, Btrfs use B-trees or B+ trees)	Databases (PostgreSQL, MySQL, SQLite, SQL Server)

The mental model builds directly from what you already know:

BST — one key, two children, branch by 2
B-tree — many keys, many children, branch by many, data anywhere
B+ tree — many keys, many children, data only at leaves, leaves linked in sorted order

The B+ tree is not a micro-optimization of the B-tree. It is a deliberate trade: give up the ability to find data in internal nodes, gain the ability to do range scans in O(k) time after a single O(log n) lookup.

That trade is the reason WHERE created_at BETWEEN '2026-01-01' AND '2026-03-31' returns in milliseconds instead of minutes.

How Databases Actually Store and Find Your Data — pages, heaps, and how indexes physically connect to rows.
PostgreSQL Indexing Deep Dive — applies B+ tree theory to a real database, plus GIN, GiST, BRIN, and hash indexes.

PostgreSQL Indexing: Internals, Types, and Trade-offs

Sun, 26 Apr 2026 00:00:00 GMT

An index is the single most impactful performance tool in a relational database. It can turn a 30-second query into a 2ms one. It can also double your write latency, bloat your storage, and confuse the query planner — all silently. This post goes deep on how indexes actually work inside PostgreSQL, every index type available, and the non-obvious rules that govern when an index helps versus hurts.

How PostgreSQL Stores Data Without an Index

To understand indexes you have to understand what they replace. PostgreSQL stores table rows in heap files — a collection of 8KB pages on disk. Rows are written in the order they arrive. There is no inherent sort order. There is no clustering. A row’s physical location is its TID (Tuple ID) — a pair (page_number, slot_offset).

When you run SELECT * FROM orders WHERE user_id = 42 against a table with no index, PostgreSQL has no choice but to read every page of the heap file, look at every row, and discard the ones where user_id != 42. That’s a sequential scan. On a table with 10 million rows and 80,000 pages, that’s 80,000 page reads.

An index is a separate data structure that maps column values to TIDs. Instead of reading 80,000 pages, you consult the index to find the handful of TIDs where user_id = 42, then fetch just those heap pages. The index does not contain the full row — only the indexed column(s) and the TID pointer back to the heap.

B-tree Internals: How the Default Index Works

The default index type in PostgreSQL is a B-tree (Balanced Tree). Understanding it deeply unlocks understanding of nearly everything else about index performance.

The Structure

A B-tree index is a tree of fixed-size pages (also 8KB, same as heap pages). There are three kinds of pages:

Root page: The entry point. Contains separator keys that route the search left or right.
Internal pages: Branch nodes. Store separator keys and child page pointers. Never store TIDs.
Leaf pages: Store the actual index entries: (indexed_value, TID) pairs sorted by value. Every leaf page is doubly linked to its neighbors — this is what makes range scans cheap.

A Point Lookup: `WHERE id = 42`

Read the root page. Binary search for 42 among separator keys. Follow the child pointer that covers 42’s range.
Read the internal page. Binary search again. Follow the child pointer.
Read the leaf page. Binary search for 42. Find the TID(s).
Use the TID to fetch the actual row from the heap.

For a table with 10 million rows, a B-tree index is typically 3-4 levels deep. That’s 3-4 page reads to find any value — vs 80,000 for a seq scan. This is the O(log N) lookup.

A Range Scan: `WHERE id BETWEEN 100 AND 200`

Find the leaf page containing 100 (same point lookup process).
Scan forward across leaf pages (following the linked list) collecting TIDs until the value exceeds 200.
Fetch heap rows for all collected TIDs.

The doubly-linked leaf chain makes this efficient — no need to go back to the root for each subsequent value. This is why B-trees support ORDER BY, BETWEEN, <, > natively.

Page Splits — The Hidden Cost of Inserts

Every leaf page has a fill factor (default 90% for indexes). When a new index entry arrives and the target leaf page is full, PostgreSQL splits the page: creates a new page, moves half the entries there, and updates the parent page to add a new separator key pointing to the new page.

If the parent is also full, it splits too — a cascade that can propagate up to the root. Root splits increase the tree height by one. This is rare but means a B-tree grows in height slowly over time.

Why this matters for UUIDs — we’ll cover this in depth later, but the short version is: random values cause every insert to land on a random page, requiring that page to be in the buffer cache. Monotonically increasing values always append to the rightmost leaf — only one page needs to be hot.

The Visibility Map and Index-Only Scans

Every heap page has a corresponding bit in the visibility map — a compact structure that tracks whether all rows on a page are visible to all transactions (fully frozen/vacuumed). When PostgreSQL does an index scan, it normally has to fetch the heap page to check row visibility. But if the visibility map says a page is all-visible, it can skip the heap fetch entirely and return the indexed values directly.

This is an index-only scan — the fastest possible read path. You see Heap Fetches: 0 in EXPLAIN ANALYZE output. It requires:

All SELECTed columns are in the index.
The visibility map is current (i.e., VACUUM has run recently on the table).

Tables with high write volume that haven’t been vacuumed recently will have many pages not marked all-visible, causing index-only scans to degrade back to regular index scans.

Index Types in PostgreSQL

PostgreSQL has six built-in index types plus extensions, each optimized for different access patterns and data shapes.

B-tree (Default)

CREATE INDEX idx_orders_user_id ON orders (user_id);
-- equivalent to:
CREATE INDEX idx_orders_user_id ON orders USING btree (user_id);

Supports: =, <, <=, >, >=, BETWEEN, IN, IS NULL, IS NOT NULL, LIKE 'prefix%', ORDER BY, DISTINCT.

Works with: Any type that has a defined ordering — integers, text, timestamps, enums, UUIDs, numeric, etc.

Use for: Primary keys, foreign keys, range queries, sorting, prefix string matches. When in doubt, start here.

Does not support: Full-text search, containment queries (@>), geometric overlap, JSONB key existence.

Hash

CREATE INDEX idx_orders_status_hash ON orders USING hash (status);

Computes a hash of the column value and stores (hash_bucket, TID). There is no tree — lookup is O(1) for equality.

Supports: Only = (equality). Nothing else.

Performance vs B-tree for equality: Hash indexes are theoretically faster for pure equality lookups on high-cardinality columns. In practice, the difference is marginal — B-tree lookup is O(log N) but with a very small constant (3-4 page reads), and PostgreSQL’s buffer cache often keeps hot pages warm anyway.

Downsides:

Cannot be used for <, >, BETWEEN, ORDER BY, LIKE, range anything.
Cannot be used in a composite index.
Before PostgreSQL 10, hash indexes were not WAL-logged and would be lost on crash — they had to be manually rebuilt. Since PostgreSQL 10 they are fully crash-safe, but the stigma remains in older documentation.
The planner rarely chooses a hash index over a B-tree even for equality unless you force it.

When to use: Almost never in practice. The one valid use case is an equality-only lookup on a column with very long values (e.g., a SHA-256 hash column) where storing the full value in a B-tree entry is wasteful — a hash index stores only the hash of the hash, making index pages much denser. Even then, most engineers reach for B-tree and a text_pattern_ops or just accept the slightly larger index.

GIN (Generalized Inverted Index)

CREATE INDEX idx_articles_tags ON articles USING GIN (tags);
CREATE INDEX idx_products_attrs ON products USING GIN (attributes);  -- JSONB
CREATE INDEX idx_articles_fts ON articles USING GIN (to_tsvector('english', body));

A GIN index is an inverted index — it maps from individual element values to the rows that contain them. For an array column tags = ['go', 'performance', 'database'], the GIN index contains three entries: go → {row 5, row 12, row 30}, performance → {row 5, row 88}, etc.

Supports:

Arrays: @> (contains), <@ (contained by), && (overlap), = ANY()
JSONB: @> (contains), ? (key exists), ?| (any key exists), ?& (all keys exist)
Full-text search: @@ (tsvector matches tsquery)
Range types: with range_ops
Trigrams (with pg_trgm): LIKE '%pattern%', ILIKE, similarity (%)

Characteristics:

Lookup is very fast — find elements in O(log N), intersect posting lists.
Insert is expensive — every new element in an array/JSONB must update potentially many posting lists. GIN uses a “pending list” buffer to batch writes (controlled by gin_pending_list_limit), but the eventual merge is costly.
Index size is large — an inverted index with many distinct elements can be several times larger than the source data.
Not suitable for columns where every row has a unique value — there’s nothing to invert.

The pending list: GIN batches inserts into a pending list (default 4MB) and merges into the main index structure lazily (during VACUUM or when the pending list fills). This makes individual inserts fast, but bulk loads can trigger expensive background merges. For bulk inserts, use CREATE INDEX after loading rather than maintaining a GIN index during load.

When to use:

Any array column you query with && or @>.
JSONB columns where you query by key existence or containment.
Full-text search (tsvector columns or to_tsvector() expressions).
Trigram search for LIKE '%substring%' or fuzzy matching.

GiST (Generalized Search Tree)

CREATE INDEX idx_locations_geom ON locations USING gist (coordinates);  -- PostGIS
CREATE INDEX idx_events_period ON events USING gist (during);           -- tsrange
CREATE INDEX idx_articles_fts ON articles USING gist (to_tsvector('english', body));

GiST is a framework for building balanced search trees for non-standard data types. Where B-tree needs a total ordering, GiST needs only a set of user-defined predicates (consistent, union, penalty, picksplit). This makes it extensible to types that don’t have a natural linear order.

Supports:

Geometric types: PostGIS (geometry, geography), built-in PostgreSQL geometric types (point, polygon, circle). Operators: && (overlap), @> (contains), ~ (same), <-> (distance).
Range types: int4range, tsrange, daterange, etc. Operators: &&, @>, -|- (adjacent).
Full-text search: tsvector @@ tsquery (but GIN is generally faster for this).
Nearest-neighbor search: GiST supports ORDER BY column <-> value LIMIT N — find the N closest points to a given coordinate. GIN cannot do this.

GiST vs GIN for full-text:

GiST: faster to build, smaller index, supports <-> nearest-neighbor. Slower at search time.
GIN: larger index, slower to build, much faster search (especially with many matching documents).

For read-heavy full-text search, GIN almost always wins. GiST is preferred when you need nearest-neighbor or when write performance matters more.

When to use: Geospatial queries (PostGIS), date/time range overlap queries, nearest-neighbor lookups, exclusion constraints (EXCLUDE USING gist).

BRIN (Block Range INdex)

CREATE INDEX idx_logs_created_at ON logs USING brin (created_at);
CREATE INDEX idx_logs_created_at ON logs USING brin (created_at) WITH (pages_per_range = 64);

A BRIN index is radically different from everything else. Instead of indexing individual values, it divides the heap into ranges of consecutive pages (default: 128 pages per range) and stores only the min and max value of the indexed column within that range.

A BRIN index on a 100GB table might be only 256KB — tens of thousands of times smaller than a B-tree index on the same column.

How a lookup works: For WHERE created_at > '2025-01-01', scan the BRIN index (a tiny read) and discard any page ranges whose max < '2025-01-01'. Then do a sequential scan of only the surviving page ranges. This is still a scan — BRIN doesn’t give you direct row-level access — but it skips huge chunks of the table.

The critical requirement: physical correlation. BRIN only works when the physical order of rows in the heap matches the order of the indexed values. If you insert rows in timestamp order (which is natural for logs, events, IoT data), then page 1 has the oldest rows, page N has the newest — the min/max ranges are tight and exclusive. If rows are inserted randomly, the min/max of every range will be nearly the full column domain, and BRIN can’t skip anything.

-- Check physical correlation before creating a BRIN index
SELECT correlation FROM pg_stats WHERE tablename = 'logs' AND attname = 'created_at';
-- correlation near 1.0 = good for BRIN
-- correlation near 0.0 = BRIN will be useless

When to use:

Append-only or mostly-append tables (logs, events, time-series, audit trails).
created_at, inserted_at, serial IDs — any column that grows monotonically with insertion order.
Very large tables (100GB+) where a B-tree index would be several GB itself.
When you need only a small fraction of the table for a time-range query and can tolerate a scan of the relevant page ranges.

When NOT to use:

Random insert patterns (correlation near 0) — BRIN will be ignored by the planner.
High-selectivity point lookups — BRIN will still scan hundreds of pages; a B-tree is far better.
Columns that are updated frequently — BRIN summaries can become stale (use brin_summarize_new_values() or autovacuum to refresh).

SP-GiST (Space-Partitioned GiST)

CREATE INDEX idx_users_phone ON users USING spgist (phone_number);
CREATE INDEX idx_network_cidr ON networks USING spgist (cidr);

SP-GiST builds space-partitioned trees — structures like tries, quad-trees, k-d trees, and radix trees. Unlike B-tree and GiST which are balanced, SP-GiST trees can be unbalanced and work well for data with natural partition structure.

Supports:

Text prefix matching (trie structure) — similar to B-tree for LIKE 'prefix%' but faster on high-cardinality text.
IP addresses and CIDR containment (inet, cidr types) — << (subnet), >>= (supernet), etc.
Geometric types (quad-tree for points) — 2D &&, @>, <->.
Phone numbers and other hierarchically structured strings.

When to use: IP/CIDR containment queries, text prefix queries on very long strings, 2D point data when GiST is too slow.

In practice, SP-GiST is the most specialized index type and you’ll encounter it mainly when working with IP ranges, hierarchical codes, or specific geometric workloads.

Composite Indexes

A composite (multi-column) index stores entries for multiple columns together.

CREATE INDEX idx_orders_user_status ON orders (user_id, status);

The Prefix Rule

A composite index (a, b, c) can be used by queries that filter on:

a alone
a, b
a, b, c

But not by queries that filter on:

b alone
c alone
b, c
c, a

The planner can only use the index from the left. This is because the index is physically sorted first by a, then by b within each a value, then by c within each (a, b) combination. Searching for b = 'pending' without knowing a is equivalent to searching a phone book by first name — the sort order is useless.

-- These use the index:
WHERE user_id = 42
WHERE user_id = 42 AND status = 'pending'
WHERE user_id = 42 AND status = 'pending' AND created_at > '2025-01-01'
WHERE user_id = 42 AND created_at > '2025-01-01'  -- uses user_id part only

-- These do NOT use the index:
WHERE status = 'pending'
WHERE created_at > '2025-01-01'
WHERE status = 'pending' AND created_at > '2025-01-01'

Exception: index skip scan. PostgreSQL 18+ (skip scans for skipping leading equality columns; not yet released as of this writing — verify against your target version) introduced skip scans for some cases, and even earlier versions can sometimes use a composite index for the second column alone via a bitmap scan — but you cannot rely on this.

Column Order in a Composite Index

The rule of thumb: put equality conditions first, range conditions last.

-- GOOD: equality first, range last
CREATE INDEX ON orders (user_id, status, created_at);
-- Query: WHERE user_id = 42 AND status = 'pending' AND created_at > '2025-01-01'
-- Narrows to one user, one status, then scans the date range

-- BAD: range in the middle
CREATE INDEX ON orders (user_id, created_at, status);
-- Query: WHERE user_id = 42 AND status = 'pending' AND created_at > '2025-01-01'
-- After a range condition (created_at BETWEEN ...), columns to its right (status)
-- cannot be used as further index conditions — but they can still be applied as a
-- filter on the rows the index retrieves. The index is used; the filter just runs
-- after the index lookup, post-fetching all rows that match the range.

Why? Once the index encounters a range predicate (>, <, BETWEEN, LIKE 'x%'), it can narrow to a start point but then must scan all subsequent entries. Columns after a range column in a composite index cannot be used to skip entries.

Covering Indexes (Index-Only Scans)

If you add the SELECTed columns to the index, PostgreSQL can serve the entire query from the index without touching the heap:

-- Query needs user_id (filter) and name, email (select)
-- Without this index: index scan + heap fetch per row
-- With this index: index-only scan (Heap Fetches: 0)
CREATE INDEX idx_users_covering ON users (user_id) INCLUDE (name, email);

The INCLUDE syntax (PostgreSQL 11+) adds columns to the leaf pages only — they’re not part of the sort key, so they don’t affect the tree structure or index lookups. The included columns just ride along as payload, allowing index-only scans without the filtering confusion of making them full key columns.

Do not put frequently updated columns as key columns in a composite index — every update to those columns requires updating the index entry, which means a page write + potential page split. Frequently updated columns work better in INCLUDE (leaf-only) or in a separate index.

Partial Indexes

A partial index indexes only the rows matching a WHERE predicate.

-- Only index orders that aren't completed — active orders are the hot working set
CREATE INDEX idx_orders_active ON orders (user_id, created_at)
WHERE status != 'completed';

-- Only index non-null values
CREATE INDEX idx_users_referral ON users (referral_code)
WHERE referral_code IS NOT NULL;

-- Unique constraint only on active users (allow multiple deleted users with same email)
CREATE UNIQUE INDEX idx_users_email_active ON users (email)
WHERE deleted_at IS NULL;

Why partial indexes are powerful:

Smaller index. If 95% of orders are completed and you only ever query active orders, a partial index on the 5% that are active is 20x smaller, fits more in memory, and has faster tree traversal.
Better statistics. The planner’s estimates for a partial index cover only the indexed subset — tighter statistics, better plan choices.
Conditional uniqueness. You can enforce uniqueness only within a subset of rows (as in the email example above), which is impossible with a regular unique index.

For a query to use a partial index, the query’s WHERE clause must imply the index predicate. The planner must be able to prove that any row satisfying the query also satisfies the index’s WHERE clause.

-- Uses the partial index (query implies status != 'completed')
SELECT * FROM orders WHERE user_id = 42 AND status = 'pending';

-- Does NOT use the partial index (query doesn't constrain status)
SELECT * FROM orders WHERE user_id = 42;

UUIDs and Indexes: The Fragmentation Problem

This is one of the most important and commonly misunderstood topics in PostgreSQL performance.

Why UUIDv4 Destroys B-tree Performance

UUIDv4 values are random — 550e8400-e29b-41d4-a716-446655440000 has no relationship to the UUID inserted before or after it. When you use UUIDv4 as a primary key (which creates a B-tree index), every insert goes to a random position in the index tree.

Here’s what happens at scale:

The cascade of problems:

Cache thrash. Each insert touches a different random leaf page. On a large table, the entire leaf level of the index is larger than your buffer cache (shared_buffers is the chunk of RAM Postgres reserves as a database-wide page cache; default 128 MB, typically tuned to ~25% of system memory). Every insert requires reading a cold page from disk before writing to it — a random read + a dirty write per insert.
Index bloat. Because inserts land on random pages, leaf pages fill up and split uniformly across the entire key space. After a split, both resulting pages are around 50% full. On a sequential insert pattern, the rightmost page is the only one that fills and splits — everything else stays at 90%+ fill factor. With random UUIDs, average fill factor degrades toward 70-75%, making the index ~25% larger than necessary.
WAL amplification. Every random page touched must be written to WAL. Sequential inserts touch one hot page repeatedly — the WAL entry for a page write is amortized across many inserts. Random inserts each dirty a new page.
Write latency spikes. Under high insert volume, the buffer eviction pressure from constantly loading new random pages causes latency spikes as the OS does I/O. This is most visible on spinning disks but happens on SSDs too.

How bad is it in practice? On a table of 50 million rows with a UUIDv4 primary key:

Index size: ~3GB (B-tree on 16-byte UUID × 50M rows, with bloat)
Insert throughput: drops significantly compared to serial/BIGSERIAL as the table grows
The working set of “hot index pages” = the entire leaf level ≈ the entire 3GB index

With a BIGSERIAL primary key (sequential integers):

Index size: ~800MB (8-byte integers, no bloat — rightmost leaf stays hot)
Insert throughput: stable as the table grows — only the rightmost leaf + its path to root need to be hot

The Fixes

Option 1: Use BIGSERIAL / BIGINT instead of UUID

The simplest fix. A 64-bit integer primary key is:

Half the size per row (8 bytes vs 16 bytes)
Always sequential — no fragmentation
Faster for equality lookups (smaller comparison)

Downside: exposes sequential IDs in URLs/APIs (enumeration risk), coordination complexity in distributed systems.

Option 2: UUIDv7 (time-ordered UUID)

UUIDv7 encodes a millisecond-precision timestamp in the most significant bits:

018e3c2f-1234-7xxx-xxxx-xxxxxxxxxxxx
^^^^^^^^^
timestamp prefix (48 bits, ms precision)

Because the timestamp prefix increases monotonically, UUIDv7 inserts always go to the rightmost leaf of the B-tree — exactly like sequential integers. You get global uniqueness without coordination, URL-safe IDs, and no B-tree fragmentation.

-- gen_random_uuid() has been in core PostgreSQL since 13 (and in pgcrypto long
-- before). For UUIDv7 specifically, use the pg_uuidv7 extension or generate at
-- the application layer.
CREATE EXTENSION pg_uuidv7;
SELECT uuid_generate_v7();  -- 018e3c2f-1234-7abc-...

-- Or generate in application code (most UUID libraries support v7 now)

Option 3: ULID

Universally Unique Lexicographically Sortable Identifier — 128-bit, 26-character Crockford Base32 string. The first 48 bits are a millisecond timestamp, the remaining 80 bits are random. Same properties as UUIDv7: monotonically increasing within a millisecond, globally unique, B-tree friendly.

Option 4: Keep UUIDv4 but use fillfactor

FILLFACTOR (the percentage of each index/heap page Postgres fills before considering it ‘full’ — default is 90 for indexes, 100 for heap tables; lowering it leaves room for HOT updates and reduces page splits on write-heavy tables).

If you must keep UUIDv4 (e.g., existing system, external ID requirements), reduce the index fill factor to leave room for out-of-order inserts without triggering as many splits:

CREATE INDEX idx_orders_uuid ON orders (id) WITH (fillfactor = 70);

This trades index size (larger, less dense pages) for fewer splits. It doesn’t fix cache thrash — random pages are still loaded cold for each insert — but reduces write amplification from constant splits.

Summary:

PK Type	Insert pattern	B-tree behavior	Recommended?
`BIGSERIAL`	Sequential	Rightmost-leaf only, no bloat	Yes — for internal systems
UUIDv4	Random	Full-tree random access, ~25% bloat	Avoid for high-write tables
UUIDv7 / ULID	Monotonic (by time)	Rightmost-leaf only, like BIGSERIAL	Yes — for distributed/external IDs
UUIDv1	Random in B-tree key order	Fragments like UUIDv4	Avoid — see note below

Note on UUIDv1: Despite being timestamp-based, UUIDv1’s time bits are stored low-order-first — so it sorts roughly randomly in B-tree key order and fragments leaves like UUIDv4. UUIDv1 also leaks the host MAC address. UUIDv6 reorders v1’s bits to be monotonic but is rarely used; UUIDv7 is the right modern choice.

Data Types and Index Compatibility

Not every data type works well with every index type. Here’s the full picture.

Operator Classes

PostgreSQL links data types to index types via operator classes. An operator class defines which operators the index can answer for a given type. When you create an index, you can specify the operator class explicitly:

-- Default operator class for text (uses locale-aware collation)
CREATE INDEX ON users (last_name);

-- text_pattern_ops: ignores collation, supports LIKE 'prefix%' and pattern matching
-- Required for LIKE queries that need to use an index with non-C locale
CREATE INDEX ON users (last_name text_pattern_ops);

If your database locale is not C or POSIX, a regular B-tree index on a text column cannot support LIKE 'prefix%' — you need text_pattern_ops. This surprises many engineers who add an index on email and then find their WHERE email LIKE 'john%' query still does a seq scan.

-- Check your database locale
SHOW lc_collate;
-- If not 'C' or 'POSIX', use text_pattern_ops for LIKE support

-- Correct approach for pattern matching on non-C locale:
CREATE INDEX idx_users_email_pattern ON users (email text_pattern_ops);
SELECT * FROM users WHERE email LIKE 'john%';  -- now uses the index

Type-to-Index Quick Reference

Data Type	Best Index	Notes
`integer`, `bigint`, `numeric`	B-tree	Default. Full range/equality/order support.
`text`, `varchar`	B-tree	Use `text_pattern_ops` for `LIKE` on non-C locale.
`text` (substring/fuzzy)	GIN + `pg_trgm`	For `LIKE '%middle%'`, `ILIKE`, similarity.
`uuid`	B-tree	Works. Random UUIDv4 causes fragmentation — use UUIDv7.
`timestamp`, `timestamptz`	B-tree	Full range support. BRIN if append-only large table.
`date`	B-tree	Same as timestamp.
`boolean`	Partial index	A plain B-tree on a boolean column has only two distinct values, so it can’t narrow a query much on its own. Use a partial index (`WHERE active = true`) to index only the rows that matter, or include the boolean as a leading or trailing column of a composite index.
`enum`	B-tree	Equality and ordering works. Low cardinality — partial index often better.
`jsonb`	GIN	`@>`, `?`, `?
`text[]`, `int[]` (arrays)	GIN	`@>`, `&&`, `= ANY()`.
`tsvector`	GIN	Full-text `@@`. GiST if nearest-neighbor needed.
`geometry` (PostGIS)	GiST	Spatial operators (`&&`, `@>`, `<->`).
`tsrange`, `daterange`	GiST	Range overlap `&&`, containment `@>`.
`inet`, `cidr`	SP-GiST	Subnet containment `<<`, `>>=`. B-tree for exact equality.
`hstore`	GIN	Key existence, containment.

Low-Cardinality Columns: When B-tree Is Counterproductive

A low-cardinality column has very few distinct values — boolean, status with 3-4 values, country_code with ~250 values.

A B-tree index on status = 'pending' where 30% of rows are pending: the index will return 30% of all rows. PostgreSQL’s planner knows this and will likely choose a seq scan instead (since 30% > the ~15% threshold for preferring a seq scan). The index exists but is rarely used.

What works instead:

-- Partial index: only the rows you actually query
CREATE INDEX idx_orders_pending ON orders (created_at) WHERE status = 'pending';
-- This index is tiny (only pending rows), always selective, always used for pending queries

-- Composite index with high-cardinality leading column
CREATE INDEX idx_orders_user_status ON orders (user_id, status);
-- user_id is high-cardinality — user_id narrows to a manageable set, status filters further

JSONB Indexing: Choosing the Right Approach

JSONB is one of the most commonly over-indexed or wrongly-indexed types. There are three different approaches with very different trade-offs:

-- 1. GIN index on the whole column — supports @>, ?, ?|, ?&
--    Large index, good for existence/containment queries
CREATE INDEX idx_products_attrs_gin ON products USING GIN (attributes);
SELECT * FROM products WHERE attributes @> '{"color": "red"}';  -- uses GIN

-- 2. Functional B-tree index on a specific key — supports = and range on that key
--    Small index, only for that specific key, supports ordering
CREATE INDEX idx_products_color ON products ((attributes->>'color'));
SELECT * FROM products WHERE attributes->>'color' = 'red';  -- uses B-tree
SELECT * FROM products ORDER BY attributes->>'price'::numeric;  -- uses B-tree for sort

-- 3. GIN with jsonb_path_ops — smaller than default GIN, faster @> queries
--    But ONLY supports @>, not ?, ?|, ?&
CREATE INDEX idx_products_attrs_path ON products USING GIN (attributes jsonb_path_ops);
SELECT * FROM products WHERE attributes @> '{"color": "red"}';  -- uses GIN
-- SELECT * FROM products WHERE attributes ? 'color';  -- CANNOT use this index

Decision rule:

Querying by key existence (? 'color') or multi-key containment → full GIN.
Querying by specific key’s value with equality or range (->>'price' > 100) → functional B-tree on that key.
Only need containment (@>) and want a smaller index → jsonb_path_ops GIN.

When NOT to Create an Index

Every index has a cost. Engineers tend to over-index because the benefit (faster reads) is visible immediately and the cost (slower writes, more storage, maintenance overhead) accumulates gradually.

1. Small tables

PostgreSQL’s seq scan on a table that fits in a few pages is faster than consulting an index. The planner will ignore the index anyway. A “small table” threshold is roughly a few hundred rows, but this varies with row width. If EXPLAIN shows Seq Scan despite an index, the table might just be too small.

2. Write-heavy tables with no read latency requirement

Every INSERT updates all indexes on the table. Every UPDATE that changes an indexed column updates those indexes. Every DELETE marks index entries as dead. On a table with 10 indexes receiving 50,000 inserts/sec, 10× the write I/O is going to index maintenance. If you never query some of those indexes directly, drop them.

-- Find indexes that are never used (since last stats reset)
SELECT schemaname, tablename, indexname, idx_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY schemaname, tablename;

An index with idx_scan = 0 since the last pg_stat_reset() is a candidate for removal. Verify it’s not a constraint enforcement index (primary key, unique) before dropping.

3. Low-cardinality columns (without a partial index)

A plain B-tree index on is_deleted boolean or status varchar with 3 values provides almost no filtering benefit. The planner will use a seq scan for any query that matches a non-trivial fraction of rows. Use a partial index instead.

4. Columns that are only queried as part of a large scan anyway

If a column is always accessed with other conditions that already narrow the result to a small set, an index on it alone adds no value. The composite index with the narrowing column first is the right answer.

5. During bulk loads

Drop non-constraint indexes before a bulk insert, then rebuild afterward:

-- Before bulk load
DROP INDEX CONCURRENTLY idx_orders_user_id;
DROP INDEX CONCURRENTLY idx_orders_created_at;

-- Load data
\COPY orders FROM 'orders.csv' CSV HEADER;

-- After bulk load — building from scratch is faster than incremental maintenance
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders (user_id);
CREATE INDEX CONCURRENTLY idx_orders_created_at ON orders (created_at);

Building an index from scratch is 3-10x faster than maintaining it through incremental inserts because PostgreSQL can sort the data once and write the B-tree in one pass. Incremental maintenance requires a random insertion for each row.

Index Maintenance

Index Bloat

MVCC keeps multiple versions of every row Postgres has updated or deleted but not yet vacuumed. Older versions (called dead tuples) take space until autovacuum reclaims them. See ACID and Isolation Levels for why MVCC exists in the first place.

Dead tuples in the heap have corresponding dead entries in every index on the table. VACUUM removes dead heap tuples and marks dead index entries as reusable, but it does not compact the index pages or return space to the OS. Over time, especially after bulk deletes or updates, indexes accumulate bloat — pages that are mostly empty but still occupy disk space and memory.

-- Check index bloat (requires pgstattuple extension)
CREATE EXTENSION IF NOT EXISTS pgstattuple;
SELECT * FROM pgstatindex('idx_orders_user_id');
-- free_percent > 20-30% is a sign of bloat worth addressing

For severe bloat, rebuild the index:

-- Non-concurrent: fast but locks the table (avoid in production)
REINDEX INDEX idx_orders_user_id;

-- Concurrent: safe in production, no table lock
REINDEX INDEX CONCURRENTLY idx_orders_user_id;

-- Or: create a new index, swap, drop old (manual concurrent reindex)
CREATE INDEX CONCURRENTLY idx_orders_user_id_new ON orders (user_id);
DROP INDEX CONCURRENTLY idx_orders_user_id;
ALTER INDEX idx_orders_user_id_new RENAME TO idx_orders_user_id;

`CREATE INDEX CONCURRENTLY` — Adding Indexes Without Downtime

Always use CONCURRENTLY in production when adding an index to a table that receives active queries:

-- Locks the table during build — ALL reads and writes block until done
CREATE INDEX idx_orders_user_id ON orders (user_id);

-- Takes only brief locks — reads and writes continue the entire time
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders (user_id);

On a table with millions of rows, a non-concurrent build can hold an AccessExclusiveLock for minutes. Every query that touches the table queues behind it. In production, that’s a full outage for that table.

What CONCURRENTLY Actually Does — Three Phases

The implementation is more subtle than “just build it without locking”:

Phase 1 — Snapshot 1 + initial build: PostgreSQL takes a snapshot of the table and builds an index on all rows visible in that snapshot. It holds only a ShareUpdateExclusiveLock during this phase — concurrent reads and writes are fully allowed. New writes that arrive during Phase 1 are not yet in the index.

Phase 2 — Wait for old transactions: PostgreSQL waits until all transactions that started before Phase 1 ended have committed or rolled back. This is the safety gate. Any transaction still running from before the index build started might have a view of the table that doesn’t include the Phase 1 index. PostgreSQL must wait them out.

Phase 3 — Snapshot 2 + catch-up: Takes a second snapshot, scans the heap again, and indexes any rows written between Phase 1 and Phase 2 that were missed. Takes another brief ShareUpdateExclusiveLock.

After Phase 3, the index is fully consistent and the planner can use it.

Why It Can Fail — INVALID Indexes

CREATE INDEX CONCURRENTLY can fail and leave behind an INVALID index in two situations:

A long-running transaction never ends. Phase 2 waits for old transactions to close. If a transaction started before Phase 1 and stays open for hours (e.g., a forgotten BEGIN in a psql session, a hung application connection), the build stalls indefinitely. The build waits indefinitely until you cancel it. There is no automatic timeout — set a lock_timeout or statement_timeout if you need bounded waits. If the build is cancelled or fails, the partially-built index is left in INVALID state and must be REINDEX-ed or DROP-ed.
A deadlock or unique constraint violation. If the index is unique and a duplicate value is inserted during the build, the index is marked INVALID.

An INVALID index is not used by the planner, but it does consume write overhead — every insert/update/delete still maintains it. You must drop and recreate it.

-- Find INVALID indexes
SELECT schemaname, tablename, indexname
FROM pg_indexes
WHERE indexname IN (
    SELECT indexrelid::regclass::text
    FROM pg_index
    WHERE NOT indisvalid
);

-- Drop the invalid index (also concurrent to avoid blocking)
DROP INDEX CONCURRENTLY idx_orders_user_id;

-- Recreate — make sure no long-running transactions are open first
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders (user_id);

Constraints and Gotchas

Cannot run inside a transaction block. CONCURRENTLY requires its own transaction management across the three phases:

BEGIN;
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders (user_id);
-- ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block
ROLLBACK;

Takes longer overall. The concurrent build does more total work than a non-concurrent one (two heap scans, phase wait). On a large table it may take 2-3x as long wall-clock time, though your application doesn’t feel it.

Unique indexes need extra care. For CREATE UNIQUE INDEX CONCURRENTLY, Phase 2 also checks for duplicates in the rows written during Phase 1. If duplicates exist, the index fails as INVALID — it won’t silently ignore them.

REINDEX CONCURRENTLY for rebuilding existing indexes (PostgreSQL 12+):

-- Rebuild a bloated or corrupt index without locking
REINDEX INDEX CONCURRENTLY idx_orders_user_id;

-- Rebuild all indexes on a table concurrently
REINDEX TABLE CONCURRENTLY orders;

Same three-phase mechanics, same INVALID-index risk on failure.

Pre-flight Checklist Before a Large Concurrent Build

-- 1. Check for long-running transactions that would stall Phase 2
SELECT pid, now() - xact_start AS duration, state, query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL
ORDER BY duration DESC;
-- Kill any that have been running for hours and are idle:
-- SELECT pg_terminate_backend(pid);

-- 2. Estimate index build time by checking table size
SELECT pg_size_pretty(pg_total_relation_size('orders'));

-- 3. Monitor build progress (PostgreSQL 12+)
SELECT phase, blocks_done, blocks_total,
       round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS pct
FROM pg_stat_progress_create_index
WHERE relid = 'orders'::regclass;

The pg_stat_progress_create_index view is invaluable for large tables — it shows exactly which phase the build is in and how far through the heap scan it is.

Autovacuum and Index Health

PostgreSQL’s autovacuum process runs automatically when a table accumulates enough dead tuples (controlled by autovacuum_vacuum_threshold and autovacuum_vacuum_scale_factor). It cleans up dead heap tuples and index entries. Without autovacuum:

Index entries accumulate as dead, causing index scans to become slower (more dead entries to skip).
The visibility map becomes stale, preventing index-only scans.
Table and index size grow unboundedly.

For high-write tables, autovacuum may not keep up with the default settings. Tune it per table:

ALTER TABLE orders SET (
    autovacuum_vacuum_scale_factor = 0.01,  -- vacuum when 1% rows are dead (default 20%)
    autovacuum_vacuum_cost_delay = 2         -- less throttling for this table
);

Bloom Filter Indexes

PostgreSQL 9.6+ ships a bloom index access method as a contrib extension. It’s a specialized index type built on the classic Bloom filter data structure — useful in a narrow but real scenario that no other index type handles well.

What a Bloom Filter Is

A Bloom filter is a space-efficient probabilistic data structure that answers one question: “Has this value been seen before?” It can answer definitively “no” (the value is definitely absent) but only probabilistically “yes” (the value might be present — with a configurable false-positive rate).

Internally it’s a bit array. To insert a value, hash it with k different hash functions, and set those k bits. To test membership, hash the query value the same way and check whether all k bits are set. If any bit is unset, the value is definitely not in the set. If all are set, the value is probably in the set — some bits may have been set by other values.

Value "alice" → hash functions → set bits [3, 17, 42, 89]
Value "bob"   → hash functions → set bits [7, 17, 55, 91]

Query "alice": check bits [3, 17, 42, 89] — all set → probably present (fetch heap to confirm)
Query "carol": check bits [2, 18, 43, 89] — bit 2 unset → definitely absent (skip heap entirely)

PostgreSQL’s Bloom Index

CREATE EXTENSION IF NOT EXISTS bloom;

CREATE INDEX idx_orders_bloom ON orders
USING bloom (user_id, status, region)
WITH (length = 80, col1 = 2, col2 = 2, col3 = 3);

length — size of the bloom signature per row in bits (8–4096, default 80)
col1/col2/col3 — number of hash functions per column (1–4096, default 2)

The index stores a fixed-length bit signature for each row. More bits and more hash functions = fewer false positives, larger index.

The One Scenario Where Bloom Beats Everything Else

A bloom index is specifically designed for tables where you need equality searches across many columns in arbitrary combinations — not a fixed set of columns you always query together.

Consider a user activity log table:

CREATE TABLE activity_log (
    id          bigserial PRIMARY KEY,
    user_id     int,
    device_id   int,
    session_id  int,
    action_type int,
    country_id  int,
    partner_id  int
);

Queries come in any combination:

WHERE user_id = 42 AND country_id = 5
WHERE device_id = 99 AND action_type = 3
WHERE partner_id = 7 AND session_id = 1234 AND country_id = 5
WHERE user_id = 42 AND device_id = 99 AND partner_id = 7

To cover all possible combinations with B-tree composite indexes, you’d need potentially dozens of indexes — one for each common combination, with correct column ordering. That’s enormous write overhead and storage.

With a single bloom index covering all six columns, any equality condition on any subset of those columns can use the one index:

CREATE INDEX idx_activity_bloom ON activity_log
USING bloom (user_id, device_id, session_id, action_type, country_id, partner_id);

The index signature for each row is a bitwise OR of the per-column signatures. A query hashes its filter values, checks whether all resulting bits are set in a row’s signature. Rows that don’t match the bit pattern are skipped without ever reading the heap. Rows whose bit pattern might match are fetched and rechecked.

The Trade-offs

False positives are normal and expected. A bloom index trades a small rate of unnecessary heap fetches for dramatically smaller index size and simpler maintenance. The planner knows this — it always applies a recheck after the bloom scan. You’ll see it in EXPLAIN:

Bitmap Heap Scan on activity_log
  Recheck Cond: ((user_id = 42) AND (country_id = 5))
  Rows Removed by Index Recheck: 23      ← false positives fetched from heap
  ->  Bitmap Index Scan on idx_activity_bloom
        Index Cond: ((user_id = 42) AND (country_id = 5))

Only supports equality (=). No range, no LIKE, no ORDER BY. Bloom is an existence check — it can’t reason about ordering or ranges.

Not useful for high-selectivity single-column queries. A B-tree beats bloom for WHERE user_id = 42 alone. Bloom shines when the combination of multiple equality conditions provides selectivity that no single-column B-tree index can.

Tuning the false-positive rate:

-- More bits per row = fewer false positives, larger index
-- Default (length=80): ~1-3% false positive rate for typical data
-- Higher length trades storage for fewer wasted heap fetches

-- Measure current false-positive rate:
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM activity_log WHERE user_id = 42 AND country_id = 5;
-- Look at "Rows Removed by Index Recheck" vs total rows fetched
-- If removal rate > 20-30%, increase length

When to Reach for Bloom

Situation	Use Bloom?
Many columns, all equality, arbitrary combinations	Yes — this is exactly what it’s for
High-cardinality single column	No — B-tree is better
Range queries	No — bloom can’t help
Fixed set of columns always queried together	No — composite B-tree is better
Write-heavy table, need to minimize index count	Yes — one bloom index vs many B-trees
Need to check “has this user_id + device_id ever appeared together?”	Yes

In practice, bloom indexes appear most in analytics tables, event logs, and data warehouse-style schemas where ad-hoc multi-column equality filtering is common and you want to avoid maintaining a combinatorial explosion of B-tree indexes.

Common Mistakes Checklist

B-Trees and B+ Trees — the data structure underlying most of these index types.
How Databases Actually Store and Find Your Data — pages, heap fetches, and clustered vs heap storage.
Reading PostgreSQL EXPLAIN Output — see these indexes show up in plan nodes.
ACID and Isolation Levels — CREATE INDEX CONCURRENTLY’s phases are isolation-level work.
Postgres Partitioning: Strategies, Advantages, and Pitfalls — local indexes per partition, partition pruning, and CREATE INDEX CONCURRENTLY per-partition.

Quick Reference

Situation	Recommendation
High-cardinality equality + range	B-tree
Append-only 100GB+ table, time-range queries	BRIN (check correlation first)
Array or JSONB containment	GIN
Full-text search	GIN on `tsvector`
Fuzzy / substring string matching	GIN + `pg_trgm`
Geospatial / range overlap	GiST
IP/CIDR subnet queries	SP-GiST
5% of rows are the hot “active” working set	Partial index on those rows
UUID primary key, high insert rate	Switch to UUIDv7 or BIGSERIAL
`boolean` or enum index	Partial index (avoid plain B-tree on low-cardinality column)
`LIKE '%substring%'` or `ILIKE`	GIN + `pg_trgm` extension
`LIKE 'prefix%'` with non-C locale	B-tree with `text_pattern_ops`
JSONB key value equality/range	Functional B-tree on `(col->>'key')`
Multi-condition AND with no composite index	BitmapAnd on individual indexes (but add composite if frequent)
Index not used despite existing	Check: function on column, type mismatch, low selectivity, stale stats, `random_page_cost`
Bulk insert	Drop non-constraint indexes, load, `CREATE INDEX CONCURRENTLY`
Many columns, equality-only, arbitrary combinations	Bloom filter index (`CREATE EXTENSION bloom`)
Adding index to live production table	Always `CREATE INDEX CONCURRENTLY`
Rebuilding bloated/corrupt index in production	`REINDEX INDEX CONCURRENTLY`
Index build stalling (Phase 2 wait)	Check `pg_stat_activity` for long-running transactions

Go Profiling: From pprof to Flame Graphs

Tue, 21 Apr 2026 00:00:00 GMT

Your service is slow. You have a theory — maybe it’s the database query, maybe it’s the JSON serialization, maybe it’s something else entirely. You could guess and change things one by one, deploying each time, hoping the graph gets better. Or you could profile it, look at exactly what your program is actually doing, and know for certain where the time or memory is going.

Profiling is the practice of measuring a running program to find out where it spends resources. Go makes this exceptionally easy compared to most languages — the profiling tools are built into the standard library, the overhead is low enough to run in production, and the tooling is part of the Go CLI itself.

This post is a deep guide. We’ll start from the very beginning — what profiling is, how it works under the hood — and work up through every profile type, how to collect them, how to read them, and how to turn what you see into a fix. If you’re a junior engineer who has never profiled anything before, this is written for you.

Why Guessing Doesn’t Scale

Before we get into the tooling, let’s talk about why profiling matters in the first place.

When a program is slow, our instinct is to reason about it. “The database call is probably slow.” “We’re probably doing too much work per request.” These guesses are sometimes right. But they can also lead you down the wrong path — spending days optimizing code that wasn’t actually a bottleneck, while the real problem goes untouched.

The deeper problem is that modern programs are complex. Your request handler calls a logger, which calls a string formatter, which calls an interface method, which calls a reflection-heavy library. Any of those could be the bottleneck. Your intuition, even after years of experience, will miss things that the profiler sees immediately.

There’s a principle in performance work sometimes called Amdahl’s Law: the speedup you get from optimizing a part of the program is limited by how much of the total time that part actually takes. If json.Marshal is 5% of your request time and you make it 10× faster, your overall request time drops by only 4.5%. But if json.Marshal is actually 60% of your request time, the same 10× improvement reduces total request time by 54%. Finding the part that actually dominates is everything.

That’s what profiling gives you — the real numbers, not the guessed ones.

How Go Works: Background You Need

To read profiles intelligently, you need a mental model of what Go is doing at runtime. This isn’t deep theory — it’s the concepts you’ll see show up in every profile.

Goroutines and the Scheduler

Go is a concurrent language. Goroutines are lightweight execution threads managed by the Go runtime, not the operating system. You can have thousands or even millions of goroutines in a single process — each uses only a few kilobytes of stack memory (compared to megabytes for an OS thread).

The Go scheduler is called the M:N scheduler because it maps M goroutines onto N OS threads. The scheduler decides which goroutines run, when they run, and on which OS thread. This scheduling happens automatically, and most of the time you don’t need to think about it.

But when you’re profiling, you do need to think about it, because:

A goroutine that’s “doing work” shows up in a CPU profile
A goroutine that’s “waiting” (on I/O, a channel, a mutex, a syscall) shows up in a goroutine or block profile
The scheduler itself can become a bottleneck — this shows up in a trace

The scheduler uses units called P (processor) — you can think of a P as a slot that lets a goroutine actually run. By default, Go sets GOMAXPROCS to the number of logical CPUs on the machine. If you have 8 CPUs, you have 8 Ps, meaning 8 goroutines can run simultaneously.

The Garbage Collector

Go is a garbage-collected language. Memory that you allocate (with make, new, struct literals, slices, maps, interface values, closures, etc.) is managed by the runtime. The GC periodically scans live objects and frees memory that’s no longer reachable.

This is convenient, but it has costs:

Allocation itself costs time. Every make([]byte, 1024) is work the runtime has to do.
GC cycles cost time. When the GC runs, it uses CPU resources and can pause goroutines.
Memory pressure affects GC frequency. If you allocate a lot, the GC runs more often.

The GC uses a stop-the-world (STW) pause to do certain housekeeping tasks. In modern Go (1.14+), these pauses are extremely short — typically under 1 millisecond. But if you’re experiencing latency spikes at the 99th or 99.9th percentile, even a 1ms pause matters. The trace profiler is the tool that surfaces these.

Understanding the GC is critical for reading memory profiles. When you see a function high in the heap profile, it’s because that function is responsible for keeping memory alive — either directly or through a chain of references.

The Stack vs The Heap

In Go, every goroutine has its own stack — a contiguous region of memory where local variables and function call frames live. The stack grows and shrinks automatically. This is fast because allocating on the stack is just a pointer increment.

The heap is shared memory. When the compiler determines that a variable’s lifetime might extend beyond the function that creates it (through escape analysis), it allocates that variable on the heap instead. Heap allocations are slower and create work for the GC.

You don’t control this directly — the compiler decides. But you can influence it, and this is one of the primary levers in memory optimization. When you see “allocations per operation” in a benchmark, it’s measuring how many times the compiler decided to use the heap.

How Profiling Actually Works

Before you use a profiler, it helps to understand what it’s doing. Go’s CPU profiler is a sampling profiler.

Here’s the idea: profiling every single instruction a program executes would slow it down enormously and produce too much data to be useful. Instead, a sampling profiler sets up a timer that fires at a regular interval — by default, 100 times per second (every 10 milliseconds). Each time the timer fires, the profiler records the current call stack of every running goroutine. After you collect thousands of these samples, you have a statistical picture of where the program spends its time.

The functions that appear most often in the samples are the ones consuming the most CPU time. Functions that are called rarely or execute quickly appear infrequently.

This means a few important things:

CPU profiles are statistical, not exact. A function that shows 42.86% of samples took roughly 42.86% of CPU time — it’s an approximation, not a precise measurement.
Very fast functions may be invisible. If a function completes in under 10ms, it might never be sampled. This is why microbenchmarks (which run functions millions of times) are better for measuring tiny functions.
The overhead is low because we’re only sampling. At 100 samples/second, the profiler adds roughly 1-5% overhead, which is safe for production.

Memory (heap) profiling works differently — it instruments allocation calls directly, and the runtime samples allocations on average once per 512 KB allocated (probabilistic, not deterministic per-512KB; configurable with runtime.MemProfileRate). Any allocation can trigger a sample; it is not a size filter.

The Profile Types — What Each One Tells You

Go provides seven distinct profile types. Each answers a different question about what your program is doing.

CPU Profile

What it measures: Where goroutines spend time when they are actively running — executing code, not waiting.

How to think about it: If you imagine all the time your program spends on the CPU sliced into 10ms intervals, the CPU profile shows you which function was at the top of the call stack during each of those slices. Functions that appear in many slices are consuming significant CPU.

What it won’t tell you: Time spent waiting. If your request handler takes 500ms but 450ms of that is waiting for a database response, the CPU profile might show only 50ms of actual work spread across various functions. The other 450ms doesn’t appear in the CPU profile because the goroutine was sleeping — not running.

When to use it:

Service is slow AND CPU usage is high (close to 100% on one or more cores)
You want to optimize a tight computation
You suspect inefficient algorithms or too much work per request

Heap Profile

What it measures: Memory currently allocated on the heap — specifically, which functions are responsible for keeping that memory alive at the time you take the snapshot.

Two views:

-inuse_space: how much memory is live right now (not yet freed by GC)
-inuse_objects: how many objects are live right now
-alloc_space: total bytes allocated since the program started (even if later freed)
-alloc_objects: total number of allocations since the program started

How to think about it: -inuse_space tells you what’s holding memory right now — useful for debugging high memory usage. -alloc_space tells you what’s been allocating over time — useful for finding GC pressure even when total memory isn’t that high.

When to use it:

Service is using more memory than expected
Memory grows over time and doesn’t come down (potential leak)
GC is running frequently and you want to reduce allocations

Allocs Profile

The Allocs profile is similar to the Heap profile but defaults to showing allocation counts rather than sizes. It records where allocations happened regardless of whether they’re still live. This is the right profile when you’re investigating GC pressure — you want to know what’s causing frequent GC cycles, not just what’s using memory right now.

Goroutine Profile

What it measures: A snapshot of every goroutine that currently exists, and exactly where each one is paused — its full call stack.

How to think about it: Imagine freezing your program at an instant in time and inspecting every thread of execution. You can see which goroutines are doing useful work, which are waiting on I/O, which are blocked on a channel, and which are stuck waiting for a lock.

When to use it:

Goroutine count is unexpectedly high and growing (goroutine leak)
Suspected deadlock — the goroutine dump will show them all waiting on each other
Requests seem to hang — you can see exactly where the goroutines handling those requests are blocked

Mutex Profile

What it measures: Where goroutines are spending time waiting to acquire a mutex (lock). This is distinct from the goroutine profile because it focuses specifically on contention time — the cumulative time goroutines spent blocked on sync.Mutex or sync.RWMutex.

How to think about it: Multiple goroutines competing for the same lock is serialization — it forces concurrency into a single-threaded bottleneck. The mutex profile shows you which locks are the most contested.

Requires opt-in: Must call runtime.SetMutexProfileFraction(n) to enable. Off by default.

When to use it:

Service does not scale when you add more CPUs
CPU usage looks low but throughput isn’t improving with more goroutines
You suspect lock contention under load

Block Profile

What it measures: Where goroutines block on synchronization primitives: channel sends/receives, select, sync.Mutex, sync.RWMutex, sync.WaitGroup, sync.Cond. It records the time goroutines spent waiting.

How to think about it: While the CPU profile shows busy time, the block profile shows waiting time. If your service is slow but CPU is idle, something is blocking goroutines. The block profile shows you what.

The key distinction from Mutex: Block includes all synchronization, not just mutexes. Channel operations, select blocks, and WaitGroup waits all show up here.

These two profiles overlap on mutex contention — the mutex profile measures wait time on contended unlock paths, while the block profile records goroutines blocking on any sync primitive (channels, select, WaitGroup, Cond, and mutexes). Reach for the mutex profile when you suspect lock contention specifically; reach for the block profile when goroutines are blocking somewhere and you don’t yet know where.

Requires opt-in: Must call runtime.SetBlockProfileRate(n) to enable. Off by default.

When to use it:

Service is slow but CPU is low (goroutines are waiting, not working)
Suspected slow I/O paths you want to quantify
Channel-based communication patterns are causing bottlenecks

Trace

What it measures: A complete timeline of everything that happened: every goroutine’s lifecycle (created, started running, blocked, unblocked, destroyed), every GC phase, processor utilization over time, system calls. This is fundamentally different from other profiles — it’s a recording of events rather than a sample.

How to think about it: All other profiles are statistical summaries. The trace is a recording. If CPU profile is like “we measured CPU usage every 10ms and averaged it,” the trace is “here is the exact sequence of everything that happened, with microsecond timestamps.”

Cost: The trace has significantly higher overhead (~10-20% vs 1-5% for pprof) and generates large files. For a 5-second trace on a busy service, you might get a 100MB file.

When to use it:

Latency spikes that don’t show up in CPU profiles (because the goroutine was sleeping, not running)
You suspect GC pauses are causing tail latency issues
Understanding scheduler behavior — why goroutines aren’t getting CPU time
Diagnosing GOMAXPROCS-related issues

Setting Up Profiling in Your Service

There are three main ways to collect profiles. Which one you use depends on what you’re profiling.

Method 1: HTTP Endpoint (Best for Services)

The most common and practical setup. Import net/http/pprof and it registers a set of HTTP handlers automatically under /debug/pprof/. You then collect profiles on-demand from a running service — either locally, in staging, or in production.

package main

import (
    "log"
    "net/http"
    _ "net/http/pprof" // THIS is the key line — side-effect import registers handlers
)

func main() {
    // Option A: single server (simpler, but exposes pprof on your main port)
    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", handleUsers)
    // pprof handlers are already registered on http.DefaultServeMux
    // so if you're using DefaultServeMux, they're already available

    // Option B: separate debug server on a different port (RECOMMENDED)
    // This way profiling never reaches your public-facing load balancer
    go func() {
        // Only bind to localhost — never 0.0.0.0 for the debug server
        log.Println("Debug server listening on localhost:6060")
        if err := http.ListenAndServe("localhost:6060", nil); err != nil {
            log.Printf("debug server error: %v", err)
        }
    }()

    // Your real service
    log.Println("Service listening on :8080")
    log.Fatal(http.ListenAndServe(":8080", mux))
}

Why a separate port? Security. The /debug/pprof/ endpoint exposes information about your running program — goroutine stacks, memory contents (through goroutine dumps). You don’t want this accessible from the internet. By binding the debug server to localhost:6060, only someone with SSH access to the machine (or on the same internal network) can reach it.

After this setup, every endpoint is available:

GET /debug/pprof/              → Index page (links to all profiles)
GET /debug/pprof/profile       → CPU profile (default 30s, ?seconds=N to change)
GET /debug/pprof/heap          → Heap snapshot
GET /debug/pprof/allocs        → Allocs profile
GET /debug/pprof/goroutine     → Goroutine dump
GET /debug/pprof/mutex         → Mutex contention profile
GET /debug/pprof/block         → Block profile
GET /debug/pprof/trace         → Execution trace (?seconds=N)
GET /debug/pprof/threadcreate  → Thread creation profile
GET /debug/pprof/cmdline       → Command line of the process
GET /debug/pprof/symbol        → Look up symbol names by address

Collecting a CPU profile from a live service:

# This collects 30 seconds of CPU data and opens an interactive session
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Or download first, analyze later (useful for remote servers)
curl -o cpu.out "http://localhost:6060/debug/pprof/profile?seconds=30"
go tool pprof cpu.out

Collecting a heap profile:

# Snapshot right now
curl -o heap.out "http://localhost:6060/debug/pprof/heap"
go tool pprof -inuse_space heap.out

Method 2: Using the Framework Integration (If You Use One)

If your service uses a web framework, there are usually packages to wire pprof in cleanly:

// With gorilla/mux:
import "github.com/gorilla/mux"
import "net/http/pprof"

func registerDebugRoutes(r *mux.Router) {
    r.PathPrefix("/debug/pprof/").HandlerFunc(pprof.Index)
    r.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
    r.HandleFunc("/debug/pprof/profile", pprof.Profile)
    r.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
    r.HandleFunc("/debug/pprof/trace", pprof.Trace)
}

Method 3: Programmatic Profiling (For CLIs and Tests)

For programs that don’t have an HTTP server, or when you want to profile a specific code path directly:

package main

import (
    "os"
    "runtime"
    "runtime/pprof"
)

func main() {
    // --- Start CPU profiling ---
    cpuFile, err := os.Create("cpu.out")
    if err != nil {
        panic(err)
    }
    defer cpuFile.Close()

    if err := pprof.StartCPUProfile(cpuFile); err != nil {
        panic(err)
    }
    defer pprof.StopCPUProfile() // IMPORTANT: must call this before program exits

    // --- Do your work here ---
    doExpensiveWork()

    // --- Take a heap snapshot ---
    // Force a GC first so the snapshot reflects only actually-live objects,
    // not objects that have been allocated but not yet collected
    runtime.GC()

    memFile, err := os.Create("mem.out")
    if err != nil {
        panic(err)
    }
    defer memFile.Close()

    if err := pprof.WriteHeapProfile(memFile); err != nil {
        panic(err)
    }
}

The defer pprof.StopCPUProfile() is critical. If your program calls os.Exit or panics without recover, deferred calls don’t run and the profile file will be incomplete. Normal main returns are fine.

Method 4: Benchmark Profiles

For tight loops or functions you’re actively optimizing, benchmark profiling is the most precise approach:

// user_test.go
package user

import (
    "encoding/json"
    "testing"
)

type User struct {
    ID       int    `json:"id"`
    Name     string `json:"name"`
    Email    string `json:"email"`
    Bio      string `json:"bio"`
}

func BenchmarkMarshalUser(b *testing.B) {
    u := User{
        ID:    42,
        Name:  "Abhimanyu Nagurkar",
        Email: "a@example.com",
        Bio:   "Software engineer, builder, dad.",
    }

    b.ReportAllocs() // Show allocations per operation in the benchmark output
    b.ResetTimer()   // Don't count setup time in the benchmark

    for i := 0; i < b.N; i++ {
        _, err := json.Marshal(u)
        if err != nil {
            b.Fatal(err)
        }
    }
    // In Go 1.24+, prefer `for b.Loop()` — equivalent and clearer.
}

Run and collect:

# Run benchmark and collect both CPU and memory profiles
go test -bench=BenchmarkMarshalUser \
        -benchmem \
        -cpuprofile=cpu.out \
        -memprofile=mem.out \
        -count=3 \
        ./...

# The -benchmem flag makes b.ReportAllocs() output visible even without pprof
# Output:
# BenchmarkMarshalUser-8   1000000   1087 ns/op   232 B/op   4 allocs/op
#                          ^          ^             ^           ^
#                          iterations  time/op      bytes       allocs

The allocs/op number is the key signal for memory optimization. 4 allocations per marshal operation means every single marshal call touches the GC 4 times. If you’re marshaling thousands of responses per second, that’s millions of allocations per second creating GC pressure.

Enabling Mutex and Block Profiling

These two are opt-in because they have measurable overhead. Add this somewhere early in your program:

package main

import (
    "net/http"
    _ "net/http/pprof"
    "runtime"
)

func main() {
    // Mutex profiling: record 1 in every N mutex contention events
    // Lower N = more data but more overhead
    // 0 = disabled (default)
    // 1 = record every single contention (use only in dev/staging)
    // 5 = record 20% of contentions (reasonable for staging)
    runtime.SetMutexProfileFraction(5)

    // Block profiling: SetBlockProfileRate(rate) controls sampling.
    // The runtime samples a blocking event with probability duration / rate —
    // so passing 1 records every event, passing 1_000_000 (1 ms) makes events
    // shorter than 1 ms increasingly rare in the profile.
    // Larger values mean less overhead but coarser data.
    // 0 = disabled (default)
    // 1 = record every single blocking event (high overhead — dev only)
    // 1000 = sample at 1us granularity (reasonable for staging)
    // 1000000 = sample at 1ms granularity (safe for production)
    runtime.SetBlockProfileRate(1000000)

    go func() {
        http.ListenAndServe("localhost:6060", nil)
    }()
    // ... rest of your server
}

Production safety guidance:

Mutex fraction of 5 (20% sampling) is generally safe in production — you’ll miss some contentions but you’ll find the hot ones.
Block rate of 1000000 (1 ms granularity) is safe in production. SetBlockProfileRate(rate) controls sampling: the runtime samples a blocking event with probability duration / rate — so passing 1 records every event, passing 1_000_000 (1 ms) makes events shorter than 1 ms increasingly rare in the profile. Larger values mean less overhead but coarser data.
Heap profiling is always on — no opt-in needed, though you can control the sampling rate with runtime.MemProfileRate.

Tagging Profiles with Labels

When a service handles multiple kinds of work, an unlabeled CPU profile mixes everything together. runtime/pprof.Do lets you attach labels to a goroutine’s profile samples so you can later filter by them.

import "runtime/pprof"

func handleRequest(ctx context.Context, req *Request) {
    pprof.Do(ctx, pprof.Labels("endpoint", req.Path, "tenant", req.TenantID), func(ctx context.Context) {
        // all CPU samples and most heap allocations under this call
        // get tagged with endpoint=... and tenant=...
        process(ctx, req)
    })
}

In pprof, filter with tags and tagfocus/tagignore:

go tool pprof -tagfocus="endpoint=/api/users" cpu.out

Labels are essential when you want to know “which endpoint is dominating CPU?” instead of just “what function?”.

Reading Profiles: The CPU Profile

Let’s walk through reading a CPU profile in full detail. This is the most important skill to develop.

Starting an Interactive Session

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

After 30 seconds of collection, you’ll see:

Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile?seconds=30
Saved profile in /home/user/pprof/pprof.samples.cpu.001.pb.gz
File: myservice
Type: cpu
Time: Apr 21 2026 10:30:00 UTC (30s)
Duration: 30s, Total samples = 2.8s (9.33%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

A few things to notice:

Total samples = 2.8s — out of 30 seconds of wall clock time, the profiler captured 2.8 seconds of actual CPU time across all goroutines. This makes sense if you have, say, 4 goroutines all actively doing work.
The file is saved locally so you can re-analyze it later without re-collecting.

The `top` Command

Type top or top10 to see the hottest functions:

(pprof) top10
Showing nodes accounting for 2.41s, 86.07% of 2.8s total
Dropped 45 nodes (cum <= 14ms)
Showing top 10 nodes out of 87
      flat  flat%   sum%        cum   cum%
     1.20s 42.86% 42.86%      1.20s 42.86%  syscall.Syscall
     0.60s 21.43% 64.29%      1.80s 64.29%  runtime.mallocgc
     0.22s  7.86% 72.14%      0.22s  7.86%  encoding/json.Marshal
     0.18s  6.43% 78.57%      0.50s 17.86%  db.(*Rows).Next
     0.12s  4.29% 82.86%      0.12s  4.29%  runtime.gcBgMarkWorker
     0.05s  1.79% 84.64%      1.05s 37.50%  main.(*Handler).ServeHTTP
     0.03s  1.07% 85.71%      0.60s 21.43%  main.fetchUsers
     0.01s  0.36% 86.07%      0.01s  0.36%  sync.(*Mutex).Lock

Let’s dissect every column:

flat and flat%: Time spent executing inside this function itself — not in any function it calls. When flat is high, this function is doing the work directly. syscall.Syscall at 1.20s flat means the program spent 1.20 seconds actually executing the syscall instruction. Note that encoding/json.Marshal at 0.22s flat means Marshal itself (not counting the functions it calls internally) used 0.22s.

sum%: Running total of flat%. After syscall.Syscall (42.86%) and mallocgc (21.43%), we’ve accounted for 64.29% of the total CPU time. This helps you know when you’ve found “enough” — once sum% hits 80-90%, you’ve identified the dominant work.

cum and cum%: Cumulative time — the time spent in this function and all functions it calls. main.(*Handler).ServeHTTP has only 0.05s flat but 1.05s cum. That means ServeHTTP itself barely does any work — it just calls other functions (like fetchUsers, which calls db.(*Rows).Next, etc.) and those sub-functions are where the time is.

The key insight: When you see high cum with low flat, the function is a dispatcher — it calls things, it doesn’t do things. Follow the call tree down to find the function with high flat. That’s your target.

Sorting options:

(pprof) top10 -flat    # sort by flat time (default)
(pprof) top10 -cum     # sort by cumulative time
(pprof) top10 -name    # sort alphabetically

The `list` Command

Once you’ve identified a suspicious function, list shows you the source code with per-line timing:

(pprof) list fetchUsers
Total: 2.8s
ROUTINE ======================== main.fetchUsers in /home/user/service/handler.go
      30ms      600ms (flat, cum) 21.43% of Total
         .          .     45:func (h *Handler) fetchUsers(ctx context.Context, filter string) ([]User, error) {
         .          .     46:    query := fmt.Sprintf("SELECT id, name, email FROM users WHERE name LIKE '%%%s%%'", filter)
      10ms       10ms     47:    rows, err := h.db.QueryContext(ctx, query)
         .          .     48:    if err != nil {
         .          .     49:        return nil, err
         .          .     50:    }
         .          .     51:    defer rows.Close()
         .          .     52:
         .          .     53:    var users []User
      10ms      550ms     54:    for rows.Next() {
         .          .     55:        var u User
      10ms       10ms     56:        if err := rows.Scan(&u.ID, &u.Name, &u.Email); err != nil {
         .          .     57:            return nil, err
         .          .     58:        }
         .          .     59:        users = append(users, u)
         .          .     60:    }
         .          .     61:    return users, nil
         .          .     62:}

The dot . means no samples were recorded at that line. Line 54 (rows.Next()) shows 550ms cumulative — the iteration over the result set is where most time goes. This tells you the database round-trips (fetching rows, network I/O) are the bottleneck, not the Go code.

Also notice line 46: fmt.Sprintf to build the SQL query with %%%s%%. This is a format string building a LIKE query. Every call to this function allocates a new string. For a high-traffic endpoint, this is unnecessary. A parameterized query would be faster and also safer (no SQL injection risk).

The `web` Command — Real CPU Call Graph

(pprof) web

This opens a call graph in your browser. Each box is a function, arrow weight represents time flowing through that call, and box size corresponds to flat time. Below is the actual pprof call graph generated from the benchmark above:

Open CPU call graph (SVG)

Real pprof CPU call graph — generated with go tool pprof -svg cpu.out. Click and drag to pan, scroll to zoom.

In the graph, the large red/pink boxes are the hot functions. The runtime nodes (scheduler, GC) dominate because the benchmark allocates heavily enough to keep the GC constantly working — itself a signal that allocation rate is too high. The demo.buildReportSlow and encoding/json.Marshal boxes connect into runtime.mallocgc, showing the allocation chain clearly.

The Flame Graph: The Most Useful Visualization

go tool pprof -http=:8090 cpu.out

This opens a web UI with multiple views including the flame graph. The flame graph is often the best way to understand a profile at a glance.

       ┌─────── json.Marshal ──────┐ ┌── strconv.AppendInt ──┐
    ┌─────────────── main.encodeResponse ───────────────────────────────┐
 ┌──────────────────────── main.(*Handler).ServeHTTP ────────────────────────────────┐
 ┌──────────────────────────────── net/http.(*conn).serve ─────────────────────────────────────────────────┐
 ←─────────────────────────────────────────── 2.8s total ─────────────────────────────────────────────────→

Reading a flame graph:

The X-axis represents total time. The Y-axis represents call stack depth. The bottom of the chart is the entry point (root functions like net/http server code or main). As you go up, you go deeper into the call stack.

Width = time. A function that takes up a wide horizontal area is consuming a lot of CPU time. A narrow bar means it’s fast or rarely called.

The flat top is where the work happens. When a function bar has nothing above it — no caller higher in the stack — that’s a leaf function. It’s where the CPU is actually executing instructions. Wide leaf functions are your optimization targets.

Plateaus indicate hot spots. If you see a “plateau” — a wide flat area at a certain height with nothing above it — that’s a function spending CPU time directly. Clicking on it in the interactive view will highlight all call paths that lead to it.

Thin slivers mean fast functions. A very thin vertical bar means a function is either fast or called infrequently. Ignore these.

The color doesn’t mean hot vs cold in pprof’s default flame graph — it’s just used to distinguish different call paths visually. (Some flame graph tools like Brendan Gregg’s flamegraph.pl do use red for hot functions, but pprof uses color categorically.)

Reading Profiles: The Heap Profile

Memory profiles require a slightly different mental model from CPU profiles. You’re not asking “where does time go?” — you’re asking “where is memory being held, and who put it there?”

Before diving into the commands, it helps to understand what the heap profile is actually recording.

What the Heap Profile Records

Every time your Go program allocates memory on the heap, the runtime has an opportunity to record that allocation. By default, the runtime samples allocations on average once per 512 KB allocated (probabilistic, not deterministic per-512KB; controlled by runtime.MemProfileRate). Each sample captures the call stack at the point of allocation — so the profile tells you not just how much memory is alive, but which function allocated it and how it was called.

This is important: the heap profile attributes memory to the function that allocated it, not necessarily the function that’s using it. If json.Marshal allocates a []byte that gets passed up the call stack and stored somewhere, the heap profile will show json.Marshal as the allocating function — even though the long-lived reference is held somewhere higher up.

The Four Heap Profile Views

Every heap profile file contains four distinct datasets. You choose which one to view with a flag:

Flag	What it shows	Best used for
`-inuse_space`	Bytes currently live on the heap	Debugging high RSS / memory usage
`-inuse_objects`	Count of objects currently live	Finding which types are numerous
`-alloc_space`	Total bytes allocated since program start	Debugging GC pressure, high allocation rate
`-alloc_objects`	Total allocation count since program start	Finding frequently-allocated types

curl -o heap.out "http://localhost:6060/debug/pprof/heap"

go tool pprof -inuse_space   heap.out   # default view
go tool pprof -inuse_objects heap.out
go tool pprof -alloc_space   heap.out
go tool pprof -alloc_objects heap.out

Understanding inuse_space vs alloc_space

This distinction trips up nearly everyone the first time. Let’s be very concrete with a story.

Imagine your HTTP service has been running for 1 hour. During that hour, it processed 100,000 requests. Each request allocated a []byte response buffer of about 50KB, used it, and the GC later freed it. It also built a cache that now holds 200MB of user data.

alloc_space would show the response buffer allocations prominently: 100,000 × 50KB = ~5GB total allocated over the hour. The cache allocations would also appear here.
inuse_space would show mainly the cache: the response buffers are all gone (freed), so only the 200MB of live cache data appears.

alloc_space (total ever allocated):
   5.00GB   main.buildResponseBuffer   <-- huge, but all freed
   0.20GB   main.(*UserCache).populate

inuse_space (alive right now):
   0.20GB   main.(*UserCache).populate <-- this is what's in memory
   0.00GB   main.buildResponseBuffer   <-- gone, GC freed it

Rule of thumb:

High RSS / OOM / kubectl top shows too much memory → use -inuse_space
High CPU from GC / many GC cycles per second / slow allocation-heavy path → use -alloc_space

Both tell you something real. Sometimes you need both to get the full picture.

Collecting and Opening

# Take a snapshot of the heap right now
curl -o heap.out "http://localhost:6060/debug/pprof/heap"

# Open interactively
go tool pprof -inuse_space heap.out

# Or open the web UI with flame graph
go tool pprof -http=:8090 -inuse_space heap.out

To force a GC before snapshotting heap, call runtime.GC() from inside the process (or via a small admin endpoint you add yourself) — there is no built-in pprof endpoint that triggers GC. In your Go code:

import (
    "runtime"
    "runtime/pprof"
)

// Force GC so the snapshot reflects only truly-live objects
runtime.GC()

f, _ := os.Create("heap.out")
defer f.Close()
pprof.WriteHeapProfile(f)

Real Profile Output: What You’ll Actually See

The outputs below are real — generated by running a benchmark with intentional allocation hot spots using go test -bench=BenchmarkAll -benchmem -cpuprofile=cpu.out -memprofile=mem.out.

The benchmark exercises three patterns:

buildReportSlow: builds a string with += in a loop — O(n²) allocations
marshalSlow: calls json.Marshal which allocates a full []byte on each call
processUsers: calls fmt.Sprintf per user to build labels

Benchmark results — this is what you see before even opening pprof:

BenchmarkBuildReportSlow-11    40176    89214 ns/op    844161 B/op     800 allocs/op
BenchmarkBuildReportFast-11   187976    19255 ns/op     20008 B/op     402 allocs/op
BenchmarkMarshalSlow-11       161523    22209 ns/op     18496 B/op       2 allocs/op
BenchmarkMarshalFast-11       153967    23324 ns/op     41026 B/op       4 allocs/op
BenchmarkProcessUsers-11       69067    52088 ns/op     30093 B/op    1746 allocs/op
BenchmarkAll-11                22957   157441 ns/op    914053 B/op    1407 allocs/op

BenchmarkBuildReportSlow at 800 allocs/op and 844KB per call vs BenchmarkBuildReportFast at 402 allocs/op and 20KB per call — the strings.Builder version uses 42× less memory per call. The -benchmem flag gives you this for free, before you even open pprof.

Reading the top Output for Heap

Running go tool pprof -alloc_space mem.out then top10 -cum on the real profile above:

Type: alloc_space
Duration: 8.11s, Total samples = 36577.85MB

      flat  flat%   sum%        cum   cum%
         0     0%     0%  36574.14MB   100%  demo.BenchmarkAll
         0     0%     0%  36574.14MB   100%  testing.(*B).runN
         0     0%     0%  36573.14MB   100%  testing.(*B).launch
32935.27MB 90.04% 90.04%  33339.32MB 91.15%  demo.buildReportSlow
    0.50MB  0.00% 90.04%   2782.90MB  7.61%  demo.marshalSlow (inline)
 1558.32MB  4.26% 94.30%   2782.40MB  7.61%  encoding/json.Marshal
    0.00MB  0.00% 94.30%   1191.54MB  3.26%  encoding/json.(*encodeState).marshal
    0.00MB  0.00% 94.30%   1191.54MB  3.26%  encoding/json.arrayEncoder.encode
    0.00MB  0.00% 94.30%   1191.54MB  3.26%  encoding/json.sliceEncoder.encode
  266.42MB  0.73% 95.03%    451.92MB  1.24%  demo.processUsers

The flat and cum columns work the same as in CPU profiles, but measure memory instead of time:

demo.buildReportSlow has 32,935MB flat — it directly allocated nearly 33GB worth of string data over the benchmark run. This is the string concatenation += in a loop, each iteration copying the entire accumulated string into a new allocation.
demo.marshalSlow has only 0.50MB flat but 2,782MB cum — marshalSlow itself allocates almost nothing, but it calls json.Marshal which allocates the full response []byte on every call. The 0.50MB flat is just the slice header.
encoding/json.Marshal at 1,558MB flat — these are the actual []byte response buffers being allocated fresh each time.
demo.processUsers at 266MB flat — all those fmt.Sprintf calls building label strings.

The pattern to read: high flat = this is the allocation site. High cum, low flat = this function is a caller that owns the cost indirectly.

Sorting and filtering:

(pprof) top10 -cum         # show by cumulative (understand call chain ownership)
(pprof) top10 -flat        # show by direct allocation (find the actual allocating line)
(pprof) top -cum -n 20     # top 20 by cum

Start with -cum to find which part of the call tree is responsible, then switch to -flat to find the exact function doing the allocating.

Drilling Into Source with list

Once top points you at a function, list shows the exact lines:

(pprof) list buildReportSlow
Total: 36577.85MB
ROUTINE ======================== demo.buildReportSlow
   32.16GB    32.56GB (flat, cum) 91.15% of Total
         .          .     23:func buildReportSlow(users []User) string {
         .          .     24:    result := ""
         .          .     25:    for _, u := range users {
         .          .     26:        // Each + allocates a new string
   32.16GB    32.56GB     27:        result += fmt.Sprintf("User %d: %s <%s>\n", u.ID, u.Name, u.Email)
         .          .     28:    }
         .          .     29:    return result
         .          .     30:}

Line 27 is the entire problem — 32GB attributed to one line (this is total bytes allocated and freed across all benchmark iterations, not memory live at any one moment). Every += on a string in Go creates a brand new string by copying the old string plus the new suffix. With 200 users, that’s 200 copies of increasing length: copy 0 chars, copy ~40 chars, copy ~80 chars, … copy ~8000 chars. Total work is O(n²) in the length of the result.

The tree output confirms the call chain:

(pprof) tree buildReportSlow
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum  cum%   calls calls% + context
----------------------------------------------------------+-------------
                               33339.32MB 100% |   demo.BenchmarkAll
32935.27MB 90.04% 90.04%  33339.32MB 91.15%   |   demo.buildReportSlow
                                 404.06MB  1.21% |     fmt.Sprintf
----------------------------------------------------------+-------------
                                 404.06MB 100% |   demo.buildReportSlow
  368.02MB  1.01% 91.05%    404.06MB  1.10%   |   fmt.Sprintf
----------------------------------------------------------+-------------

buildReportSlow owns 90% of all allocations. fmt.Sprintf inside it accounts for another 404MB (the intermediate formatted strings). The string concatenation itself — the += — is the 32GB, because every loop iteration copies all previous output.

Real Memory Call Graph

Here is the actual go tool pprof -alloc_space -svg mem.out output from the same benchmark — a real memory call graph, not a diagram:

Open memory call graph (SVG)

Real pprof memory call graph — generated with go tool pprof -alloc_space -svg mem.out. Click and drag to pan, scroll to zoom.

demo.buildReportSlow dominates the graph at 90% of total allocations — its box is the largest. The arrows flow down into fmt.Sprintf and runtime.mallocgc. encoding/json.Marshal is the second cluster. The relative box sizes make the 90/8/2 split visible at a glance.

To generate this yourself from any profile:

go tool pprof -alloc_space -svg mem.out > mem-callgraph.svg
go tool pprof -svg cpu.out > cpu-callgraph.svg

Requires Graphviz (brew install graphviz on macOS).

Using the Heap Flame Graph

The interactive flame graph is the fastest way to see the allocation picture at a glance. Run:

go tool pprof -http=:8090 -alloc_space mem.out

Your browser opens at localhost:8090. Click Flame Graph in the top nav. For the profile above, you’d see a single dominant bar — buildReportSlow taking up ~90% of the horizontal width, with marshalSlow/json.Marshal occupying the remaining ~8%. The overwhelming width of buildReportSlow makes the problem impossible to miss visually; you don’t even need to read the numbers.

Here’s the real pprof flame graph captured from running this exact benchmark:

Real pprof flame graph (alloc_space) — demo.buildReportSlow is the wide bar on the left, demo.marshalSlow/json.Marshal on the right. Width = allocated bytes.

In the web UI, the flame graph for heap profiles reads exactly like the CPU flame graph, but width = memory. A wide bar means that call path is responsible for a lot of allocated memory.

Switch between views in the top navigation:

Flame graph: visual call tree
Top: the top table
Source: annotated source (same as list)
Peek: shows callers of a function

One particularly useful feature: in the flame graph, you can click a function bar to filter the view to only that function’s subtree. This helps when a big allocator like json.Marshal appears in many call paths — you can isolate just the path you care about.

The `-base` Flag: Finding What Grew

The most powerful technique for tracking memory leaks is taking two snapshots and comparing them. The diff tells you exactly what grew — not what’s large in general, but what’s larger now than before.

# Step 1: take the first snapshot before the suspected leak period
curl -o heap1.out "http://localhost:6060/debug/pprof/heap"
echo "Snapshot 1 at $(date)"

# Step 2: let the service run for a while under normal traffic
# The leak will accumulate during this time

# Step 3: take the second snapshot
curl -o heap2.out "http://localhost:6060/debug/pprof/heap"
echo "Snapshot 2 at $(date)"

# Step 4: diff them — what grew?
go tool pprof -inuse_space -base heap1.out heap2.out

The output now shows deltas — positive numbers mean that function is holding more memory in snapshot 2:

(pprof) top10
Showing nodes accounting for +52.40MB
      flat  flat%   sum%        cum   cum%
  +42.00MB 80.15% 80.15%  +42.00MB 80.15%  main.(*EventStore).Append
  +10.40MB 19.85% 100.00%  +52.40MB 100.00%  main.(*MetricsCollector).Record

If you see a function with steadily growing positive values across multiple snapshots taken over time, that’s your leak.

Taking periodic snapshots automatically:

For production leak investigations, a simple shell loop works well:

for i in 1 2 3 4 5; do
    curl -s -o "heap_$i.out" "http://prod-host:6060/debug/pprof/heap"
    echo "Snapshot $i at $(date)"
    sleep 300  # 5 minutes between snapshots
done

# Then compare snapshot 1 vs 5 to see total growth
go tool pprof -inuse_space -base heap_1.out heap_5.out

# And compare consecutive snapshots to see the rate of growth
go tool pprof -inuse_space -base heap_1.out heap_2.out
go tool pprof -inuse_space -base heap_2.out heap_3.out

If the growth is consistent across consecutive diffs, the leak is active throughout. If only some diffs show growth, the leak might be triggered by a specific condition (a particular request type, time-of-day behavior, etc.).

Understanding Memory Profile Sampling

Heap profiling is not 100% complete — it’s sampled. By default, the runtime samples allocations on average once per 512 KB allocated (probabilistic, not deterministic per-512KB). This means small allocations that happen infrequently might not appear in the profile.

You can lower the sampling threshold to catch smaller allocations:

import "runtime"

func init() {
    // Record every single allocation (maximum detail, higher overhead)
    // Default is 512*1024 (512KB)
    runtime.MemProfileRate = 1
}

Setting MemProfileRate = 1 means every allocation is recorded. This is useful when investigating a suspected small-object leak that isn’t showing up in the default profile, but it increases overhead significantly and should only be used for targeted debugging, not in production.

What “RSS vs Heap” Means and Why They Differ

One thing that confuses people: the heap profile might show 200MB live but your process’s RSS (resident set size — what top or kubectl top reports) might be 400MB. Why the gap?

Several reasons:

Go runtime overhead: The Go scheduler, goroutine stacks, internal runtime structures, and metadata all consume memory that doesn’t show up in the heap profile.
Stack memory: Each goroutine has a stack. 1,000 goroutines × 8KB minimum stack = 8MB just in stacks. Stacks can grow much larger.
Memory fragmentation: The OS allocates memory in pages (typically 4KB or larger). Even if your heap objects total 200MB, the OS might have mapped 250MB of pages to the process.
Go’s memory retention: Go doesn’t immediately return freed memory to the OS. After the GC frees objects, the memory stays in Go’s heap arena until the OS explicitly needs it back or until runtime.FreeOSMemory() is called (or MADV_FREE/MADV_DONTNEED is used on Linux).

If you need to shrink RSS without reducing actual live memory, you can call debug.FreeOSMemory(), which forces the runtime to return unused heap pages to the OS. This is a one-shot operation though — it doesn’t change the long-term allocation pattern.

import "runtime/debug"

// Force return of unused memory pages to the OS
// Rarely needed — the runtime handles this automatically over time
debug.FreeOSMemory()

Using runtime/metrics for Continuous Monitoring

Instead of taking snapshots manually, you can expose heap metrics continuously via runtime/metrics (Go 1.16+):

package main

import (
    "fmt"
    "runtime/metrics"
)

func printHeapStats() {
    // Define which metrics you want
    samples := []metrics.Sample{
        {Name: "/memory/classes/heap/objects:bytes"},   // live heap objects
        {Name: "/memory/classes/heap/released:bytes"},  // released to OS
        {Name: "/memory/classes/heap/free:bytes"},      // free but retained
        {Name: "/gc/heap/allocs:bytes"},                 // cumulative allocs
        {Name: "/gc/cycles/total:gc-cycles"},            // total GC cycles
    }

    metrics.Read(samples)

    for _, s := range samples {
        if s.Value.Kind() == metrics.KindUint64 {
            fmt.Printf("%s = %d\n", s.Name, s.Value.Uint64())
        }
    }
}

Exposing these as Prometheus or Datadog metrics gives you continuous visibility without needing to collect pprof snapshots. You’d typically alert on heap/objects:bytes growing without bound over days, and then use pprof snapshots to investigate the cause once the alert fires.

Reading Profiles: Goroutine Dumps

The goroutine profile is different from CPU and heap — it’s not a statistical sample. It’s an instant snapshot of every goroutine that exists right now, with their complete call stacks.

Collecting

# debug=1: just goroutine counts per state
curl "http://localhost:6060/debug/pprof/goroutine?debug=1"

# debug=2: full stack traces for every goroutine (this is what you want)
curl "http://localhost:6060/debug/pprof/goroutine?debug=2"

What a Goroutine Dump Looks Like

goroutine 1 [running]:
main.main()
    /home/user/service/main.go:42 +0x1a4

goroutine 18 [sleep]:
time.Sleep(0x77359400)
    /usr/local/go/src/runtime/time.go:195 +0xd2
main.backgroundJob()
    /home/user/service/jobs.go:23 +0x45
created by main.startJobs in goroutine 1
    /home/user/service/main.go:38 +0x6e

goroutine 23 [chan receive]:
main.processQueue(0xc00012a000)
    /home/user/service/worker.go:45 +0x89
created by main.startWorkers in goroutine 1
    /home/user/service/main.go:52 +0xb2

goroutine 24 [chan receive]:
main.processQueue(0xc00012a000)
    /home/user/service/worker.go:45 +0x89
created by main.startWorkers in goroutine 1
    /home/user/service/main.go:52 +0xb2

... (200 more goroutines in exactly the same state)

The goroutine state is the critical signal. Each goroutine shows its current state in brackets:

[running] — currently executing on a CPU
[sleep] — in time.Sleep() or similar
[chan receive] — blocked waiting for a channel to have data
[chan send] — blocked waiting for a channel to accept data
[IO wait] — blocked on I/O (network, file)
[semacquire] — waiting to acquire a mutex or semaphore
[select] — blocked in a select statement
[syscall] — executing a system call

What to look for:

Goroutine count that’s unexpectedly high. A healthy service might have hundreds of goroutines — one per active HTTP connection, some for background jobs, some for the runtime itself. But if you see 10,000 goroutines and you have only 100 active requests, something is leaking.
Many goroutines in the same state at the same call stack. If you see 500 goroutines all sitting at chan receive on the exact same line, that’s a single channel nobody is reading from. Everything that tries to write to it blocks.
Goroutines stuck in syscall state for a long time. System calls should be brief. A goroutine that’s been in syscall for a long time (seconds) is likely doing a blocking network operation with no timeout.
created by X pointing to the same place. If you see thousands of goroutines all created by the same function, that function is creating goroutines faster than they’re completing.

Goroutine Leak Example

Here’s a classic goroutine leak pattern and what it looks like in a dump:

// BUGGY CODE: this leaks a goroutine every time processRequest is called
func processRequest(req *Request) {
    resultCh := make(chan Result)  // unbuffered channel

    go func() {
        result := doExpensiveWork(req)
        resultCh <- result  // THIS BLOCKS if nobody reads the channel
    }()

    select {
    case result := <-resultCh:
        sendResponse(result)
    case <-time.After(100 * time.Millisecond):
        sendTimeout()
        // We return here but the goroutine is still running, blocked on resultCh <- result
        // Nobody will ever read from resultCh now
        // The goroutine is leaked
        return
    }
}

In the goroutine dump, you’d see hundreds of goroutines like:

goroutine 1842 [chan send, 45 minutes]:
main.processRequest.func1()
    /home/user/service/handler.go:67 +0x94
created by main.processRequest in goroutine 1234
    /home/user/service/handler.go:62 +0x4e

The 45 minutes shows how long the goroutine has been blocked — a clear sign it’s stuck forever.

The fix:

func processRequest(req *Request) {
    // Use a context with timeout so the goroutine gets a signal to stop
    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel() // This signals the goroutine even if we return early

    resultCh := make(chan Result, 1) // Buffered — goroutine can always send

    go func() {
        result := doExpensiveWork(ctx, req) // Pass ctx so work can abort
        select {
        case resultCh <- result:
        case <-ctx.Done(): // If timeout happened, don't block
        }
    }()

    select {
    case result := <-resultCh:
        sendResponse(result)
    case <-ctx.Done():
        sendTimeout()
    }
}

Reading Profiles: The Execution Trace

The trace is the most powerful profiling tool but also the most complex to interpret. Use it when other profiles haven’t given you the answer.

Collecting and Opening

# Collect 5 seconds of trace (use a short window — files get large fast)
curl -o trace.out "http://localhost:6060/debug/pprof/trace?seconds=5"

# Open in browser
go tool trace trace.out

This opens a browser UI at http://localhost:PORT/ with several views.

The Views and What They Tell You

Goroutine analysis — This shows time breakdown per goroutine across five states:

Execution time: goroutine was running on a CPU
Network wait: goroutine was waiting for network I/O
Sync block: goroutine was waiting on a mutex/channel/semaphore
Scheduler wait: goroutine was ready to run but waiting for a CPU slot
GC time: goroutine was paused or helping with GC

A goroutine with 10% Execution and 80% Sync block is spending most of its time waiting — likely for a slow database or a contested lock.

View trace — The raw timeline view. You see:

All goroutines as horizontal lanes
Color indicates state (running, blocked, network wait, etc.)
GC events shown as spans that overlay the goroutine timeline
You can zoom in to microsecond resolution

If you see periodic “gaps” across all goroutines at the same time — that’s a GC stop-the-world pause. The length of that gap is your STW pause latency.

Minimum Mutator Utilization (MMU) — This chart directly answers “what percentage of the time was my code actually running (as opposed to being stopped for GC)?” The Y-axis is utilization (1.0 = 100% of time running), the X-axis is window size (looking at any N-second window, what’s the minimum utilization?). If the MMU at 1ms is 0.80, it means in the worst case, your code was only running 80% of the time in any 1ms window — 20% was GC pauses.

Mutex and Block Profiles in Detail

Mutex Profile

After enabling with runtime.SetMutexProfileFraction(5):

go tool pprof http://localhost:6060/debug/pprof/mutex

(pprof) top5
Showing nodes accounting for 8.50s, 95.25% of 8.92s total
      flat  flat%   sum%        cum   cum%
     4.20s 47.09% 47.09%      4.20s 47.09%  sync.(*Mutex).Unlock
     2.10s 23.54% 70.63%      6.30s 70.63%  main.(*Cache).Get
     1.50s 16.82% 87.45%      1.50s 16.82%  main.(*Cache).Set

The times here represent how long other goroutines waited because of contention on this mutex. Cache.Get causing 6.3s of contention means goroutines collectively waited 6.3 seconds trying to acquire the cache’s lock.

The fix pattern: if reads far outnumber writes, switch from sync.Mutex to sync.RWMutex and use RLock()/RUnlock() for reads. Multiple goroutines can hold an RLock simultaneously — only writers need exclusive access.

// Before: every operation, read or write, blocks everyone
type Cache struct {
    mu    sync.Mutex
    items map[string]string
}

func (c *Cache) Get(key string) (string, bool) {
    c.mu.Lock()
    defer c.mu.Unlock()
    v, ok := c.items[key]
    return v, ok
}

// After: reads can happen concurrently, only writes are exclusive
type Cache struct {
    mu    sync.RWMutex
    items map[string]string
}

func (c *Cache) Get(key string) (string, bool) {
    c.mu.RLock()         // Multiple goroutines can hold RLock simultaneously
    defer c.mu.RUnlock()
    v, ok := c.items[key]
    return v, ok
}

func (c *Cache) Set(key, value string) {
    c.mu.Lock()          // Only one writer at a time
    defer c.mu.Unlock()
    c.items[key] = value
}

Block Profile

After enabling with runtime.SetBlockProfileRate(1):

go tool pprof http://localhost:6060/debug/pprof/block

(pprof) top5
Showing nodes accounting for 15.2s, 87.36% of 17.4s total
      flat  flat%   sum%        cum   cum%
     8.40s 48.28% 48.28%      8.40s 48.28%  main.(*DB).query
     4.20s 24.14% 72.41%      4.20s 24.14%  runtime.selectgo
     2.10s 12.07% 84.48%      2.10s 12.07%  sync.(*WaitGroup).Wait

DB.query blocking 8.4s means goroutines spent 8.4 seconds waiting in that function. Combined with looking at the source code, this usually means waiting on a connection pool (not enough connections available) or slow query results.

runtime.selectgo blocking means goroutines are blocking in select statements — either waiting for a channel or a timeout. This might be expected (workers waiting for jobs) or a sign of slow producers.

Common Scenarios: Step-by-Step Walkthroughs

Scenario 1: The “Everything Is Slow” Investigation

You get a report: “The API is slow.” P50 is 200ms, P99 is 2 seconds. CPU is at 60%. Where do you start?

Step 1: CPU profile first because CPU is elevated.

# Generate some load first (use hey, wrk, or k6)
hey -n 10000 -c 50 http://localhost:8080/api/users

# In another terminal, collect the profile
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

(pprof) top10
      flat  flat%   sum%        cum   cum%
     1.80s 64.29% 64.29%      1.80s 64.29%  runtime.mallocgc
     0.40s 14.29% 78.57%      0.40s 14.29%  encoding/json.Marshal
     0.30s 10.71% 89.29%      2.50s 89.29%  main.(*Handler).getUsers

mallocgc is the garbage collector’s allocation function. It being at 64% means the program is spending most of its CPU time allocating memory, not doing actual work. This is GC pressure — the allocation rate is so high that the GC can’t keep up.

Step 2: Find what’s allocating.

# Get the allocs profile (not heap — allocs shows total allocation rate)
go tool pprof http://localhost:6060/debug/pprof/allocs

(pprof) top10 -alloc_space
      flat  flat%   sum%        cum   cum%
  1.20GB 48.00% 48.00%    1.20GB 48.00%  encoding/json.Marshal
  0.80GB 32.00% 80.00%    2.00GB 80.00%  main.(*Handler).getUsers
  0.30GB 12.00% 92.00%    0.30GB 12.00%  fmt.Sprintf

json.Marshal is allocating 1.2GB total (this is over the 30 second period). For 10,000 requests, that’s 120KB of allocation per request just for JSON serialization.

Step 3: List the hot function.

(pprof) list getUsers
...
     .     1.2GB     78:    data, err := json.Marshal(users)

Every request is marshaling a potentially large users slice into JSON from scratch. Fixes:

Use json.NewEncoder(w).Encode(users) — streams directly to the response writer, avoids allocating the []byte intermediate
If the data is cacheable, cache the serialized result
Use sync.Pool for the encoder itself

Step 4: Implement, re-profile, confirm.

// Before
func (h *Handler) getUsers(w http.ResponseWriter, r *http.Request) {
    users, _ := h.store.ListUsers()
    data, err := json.Marshal(users)    // allocates a []byte
    if err != nil {
        http.Error(w, err.Error(), 500)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    w.Write(data)
}

// After
func (h *Handler) getUsers(w http.ResponseWriter, r *http.Request) {
    users, _ := h.store.ListUsers()
    w.Header().Set("Content-Type", "application/json")
    // json.NewEncoder streams to w directly — no intermediate allocation
    if err := json.NewEncoder(w).Encode(users); err != nil {
        // Can't set status code here (headers already sent), so log it
        log.Printf("encode error: %v", err)
    }
}

Re-profile and compare. The mallocgc entry should drop significantly.

Scenario 2: Memory Leak Investigation

You notice your service’s memory usage grows by about 50MB per hour and never comes back down. The service runs fine for a day but needs to be restarted weekly.

Step 1: Capture two snapshots separated by time.

# Capture the first snapshot (note the time)
curl -o heap1.out "http://localhost:6060/debug/pprof/heap"
echo "Snapshot 1 taken at $(date)"

# Wait 1 hour, or until you see significant memory growth
sleep 3600

# Capture the second snapshot
curl -o heap2.out "http://localhost:6060/debug/pprof/heap"
echo "Snapshot 2 taken at $(date)"

Step 2: Compare.

go tool pprof -inuse_space -base heap1.out heap2.out

(pprof) top5
Showing nodes accounting for +52.40MB
      flat  flat%   sum%        cum   cum%
  +42.00MB 80.15% 80.15%  +42.00MB 80.15%  main.(*EventStore).Append
  +10.40MB 19.85% 100.00%  +52.40MB 100.00%  main.(*MetricsCollector).Record

EventStore.Append grew by 42MB. This is your leak.

Step 3: Inspect the code.

// Probable buggy code
type EventStore struct {
    events []Event  // this slice grows unboundedly
    mu     sync.Mutex
}

func (s *EventStore) Append(e Event) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.events = append(s.events, e)  // never trimmed, never rotated
}

The events slice keeps all events forever with no eviction, rotation, or size limit.

Fix options:

Keep only the last N events with a ring buffer
Write events to an external store (database, message queue) and don’t keep them in memory
Add a TTL-based eviction: periodically trim events older than some duration

Scenario 3: Diagnosing p99 Latency Spikes

P50 is 10ms, P99 is 500ms. The requests that are slow seem random — no pattern in the payload. CPU is low, memory is stable. This is a classic GC pause pattern.

Step 1: Capture a trace during load.

# Apply load
hey -n 5000 -c 100 http://localhost:8080/api/data

# Capture trace simultaneously (5 second window is usually enough)
curl -o trace.out "http://localhost:6060/debug/pprof/trace?seconds=5"
go tool trace trace.out

In the browser, open “View trace” and zoom in on the timeline. Look for moments where all goroutines stop simultaneously. These are GC stop-the-world pauses.

You might see:

STW mark termination: 2.3ms
STW sweep termination: 0.1ms
STW mark termination: 1.8ms
... repeating every 200ms ...

2.3ms STW pauses every 200ms. At 100 concurrent requests, a 2.3ms pause would cause all in-flight requests to stall. For a request that took 8ms without the pause, it now takes 10.3ms. But if a request is unlucky and hits two pauses, it takes 12.6ms. At p99, you’re seeing multiple pauses accumulate.

Step 2: Check allocation rate.

In the trace UI, look at “Goroutine analysis” and look at the GC goroutines. If gcBgMarkWorker is constantly present, the GC is running continuously — a sign of very high allocation rate.

Step 3: Reduce allocation rate.

The most effective fix is using sync.Pool for frequently-allocated objects:

// Before: every request allocates a new buffer
func handleRequest(w http.ResponseWriter, r *http.Request) {
    buf := new(bytes.Buffer)  // new allocation every request
    buildResponse(buf, r)
    w.Write(buf.Bytes())
}

// After: reuse buffers from a pool
var bufPool = sync.Pool{
    New: func() any {
        return new(bytes.Buffer)
    },
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
    buf := bufPool.Get().(*bytes.Buffer)
    buf.Reset()              // clear the buffer before use
    defer bufPool.Put(buf)   // return to pool when done

    buildResponse(buf, r)
    w.Write(buf.Bytes())
}

sync.Pool maintains a pool of objects that can be reused across goroutines. The GC clears the pool on each GC cycle (so pool objects don’t prevent garbage collection), but in the steady state between GC cycles, goroutines reuse the same buffer objects instead of allocating new ones.

Step 4: Tune the GC target if needed.

If you’ve reduced allocations and still see GC pressure, you can tune when the GC triggers:

import "runtime/debug"

func main() {
    // GOGC controls when GC triggers. Default is 100, meaning GC triggers
    // when heap grows 100% beyond the size after the last GC.
    // Setting to 200 means GC triggers less often but uses more memory.
    // This is a trade-off: less GC CPU, more RAM usage.
    debug.SetGCPercent(200)

    // GOMEMLIMIT (Go 1.19+) sets a soft memory limit.
    // The GC will work harder to stay under this limit.
    // This prevents the service from using so much memory it gets OOM-killed.
    debug.SetMemoryLimit(1 << 30) // 1 GB

    // ... rest of your program
}

Or via environment variables (no code change needed):

GOGC=200 GOMEMLIMIT=1073741824 ./myservice

Scenario 4: The Service Doesn’t Scale Under Load

Adding more CPUs (or increasing GOMAXPROCS) doesn’t improve throughput. This suggests serialization — something is forcing work to happen one at a time.

Profile: Mutex contention.

# Make sure SetMutexProfileFraction was called at startup
go tool pprof http://localhost:6060/debug/pprof/mutex

(pprof) top5
      flat  flat%   sum%        cum   cum%
    12.30s 68.33% 68.33%     12.30s 68.33%  main.(*RequestCounter).Increment
     4.20s 23.33% 91.67%      4.20s 23.33%  main.(*RateLimiter).Allow

A RequestCounter that’s being incremented on every request from every goroutine is a global shared state with a mutex. Every goroutine contends on it.

Fix: Use sync/atomic for simple counters.

// Before: mutex-protected counter is a bottleneck
type RequestCounter struct {
    mu    sync.Mutex
    count int64
}

func (c *RequestCounter) Increment() {
    c.mu.Lock()
    c.count++
    c.mu.Unlock()
}

func (c *RequestCounter) Get() int64 {
    c.mu.Lock()
    defer c.mu.Unlock()
    return c.count
}

// After: atomic operations require no lock
type RequestCounter struct {
    count int64 // must be 64-bit aligned for atomic ops (automatic on 64-bit platforms when the field is the first in the struct; matters on 32-bit ARM/x86 if you reorder fields)
}

func (c *RequestCounter) Increment() {
    atomic.AddInt64(&c.count, 1) // lock-free, safe from multiple goroutines
}

func (c *RequestCounter) Get() int64 {
    return atomic.LoadInt64(&c.count)
}

For more complex shared data structures, consider sharding — splitting one lock into N locks by key:

// Instead of one global cache with one lock:
type ShardedCache struct {
    shards [256]struct {
        mu    sync.RWMutex
        items map[string]string
    }
}

func (c *ShardedCache) shard(key string) int {
    // Simple hash to pick a shard
    h := fnv.New32a()
    h.Write([]byte(key))
    return int(h.Sum32()) % 256
}

func (c *ShardedCache) Get(key string) (string, bool) {
    s := c.shard(key)
    c.shards[s].mu.RLock()
    defer c.shards[s].mu.RUnlock()
    v, ok := c.shards[s].items[key]
    return v, ok
}

Now instead of all 256 goroutines contending on a single lock, each goroutine contends with only ~1 other goroutine on average (because keys are distributed across 256 shards).

Visual Guide: Reading Flame Graphs and Memory Diagrams

This section walks through what the go tool pprof web UI actually looks like and how to navigate it. Since screenshots from a live pprof session depend on your specific program, the annotated diagrams below replicate the structure and layout of what you’ll see — so when you open the real UI, you’ll know exactly what you’re looking at.

Opening the Web UI

go tool pprof -http=:8090 cpu.out

Your browser opens to http://localhost:8090. The top navigation bar has these views:

[ Top ] [ Graph ] [ Flame Graph ] [ Peek ] [ Source ] [ Disassemble ]
                                      ^
                                  start here

Top — the top table you’ve already seen in the interactive CLI. Good first stop.

Graph — a directed call graph. Boxes are functions, arrows show call relationships, box size corresponds to flat cost, arrow weight corresponds to flow. Gets cluttered on large profiles but good for seeing call relationships.

Flame Graph — the flame graph. Best for understanding the full picture at a glance.

Peek — shows callers of the selected function. Useful for understanding who is responsible for a hot function you didn’t write.

Source — annotated source, same as the list command.

Anatomy of a CPU Flame Graph

Here’s what a CPU flame graph looks like for a typical HTTP service, annotated:

 WIDE AND AT THE TOP = hot leaf function (where CPU is burning)
 │
 │    ┌──────────────────────────────────────────────┐
 │    │          encoding/json.(*Encoder).Encode     │  ← 38% CPU, this is hot
 │    └──────────────────────────────────────────────┘
 │    ┌─────────────────────────────────────────────────────────────────┐
 │    │                   main.buildAPIResponse                         │
 │    └─────────────────────────────────────────────────────────────────┘
 │    ┌──────────────────────────────────────────────────────────────────────────────────┐
 │    │                           main.(*Handler).ServeHTTP                              │
 │    └──────────────────────────────────────────────────────────────────────────────────┘
 │    ┌──────────────────────────────────────────────────────────────────────────────────────────┐
 │    │                              net/http.(*conn).serve                                       │
 │    └──────────────────────────────────────────────────────────────────────────────────────────────┘
 ↓
 WIDE AND AT THE BOTTOM = root of the call stack (entry point)

Key patterns to recognize:

Pattern 1 — Flat top (direct hot spot)

 ┌────────────────────────────────────┐
 │         syscall.RawSyscall         │  ← wide, nothing above it
 └────────────────────────────────────┘  ← this IS the bottleneck
 ┌─────────────────────────────────────────┐
 │         net.(*netFD).Read               │
 └─────────────────────────────────────────┘
 ┌──────────────────────────────────────────────────────┐
 │             bufio.(*Reader).fill                     │
 └──────────────────────────────────────────────────────┘

syscall.RawSyscall has no children (nothing above it). The CPU is spending time executing the system call directly. The fix is either to reduce the number of syscalls (buffering, batching) or accept that I/O is the bound.

Pattern 2 — Wide middle (delegating hot spot)

                         ┌──────┐ ┌──────┐
                         │ fmt  │ │strco │  ← many small children
                         └──────┘ └──────┘
 ┌─────────────────────────────────────────────────────────────┐
 │                  encoding/json.Marshal                       │  ← wide, has children
 └─────────────────────────────────────────────────────────────┘

json.Marshal is wide but has children above it. The work is split across the children — you can’t optimize Marshal directly; you need to reduce how often it’s called or switch to a faster encoder.

Pattern 3 — “Tall spike” (rare deep call)

         ┌────┐
         │    │  ← narrow spike going deep
         └────┘
         │    │
         └────┘
         │    │
         └────┘
 ┌───────────────────────────────────────────────────────────┐
 │                                                           │
 └───────────────────────────────────────────────────────────┘

A narrow, tall column means a function that’s rarely called but goes deep into the call stack when it is. Usually not a performance bottleneck unless the narrow bar is also very wide (meaning it runs for a long time per call).

Pattern 4 — GC work (purple/distinct color)

 ┌──────────────────────────────┐
 │  runtime.gcBgMarkWorker      │  ← GC background mark worker
 └──────────────────────────────┘
 ┌────────────────────────────────┐
 │  runtime.mallocgc              │  ← allocator (if wide, you have alloc pressure)
 └────────────────────────────────┘

In pprof’s web UI, runtime functions are often shown in a distinct color. gcBgMarkWorker appearing wide means the GC is doing a lot of marking — your heap is large or changing fast. mallocgc wide means the allocator is being called constantly. Both are signals to investigate the allocs profile.

Anatomy of a Heap Flame Graph

The heap flame graph looks the same structurally, but the width represents memory, not time. Here’s an annotated version for -inuse_space:

 ┌───────────────────────────────────────────────────────────────────────┐
 │              bytes.(*Buffer) internal backing array                   │  ← 412MB live
 └───────────────────────────────────────────────────────────────────────┘
 ┌──────────────────────────────────────────────────────────────────────────────┐
 │                       main.buildResponse                                     │  ← 490MB cum
 └──────────────────────────────────────────────────────────────────────────────┘
 ┌──────────────────────────────────────────────────────────────────────────────────────────┐
 │                            net/http.(*conn).serve                                         │  ← everything below
 └──────────────────────────────────────────────────────────────────────────────────────────────┘

What to look for in a heap flame graph:

Wide bars near the top — the allocating functions. These are where your memory is actually coming from. If bytes.(*Buffer).Write is 60% wide, 60% of your live heap is buffer backing arrays.
Runtime functions — runtime.mallocgc, runtime.newobject, runtime.makeslice, runtime.makemap. If these appear prominently in the alloc view, your hot paths are allocating frequently.
Unexpected wide bars — functions you didn’t expect to be allocating much. A logging library appearing with 20% of allocations is worth investigating.
The “diff” flame graph — when you use -base, the flame graph shows bars colored by growth: green bars shrank (less memory), red bars grew. Wide red bars are your leak candidates.

The Heap Memory Flow Diagram

To really understand what the heap profile captures, it helps to trace the lifecycle of a single allocation through the system:

This sequence explains why heap profiles can sometimes show more memory than is actually in use: the profile is sampled at allocation time, and the GC hasn’t run yet to clean up the objects that have become unreachable since the last sample. This is why forcing a GC (runtime.GC()) before taking a heap snapshot gives you a cleaner picture.

Comparing Two Heap Snapshots Visually

The diff view (using -base) is one of the most important techniques. Here’s what to expect at each stage of a leak investigation:

Stage 1: Immediately after startup

inuse_space top5:
     flat   cum
   50.0MB  50.0MB  main.(*Config).load         ← config loaded into memory
   20.0MB  20.0MB  main.initRoutes             ← routing table
    5.0MB   5.0MB  sync.(*Map).Store           ← initial cache warmup

This is your baseline. Most of this is expected startup allocation.

Stage 2: After 1 hour of traffic (snapshot 2)

inuse_space top5:
     flat   cum
  892.0MB 892.0MB  main.(*EventStore).Append   ← WAS 0MB at startup
   50.0MB  50.0MB  main.(*Config).load         ← same as before (stable)
   20.0MB  20.0MB  main.initRoutes             ← same

EventStore.Append went from 0 to 892MB. That’s your leak.

Stage 3: The diff view (heap2 - heap1)

go tool pprof -inuse_space -base heap1.out heap2.out
(pprof) top5

      flat  flat%   cum   cum%
  +892.0MB 99.4%   +892.0MB 99.4%  main.(*EventStore).Append
    +0.0MB  0.0%     +0.0MB  0.0%  main.(*Config).load         ← no change

The diff clearly isolates the growth. Now use list to find the exact line:

(pprof) list EventStore.Append
ROUTINE ======================== main.(*EventStore).Append
  +892.0MB  +892.0MB (flat, cum)
         .          .     15: func (s *EventStore) Append(e Event) {
         .          .     16:     s.mu.Lock()
         .          .     17:     defer s.mu.Unlock()
  +892.0MB  +892.0MB     18:     s.events = append(s.events, e)  ← every call adds forever
         .          .     19: }

Line 18 is the culprit. The s.events slice grows on every request and is never trimmed. Fix: cap the slice size, use a ring buffer, or write events to an external store.

When you open go tool pprof -http=:8090 profile.out, here’s every URL and what it shows:

URL	Description
`/`	Redirects to `/ui/`
`/ui/`	Default view (usually Top)
`/ui/top`	Top table — flat and cum costs
`/ui/flamegraph`	Flame Graph — the most useful view
`/ui/graph`	Call graph (can get cluttered)
`/ui/peek`	Callers/callees of a selected function
`/ui/source`	Annotated source for a function
`/ui/disasm`	Assembly-level view (for low-level work)
`/ui/weblist`	Source code in browser with cost annotations

Keyboard shortcuts in the flame graph view:

Click a bar — zoom in to that function’s subtree
Click the root bar — zoom back out
Search box — type a function name to highlight matching bars across the chart
Mouse wheel / pinch — zoom in/out on the timeline axis

The search box is particularly useful. If you suspect json.Marshal is involved across multiple call paths, typing json will highlight every bar that contains the word “json” in its name — showing you all the different places JSON marshaling happens.

Continuous Profiling

The patterns above assume you collect a profile when something looks wrong. Modern teams flip this: profile all the time in production at low overhead, store profiles in a time-series backend, and query them by tag, time range, or commit hash — like metrics, but for code.

The major options:

Pyroscope (open source, now part of Grafana) — works with Go’s pprof endpoints out of the box, integrates with Grafana for flame-graph queries.
Grafana Phlare — Grafana Labs’ purpose-built profile storage, similar tradeoffs.
Polar Signals Parca — open source, eBPF-based for system-wide CPU profiling without code changes.
Datadog Continuous Profiler / Pixie — commercial offerings with deeper integration.

The win: when an incident hits, you don’t need to reproduce it under a profiler — yesterday’s profile is already there. The cost: a small steady CPU/network overhead and a profile-storage tier to operate.

The Decision Flowchart

When you observe a problem, here’s how to pick the right profile:

Optimization Techniques Reference

Once you’ve found a bottleneck, here are the most common fixes organized by category.

Reducing Allocations

Use strings.Builder instead of + concatenation:

// Bad: each + creates a new string (O(n²) total allocations)
result := ""
for _, s := range parts {
    result += s + ", "
}

// Good: Builder amortizes allocations
var b strings.Builder
b.Grow(estimatedSize) // optional: pre-allocate if you know the size
for i, s := range parts {
    b.WriteString(s)
    if i < len(parts)-1 {
        b.WriteString(", ")
    }
}
result := b.String()

Pre-allocate slices:

// Bad: append keeps reallocating the backing array
var users []User
for _, row := range rows {
    users = append(users, scanUser(row))
}

// Good: allocate once if you know the count
users := make([]User, 0, len(rows)) // capacity = len(rows)
for _, row := range rows {
    users = append(users, scanUser(row))
}

Use sync.Pool for hot paths:

var jsonEncoderPool = sync.Pool{
    New: func() any {
        return json.NewEncoder(io.Discard) // placeholder writer
    },
}

// In the actual handler, we need a new encoder (can't reuse across different writers)
// But we CAN pool the underlying buffer
var bufPool = sync.Pool{
    New: func() any { return new(bytes.Buffer) },
}

Avoid interface boxing for simple values:

// Bad: creates an interface value (allocation if value doesn't fit in pointer)
var x interface{} = 42
fmt.Println(x)

// Good for hot paths: use concrete types
func processInt(x int) { fmt.Println(x) }

Reducing Lock Contention

Prefer sync.RWMutex for read-heavy maps:

// Use RLock for reads (multiple goroutines can hold RLock simultaneously)
// Use Lock only for writes

Use sync/atomic for counters and flags:

var requestCount int64
atomic.AddInt64(&requestCount, 1)

Consider sync.Map for read-heavy, write-rarely maps:

// sync.Map is optimized for the case where entries are written once
// and read many times, or when different goroutines access disjoint keys
var cache sync.Map
cache.Store("key", value)
v, ok := cache.Load("key")

Reducing GC Pressure

Tune GOGC via environment:

# Default is GOGC=100 (GC when heap doubles from last collection size)
# Higher = less frequent GC, more memory usage
GOGC=200 ./myservice

# Set a hard memory limit (Go 1.19+)
# The runtime will run GC more aggressively if approaching this limit
GOMEMLIMIT=500MiB ./myservice

Avoid closures that capture large values:

// Bad: closure captures entire largeConfig (keeps it alive)
func makeHandler(largeConfig *Config) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // uses only largeConfig.Timeout
        doWork(largeConfig.Timeout)
    }
}

// Good: capture only what you need
func makeHandler(largeConfig *Config) http.HandlerFunc {
    timeout := largeConfig.Timeout  // only this escapes to the heap
    return func(w http.ResponseWriter, r *http.Request) {
        doWork(timeout)
    }
}

Quick Reference

Collecting Profiles

# CPU — 30 second sample (adjust ?seconds=N as needed)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Heap — live objects right now
go tool pprof -inuse_space http://localhost:6060/debug/pprof/heap

# Heap — everything allocated since start (for GC pressure)
go tool pprof -alloc_space http://localhost:6060/debug/pprof/heap

# Heap diff — what grew between two snapshots
curl -o heap1.out http://localhost:6060/debug/pprof/heap
# ... wait ...
curl -o heap2.out http://localhost:6060/debug/pprof/heap
go tool pprof -inuse_space -base heap1.out heap2.out

# Goroutine dump — full stacks of every goroutine
curl "http://localhost:6060/debug/pprof/goroutine?debug=2" > goroutines.txt

# Mutex contention (requires SetMutexProfileFraction)
go tool pprof http://localhost:6060/debug/pprof/mutex

# Block profile (requires SetBlockProfileRate)
go tool pprof http://localhost:6060/debug/pprof/block

# Execution trace
curl -o trace.out "http://localhost:6060/debug/pprof/trace?seconds=5"
go tool trace trace.out

# Benchmark profiles
go test -bench=BenchmarkMyFunc -benchmem -cpuprofile=cpu.out -memprofile=mem.out ./...

Interactive pprof Commands

(pprof) top10              # top 10 by flat time (CPU) or flat size (heap)
(pprof) top10 -cum         # top 10 by cumulative time/size
(pprof) list FunctionName  # annotated source with per-line costs
(pprof) web                # call graph in browser
(pprof) weblist FuncName   # annotated source in browser
(pprof) tree               # call tree rooted at top nodes
(pprof) png                # save call graph as PNG
(pprof) svg                # save call graph as SVG
(pprof) help               # list all commands

Opening the Web UI (Flame Graph)

# From a saved file
go tool pprof -http=:8090 cpu.out

# Directly from a live service
go tool pprof -http=:8090 http://localhost:6060/debug/pprof/profile?seconds=30

A Note on Profiling in Production

Running profiling in production is safe and common. The key points:

CPU profiling at the default 100Hz sampling rate adds about 1-5% overhead. This is acceptable for most services.
Heap profiling is always on by default (you just need to fetch the snapshot). No meaningful overhead.
Goroutine dumps are cheap to collect but the goroutine dump itself stops the world briefly — on a service with 100,000 goroutines, this could take tens of milliseconds. Don’t collect these in a tight loop.
Mutex and block profiling at high sampling rates have measurable overhead. Use conservative rates in production (SetMutexProfileFraction(5), SetBlockProfileRate(1000000)).
Traces have the highest overhead. Collect only for short intervals (5s) and only when investigating an active issue.
Never expose /debug/pprof/ publicly. Bind the debug server to localhost and use SSH port forwarding or an internal VPN to access it.

The habit to build is this: when something is slow or using too much memory, profile before you optimize. Look at the data before you change the code. The 20 minutes spent profiling will almost always save you hours of guessing and reverting.

The more you use these tools, the faster you get. Eventually profiling becomes a reflex — the first thing you reach for, not the last.

PostgreSQL EXPLAIN: Reading and Understanding Query Plans

Tue, 21 Apr 2026 00:00:00 GMT

When a query is slow, your first move should always be EXPLAIN. Most developers have seen the output, stared at it briefly, and moved on without really understanding what they were looking at. This post changes that. We’ll go from zero to being able to read a plan with confidence — including the gotchas that routinely fool even experienced engineers.

What EXPLAIN Actually Does

When you submit a SQL query, PostgreSQL doesn’t execute it immediately. First, the query planner analyzes all the ways it could satisfy the query and estimates which is cheapest. EXPLAIN shows you the plan the planner chose, without actually running the query.

EXPLAIN ANALYZE runs the query and shows you the plan alongside the actual measured times. That distinction matters enormously, as we’ll see.

-- Shows the plan only (no execution)
EXPLAIN SELECT * FROM orders WHERE user_id = 42;

-- Runs the query and shows plan + actual timings
EXPLAIN ANALYZE SELECT * FROM orders WHERE user_id = 42;

-- Most useful form: adds buffers (I/O stats) and verbose node info
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) SELECT * FROM orders WHERE user_id = 42;

The planner’s estimates are based on statistics — table row counts, column value distributions, and correlation info stored in pg_statistic (surfaced via pg_stats). If your statistics are stale (last ANALYZE ran a while ago), the planner will make poor decisions.

Reading the Plan Tree

A query plan is a tree of nodes. Each node is an operation — a scan, a join, a sort, an aggregate. The tree is read bottom-up: leaves execute first, results flow upward to parent nodes.

Here’s a simple plan to orient us:

Nested Loop  (cost=0.43..16.50 rows=10 width=64)
  ->  Index Scan using users_pkey on users  (cost=0.43..8.45 rows=1 width=32)
        Index Cond: (id = 42)
  ->  Seq Scan on orders  (cost=0.00..8.00 rows=10 width=32)
        Filter: (user_id = 42)

How to read this:

The bottom-most nodes (Index Scan on users, Seq Scan on orders) run first.
Their results feed up into Nested Loop, which joins them.
Indentation = depth in the tree. More indented = runs earlier.

The Cost Fields

Every node shows (cost=startup..total rows=N width=W):

Field	Meaning
`startup`	Cost before the first row can be returned (e.g. building a sort buffer)
`total`	Estimated total cost if all rows are consumed
`rows`	Estimated number of rows this node will output
`width`	Estimated average row size in bytes

Cost is unitless — it’s in abstract “cost units” that PostgreSQL uses internally to compare options. A cost of 1.0 roughly corresponds to one sequential page read. It’s meaningless as an absolute number but useful for comparing nodes within the same plan.

The total cost of the root node is what the planner minimized when choosing this plan. Higher cost = planner thinks it’s more expensive.

EXPLAIN ANALYZE adds actual timings

With ANALYZE, each node also shows:

Seq Scan on orders  (cost=0.00..8820.00 rows=200000 width=40)
                    (actual time=0.012..412.340 rows=198234 loops=1)

(actual time=startup..total rows=N loops=N)

Field	Meaning
`actual time`	Real measured time in milliseconds (startup..total)
`actual rows`	How many rows were actually produced
`loops`	How many times this node executed (important in joins!)

The loops field is critical. In a nested loop join, the inner side runs once per row from the outer side. If loops=500 and actual time=0.05..0.10, the total contribution of that node is 0.10ms × 500 = 50ms — not 0.10ms. PostgreSQL reports per-loop averages, so you must multiply by loops yourself.

BUFFERS shows I/O

EXPLAIN (ANALYZE, BUFFERS) adds buffer hit/miss stats to each node:

Seq Scan on orders  (cost=0.00..8820.00 rows=200000 width=40)
                    (actual time=0.012..412.340 rows=198234 loops=1)
  Buffers: shared hit=1240 read=6780

Field	Meaning
`shared hit`	Pages found in PostgreSQL’s shared buffer cache (fast)
`shared read`	Pages read from disk or OS page cache (slow)
`shared written`	Dirty pages written during the query
`temp read/written`	Temporary disk I/O (spilled sorts, hash joins)

High read vs hit ratio = your working set doesn’t fit in shared_buffers. High temp written = your sort/hash operations don’t fit in work_mem (the per-operation memory budget for sorts, hashes, and bitmap allocations — defaults to 4 MB) and are spilling to disk.

The Full EXPLAIN Output — Annotated

Let’s look at a realistic plan with ANALYZE, BUFFERS, and FORMAT TEXT:

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT u.name, COUNT(o.id) as order_count
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.created_at > '2025-01-01'
GROUP BY u.id, u.name
ORDER BY order_count DESC
LIMIT 10;

Limit  (cost=18420.32..18420.34 rows=10 width=40)
       (actual time=523.812..523.814 rows=10 loops=1)
  ->  Sort  (cost=18420.32..18445.32 rows=10000 width=40)
            (actual time=523.811..523.812 rows=10 loops=1)
        Sort Key: (count(o.id)) DESC
        Sort Method: top-N heapsort  Memory: 25kB
        ->  HashAggregate  (cost=18000.00..18100.00 rows=10000 width=40)
                           (actual time=519.234..521.456 rows=9823 loops=1)
              Group Key: u.id, u.name
              Batches: 1  Memory Usage: 2048kB
              ->  Hash Join  (cost=1240.00..16500.00 rows=100000 width=16)
                             (actual time=8.432..480.231 rows=98432 loops=1)
                    Hash Cond: (o.user_id = u.id)
                    Buffers: shared hit=4820 read=2340
                    ->  Seq Scan on orders o  (cost=0.00..8820.00 rows=200000 width=8)
                                              (actual time=0.012..210.340 rows=200000 loops=1)
                          Buffers: shared hit=2240 read=2340
                    ->  Hash  (cost=980.00..980.00 rows=20800 width=16)
                               (actual time=8.120..8.120 rows=20812 loops=1)
                          Buckets: 32768  Batches: 1  Memory Usage: 1280kB
                          ->  Index Scan using users_created_at_idx on users u
                                (cost=0.43..980.00 rows=20800 width=16)
                                (actual time=0.032..6.234 rows=20812 loops=1)
                              Index Cond: (created_at > '2025-01-01')
                              Buffers: shared hit=580

Reading this bottom-up:

Index Scan on users — uses the created_at index to find the 20,812 users created after 2025-01-01. Fast, 6ms.
Hash — builds a hash table from those users in memory (1280kB). One-time cost.
Seq Scan on orders — reads all 200,000 orders. The bottleneck: 210ms, 2340 pages read from disk.
Hash Join — probes the hash table with each order row to find matches. Total 480ms.
HashAggregate — groups and counts. 519ms cumulative.
Sort — sorts by count descending using a top-N heapsort (only keeps 10 rows, very cheap).
Limit — returns the top 10 rows.

The problem is obvious: the Seq Scan on orders with 2340 disk reads is the bottleneck. An index on orders.user_id would change this to an index scan, potentially eliminating most of those reads.

Seq Scan vs Index Scan vs Bitmap Index Scan

This is the section most engineers want to understand. PostgreSQL has three fundamentally different ways to read rows from a table, each with different trade-offs.

Sequential Scan (Seq Scan)

Reads every page of the table from start to finish, in physical disk order.

Seq Scan on orders  (cost=0.00..8820.00 rows=200000 width=40)
                    (actual time=0.012..412.340 rows=198234 loops=1)
  Filter: (status = 'pending')
  Rows Removed by Filter: 1766

When PostgreSQL chooses it:

No usable index exists.
The query will return a large fraction of the table (typically >5-20% of rows). Reading the index + jumping around the heap for each row costs more than just reading the whole table sequentially.
The table is small — a seq scan on a 50-row table is always faster than any index lookup.
random_page_cost is set high (spinning disks) making index random I/O expensive relative to sequential I/O.

The key insight: Seq scan has low overhead per page. Index scan has high overhead per row (random I/O). The crossover point is usually around 5-15% of the table’s rows depending on your storage and random_page_cost setting.

When it surprises you: You added an index but the planner still chooses a seq scan. This usually means either (a) you’re fetching more rows than you think, (b) your statistics are stale (ANALYZE needed), or (c) random_page_cost is too high for SSD storage (default is 4.0, should be ~1.1 on SSD).

-- Check what fraction of rows your filter selects
SELECT COUNT(*) FILTER (WHERE status = 'pending') * 100.0 / COUNT(*) AS pct
FROM orders;
-- If pct > 10%, seq scan is probably right

Index Scan

Uses a B-tree (or other) index to find row locations, then fetches each row from the heap (the main table storage) one by one.

Index Scan using orders_user_id_idx on orders
  (cost=0.43..42.50 rows=10 width=40)
  (actual time=0.032..0.218 rows=8 loops=1)
  Index Cond: (user_id = 42)

How it works:

Traverse the B-tree index to find all entries matching user_id = 42.
For each index entry, follow the heap pointer to fetch the actual row from the table — these are TIDs (tuple identifiers — the (page, slot) physical addresses we covered in How Databases Actually Store and Find Your Data).
Each heap fetch is a random I/O — potentially a different page each time.

When PostgreSQL chooses Index Scan:

A small fraction of rows match the predicate (low selectivity in the result-set sense).
The index columns and the physical row order are well-correlated — i.e. nearby keys in the index map to nearby pages on disk. Postgres tracks this via pg_stats.correlation (1.0 = perfect, 0.0 = random). High correlation makes Index Scan cheap because heap fetches stay sequential.

When correlation is low, Postgres prefers Bitmap Index Scan instead — same index, but it sorts the matching tuple IDs by physical page before fetching, turning random heap I/O into a sequential pass.

-- Check correlation for a column
SELECT tablename, attname, correlation
FROM pg_stats
WHERE tablename = 'orders' AND attname = 'user_id';

Index-only scan — a variant where the index contains all the columns the query needs. PostgreSQL can return results without ever touching the heap:

Index Only Scan using orders_user_status_idx on orders
  (cost=0.43..12.50 rows=10 width=8)
  (actual time=0.018..0.045 rows=8 loops=1)
  Index Cond: (user_id = 42)
  Heap Fetches: 0

Heap Fetches: 0 means pure index reads. This is the fastest possible scan for selective queries. Requires that all SELECTed columns (and the WHERE columns) are in the index. PostgreSQL also needs the visibility map (a per-table bitmap that tracks which pages contain only tuples visible to all transactions — see PostgreSQL Indexing Deep Dive for the mechanics) to confirm rows are visible (otherwise it must fetch from the heap), so VACUUM must have run recently.

Bitmap Index Scan

Bitmap Index Scan is PostgreSQL’s most clever scan strategy — a two-phase approach designed specifically for the middle-ground situation where too many rows match for an index scan, but not enough to justify reading the entire table. It trades an extra bookkeeping step for much cheaper I/O.

The Two Phases in Detail

Bitmap Heap Scan on orders  (cost=240.32..4820.10 rows=12000 width=40)
                             (actual time=4.231..82.340 rows=11984 loops=1)
  Recheck Cond: (status = 'pending')
  Heap Blocks: exact=3420
  ->  Bitmap Index Scan on orders_status_idx
        (cost=0.00..237.32 rows=12000 width=0)
        (actual time=3.890..3.890 rows=11984 loops=1)
        Index Cond: (status = 'pending')

Phase 1 — Bitmap Index Scan:

PostgreSQL walks the index (just like a regular index scan) and collects the physical address (TID — tuple identifier) of every row matching the condition. But instead of immediately fetching those rows from the heap, it builds a bitmap in memory. Each bit in the bitmap represents one heap page. If any matching row lives on a page, that page’s bit is set to 1.

At the end of Phase 1, we know which pages contain matching rows — but we haven’t touched the heap at all yet. Crucially, if 50 matching rows all happen to live on page 4820, that page’s bit is set exactly once, not 50 times.

Phase 2 — Bitmap Heap Scan:

PostgreSQL scans the heap, but only reads pages whose bit is set. Because it processes pages in ascending physical page order (not in the random order the index would produce), this is sequential I/O — the OS and storage layer can prefetch ahead efficiently. Each flagged page is read once, all matching rows on that page are extracted, and the Recheck Cond filter is applied.

Why This Is Faster Than Index Scan at Medium Selectivity

If 50,000 rows match a predicate over a 1M-row table on 10K pages, an Index Scan can hit up to ~10K distinct pages in index order, which on a poorly-correlated index means thousands of random I/Os. A Bitmap Index Scan first builds a bitmap of matching row IDs, sorts them by physical page, and reads each page once — turning thousands of random reads into ~10K sequential ones.

The bitmap converts random I/O into sequential I/O by batching and sorting all the access locations first.

Exact vs Lossy Bitmaps

The bitmap lives in memory — specifically in work_mem. This is where PostgreSQL’s adaptiveness becomes especially interesting.

When there’s enough work_mem: The bitmap tracks individual rows within each page (bit-per-tuple). This is an exact bitmap. When the heap scan reads a page, it knows exactly which rows to return — no extra filtering needed.

When work_mem is exhausted: PostgreSQL degrades the bitmap from per-tuple to per-page resolution. Each bit now means “this page might have a matching row.” This is a lossy bitmap. To compensate, the Recheck Cond filter is applied to every row on every flagged page to discard false positives. The query still produces correct results, but more heap rows are fetched and filtered.

You can see this distinction in the plan output:

Bitmap Heap Scan on orders
  Recheck Cond: (status = 'pending')
  Heap Blocks: exact=3200 lossy=820    -- 820 pages went lossy
  Rows Removed by Index Recheck: 14200 -- extra rows fetched and discarded

lossy=820 means 820 pages were read without knowing exactly which rows were valid — all rows on those pages had to be rechecked. If lossy is large relative to exact, you’re reading significantly more heap data than necessary. Fix: increase work_mem for the session or globally.

-- Test the effect
SET work_mem = '64MB';
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
-- If 'lossy' drops to 0, you've found your sweet spot

BitmapAnd — Using Multiple Indexes Simultaneously

This is one of the most powerful capabilities of the bitmap approach. When a query has AND conditions on multiple indexed columns, PostgreSQL can build a separate bitmap for each index and intersect them:

Bitmap Heap Scan on orders  (cost=340.15..2840.10 rows=500 width=40)
                             (actual time=5.230..18.420 rows=487 loops=1)
  Recheck Cond: ((user_id = 42) AND (status = 'pending'))
  Heap Blocks: exact=420
  ->  BitmapAnd  (cost=340.15..340.15 rows=500 width=0)
                  (actual time=5.100..5.100 rows=0 loops=1)
        ->  Bitmap Index Scan on orders_user_id_idx
              (cost=0.00..120.43 rows=5000 width=0)
              (actual time=1.230..1.230 rows=4823 loops=1)
              Index Cond: (user_id = 42)
        ->  Bitmap Index Scan on orders_status_idx
              (cost=0.00..218.97 rows=50000 width=0)
              (actual time=3.120..3.120 rows=49320 loops=1)
              Index Cond: (status = 'pending')

How BitmapAnd works:

Walk orders_user_id_idx → bitmap A (pages containing user 42’s orders, 5000 rows → maybe 200 pages set).
Walk orders_status_idx → bitmap B (pages containing pending orders, 50000 rows → maybe 2000 pages set).
AND the two bitmaps (bitwise AND) → result bitmap has only pages that appear in both. From 2200 candidate pages, only ~420 pages actually contain rows matching both conditions.
Read those 420 pages sequentially.

Without a composite index, this is the next best thing — two mediocre single-column indexes combine to behave almost like a targeted composite index. The AND step is O(pages/64) — extremely cheap, just a bitwise AND across machine words.

When BitmapAnd is chosen over a composite index: If you have a composite index (user_id, status), the planner will usually prefer a direct Index Scan or Bitmap Index Scan on it. BitmapAnd is used when no single index covers all conditions but multiple individual indexes each filter the data significantly.

BitmapOr — Satisfying OR Conditions with Multiple Indexes

For OR conditions, PostgreSQL builds two bitmaps and takes their union:

Bitmap Heap Scan on orders
  Recheck Cond: ((user_id = 42) OR (status = 'urgent'))
  ->  BitmapOr
        ->  Bitmap Index Scan on orders_user_id_idx
              Index Cond: (user_id = 42)
        ->  Bitmap Index Scan on orders_status_idx
              Index Cond: (status = 'urgent')

Each index produces a bitmap; BitmapOr sets a page’s bit if either bitmap has it set. The heap scan then reads all flagged pages. This is far better than falling back to a seq scan just because of an OR condition.

An important nuance: BitmapOr can only avoid a seq scan if both sides of the OR have usable indexes. If one condition has no index, the planner may discard the entire strategy and choose a seq scan for the whole table — because it can’t guarantee it won’t miss rows that match the un-indexed condition.

When Does PostgreSQL Choose Bitmap Over Regular Index Scan?

The planner models the expected cost of random I/O (index scan) vs. the bitmap setup cost + sequential I/O (bitmap scan). The decision depends on:

Factor	Pushes toward Index Scan	Pushes toward Bitmap Scan
Matching rows	Very few (< ~1%)	More (1-20%)
Table size	Small	Large
`random_page_cost`	Low (SSD: 1.1)	High (disk: 4.0)
Physical correlation	High (rows clustered together)	Low (rows scattered)
Multiple filter conditions	Single-column index covers all	Multiple single-column indexes

You can observe this tipping point experimentally:

-- With low work_mem or high selectivity, see what the planner chooses
SET enable_indexscan = off;         -- force bitmap to test its cost
EXPLAIN SELECT * FROM orders WHERE status = 'pending';

SET enable_bitmapscan = off;        -- force seq scan to test its cost  
EXPLAIN SELECT * FROM orders WHERE status = 'pending';

-- Reset
SET enable_indexscan = on;
SET enable_bitmapscan = on;

enable_* settings don’t forbid a node — they add a huge cost penalty. They’re a debugging tool, not a production knob. Always reset them after diagnosing a plan.

Bitmap Scan and GIN/GiST Indexes

B-tree isn’t the only index type that feeds the bitmap machinery. GIN (used for full-text search, arrays, JSONB) and GiST (used for geometry, ranges, full-text) always use the bitmap approach when scanning — they produce a set of heap locations that gets fed into a Bitmap Heap Scan:

Bitmap Heap Scan on articles
  Recheck Cond: (to_tsvector('english', body) @@ 'postgresql'::tsquery)
  ->  Bitmap Index Scan on articles_body_fts_idx
        Index Cond: (to_tsvector('english', body) @@ 'postgresql'::tsquery)

For JSONB and array containment queries, the same pattern applies:

-- GIN index on tags array
CREATE INDEX orders_tags_gin_idx ON orders USING GIN (tags);

-- Query uses Bitmap Index Scan via GIN
SELECT * FROM orders WHERE tags @> ARRAY['urgent', 'international'];

GIN indexes are inherently multi-valued — one index entry can point to many rows, and one row can have many index entries. The bitmap approach naturally handles this many-to-many relationship without duplication.

The Decision Tree

Join Strategies

Plans with joins show how PostgreSQL combined the two row sets. There are three strategies:

Nested Loop Join

Nested Loop  (cost=0.43..45.80 rows=10 width=64)
  ->  Index Scan using users_pkey on users  (rows=1)
  ->  Index Scan using orders_user_id_idx on orders  (rows=10)
        Index Cond: (user_id = users.id)

For each row from the outer side (users), scan the inner side (orders) for matching rows. loops on the inner node = number of outer rows. Best when outer side is small and inner side has a good index.

Trap: If the outer side returns thousands of rows and the inner has no index, this becomes O(N×M) — catastrophic.

Hash Join

Hash Join  (cost=1240.00..16500.00 rows=100000 width=16)
  Hash Cond: (o.user_id = u.id)
  ->  Seq Scan on orders o
  ->  Hash
        ->  Index Scan on users u

Build a hash table from the smaller side (users), then probe it with each row from the larger side (orders). One pass over each side. Memory cost is the hash table — if it doesn’t fit in work_mem, it spills to disk (Batches: N where N > 1).

Best for large joins where neither side is tiny.

Merge Join

Merge Join  (cost=24000.00..32000.00 rows=50000 width=64)
  Merge Cond: (o.user_id = u.id)
  ->  Sort on users.id
  ->  Sort on orders.user_id

Sort both sides on the join key, then merge them like merge sort’s combine step. Efficient when both inputs are already sorted (e.g., from an index scan on the join key) or when both sides are large and a hash table wouldn’t fit in memory.

Other Node Types You’ll See

Node	What It Does
`Sort`	Sorts rows. Shows `Sort Method` and memory used. If `external merge` appears, it spilled to disk — increase `work_mem`.
`HashAggregate`	GROUP BY using a hash table. Shows `Batches` — if > 1, it spilled.
`GroupAggregate`	GROUP BY on pre-sorted input. Lower memory, requires sorted input.
`Aggregate`	Simple aggregates (COUNT, SUM) with no grouping.
`Limit`	Stops when N rows collected. Affects how much of the plan below it actually runs.
`Materialize`	Caches a node’s output in memory so it can be scanned multiple times (e.g., inner side of a nested loop).
`Subquery Scan`	Scans the result of a subquery as if it’s a table.
`CTE Scan`	Reads from a materialized CTE. PostgreSQL 12+ inlines most CTEs unless you use `MATERIALIZED`.
`Function Scan`	Result of a set-returning function, e.g. `generate_series()`.
`Gather` / `Gather Merge`	Collects results from parallel workers.
`Parallel Seq Scan`	Seq scan split across multiple parallel workers.

EXPLAIN Options Reference

-- Basic plan (no execution)
EXPLAIN SELECT ...;

-- Execute and time
EXPLAIN ANALYZE SELECT ...;

-- Full diagnostics — use this by default
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;

-- JSON output — useful for tooling, pg_query, etc.
EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) SELECT ...;

-- Show verbose node output (output columns, schema-qualified names)
EXPLAIN (ANALYZE, VERBOSE) SELECT ...;

-- Show row-level settings (e.g. if JIT kicked in)
EXPLAIN (ANALYZE, BUFFERS, SETTINGS) SELECT ...;

-- Include WAL stats (useful for INSERT/UPDATE/DELETE plans)
EXPLAIN (ANALYZE, WAL) UPDATE orders SET status = 'done' WHERE id = 42;

Note: WAL, BUFFERS, TIMING, and VERBOSE only emit data when run with ANALYZE — without it, the query is only planned, never executed.

One gotcha with EXPLAIN ANALYZE on write queries: it actually executes the DML. Wrap in a transaction and roll back:

BEGIN;
EXPLAIN ANALYZE UPDATE orders SET status = 'done' WHERE user_id = 42;
ROLLBACK;

Identifying the Bottleneck

The plan can have dozens of nodes. Here’s a systematic way to find what’s costing you:

1. Find the slowest node. Look at actual time on each node. The difference between a node’s time and its children’s times is the work that node itself did. A node with actual time=480ms whose children took actual time=210ms spent 270ms doing its own work (e.g., the hash join probe).

2. Compare estimated vs actual rows. A large discrepancy indicates the planner had bad statistics and may have chosen the wrong plan:

Seq Scan on orders  (cost=0.00..8820.00 rows=200 width=40)   -- planner estimated 200
                    (actual time=0.012..412.340 rows=198234 loops=1)  -- actually 198k!

The planner thought it was dealing with 200 rows and may have chosen a nested loop join that’s catastrophic with 198k rows. Fix: ANALYZE orders; or investigate why statistics are wrong.

3. Look for loops multiplied cost. A node showing actual time=0.05 loops=5000 costs 5000 × 0.05 = 250ms total.

4. Look for spills.

Sort Method: external merge — sort spilled to disk. Increase work_mem.
HashAggregate: Batches: 4 — hash aggregate spilled. Increase work_mem.
Bitmap Heap Scan: Heap Blocks: exact=100 lossy=4000 — bitmap went lossy. Increase work_mem.

5. Look for filter removals.

Filter: (status = 'active')
Rows Removed by Filter: 199500

Reading 200k rows and keeping 500 means a seq scan is doing huge unnecessary work. An index on status would help — but only if status is selective enough.

The Gotchas: When Indexes Get Ignored

These are the situations that catch engineers off guard. You add an index, but EXPLAIN still shows a seq scan or the index isn’t used the way you expect.

1. `LIKE` with a leading wildcard

-- Uses the index (prefix match)
SELECT * FROM users WHERE email LIKE 'john%';

-- IGNORES the index (leading wildcard can't be found in a B-tree)
SELECT * FROM users WHERE email LIKE '%@example.com';

B-tree indexes store values in sorted order. A leading wildcard means “could be anything before this pattern” — there’s no sorted prefix to binary-search on. The full table must be scanned.

Fix: For suffix/substring matching, use pg_trgm (trigram) index:

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX users_email_trgm_idx ON users USING GIN (email gin_trgm_ops);
-- Now LIKE '%@example.com' can use the trigram index

2. Function applied to the indexed column

-- Ignores the index on created_at
SELECT * FROM orders WHERE DATE(created_at) = '2025-01-01';

-- Uses the index
SELECT * FROM orders WHERE created_at >= '2025-01-01' AND created_at < '2025-01-02';

When you wrap a column in a function, PostgreSQL can’t use a plain B-tree index on the column — the index stores raw created_at values, not DATE(created_at) values.

Fix option 1: Rewrite the query to not use the function on the column side (as above).

Fix option 2: Create a functional index that stores the expression result:

CREATE INDEX orders_date_idx ON orders (DATE(created_at));
-- Now DATE(created_at) = '2025-01-01' uses the index

The same pattern applies to LOWER(email), UPPER(name), EXTRACT(year FROM date), etc.

3. Type mismatch / implicit cast

-- Column is INTEGER, but you pass a string
SELECT * FROM orders WHERE user_id = '42';  -- implicit cast may prevent index use

-- Safer: explicit type match
SELECT * FROM orders WHERE user_id = 42;

It’s the column-side cast that breaks index use. WHERE id = '42' (literal cast) works — Postgres folds the constant to int. WHERE id::text = '42' (column cast) doesn’t — Postgres has to compute id::text for every row before comparing, defeating the index.

Particularly common with: numeric columns passed string literals from ORMs, VARCHAR vs TEXT, TIMESTAMP vs TIMESTAMPTZ.

-- Check this in pg_stats: if you're joining two columns with different types,
-- the planner may not be able to estimate selectivity well
SELECT tablename, attname, atttypid::regtype
FROM pg_attribute JOIN pg_class ON attrelid = pg_class.oid
WHERE relname = 'orders' AND attname = 'user_id';

4. OR conditions

-- Can't use a single index efficiently — often falls back to seq scan
SELECT * FROM orders WHERE user_id = 42 OR status = 'pending';

PostgreSQL can handle this with a BitmapOr if both columns are indexed, but it requires two separate index scans and a bitmap combine. If the individual conditions are not very selective, a seq scan may still win. Rewrite as UNION ALL when you need two genuinely independent conditions:

SELECT * FROM orders WHERE user_id = 42
UNION ALL
SELECT * FROM orders WHERE status = 'pending' AND user_id != 42;

5. `NOT IN` and `!=` on indexed columns

-- Ignores the index — negation has no useful range in a B-tree
SELECT * FROM orders WHERE status != 'cancelled';

-- Also ignores the index
SELECT * FROM orders WHERE user_id NOT IN (1, 2, 3);

A B-tree is optimized for finding values, not for finding the absence of a value. NOT IN with a large subquery is particularly dangerous — it can prevent index use entirely and generate a cross-product in the plan.

Fix: Rewrite using LEFT JOIN ... WHERE right_id IS NULL instead of NOT IN with subqueries:

-- Slow: NOT IN with subquery — disables index, doesn't handle NULLs well
SELECT * FROM users WHERE id NOT IN (SELECT user_id FROM banned_users);

-- Fast: anti-join
SELECT u.* FROM users u
LEFT JOIN banned_users b ON b.user_id = u.id
WHERE b.user_id IS NULL;

6. NULL comparisons

-- IS NULL / IS NOT NULL can use an index, but only if the index includes NULL values
-- B-tree indexes DO include NULLs (unlike some other databases)
SELECT * FROM orders WHERE completed_at IS NULL;  -- can use index on completed_at

PostgreSQL B-tree indexes include NULL entries, so IS NULL and IS NOT NULL can use them. However, partial indexes specifically excluding or including NULLs are more efficient for heavily NULL-skewed columns:

-- Partial index: only rows where completed_at IS NULL
-- Much smaller index, only used by queries filtering for NULL
CREATE INDEX orders_pending_idx ON orders (id) WHERE completed_at IS NULL;

7. Statistics are stale

-- Force a statistics refresh on a table
ANALYZE orders;

-- Check when statistics were last collected
SELECT schemaname, tablename, last_analyze, last_autoanalyze
FROM pg_stat_user_tables
WHERE tablename = 'orders';

-- Check the n_distinct and correlation for a column
SELECT attname, n_distinct, correlation
FROM pg_stats
WHERE tablename = 'orders' AND attname = 'user_id';

n_distinct tells the planner how many distinct values exist. If it’s wrong (e.g., says 200 but there are actually 200,000), the planner will massively under-estimate the selectivity of user_id = 42 and may choose a seq scan.

For columns with very many distinct values, the default default_statistics_target = 100 may be too low. Increase it for a specific column:

ALTER TABLE orders ALTER COLUMN user_id SET STATISTICS 500;
ANALYZE orders;

8. The planner disables an index due to cost settings

Two settings control when the planner chooses random I/O:

-- Default: 4.0 (calibrated for spinning disks)
-- On SSDs, set to 1.1 — makes index scans much more attractive
SET random_page_cost = 1.1;

-- You can also set per-session for testing
SET random_page_cost = 1.1;
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
RESET random_page_cost;  -- or use SET LOCAL inside a transaction

If your server has SSDs but random_page_cost is still 4.0, the planner is systematically undervaluing index scans. Many cloud PostgreSQL setups (RDS, Cloud SQL) run on SSDs but don’t override this default.

9. Partial indexes not being used

CREATE INDEX orders_pending_idx ON orders (created_at) WHERE status = 'pending';

-- Uses the partial index
SELECT * FROM orders WHERE status = 'pending' AND created_at > NOW() - INTERVAL '7 days';

-- Does NOT use it (status is not in the WHERE clause)
SELECT * FROM orders WHERE created_at > NOW() - INTERVAL '7 days';

A partial index only applies to queries whose WHERE clause implies the index’s predicate. If your query doesn’t include the filter condition from the index definition, PostgreSQL won’t consider it.

10. `ILIKE` and case-insensitive patterns

-- ILIKE uses a case-insensitive match — ignores a plain B-tree index
SELECT * FROM users WHERE email ILIKE 'john%';

A plain B-tree index on email is case-sensitive. ILIKE requires comparing lowercase-folded values, which the raw index doesn’t store.

Fix options:

-- Option 1: functional index on lower()
CREATE INDEX users_email_lower_idx ON users (LOWER(email));
SELECT * FROM users WHERE LOWER(email) LIKE 'john%';

-- Option 2: citext extension
CREATE EXTENSION citext;
ALTER TABLE users ALTER COLUMN email TYPE citext;
-- Now regular = and LIKE are case-insensitive and use the B-tree index

-- Option 3: trigram index with gin_trgm_ops (handles ILIKE '%pattern%' too)
CREATE INDEX users_email_trgm_idx ON users USING GIN (email gin_trgm_ops);

Practical Workflow: Diagnosing a Slow Query

When a query is slow, here’s the exact workflow to follow:

Step 1: Get the full plan with BUFFERS

EXPLAIN (ANALYZE, BUFFERS) <your query>;

Step 2: Find the most expensive node. Look at the actual time on each node. The root node’s time minus its children’s times = that node’s own work.

Step 3: Check for row estimate mismatches. If rows=200 in the plan but actual rows=200000, your statistics are stale or the planner has wrong assumptions. Run ANALYZE <table>.

Step 4: Look for seq scans on large tables. Check if there’s a filter removing most rows — that’s your index opportunity.

Step 5: Check Buffers: read — high read counts mean disk I/O. Consider whether the working set fits in shared_buffers.

Step 6: Look for spills — external merge in sort, Batches > 1 in hash operations. Increase work_mem for the session and re-run:

SET work_mem = '256MB';
EXPLAIN (ANALYZE, BUFFERS) <your query>;

Step 7: If index exists but isn’t used, check:

Is the column wrapped in a function?
Is there a type mismatch?
Is selectivity too low? (Query returns >15% of rows)
Is random_page_cost misconfigured for your storage?
Run ANALYZE <table> and check again.

Using explain.depesz.com and pev2

Reading raw text output works, but for complex plans there are better tools:

explain.depesz.com — paste your EXPLAIN ANALYZE output and get a color-coded breakdown. Highlights the slowest nodes, shows estimated vs actual rows side by side. The go-to tool for sharing plans with teammates.

PEV2 (Postgres Explain Visualizer 2) — a more modern visual tree. Shows the plan as a node graph with timing proportional to node width. Particularly good for plans with many joins.

pgBadger / pg_stat_statements — for identifying which queries are slow in aggregate over time (not just one-off analysis):

-- Enable pg_stat_statements in postgresql.conf:
-- shared_preload_libraries = 'pg_stat_statements'

-- Find your top 10 slowest queries by total time
SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

Quick Reference

Plan output	What to look for
`Seq Scan` + large `Rows Removed by Filter`	Missing index on filter column
`rows=100` but `actual rows=100000`	Stale statistics — run `ANALYZE`
`loops=500` on inner node	Nested loop with large outer side — check for index on join column
`Sort Method: external merge`	Sort spilled to disk — increase `work_mem`
`Batches: 4` on HashAggregate/Hash Join	Hash spilled to disk — increase `work_mem`
`Heap Blocks: lossy=N`	Bitmap went lossy — increase `work_mem`
`Heap Fetches: 0`	Index-only scan — best case, keep your visibility map current
`shared read=N` (large)	Heavy disk I/O — check `shared_buffers`, consider caching
`Buffers: temp read/written`	Operator spilled to temp files
`Gather` / `Parallel`	Query is using parallel workers — check `max_parallel_workers_per_gather`
Index exists, seq scan chosen	Check selectivity, `random_page_cost`, stale stats, function on column

PostgreSQL Indexing Deep Dive — the index types whose plan nodes you’re now reading.
How Databases Actually Store and Find Your Data — pages, heaps, TIDs — the substrate of every plan.
B-Trees and B+ Trees — why “Index Scan” is so cheap on B+ tree leaves.
ACID, Read Phenomena, Isolation Levels — when query plans interact with concurrent transactions.
Postgres Partitioning: Strategies, Advantages, and Pitfalls — how Append, partition pruning, and Subplans Removed show up in EXPLAIN.

How Databases Actually Store and Find Your Data

Mon, 20 Apr 2026 00:00:00 GMT

When you write SELECT * FROM orders WHERE id = 42, you’re thinking in tables and rows. The database is thinking in pages, heaps, and B-trees. Understanding the gap between those two views explains every performance decision you’ll ever make about indexes.

Let’s build the picture from the ground up.

The Table: A Logical Abstraction

A table is a logical concept — a named collection of rows with defined columns. It says nothing about how data is physically stored. Two databases can expose the exact same CREATE TABLE syntax and store the data in completely different ways underneath.

The physical reality lives one level down, in pages.

Pages: The Unit of I/O

Databases never read a single row from disk. They read a page — a fixed-size block of data, typically 8KB (PostgreSQL, SQL Server) or 16KB (MySQL InnoDB). A page is the smallest unit the storage engine reads or writes.

A page contains a header, a slot array pointing to row locations, and the actual row data:

Rows are written from the bottom of the page upward. The slot array at the top grows downward. This design means rows can be moved within a page (for compaction) without changing the slot number — callers reference (page, slot), not a raw byte offset.

Why this matters: If your query touches 1 row that happens to share a page with 99 other rows, the database reads all 100 rows’ worth of data into memory regardless. Page size is why “one row lookups” are cheap only when paired with an index — without one, you may read every page in the table.

The Heap: An Unordered Table

The default table structure in most databases is a heap. A heap is simply a collection of pages with no inherent ordering — rows are inserted wherever there is free space, not in any sorted order.

Notice: the rows are in no particular order. id=1 is on page 1, id=2 is on page 2, id=3 is on page 2. There is no relationship between the key value and where the row physically lives.

Consequence: To find id = 4 without an index, the database must read every page from start to finish — a sequential scan or full table scan. For a 10-million-row table, that means reading millions of pages from disk.

Row IDs: The Physical Address of a Row

Every row in a heap has a physical address — a pointer to exactly which page and which slot it occupies. Different databases name this differently:

PostgreSQL	Oracle	SQL Server
ctid	ROWID	RID
e.g. (2, 3)	e.g. AAABXSAABAAAACFAAH	e.g. 1:2:3
= page 2, slot 3	encoded page + slot	= file 1, page 2, slot 3

These row IDs are physical, not logical. They change when a row is moved — during VACUUM FULL (VACUUM is PostgreSQL’s process for reclaiming space from deleted/updated row versions), CLUSTER, or other operations that physically rewrite tuples. Standard VACUUM does not move existing tuples and does not change ctids. This has critical implications for how indexes work, as we’ll see next.

Row IDs are not normally exposed to application code, but they are the mechanism that makes indexes fast.

Indexes and How They Use Row IDs

An index is a separate data structure — almost always a B-tree or B+ tree — that maps a column’s values to the physical location of the corresponding rows in the heap.

When you query WHERE id = 7:

The database traverses the B-tree from root to leaf — typically 2–4 page reads for a large index.
The leaf node contains the row ID: (2, 2) — page 2, slot 2.
The database reads page 2 from the heap and extracts the row at slot 2.

This is a double lookup: index traversal, then a heap fetch. For a single row, this is very fast. At scale, it’s the foundation of every indexed query plan.

The Clustered Index: Data Ordered by the Index

A clustered index eliminates the double lookup by storing the actual row data inside the B-tree leaf nodes — not a pointer to a heap, but the row itself.

The leaf level of a clustered index is the table. There is no separate heap. Rows are physically stored in sorted order by the index key.

Benefits:

Range scans are extremely fast — WHERE id BETWEEN 4 AND 7 reads three contiguous leaf pages, no heap fetch.
No double lookup — one B-tree traversal retrieves the full row.

Cost:

Inserts must go into the right sorted position. Random inserts (e.g., UUIDs as keys) cause page splits — when an insert won’t fit in the target page, the database allocates a new page and moves half the rows to it, an expensive operation that fragments the index over time.
There can only be one clustered index per table — the data can only be physically sorted one way.

Aside — FILLFACTOR: Most databases let you reserve free space on each page at index creation time (PostgreSQL’s WITH (fillfactor = 70), MySQL’s similar setting). A 70% fillfactor leaves 30% of each page empty for future inserts and HOT updates, dramatically reducing page splits on write-heavy tables. The tradeoff is read amplification — the index occupies more pages.

Primary Key vs Clustered Index

This is one of the most misunderstood distinctions in databases. They are related but not the same thing.

MySQL InnoDB

InnoDB always has a clustered index. If you define a primary key, it becomes the clustered index. If you don’t, InnoDB finds the first unique NOT NULL column. If none exists, InnoDB creates a hidden 6-byte row ID and clusters on that. You cannot opt out.

SQL Server

A primary key creates a clustered index by default — but you can override this. You can create a primary key as a non-clustered index, and separately define a different column as the clustered index. Primary key and clustered index are independent choices.

PostgreSQL

PostgreSQL has no clustered indexes in the traditional sense. All tables are heaps. A primary key creates a regular B-tree index that stores row IDs pointing back to the heap. The CLUSTER command can physically reorder a table’s pages to match an index — but this ordering is not maintained on subsequent writes. It’s a one-time reorganization, not a structural property.

Secondary Indexes

A secondary index is any index that is not the clustered index. How it works depends on the table’s storage structure.

Secondary Index on a Heap Table (PostgreSQL)

Both the primary key index and any secondary indexes are B-trees pointing to physical row IDs (ctid).

Problem with heap + secondary index: If a row is vacuumed and moves to a different page, every index pointing to that row ID must be updated. PostgreSQL’s HOT (Heap-Only Tuple) optimization handles a different case: when an UPDATE doesn’t change any indexed column, Postgres places the new row version on the same page and chains it from the old one — so no new index entries need to be created. This avoids index bloat from frequent updates to non-indexed columns and keeps secondary indexes pointing at the page rather than at every individual row version.

Secondary Index on a Clustered Table (MySQL InnoDB)

Secondary indexes don’t store physical row IDs — they store the primary key value instead. To fetch the full row, InnoDB does a secondary index lookup followed by a primary key (clustered index) lookup.

Why primary key value instead of physical address? When a clustered index page splits (due to an insert), rows shift around. If secondary indexes stored physical addresses, every secondary index would need updating on every page split. By storing the primary key, secondary indexes remain stable — only the clustered index itself needs to maintain the physical layout.

Cost: Every secondary index lookup on InnoDB requires two B-tree traversals — one in the secondary index, one in the clustered index. This is called a double lookup (also called a bookmark lookup in SQL Server terminology).

Mitigation — Covering Index: If the query only needs columns that are already in the index, the database doesn’t need to look up the full row at all. A covering index includes all columns needed by the query, allowing the lookup to be resolved entirely from the index leaf node.

-- Without covering index: secondary index lookup + clustered index lookup
SELECT email, created_at FROM users WHERE email = 'alice@x.com';

-- With covering index on (email, created_at): resolved from index alone
CREATE INDEX idx_email_created ON users (email, created_at);

How It All Connects

B-Trees and B+ Trees — the data structure that powers nearly every index in this post.
Database Normalization — the logical model these physical structures support.
PostgreSQL Indexing Deep Dive — applies the heap/clustered/secondary distinction to a real database.
Reading PostgreSQL EXPLAIN Output — see these structures show up in real query plans.
Postgres Partitioning: Strategies, Advantages, and Pitfalls — how heap files and TIDs split across multiple physical tables.

Key Takeaways

Pages are the unit of I/O. Indexes reduce the number of pages read; they don’t make individual page reads faster.
Heaps store rows in insertion order. Without an index, finding a row requires reading every page.
Row IDs (ctid, ROWID, RID) are physical addresses. They are what B-tree indexes store to bridge the index and the heap.
A clustered index stores the actual rows in the B-tree leaf nodes. The leaf level is the table. There is no heap.
A primary key is a logical uniqueness constraint. Whether it becomes a clustered index depends on the database: always in InnoDB, by default in SQL Server, never in PostgreSQL.
Secondary indexes on a clustered table store the primary key value, not a physical address, to stay stable across page splits — at the cost of a double lookup.
A covering index avoids the second lookup entirely by including all the columns a query needs directly in the index.

The next time you wonder whether to add an index, you’re really asking: is it worth the cost of maintaining a separate B-tree and potentially doing a second lookup, in exchange for skipping a sequential scan of all those pages?

ACID, Read Phenomena, and Isolation Levels: What to Use When

Sun, 19 Apr 2026 00:00:00 GMT

Most concurrency bugs in production aren’t application bugs — they’re isolation bugs. Two requests arrive at the same instant, both read the same row, both update it, and one update silently disappears. Or a customer is shown a price that’s stale by 50 milliseconds and gets charged a different amount. Or a hotel room gets double-booked because two transactions saw “one room available” at the same time.

The fix isn’t more retries or better logging. It’s understanding what your database actually guarantees, what it doesn’t, and what knobs let you trade one for the other. This post walks through ACID, the five concurrency anomalies you’ll meet in production, the four SQL isolation levels and what each one costs you, how Postgres implements them, and a decision tree for picking the right level — without paying for SERIALIZABLE everywhere.

The Vocabulary

Before any code, the words. Most isolation-level confusion comes from these terms being used loosely.

Transaction — a unit of work grouping one or more operations into a single logical action. Either all succeed (COMMIT) or none do (ROLLBACK). Started with BEGIN.

ACID — the four guarantees a “safe” transaction provides: Atomicity (all-or-nothing), Consistency (constraints hold), Isolation (no interference with concurrent transactions), Durability (committed data survives crashes).

Read phenomenon — a specific kind of anomaly that becomes possible when isolation is relaxed. Dirty reads, non-repeatable reads, phantom reads, lost updates, write skew — each one a real bug pattern, named after the SQL standard’s classification.

Isolation level — the dial controlling which phenomena your transactions are allowed to observe. Higher level = fewer anomalies = more contention. Four standard levels: Read Uncommitted, Read Committed, Repeatable Read, Serializable.

MVCC (Multi-Version Concurrency Control) — the mechanism that lets readers and writers not block each other. Every row keeps multiple versions; each transaction sees the version that was committed when it started. Used by Postgres, MySQL InnoDB, Oracle.

Snapshot — a point-in-time view of the database visible only to one transaction. Under MVCC, every transaction operates against its own snapshot. Reads inside the transaction stay consistent even as other transactions commit.

Lock — a database-managed token a transaction holds on a resource (row, range, table) to control concurrent access. Shared (read) locks allow other readers but block writers. Exclusive (write) locks block everyone else. Predicate locks cover a query condition (WHERE status = 'available'), not just specific rows.

SSI (Serializable Snapshot Isolation) — Postgres’s implementation of SERIALIZABLE. Tracks read-write dependency cycles between concurrent transactions and aborts the one that would cause an anomaly, instead of blocking them with predicate locks. Cheaper than classical serializable, but you must handle serialization-failure errors and retry.

Pessimistic vs optimistic concurrency control — pessimistic locks resources up front (SELECT … FOR UPDATE). Optimistic doesn’t lock, but checks at commit time whether the data you read has been modified by someone else, and retries if so. Pessimistic is safer under contention; optimistic is faster when contention is rare.

WAL (Write-Ahead Log) — every change is appended to a log file before the in-memory page is touched. Durability comes from fsync’ing the WAL before COMMIT returns success. Recovery replays the WAL from the last checkpoint.

ACID, Briefly

The four properties, in one paragraph each.

Atomicity — all-or-nothing

Every operation inside a transaction is committed together, or the entire transaction is rolled back on failure. Alice transfers $500 to Bob:

BEGIN;
  UPDATE accounts SET balance = balance - 500 WHERE id = 'alice';
  UPDATE accounts SET balance = balance + 500 WHERE id = 'bob';
COMMIT;

If the server crashes after the first UPDATE but before the second, atomicity ensures the rollback on recovery restores Alice’s balance. Money never disappears mid-transfer.

Consistency — constraints hold

Every transaction must leave the database satisfying all declared rules — foreign keys, unique constraints, CHECK constraints, triggers. If Alice has $200 and a debit of $500 would violate CHECK (balance >= 0), the entire transaction is rejected. No partial state, no application bypass.

Isolation — concurrent transactions don’t interfere

This is the most nuanced of the four. Full isolation is expensive, so databases let you tune it via isolation levels. The rest of this post is mostly about this property.

Durability — committed data survives crashes

Once COMMIT returns success, the change is persisted to disk via the WAL, even if the power cuts out a millisecond later. This is why commits have an I/O cost — durability isn’t free.

The Five Read Phenomena

When isolation is relaxed, transactions can observe these anomalies. The SQL standard names three; two more (lost update and write skew) are real bugs you’ll hit in practice that don’t appear in the original definitions.

1. Dirty Read

A transaction reads data written by another transaction that has not yet committed. If the writer rolls back, the reader has acted on data that never officially existed.

Transaction B read a value that was rolled back. The seat was available all along — but the customer was already told it wasn’t.

2. Non-Repeatable Read

A transaction reads the same row twice and gets different values, because another committed transaction modified that row in between.

Same row, same transaction, two different values. Causes silent billing discrepancies that only surface in customer complaints.

3. Phantom Read

A transaction re-executes a query with a filter and gets a different set of rows — not because existing rows changed, but because another transaction inserted or deleted matching rows.

No existing row was modified between Transaction A’s two reads — a new row appeared and changed the result. Especially dangerous in booking, ticketing, and inventory systems.

4. Lost Update

Two transactions read the same value, independently compute an update, and write it back — one silently overwrites the other.

Two orders shipped, stock only dropped by 1. At scale this leads to overselling — shipping products that don’t exist. The safe fix is an atomic update: SET stock = stock - 1 instead of a read-modify-write cycle.

5. Write Skew

Two transactions read the same rows, see a consistent snapshot, and each updates different rows — but the combined result violates a constraint that no single transaction would have broken on its own.

Neither transaction modified a row the other read, so standard MVCC snapshot rules don’t see a conflict. Snapshot isolation (Postgres Repeatable Read) does not prevent this. Only true SERIALIZABLE does.

The Four Isolation Levels

The SQL standard defines four levels. Each prevents a progressively stricter set of phenomena.

Read Uncommitted

The weakest level. Transactions can read uncommitted data from others.

Prevents: nothing
Allows: all five phenomena
When to use: almost never. Possibly rough approximate analytics where slightly dirty counts are acceptable and you need maximum read throughput.
When to avoid: any transactional system. Any read where the answer drives a decision. Anywhere correctness matters.
Postgres note: Postgres internally treats READ UNCOMMITTED as READ COMMITTED — it never returns dirty reads even when you ask for them.

Read Committed

Transactions only see committed data. Each statement gets a fresh view of the latest committed state, so the same query inside one transaction may return different rows if run twice.

Prevents: dirty reads
Allows: non-repeatable reads, phantom reads, lost updates, write skew
When to use: the safe baseline for most OLTP applications — dashboards, user profile reads, standard CRUD. Default in Postgres, Oracle, SQL Server.
When to avoid: multi-step logic that reads the same row twice (you’ll see shifting values mid-transaction). Operations that compute derived results from multiple rows that need to be a consistent set.

Repeatable Read

Once a transaction reads a row, the transaction continues to see the snapshot from when it began — even if other transactions commit changes to that row in the meantime.

Prevents: dirty reads, non-repeatable reads
Allows: phantom reads (in standard implementations), write skew
When to use: multi-step business logic that reads the same data multiple times and needs a consistent view. Generating invoices, calculating totals, multi-query reports.
When to avoid: anywhere write skew matters (on-call rotations, multi-row constraint enforcement). Use SERIALIZABLE instead.
Postgres note: Repeatable Read in Postgres also prevents lost updates by aborting the second transaction with a 40001 serialization error. MySQL’s InnoDB doesn’t — use SELECT … FOR UPDATE or atomic updates if you’re on MySQL.
MySQL note: MySQL InnoDB’s Repeatable Read prevents phantom reads via gap locks on locking reads (SELECT … FOR UPDATE). Plain SELECT uses snapshot reads, which can still observe phantoms in some edge cases.

Serializable

The strictest level. Transactions execute as if they ran one at a time, sequentially. No concurrency anomalies are possible.

Prevents: all five phenomena
Allows: nothing — at the cost of more transaction aborts under contention
When to use: financial ledgers, inventory reservation, double-entry bookkeeping, regulatory compliance — anywhere a bug means lost money, corrupted data, or violated invariants. Use it precisely where you need it.
When to avoid: as a blanket “be safe” setting. The retry cost on contention is real, and most application code doesn’t handle serialization-failure errors correctly. Use Read Committed by default and raise to SERIALIZABLE only on the transactions that genuinely need it.
Postgres note: Implemented via SSI (Serializable Snapshot Isolation) — see below.

The phenomena-by-level matrix

Isolation Level	Dirty Read	Non-Repeatable	Phantom	Lost Update	Write Skew
Read Uncommitted	Possible	Possible	Possible	Possible	Possible
Read Committed	Prevented	Possible	Possible	Possible	Possible
Repeatable Read	Prevented	Prevented	Possible *	Prevented (PG) †	Possible
Serializable	Prevented	Prevented	Prevented	Prevented	Prevented

* MySQL InnoDB’s Repeatable Read also prevents phantom reads via gap locks. † PostgreSQL’s Repeatable Read prevents lost updates by aborting the conflicting transaction. MySQL InnoDB’s does not.

How Postgres Implements Isolation

Two mechanisms, layered.

MVCC — readers and writers don’t block each other

Every UPDATE writes a new row version; the old version stays until vacuum reclaims it. Each transaction’s snapshot determines which version it sees. Consequences:

Readers never block writers and vice versa. A long-running SELECT doesn’t lock anyone out of writing.
Repeatable Read is implemented by simply pinning the snapshot for the entire transaction.
The cost is bloat — dead row versions accumulate, and autovacuum reclaims them in the background.

SSI — Serializable without predicate locking

Classical serializable isolation requires predicate locks: locks not on rows, but on query conditions. They’re correct but block aggressively. Postgres instead uses Serializable Snapshot Isolation (SSI):

Each transaction runs at Repeatable Read (its own snapshot).
Postgres tracks read-write dependencies between concurrent transactions in the background.
At commit time, if Postgres detects a dependency cycle that would produce a non-serializable schedule, it aborts the offending transaction with a serialization-failure error.

The trade: less blocking, more aborts. Your application must be prepared to retry transactions that fail with SQLSTATE 40001:

-- In application code:
BEGIN ISOLATION LEVEL SERIALIZABLE;
-- ... do work ...
COMMIT;
-- If COMMIT returns serialization_failure, sleep briefly and retry
-- the whole transaction from BEGIN.

This retry loop is non-negotiable for SERIALIZABLE. Skipping it means random transaction failures under load.

A note on other databases

Oracle’s “Serializable” is actually snapshot isolation — it permits write skew, despite the name.
SQL Server has “Serializable” (predicate locking) and “Snapshot” (MVCC) as distinct levels.
MySQL InnoDB’s Repeatable Read is unusual: it prevents phantoms with gap locks but not lost updates with snapshot conflict detection.

Treat “Serializable” with skepticism across systems — read your DB’s actual semantics.

Concurrency Tools in Application Code

Isolation levels handle concurrency at the database level. In application code, three more tools cover the cases isolation alone won’t fix.

`SELECT … FOR UPDATE` (Pessimistic Locking)

Acquires a row-level write lock as part of the read. Other transactions that try to read the same row with FOR UPDATE block until you commit.

BEGIN;
  SELECT stock FROM products WHERE sku = 'WIDGET' FOR UPDATE;
  -- application logic decides what to write
  UPDATE products SET stock = stock - 1 WHERE sku = 'WIDGET';
COMMIT;

Use when you need to read, decide, and write atomically and the read must reflect the latest committed value, not a snapshot. Common in inventory deduction, ticket reservation, money transfer.

Watch out: holding a row lock across an HTTP boundary or a slow operation pins the lock for the duration. A SELECT … FOR UPDATE followed by a 2-second external API call followed by COMMIT means every concurrent request for that row waits 2 seconds.

Atomic Updates

When the new value is a pure function of the old one, skip the lock and let the database compute it:

UPDATE products
SET stock = stock - 1
WHERE sku = 'WIDGET' AND stock > 0;

Single statement, no read-modify-write race. The AND stock > 0 predicate prevents negative stock without an extra check. The simplest correct solution to lost updates when applicable.

Optimistic Concurrency Control (Version Columns)

Add a version column. Every update increments it. Every update conditions the write on the version it last read:

SELECT id, balance, version FROM accounts WHERE id = 42;
-- application: balance=100, version=7
UPDATE accounts
SET balance = 80, version = 8
WHERE id = 42 AND version = 7;

If another transaction wrote first, the row’s version is now 8 and your WHERE version = 7 matches zero rows. The application sees rowcount = 0 and retries the read-modify-write cycle.

Pros: no blocking, no deadlocks, works across HTTP boundaries (the version travels in the request). Cons: requires retry logic, performs poorly under high contention.

The decision between pessimistic and optimistic comes down to contention rate. Rare contention → optimistic. Frequent contention → pessimistic.

Decision Tree

Scenario	Reach for	Why
Standard web app CRUD, dashboards, profile reads	`READ COMMITTED`	Default. Cheap. Anomalies are real but acceptable in this domain.
Multi-step report or invoice — query the same data multiple times in one transaction	`REPEATABLE READ`	Consistent snapshot. Avoids the price-changed-mid-transaction class of bug.
Concurrent decrement of a counter / stock / balance	Atomic `UPDATE ... SET x = x - 1` at `READ COMMITTED`	Single statement, no race. Don’t reach for higher isolation.
Need to read, decide, then write atomically; long-lived contention expected	`SELECT … FOR UPDATE` at `READ COMMITTED`	Pessimistic lock on the row. Easy mental model. Watch lock duration.
Same, but contention is rare and you can retry	Optimistic version column at `READ COMMITTED`	No blocking, works across HTTP boundaries, just needs retry logic.
Multi-row invariant (“at least one doctor on call,” “no double-booking”)	`SERIALIZABLE`	Only level that catches write skew. Implement retry on serialization failure.
Financial transfers, double-entry ledger	`SERIALIZABLE`	The lost-money class of bug. Worth the aborts.

The pattern: default to READ COMMITTED, raise selectively where the workload demands it. Most applications need higher isolation only on specific endpoints — not as a global setting.

Common Mistakes

Closing

Isolation level isn’t a “set and forget” knob. It’s a per-workload decision that interacts with how you structure your transactions, how long you hold them open, and what your application does when a commit fails. The teams that get this right are usually the ones who default to READ COMMITTED, identify which specific transactions actually need stronger guarantees, raise those to REPEATABLE READ or SERIALIZABLE, and handle retries in code. The teams that get it wrong either set SERIALIZABLE everywhere and wonder why their throughput collapsed, or stay on READ COMMITTED and write SELECT … FOR UPDATE everywhere as a substitute.

Getting these wrong is how production databases produce corrupted data, double-bookings, billing discrepancies, and race conditions that only show up at 3 AM under peak load. Getting them right is mostly about understanding which anomaly your code can encounter, choosing the cheapest level that prevents it, and writing the retry path for when the database asks you to.

Database Normalization — the schema-level rules transactions defend.
How Databases Actually Store and Find Your Data — what “a row” physically is, the substrate for everything above.
PostgreSQL Indexing Deep Dive — including how CREATE INDEX CONCURRENTLY interacts with isolation.
Database Replication — how isolation guarantees travel (or don’t) across replicas.

Database Normalization: 1NF through 5NF Explained

Sun, 19 Apr 2026 00:00:00 GMT

Database normalization is the process of organizing tables to reduce redundancy and prevent data anomalies. Each “normal form” is a progressively stricter rule about how data should be structured. For each form, we’ll look at a table that violates it, explain exactly why, then show the corrected design with real data.

First Normal Form (1NF)

The Rule: Every column holds atomic (indivisible) values. No repeating groups. Every row is uniquely identifiable by a primary key. Column data types are consistent.

The Violating Table — Insured Customer Family Data:

(Imagine the columns repeating: Child_1, Age_1, Child_2, Age_2, ... — the duplicate column names below render as a single header in markdown but represent the unbounded repeating-group violation.)

Customer Code	Customer Surname	Child	Age	Child	Age ···
DENT04	Denton	Billy	3	Olivia	10
PARK31	Park	Justin	7
PARK31	Park	Phoebe	5
KELL17	Keller	Alice	4	Ivy	Newborn ···

This table violates 1NF in four separate ways:

Row order is meaningful — the header says “from most-recently-entered (top) to least-recently-entered (bottom)”. Data meaning should never depend on row order.
Inconsistent data types — the Age column holds integers in some rows (“10”) and strings in others (“Newborn”). A column must have one consistent type.
No primary key — Customer Code is not unique: PARK31 appears twice. Uniqueness is not being enforced.
Repeating groups — Child and Age columns repeat indefinitely across the row. The number of children is unbounded, so there’s no way to design a fixed schema.

The Fix — Two tables, both in 1NF (and well-normalized for this schema — we’ll see what 5NF actually requires further down):

Customer_Code	Customer_Surname
DENT04	Denton
PARK31	Park
KELL17	Keller

Customer_Code	Child_Seq_No	Child_Name	Child_Age	Date_Recorded
PARK31	1	Phoebe	5	2026-01-27
PARK31	2	Justin	7	2026-01-28
DENT04	1	Billy	3	2026-02-01
DENT04	2	Olivia	10	2026-02-01
KELL17	1	Ivy	0	2026-01-19
KELL17	2	Alice	4	2026-01-19
KELL17	3	Jonah	7	2026-01-19
KELL17	4	Billy	9	2026-01-19

Primary key of Customer = Customer_Code
Primary key of Customer_Dependent_Child = { Customer_Code, Child_Seq_No }
Customer_Code on the child table is a foreign key to the Customer table
Date_Recorded replaces row ordering as the way to track recency
In a real database, Child_Age would be replaced with Child_Birth_Date — age changes, birth dates don’t

A Quick Note on Functional Dependencies

Before 2NF, here’s the notation we’ll use throughout. A functional dependency X → Y means: “given X, you can determine Y uniquely.” If you know an Employee_Code, you know exactly one Employee_Surname. So Employee_Code → Employee_Surname.

The normal forms are progressively stricter rules about which functional dependencies are allowed in a single relation. The mnemonic for 1NF/2NF/3NF is:

Every non-key column depends on the key, the whole key, and nothing but the key.

1NF — every value is atomic; the table has a key.
2NF — non-key columns depend on the whole key (no partial dependencies on part of a composite key).
3NF — non-key columns depend on nothing but the key (no transitive dependencies through other non-key columns).

BCNF, 4NF, and 5NF generalize this further to handle edge cases. We’ll get to those.

Second Normal Form (2NF)

The Rule: Must be in 1NF. Every non-key column must depend on the entire primary key — not just a part of it. This only applies when the primary key is composite (made up of more than one column).

The Violating Table — Employee Working Days:

Employee_Code	Working_Day	Employee_Surname
WREN02	TUESDAY	Wren
WREN02	WEDNESDAY	Wren
WREN02	THURSDAY	Wren
MILL01	MONDAY	Miller
MILL01	TUESDAY	Miller

The primary key is { Employee_Code, Working_Day }. But Employee_Surname only depends on Employee_Code — it tells us about the employee, not about which day they work. This is a partial dependency: a non-key column depends on only part of the composite key.

The result is obvious in the data: “Wren” is repeated three times and “Miller” twice. If the employee changes their surname, every row must be updated — miss one and the data becomes inconsistent.

The Fix:

Employee_Code	Employee_Surname
WREN02	Wren
MILL01	Miller

Employee_Code	Working_Day
WREN02	TUESDAY
WREN02	WEDNESDAY
WREN02	THURSDAY
MILL01	MONDAY
MILL01	TUESDAY

Primary key of Employee = Employee_Code
Primary key of Employee_Working_Days = { Employee_Code, Working_Day }
Employee_Code on Employee_Working_Days is a foreign key to Employee
Surname is stored once — update it in one place, it’s consistent everywhere

Third Normal Form (3NF)

The Rule: Must be in 2NF. No non-key column should depend on another non-key column. Every non-key attribute must depend on the key, the whole key, and nothing but the key.

The Violating Table — Product:

Product_ID	Product_Name	Product_Subcategory	Product_Category
103	400-Piece Autumn Foliage Jigsaw	JIGSAWS	PUZZLES
104	250-Piece Magic Bookshop Jigsaw	JIGSAWS	PUZZLES
105	Standard Rubik’s Cube	MECHANICAL-PUZZLES	PUZZLES
106	Kendrick Koala	CUDDLY-TOYS	TOYS
107	Adventurer Walkie Talkie Set	ELECTRONIC-TOYS	TOYS

The primary key is Product_ID. Product_Category depends on Product_ID only transitively — through Product_Subcategory. The real determinant is Product_Subcategory, and Product_Subcategory is not a candidate key of this table. This is a transitive dependency: Product_ID → Product_Subcategory → Product_Category.

The redundancy is visible: “PUZZLES” appears three times and “TOYS” twice. If we rename the “PUZZLES” category, we have to update multiple rows. Miss one, the data becomes contradictory.

The Fix:

Product_ID	Product_Name	Product_Subcategory
103	400-Piece Autumn Foliage Jigsaw	JIGSAWS
104	250-Piece Magic Bookshop Jigsaw	JIGSAWS
105	Standard Rubik’s Cube	MECHANICAL-PUZZLES
106	Kendrick Koala	CUDDLY-TOYS
107	Adventurer Walkie Talkie Set	ELECTRONIC-TOYS

Product_Subcategory	Product_Category
JIGSAWS	PUZZLES
MECHANICAL-PUZZLES	PUZZLES
CUDDLY-TOYS	TOYS
ELECTRONIC-TOYS	TOYS

Primary key of Product = Product_ID
Primary key of Product_Subcategory = Product_Subcategory
Product_Subcategory on Product is a foreign key to Product_Subcategory table
Renaming “PUZZLES” to “BRAIN-GAMES” now requires updating exactly one row

3NF is the practical target for most production databases. It eliminates the most common redundancy and anomaly problems without over-fragmenting the schema.

Boyce-Codd Normal Form (BCNF)

The Rule: Must be in 3NF. For every functional dependency X → Y, X must be a candidate key. Every determinant must be a superkey.

BCNF is a stricter variant of 3NF that catches edge cases involving tables with multiple overlapping candidate keys.

The Violating Table — Course Scheduling:

Teacher	Subject	Room
Dr. Lee	Databases	Room 101
Dr. Lee	Algorithms	Room 202
Dr. Kim	Databases	Room 303

Business rules:

Each teacher-room pairing has exactly one subject (so {Teacher, Room} → Subject).
Each room hosts only one subject (so Room → Subject).

Candidate keys: {Teacher, Room}. Note that {Teacher, Subject} is not a candidate key — multiple rooms can host the same teacher/subject pair.

But Room → Subject is a valid functional dependency — and Room alone is not a candidate key. This violates BCNF. The consequence: if Room 101 is reassigned to “Algorithms”, you might update one row but leave the teacher’s schedule inconsistent.

The Fix:

Room	Subject
Room 101	Databases
Room 202	Algorithms
Room 303	Databases

Teacher	Room
Dr. Lee	Room 101
Dr. Lee	Room 202
Dr. Kim	Room 303

Every determinant is now a candidate key. The Room → Subject dependency lives in its own table where Room is the primary key.

BCNF decomposition is not always dependency-preserving — splitting Room → Subject into its own table loses the ability to enforce {Teacher, Room} → Subject with a simple foreign key. This is the classic BCNF tradeoff: stronger normalization, weaker constraint enforcement.

Fourth Normal Form (4NF)

The Rule: Must be in BCNF. For every non-trivial multi-valued dependency X →→ Y, X must be a superkey of the relation.

A multi-valued dependency exists when one attribute independently determines a set of values for two other attributes in the same table — and those two sets are unrelated to each other.

The Violating Table — Course Assignments:

Course	Textbook	Teacher
Databases	SQL Mastery	Dr. Lee
Databases	SQL Mastery	Dr. Kim
Databases	DB Internals	Dr. Lee
Databases	DB Internals	Dr. Kim

A course has multiple textbooks AND multiple teachers — independently. Any teacher can use any textbook. The table is forced to store every possible (Textbook, Teacher) combination. Adding a new teacher requires adding one row per textbook. Adding a textbook requires adding one row per teacher. Miss one and the data is inconsistent.

The Fix:

Course	Textbook
Databases	SQL Mastery
Databases	DB Internals

Course	Teacher
Databases	Dr. Lee
Databases	Dr. Kim

Each independent multi-valued fact lives in its own table. Adding Dr. Park as a teacher is now one row — not one row per textbook.

Fifth Normal Form (5NF)

5NF is rarely the deciding factor in practical schema design — most production databases stop at 3NF or BCNF. We cover it here because the underlying idea (avoiding fabricated tuples on rejoin) is genuinely useful even when you’re not formally proving 5NF compliance.

The Rule: Must be in 4NF. The table cannot be decomposed further without losing information. Eliminates join dependencies that are not implied by candidate keys.

5NF applies when the only lossless way to decompose a table is into three or more smaller tables. Any two-table split introduces spurious (incorrect) rows when rejoined.

The Violating Table — Supplier / Product / Project:

Supplier	Product	Project
TechCorp	Laptop	Apollo
TechCorp	Laptop	Hermes
TechCorp	Monitor	Apollo
GadgetCo	Keyboard	Apollo

The constraint that makes this 5NF-relevant: the table is only valid if the following business rule holds — if Supplier S sells Product P, AND Product P is used in Project J, AND Supplier S is engaged on Project J, then (S, P, J) must appear in the table. In other words, the relation is the join of three binary relations: SuppliersProducts, ProductsProjects, and SuppliersProjects. Without that constraint, the table is just a normal ternary relationship and no decomposition would lose information.

The rule: a (Supplier, Product, Project) row is only valid if all three pairwise relationships independently hold — the supplier sells that product, the product is used in that project, and the supplier is engaged on that project. No two-way split can enforce this correctly without creating spurious combinations.

The Fix — three pairwise tables:

Supplier	Product
TechCorp	Laptop
TechCorp	Monitor
GadgetCo	Keyboard

Product	Project
Laptop	Apollo
Laptop	Hermes
Monitor	Apollo
Keyboard	Apollo

Supplier	Project
TechCorp	Apollo
TechCorp	Hermes
GadgetCo	Apollo

The original table is reconstructed by joining all three. Any pair-wise decomposition would introduce rows that were never true.

Why three tables, not two? If we decomposed into just Supplier_Product (S, P) and Product_Project (P, J), rejoining on P would manufacture rows that never appeared in the original table. For example, if Supplier A sells the Keyboard and Project Apollo uses Keyboards, the join produces (A, Keyboard, Apollo) — even if Supplier A is not actually engaged on Apollo. The third table Supplier_Project (S, J) is exactly the constraint that prevents this fabrication: only (S, P, J) triples that survive joining all three tables are valid.

The Progression at a Glance

3NF / BCNF is the right target for most production systems. Go further only when you have genuinely independent multi-valued facts (4NF) or complex three-way join dependencies (5NF).

When to Denormalize

Normalization is the right starting point — not always the ending point. Intentional redundancy is valid when:

Read performance is critical — heavily joined queries across many normalized tables are expensive. A materialized view or denormalized reporting table can dramatically speed up reads.
You’re building a data warehouse — OLAP workloads favor wide flat tables (star/snowflake schemas) over normalized OLTP schemas.
The data rarely changes — if a field updates once a year, the update anomaly risk is low.
A query is too hot to afford joins — on high-throughput endpoints, a cache field with an explicit refresh strategy can be worth the complexity.

Normalize first, denormalize deliberately. Start clean, measure where bottlenecks actually are, and introduce redundancy with intention — not by accident.

Pitfalls and Common Mistakes

The rules above are clear, but applying them in practice is where teams get into trouble. The recurring patterns worth watching for:

Confusing 3NF with BCNF. They sound interchangeable but aren’t. A relation can be in 3NF and still violate BCNF if a non-key attribute determines another candidate-key attribute. Check every functional dependency — not just the ones from the primary key.
Treating 4NF and 5NF as required. They apply only when independent multi-valued facts (4NF) or three-way join dependencies (5NF) actually exist in your data. Most production schemas don’t hit either situation. Don’t decompose chasing a textbook target you don’t have.
Over-normalizing read-mostly schemas. Splitting tables to satisfy 3NF only pays off if updates happen. A table where rows are written once and never modified rarely benefits from being split into three with foreign keys — you’re paying join cost for no anomaly-prevention upside.
Mistaking “atomic” for “small.” 1NF demands no repeating groups, not “no complex values.” A JSONB document, a PostGIS geometry, or an array column is 1NF-compliant if treated as a single indivisible value. Don’t split data that the application always reads and writes as a unit.
Dropping foreign keys “for performance.” Foreign keys cost very little on modern Postgres and are the only mechanism that guarantees the referential integrity normalization assumes. Disabling them turns “guaranteed valid” into “valid as long as no bug ever ships.”
Storing derived values that violate 3NF. A row holding unit_price, quantity, and total_price creates a transitive dependency (total_price is derivable from the other two). Either compute it on read or accept the redundancy with an explicit refresh strategy and write the trigger / view that keeps it consistent.
Adding nullable “extension” columns instead of normalizing. Adding secondary_email, tertiary_email, home_phone, work_phone to a users table — most of them null for most users — is a repeating group disguised as separate columns. The right move is a user_contact_methods child table.
Premature normalization before knowing the access patterns. Splitting a domain into eight tables before you know how the application queries it produces a schema optimized for nothing. Sketch the read paths first; let them inform the table boundaries.
Forgetting that BCNF decomposition isn’t always dependency-preserving. Decomposing to satisfy BCNF can scatter a multi-attribute functional dependency across two tables in a way that makes it un-enforceable with a simple FK. Decide explicitly whether you’d rather lose the BCNF guarantee or the dependency-preservation guarantee — there’s no fully correct answer.
Using surrogate keys to “skip” normalization. Adding an id BIGSERIAL to a table doesn’t make it normalized — it just gives the existing duplicates a new column. Normalize against the real candidate keys; add surrogate IDs as an implementation detail for foreign-key ergonomics.
Ignoring functional dependencies in legacy schemas. When refactoring a denormalized table, list every X → Y you can identify in the data before designing the new shape. The dependencies tell you where the tables want to split — guessing produces a schema that fights you at the next migration.

ACID and Isolation Levels — what transactions defend the schema you’ve now normalized.
How Databases Actually Store and Find Your Data — what your normalized rows physically become.
Reading PostgreSQL EXPLAIN Output — how a normalized schema’s joins translate to plan nodes.

Write Ahead

Postgres Concurrency Internals: Connections, Locks, and MVCC

1. The Connection Model

Why this matters

2. The Vocabulary

3. Connection Pooling

PgBouncer pool modes

Transaction-pooling caveats (the real-world list)

Pool sizing

What’s after PgBouncer

4. The Lock Manager

Heavyweight locks — the user-visible layer

Acquisition: which statement takes which mode

Compatibility: which modes can share an object

Row-level locks (a different mechanism)

Lightweight locks — the internal layer

5. MVCC Under the Hood

The tuple header

Snapshots

HOT updates (the optimization that matters)

Vacuum: the garbage collector

XID wraparound

6. Deadlocks and How They’re Resolved

The classic cause: inconsistent lock ordering

When deadlocks are inevitable

deadlock_timeout tuning

7. Observing It Live

pg_stat_activity — what every backend is doing

pg_locks — every lock in the cluster

pg_blocking_pids() — the one-shot diagnostic

pg_stat_statements — workload accounting (different but related)

8. Advisory Locks

When to reach for them

What NOT to use them for

The session-pool footgun

9. Buffer Pool Concurrency (Briefly)

10. Decision Tree

11. Common Mistakes

12. Closing

Related reading

Database Replication: Strategies, Topologies, and What to Use When

1. Why Replicate

2. The Vocabulary

3. The Three Real Replication Strategies

Statement-based (historical, mostly avoided)

Physical / WAL Streaming (Postgres default)

Logical Replication (publication / subscription)

Trigger-based (last-resort, application-level)

4. Sync Modes — The Latency / Durability Trade

5. Topologies — Single-Leader, Multi-Leader, Leaderless

Single-Leader (95% of systems)

Multi-Leader

Leaderless

6. Consistency Models in Plain Language

7. Replica Reads — The Rules

Pattern 1: Read primary after write

Pattern 2: Sticky sessions

Pattern 3: LSN tracking (read-your-writes)

Pattern 4: Read-only replica with a freshness contract

8. Postgres Replication, End to End

9. Decision Tree

10. Common Mistakes Checklist

11. Closing

Related reading

Postgres Partitioning: Strategies, Real Advantages, and Pitfalls

1. What Partitioning Actually Is

2. The Three Partitioning Strategies

3. Partition Pruning — The Mechanism That Makes It Worth Anything

4. Local Indexes — and the Absence of Global Indexes

5. The Real Advantages (with concrete examples)

5.1 The active working set fits in shared_buffers

5.2 Retention becomes O(1)

5.3 Operational isolation for DDL

5.4 Autovacuum parallelism

5.5 Per-partition statistics, better query plans

5.6 Index-only scans for old data

5.7 Tiered storage via tablespaces

5.8 CREATE INDEX CONCURRENTLY scales linearly

5.9 Capacity planning is visible

5.10 Selective backups and archival

`deadlock_timeout` tuning

`pg_stat_activity` — what every backend is doing

`pg_locks` — every lock in the cluster

`pg_blocking_pids()` — the one-shot diagnostic

`pg_stat_statements` — workload accounting (different but related)

5.1 The active working set fits in `shared_buffers`

5.8 `CREATE INDEX CONCURRENTLY` scales linearly

MVCC, dead tuples, and `xid` wraparound

7.5 `ATTACH` and `DETACH` partition locks

7.15 `pg_dump` dumps each partition separately

5.1 Cross-shard `JOIN`s