Your next migration will lock the users table.
Here is how to ship schema changes nobody notices.

I'm a senior backend tech lead and I run Belmo, a small European PaaS. I'll mention it once near the end with a clear disclosure. Everything before that works on any platform and any Postgres you deploy to.

You add a column. It seems like the smallest possible change. One line in a migration file, a new field on a form, you push it, and the deploy hangs. Then the error reports start: every request that touches the users table is timing out. Your app is not down because the code is broken. It is down because a single ALTER TABLE is holding a lock that every other query is now waiting behind, and the queue is growing faster than it drains.

This is one of the few production incidents that hits you precisely when you are being careful. You are not pushing risky code. You are doing database hygiene, adding a field, making something required, renaming a badly named column. And the migration that looked free on your laptop with forty rows takes the whole table offline at four hundred thousand. I have watched this happen to good engineers and to non-tech founders alike, and the fix is the same idea in both cases. You stop treating a schema change as one atomic event and start treating it as a sequence of safe, boring steps.

Why your laptop lied to you

The reason this surprises people is that migrations are tested in exactly the conditions where they cannot fail. On your machine the table is tiny. The migration runs in single-digit milliseconds, so even if it takes an aggressive lock, nothing else is trying to read the table at that instant. It looks instant because it is, locally. There is no concurrency, no traffic, no queue to block.

Production is the opposite environment. The table is large, and there are dozens of queries per second already touching it when your migration arrives. Some database operations need an exclusive lock, which means they have to wait for every in-flight query on that table to finish, and while they wait, they block every new query from starting. So a migration that needs the lock for two seconds does not cost you two seconds. It costs you two seconds plus however long the slowest current query takes to finish, and during that whole window every new request piles up behind it. On a busy table, "two seconds" becomes a thirty-second outage, and your connection pool fills with waiting queries until the app falls over entirely.

The dangerous part is that not all migrations behave this way, so you cannot learn the lesson from a near miss. Adding a nullable column in modern Postgres is nearly free. Adding a column with a volatile default, or a NOT NULL constraint that forces a full table rewrite, or an index built the normal way, can all lock the table for as long as the rewrite or scan takes. You will run ten safe migrations, conclude that migrations are safe, and the eleventh will be the one that rewrites a million rows.

The pattern that fixes almost all of it: expand and contract

The core mistake is wanting the schema and the code to change at the same moment. They feel like they should be one step. But a running app means old code and new code coexist for at least a few seconds during a deploy, and a single locking change forces the database to flip all at once underneath both of them. The way out is to break every change into additive steps that are each safe on their own, and to never make the database and the code depend on each other being updated in the same instant.

The pattern has a name, expand and contract, and it has three phases. First you expand: you add the new thing without removing or constraining the old thing, so the database now supports both the old and new shape. Then you migrate the data and ship the code that uses the new shape, while the old shape still works as a fallback. Finally you contract: once nothing reads the old shape anymore, you remove it. Each phase is a separate deploy. None of them locks the table for any meaningful time. The schema is never in a state that the currently-running code cannot handle.

It sounds slower because it is more steps, and that is the actual trade. You exchange one scary five-minute deploy for three boring ones spread over a day or two. For a side project that nobody is paying for, maybe you take the risk. For anything with real users, the boring version is the cheap version, because the exciting version eventually bills you an outage.

Walking through the two changes that bite hardest

Take the classic example: you want to add a country column to users and make it required. The naive migration is one statement.

-- The version that locks your table on Postgres
ALTER TABLE users ADD COLUMN country text NOT NULL DEFAULT 'unknown';

On an old enough Postgres, or with a volatile default, this rewrites every row while holding an exclusive lock. The expand-and-contract version is three deploys, and each individual step is safe.

-- Deploy 1 (expand): add it nullable, no default rewrite, instant
ALTER TABLE users ADD COLUMN country text;

-- Deploy 2 (backfill): fill existing rows in batches, off the hot path
UPDATE users SET country = 'unknown'
WHERE country IS NULL AND id BETWEEN 1 AND 10000;
-- repeat the range in a loop, a few thousand rows at a time

-- Deploy 3 (contract): now that every row has a value, add the constraint
-- Validate it without a long lock by adding NOT VALID first, then validating
ALTER TABLE users ADD CONSTRAINT users_country_not_null
  CHECK (country IS NOT NULL) NOT VALID;
ALTER TABLE users VALIDATE CONSTRAINT users_country_not_null;

The NOT VALID trick matters. Adding the constraint as NOT VALID takes a brief lock and skips the full-table scan, so it returns immediately. The separate VALIDATE CONSTRAINT step does the scan, but it takes a weaker lock that lets reads and writes continue while it runs. You have turned one long exclusive lock into two short ones with the slow part happening in the background.

Indexes have the same shape of problem and the same shape of fix. The default CREATE INDEX locks the table against writes for the entire build, which on a large table is minutes. Postgres gives you a concurrent version that does not.

-- Locks writes for the whole build. Do not do this on a live table.
CREATE INDEX idx_users_country ON users (country);

-- Builds in the background, lets writes continue. Do this instead.
CREATE INDEX CONCURRENTLY idx_users_country ON users (country);

The concurrent build is slower in wall-clock time and cannot run inside a transaction, which trips up some migration tools that wrap everything in one. That is worth knowing before it surprises you mid-deploy, because if the tool insists on a transaction you have to mark that migration to run outside one.

The worst offender of all is renaming a column, because there is no safe atomic rename when old code is still reading the old name. The honest answer is that you do not rename. You add the new column, write to both for a release, backfill, point reads at the new one, and only then drop the old one. A rename is just an expand-and-contract where people forget that the old code is still running for the next thirty seconds.

The part that has nothing to do with SQL

Even a perfect migration can take you down if the deploy choreography is wrong. The order of operations between your schema and your code is its own source of outages, and it is the part people skip.

The rule that prevents most of it: a migration must be compatible with both the currently-running code and the code you are about to deploy. If you drop a column and then deploy the code that stopped using it, there is a window where the old code is still live and querying a column that no longer exists. Every request in that window errors. So additive changes go before the code that needs them, and destructive changes go after the last code that used them, usually a full release later. Expand before deploy, contract after.

Two more habits save real incidents. Set a short lock timeout on your migrations so that a migration which cannot get its lock quickly fails fast instead of hanging and blocking everything behind it. A failed migration you can retry at 2am is infinitely better than a migration that wedges the whole app for ten minutes while it waits.

-- At the top of a migration: give up fast instead of queueing behind traffic
SET lock_timeout = '3s';

And always run a migration as its own deploy step, before the application starts, not lazily on the first request that happens to need it. I have seen apps where the migration runs on boot inside the web process, so a slow migration means the health check fails, the platform thinks the deploy is broken, and it starts cycling instances while the migration is still running. Separate the two.

What I built, and the honest disclosure

I run Belmo, a small European PaaS, and I kept doing the same dance by hand for clients: run the migration as its own step, before the code that needs it, never inside the web process on boot. On Belmo Pro that step is an on-demand script, a one-off job that runs separately from your web service and exits. That is the disclosure, that is the one mention. It does not do the hard part for you. Expand and contract is a discipline, not a feature, and no platform can decide for you whether a given ALTER is going to rewrite your table, or what lock_timeout suits your traffic. The tooling can make the safe path easier to take, but you still have to know what you are asking the database to do.

What to do tonight, whatever platform you are on

1. Find your three or four biggest tables and treat any change to them as a multi-step change by default. Small tables you can be careless with. Big ones you cannot.
2. Add SET lock_timeout to the top of your migrations so a stuck migration fails fast instead of taking the app with it.
3. Switch every index you create on a populated table to CREATE INDEX CONCURRENTLY, and check that your migration tool will let it run outside a transaction.
4. Stop adding NOT NULL columns in one shot. Add nullable, backfill in batches, then validate the constraint with the NOT VALID then VALIDATE two-step.
5. Never rename a column on a live table. Add, dual-write, backfill, switch reads, drop. Five steps, zero downtime.
6. Make migrations a separate deploy step that runs before the new code takes traffic, not something the web process does on boot.
7. Rehearse one scary migration against a copy of production-sized data before you run it for real. The number of rows is the entire difference between safe and not, and you want to find that out on a clone.

One open question

The thing I go back and forth on is how far to take this on a small app. Expand and contract is unambiguously correct at scale, but it triples the number of deploys, and for a project with a few hundred users a locking migration at 3am Paris time might cost you nothing because nobody is awake. I have a real bias toward doing it the safe way every time so the muscle memory is there when it matters, but I have also watched solo founders burn an entire evening on a three-phase migration for a table that would have rewritten in eight seconds during a maintenance window.

Where do you draw that line? Do you run every change the safe way regardless of size, or do you keep a threshold in your head, some row count above which you switch into the careful mode? I would like to hear how people who actually ship alone decide this, because the textbook says always and reality says it depends.

Related reads on this blog: Your indie SaaS has zero working Postgres backups. Here's the 20-minute fix (and the drill you need to run before you sleep tonight). · Your indie SaaS has no staging environment. Here's the 30-minute setup (and the 4 production bugs it catches before your users do). · Your query was instant at launch. At 50,000 rows it takes 8 seconds. Here is the indexing guide nobody handed you.