The Code Rolled Back. The Schema Could Not.

The previous release still has to run against the new schema, or rollback is not an option.

A team ships a module on a Tuesday. The new release renames a column, the migration runs as part of the deploy, and an hour later something is wrong, so someone clicks rollback. The old container comes back in seconds. It then queries a column that no longer exists and fails on every request. The rollback worked and the system is down.

Shipping without rollbacks is an engineering goal about how changes get released. The only rollback that stays cheap is one where the previous version of the code can still run against the current version of the database. That is a property you design into each migration before the deploy, and a pipeline cannot supply it afterwards.

Rolling back code does not roll back data

Code is replaceable. You redeploy an older artifact and the process is the same as it was. A database holds state that the previous release never wrote and may not understand: new columns, dropped columns, rewritten rows, new constraints. A deploy pipeline treats “release N” as one unit, but release N is two things with different undo properties.

A “down” migration looks like the answer and mostly is not. Reversing a DROP COLUMN cannot bring back the data. Reversing a backfill means guessing which rows were touched. Reversing a type change can truncate values written since. A down script that was never run against production-shaped data is a plan, not a safety net.

The failure mode is a migration that only the new code can survive

The mechanism is simple. Release N+1 ships a migration that is incompatible with release N, such as a rename, a drop, a new NOT NULL without a default, or a tightened constraint. Between the moment the migration runs and the moment you decide to roll back, the system is in a state release N cannot serve. Rollback then means running a destructive reverse migration under pressure, or rolling forward with a hotfix at the worst possible time.

I would treat a migration that breaks the previous release as a blocked change in review. It is a hard error in the same way an import block for a resource that does not exist is a hard error, not a quiet no-op. If the previous release cannot run against the new schema, the pull request does not merge in that shape.

Expand, deploy, contract

The standard answer is to split one breaking change into compatible steps, each deployed separately. Here is an example for renaming customer_ref to account_ref:

// Example: step 1, expand. Additive only, safe for the old release.
export async function up(db: Db) {
  await db.query(`ALTER TABLE orders ADD COLUMN account_ref text`);
  // Example: new rows are kept in sync by the database, not by app code,
  // so the old release keeps working and the new column never lags.
  await db.query(`
    CREATE FUNCTION sync_account_ref() RETURNS trigger AS $$
    BEGIN
      IF TG_OP = 'INSERT' THEN
        NEW.account_ref := COALESCE(NEW.account_ref, NEW.customer_ref);
        NEW.customer_ref := COALESCE(NEW.customer_ref, NEW.account_ref);
      ELSIF NEW.customer_ref IS DISTINCT FROM OLD.customer_ref THEN
        NEW.account_ref := NEW.customer_ref;
      ELSIF NEW.account_ref IS DISTINCT FROM OLD.account_ref THEN
        NEW.customer_ref := NEW.account_ref;
      END IF;
      RETURN NEW;
    END; $$ LANGUAGE plpgsql`);
  await db.query(`
    CREATE TRIGGER orders_sync BEFORE INSERT OR UPDATE ON orders
    FOR EACH ROW EXECUTE FUNCTION sync_account_ref()`);
}

The sequence is:

Expand. Add the new column, nullable, with the trigger above. Release N still works because nothing it uses has changed.
Backfill. Copy old values into the new column in batches, not in one transaction that locks the table.
Deploy the code that reads and writes the new column. Now rollback is real. If release N+1 misbehaves, release N runs against the same schema, because both columns are populated.
Contract. Only after the new release has been stable for as long as you are willing to wait, drop the old column and the trigger, in a later release with its own review.

Every step is compatible in both directions, so no step needs a down migration to be undone safely. The old column stays until you have decided you will never go back to code that needs it.

Constraints are where this gets stricter, not looser

The tempting shortcut is adding NOT NULL or a CHECK in the same release as the code that satisfies it. The old release does not satisfy it, so rolling back turns every insert into an error. The safe order is to add it as a CHECK ... NOT VALID (for a required column, CHECK (col IS NOT NULL) NOT VALID), backfill, then VALIDATE CONSTRAINT, which checks existing rows without holding the heavy lock for the whole scan, and only then rely on it.

I care about this because, in ledger work in Node/TypeScript on Postgres, I put the rules in the database: constraints that refuse an unbalanced write, integer minor units, an append-only audit. That makes the schema the strictest part of the system, and the part you least want to change in a hurry. A constraint is a promise every future writer has to keep, including the release you roll back to. The same instinct applies to compute: on the infrastructure platform I work on, Cloud Run is pinned to the live image rather than a moving tag, so that “go back” means a specific artifact and not a guess. A schema needs the same treatment: go back to something that was known to work against the current state.

What a pipeline needs to keep rollbacks rare

Rollback frequency is a measure of how often a release surprised you. You will not get it to zero by trying harder. You get it near zero by making surprises cheap and by catching incompatibility before the deploy:

Run the previous release’s test suite against the new schema in CI. If it fails, the migration is not backward compatible and the build says so.
Run migrations as a separate, earlier step from the code deploy, so the schema change is visible and reversible on its own.
Keep feature flags for behaviour, not for schema. A flag can turn off new code but cannot un-drop a column.

What I would tell someone building this

Make the previous release a test target. If release N cannot pass its own tests against the N+1 schema, the migration is not safe to ship, and you want to learn that in CI.

Never drop or rename in the same release that stops using it. Expand first, contract in a later release once you are sure you are not going back.

Add constraints in two steps. Create them NOT VALID, backfill, validate, then depend on them.

Treat a down migration as untested until it has run on realistic data. If you would not run it at 2 a.m. under pressure, it is not your rollback plan.

Decide what rollback means before the release. Write down which artifact you return to and which schema state it needs. If you cannot, the change is not ready to ship.