Database Migrations

The job that wakes you up at 3am

Database migrations are where deploys go to die. Your code rolls forward in seconds; the schema lives until you remove it. Get the migration order wrong and you have an outage that cannot be reverted by rolling back the deploy.

The expand-and-contract pattern

For any migration that touches data used by both old and new code:

Expand — add the new column / table / index. Old code ignores it; new code can use it. Backwards compatible.
Migrate — backfill data into the new structure. Often a separate background job, not the deploy.
Switch — deploy code that reads/writes the new structure. Old code still tolerated.
Contract — once 100% of code is on the new structure (and stays that way for a release cycle), drop the old column / table.

Migration runner placement

Pre-deploy job — run migrations before any new code reaches users. Safe for additive (expand) migrations.
App startup — code runs migrations on boot. Risky: race conditions if multiple instances start simultaneously, longer cold-start.
Manual / on-demand — destructive migrations (drop column) gated behind a manual approval, run separately from deploy.

Code

Pre-deploy migration job·yaml

  migrate:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: production
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123:role/gha-migrate
          aws-region: us-east-1
      - run: |
          # Run only additive migrations here.
          # Destructive ones live in a separate manual workflow.
          DATABASE_URL=$(aws ssm get-parameter --name /prod/db-url --with-decryption --query Parameter.Value --output text)
          DATABASE_URL="$DATABASE_URL" alembic upgrade head

  deploy:
    needs: migrate
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: ./deploy.sh

The job that wakes you up at 3am

The expand-and-contract pattern

Migration runner placement

Code

External links

Exercise

Progress

Comments 0