Malav Naik ગુજરાતી Résumé

Data engineering

Why '2 Hours of Maintenance' Becomes 3 Days: The Engineering Behind Failed Deployments and Rollbacks

What turns a short maintenance window into days? A visual guide to failed deployments, database migrations and rollback, with Employment Permits Online as a case study.

Published · Updated · 8 min read · by Malav Naik

Malav Naik in a black kurta in front of colourful umbrellas and marigold garlands – Why '2 Hours of Maintenance' Becomes 3 Days: The Engineering Behind Failed Deployments and Rollbacks

As I write this on 27 October 2026, Ireland’s Employment Permits Online portal has shown this page for about three days:

“Site Unavailable. This site is unavailable due to essential maintenance. We apologise for any inconvenience caused and hope to resume normal service as soon as possible.”

An earlier notice on the portal announced a three-hour maintenance window (Tuesday 6 October, 6pm to 9pm). That’s what planned maintenance usually looks like. So what can turn hours into days? We don’t know the reason in this case, so this post doesn’t guess. Instead it walks through one common what if: a failed deployment that can’t be cleanly rolled back, especially when a database migration is involved.

The portal in brief: a cloud-based system for Irish work permit applications, live since 28 April 2025, when it replaced the older EPOS system. It uses email verification and multi-factor authentication (MFA), and appears to be hosted on Microsoft Azure App Service.

A normal maintenance window

A well-run window follows a script. About two hours of work, plus an hour of buffer for small surprises.

A normal three-hour maintenance window Eight steps in order: announce (days before), freeze changes (day before), then inside the window: holding page on (about 5 minutes), backup and snapshot (about 20), deploy the new version (about 15), run database migrations (about 45), smoke tests of login, MFA, submit and pay (about 30), and reopen (about 5). That is about two hours of work plus an hour of buffer, so roughly three hours in total. Announce 1 days before Freeze changes 2 day before Holding page on 3 ~5 min Backup & snapshot 4 ~20 min Deploy new version 5 ~15 min Run migrations 6 ~45 min Smoke tests 7 ~30 min Reopen 8 ~5 min Before The announced window: ≈ 3 hours (≈ 2 hours of work + 1 hour buffer) A normal three-hour maintenance window Eight steps in order: announce (days before), freeze changes (day before), then inside the window: holding page on (about 5 minutes), backup and snapshot (about 20), deploy the new version (about 15), run database migrations (about 45), smoke tests of login, MFA, submit and pay (about 30), and reopen (about 5). That is about two hours of work plus an hour of buffer, so roughly three hours in total. 1 Announce days before 2 Freeze changes day before 3 Holding page on ~5 min 4 Backup & snapshot ~20 min 5 Deploy new version ~15 min 6 Run migrations ~45 min 7 Smoke tests ~30 min 8 Reopen ~5 min Window ≈ 3 hours ≈ 2 hours of work + 1 hour buffer
Figure 1 — A normal maintenance window: announce, freeze, holding page, backup, deploy, migrate, test, reopen.
Deep dive: what each step is for
  • Announce, freeze: users plan around the date; no other changes go in, so if something breaks there’s one suspect.
  • Holding page: the load balancer or gateway (the front door all requests pass through) shows a static page, so no new data arrives mid-change.
  • Backup and snapshot: taken after traffic stops, so it’s complete and consistent.
  • Deploy, then migrations: a schema migration changes the database’s structure (add a column); a data migration changes its contents (convert every phone number).
  • Smoke tests: quick checks of the journeys people actually use: log in, get an MFA code, submit, pay.
  • Reopen and monitor: watch error rates and support channels for the next few hours.

Where it breaks

The buffer is sized for small surprises, not for a step that fails outright. These are the usual culprits:

Where a maintenance window breaks The same eight-step timeline with six failure points pinned to the steps they hit. Deploy: config drift, such as an expired certificate or secret (warning). Migrations: a migration that is far slower at production scale (warning) and discovered data corruption (failure). Smoke tests: MFA codes don't arrive (failure) and a broken payment or SMS integration (failure). Reopen: a security finding that must be fixed first (failure), so reopening is blocked. Every fix also needs a change approval, a vendor ticket and a sign-off. Announce 1 Freeze changes 2 Holding page on 3 Backup & snapshot 4 Deploy new version 5 Run migrations 6 Smoke tests 7 Reopen blocked 8 1 2 3 4 5 6 1 Config drift Expired certificate or secret 2 Slow migration Hours, not minutes, at scale 3 Data corruption Bad rows stop the migration 4 Failed smoke test MFA codes don't arrive 5 Broken integration Payment or SMS calls fail 6 Security finding Must be fixed before reopen Each fix also waits for: change approval → vendor ticket → sign-off Organisational overhead turns a 20-minute fix into half a day. Where a maintenance window breaks The same eight-step timeline with six failure points pinned to the steps they hit. Deploy: config drift, such as an expired certificate or secret (warning). Migrations: a migration that is far slower at production scale (warning) and discovered data corruption (failure). Smoke tests: MFA codes don't arrive (failure) and a broken payment or SMS integration (failure). Reopen: a security finding that must be fixed first (failure), so reopening is blocked. Every fix also needs a change approval, a vendor ticket and a sign-off. Announce Freeze changes Holding page on Backup & snapshot Deploy new version 1 Run migrations 2 3 Smoke tests 4 5 Reopen blocked 6 1 Config drift Expired certificate or secret 2 Slow migration Hours, not minutes, at scale 3 Data corruption Bad rows stop the migration 4 Failed smoke test MFA codes don't arrive 5 Broken integration Payment or SMS calls fail 6 Security finding Must be fixed before reopen Each fix waits for approval, a vendor ticket and sign-off
Figure 2 — Where it breaks. Amber "!" means slow or degraded; red "✕" means blocked.

One of these adds hours. Two or three together add days.

Deep dive: the failure points in detail
  • Config drift: production quietly stops matching test. A secret rotated by hand, an expired TLS certificate (the file that lets browsers trust the site), a firewall rule that only exists in one place. On Azure App Service, settings can be marked as slot settings, which stay with an environment during a swap; mark the wrong one and the new version points at the wrong database.
  • Slow migration: see the next section.
  • Data corruption: migrations touch every row, so they find years-old problems: duplicates, impossible dates, broken links between tables.
  • Failed smoke test / broken integration: identity and MFA, SMS gateways, payments, email and registry lookups are each run by someone else, with their own credentials, rate limits and IP allow-lists.
  • Security finding: a scan during the window finds something that must be fixed before reopening.
  • Organisational overhead: in a large organisation every new fix is a new change, with approval, vendor tickets and sign-off.

Why migrations get slow at production scale

A migration that ran in seconds on test data can take most of an hour on the real thing, and that’s before anything goes wrong.

Why a migration that took seconds in testing takes most of an hour in production Two bars: on a 50,000-row test copy a conversion takes about 25 seconds; on 5 million production rows the same script takes about 42 minutes, a hundred times longer. Then four hidden costs: rewriting rows copies the whole table, building indexes reads and sorts it all, writing the transaction log makes replicas fall behind, and waiting for locks makes every query queue. Test copy · 50,000 rows 25 s Production · 5,000,000 rows ≈ 42 min Same script, 100× the rows, 100× the time. Then four hidden costs: Rewrite rows copies the whole table 1 Build indexes reads and sorts it all 2 Write the log replicas fall behind 3 Wait for locks every query queues 4 Why a migration that took seconds in testing takes most of an hour in production Two bars: on a 50,000-row test copy a conversion takes about 25 seconds; on 5 million production rows the same script takes about 42 minutes, a hundred times longer. Then four hidden costs: rewriting rows copies the whole table, building indexes reads and sorts it all, writing the transaction log makes replicas fall behind, and waiting for locks makes every query queue. Test copy · 50,000 rows 25 s Production · 5,000,000 rows ≈ 42 min Plus four hidden costs: Rewrite rows copies the whole table 1 Build indexes reads and sorts it all 2 Write the log replicas fall behind 3 Wait for locks every query queues 4
Figure 3 — Same script, 100× the rows, 100× the time. Then four hidden costs on top.

The fourth cost, locks, is the one that surprises even experienced teams. A schema change can freeze the whole site before it has started:

How one ALTER TABLE can block every query A long-running report query holds a read lock on a table. An ALTER TABLE needs an exclusive lock, so it waits for the report to finish. Every new query on that table, such as login, submit and status checks, queues behind the waiting ALTER TABLE and is blocked, even though the ALTER has not started yet. The fix: set lock_timeout to a few seconds, so the ALTER gives up, the queue drains, and the migration retries later. Nightly report long SELECT, holds a lock ALTER TABLE waits for exclusive lock waits Every new query on the table Login query Submit query Status query queued Fix: SET lock_timeout = '3s' before the ALTER The ALTER gives up after 3 seconds, the queue drains, and the migration tool retries later. How one ALTER TABLE can block every query A long-running report query holds a read lock on a table. An ALTER TABLE needs an exclusive lock, so it waits for the report to finish. Every new query on that table, such as login, submit and status checks, queues behind the waiting ALTER TABLE and is blocked, even though the ALTER has not started yet. The fix: set lock_timeout to a few seconds, so the ALTER gives up, the queue drains, and the migration retries later. Nightly report long SELECT, holds a lock waits ALTER TABLE waits for exclusive lock queued behind it Login query Submit query Status query Fix: lock_timeout = '3s' The ALTER gives up after 3 s, the queue drains, retry later.
Figure 4 — The lock queue. The ALTER waits for the report; every new query waits for the ALTER.
Deep dive: rewrites, indexes, logs and locks (with SQL)

Rewrites. Some changes only update the database’s catalogue (its record of what tables look like). In PostgreSQL 11+, adding a column with a constant default is one of those and finishes almost at once. A volatile default like now(), or changing a column’s type, rewrites the whole table.

Indexes. A plain CREATE INDEX blocks writes while it reads and sorts the table. CREATE INDEX CONCURRENTLY doesn’t, but it’s slower, can’t run inside a transaction, and if it fails it leaves an invalid index to drop and rebuild.

Logs. Every change is written to the transaction log first (the WAL in PostgreSQL). Millions of updated rows mean gigabytes of log, full disks and replicas (read-only copies) falling behind.

Locks. Most ALTER TABLE statements need an ACCESS EXCLUSIVE lock: nobody can even read the table. Waiting locks form a queue, so everything behind the ALTER waits too. The fix is to give up fast and retry:

-- PostgreSQL. If the lock isn't free within 3 seconds, cancel: the queue drains
-- and the migration tool retries a little later.
SET lock_timeout = '3s';

-- PG 11+: no default, so no table rewrite. Holds the lock only for a moment.
ALTER TABLE applicant ADD COLUMN mobile_e164 varchar(16);

-- Doesn't block writes, but must run outside a transaction block.
-- If it fails, drop the INVALID index it leaves behind and run it again.
CREATE INDEX CONCURRENTLY idx_applicant_mobile ON applicant (mobile_e164);

Can you cancel half-way? In PostgreSQL, cancelling a transaction is near-instant; VACUUM cleans up later. In SQL Server without Accelerated Database Recovery (ADR), undoing can take about as long as the work already done. Azure SQL Database always has ADR on, so rollback of a cancelled transaction is near-instant. On every engine, batches that already committed stay committed.

Rolling back: code is easy, data is hard

Code has no memory: redeploy yesterday’s version and it behaves like yesterday. A database remembers everything written since the change.

Rolling back code versus rolling back data Left, rolling back code: minutes, redeploy the old version, data untouched, low risk. Right, rolling back data: hours to days, restore a backup, recent writes are lost, then reconcile by hand. Below, a timeline: last backup, then the failure, then restore complete. The span from the last backup to the failure is data lost, measured by the RPO. The span from the failure to restore complete is downtime, measured by the RTO. Rolling back code Time Minutes How Redeploy the old version Data Untouched Risk Low Rolling back data Time Hours to days How Restore a backup Cost Recent writes are lost Then Reconcile by hand What a restore costs: RPO and RTO Data lost (RPO) Site down (RTO) Last backup e.g. 02:00 Failure e.g. 20:00 Restore complete e.g. +18 hours Writes between the backup and the failure are gone unless you can replay them. Rolling back code versus rolling back data Left, rolling back code: minutes, redeploy the old version, data untouched, low risk. Right, rolling back data: hours to days, restore a backup, recent writes are lost, then reconcile by hand. Below, a timeline: last backup, then the failure, then restore complete. The span from the last backup to the failure is data lost, measured by the RPO. The span from the failure to restore complete is downtime, measured by the RTO. Rolling back code Time Minutes How Redeploy the old version Data Untouched Risk Low Rolling back data Time Hours to days How Restore a backup Cost Recent writes are lost Then Reconcile by hand What a restore costs Data lost (RPO) Site down (RTO) Last backup Failure Restore complete Writes after the backup are gone.
Figure 5 — Code vs. data rollback. RPO is how much data you can afford to lose; RTO is how long you can afford to be down.

A real example. In January 2017, GitLab accidentally deleted about 300 GB from its production database. Its regular backups turned out not to work. It restored from a copy an engineer had happened to take six hours earlier, and copying it back took around 18 hours on slow disks. About six hours of data were lost. GitLab published a very open postmortem, and its lesson is simple: a backup you haven’t restored isn’t a backup.

Deep dive: what an irreversible migration looks like

A realistic change: store phone numbers in one international format, so SMS codes send reliably.

-- One release, one step. Simple to write, impossible to undo safely.
ALTER TABLE applicant ADD COLUMN mobile_e164 varchar(16);

-- Rewrites every row in one transaction: slow on millions of rows.
-- normalise_phone() stands in for your own conversion function.
UPDATE applicant SET mobile_e164 = normalise_phone(phone);

-- The point of no return. The old app version still reads "phone" and breaks at once,
-- and a "down" migration can't rebuild the original values from the converted ones.
ALTER TABLE applicant DROP COLUMN phone;

If the new SMS integration then fails, you can’t redeploy the old version: its column is gone. Your only routes are fix forward or restore.

Restore or repair?

Restoring a backup over production also deletes the good work done since. Experienced teams often prefer to repair just the damaged rows.

Full restore versus surgical repair Before: the database holds good data, rows damaged by the migration, and new data written since. A full restore over production keeps only the good data from before the migration: the damaged rows and all new data are lost, so real work must be redone. A surgical repair restores a copy alongside production and copies back only the damaged rows: good data stays, damaged rows are repaired, and new data is kept. It is slower and needs care. Full restore restore over production Before Good data Damaged New data After Good data Lost Lost Real work lost users must redo it Surgical repair restore alongside, copy back Before Good data Damaged New data After Good data Repaired Kept Every good write kept slower, needs care Full restore versus surgical repair Before: the database holds good data, rows damaged by the migration, and new data written since. A full restore over production keeps only the good data from before the migration: the damaged rows and all new data are lost, so real work must be redone. A surgical repair restores a copy alongside production and copies back only the damaged rows: good data stays, damaged rows are repaired, and new data is kept. It is slower and needs care. Full restore restore over production Before Good data Damaged New data After Good data Lost Lost Real work lost users must redo it Surgical repair restore alongside, copy back Before Good data Damaged New data After Good data Repaired Kept Every good write kept slower, needs care
Figure 6 — Full restore vs. surgical repair. A repair keeps every good write, at the cost of more care.
Deep dive: point-in-time restore and a surgical repair in SQL

Most production databases combine full, differential and transaction-log backups. Replaying the log lets you restore to an exact moment, say one minute before the migration: a point-in-time restore (PITR). Three costs bite during an outage:

  1. Restore time grows with database size and log volume. Your real RTO is whatever that takes; rehearse it.
  2. It often creates a new database. Azure SQL Database restores a point in time into a new database (history kept 1–35 days, 7 by default), so every connection string must be repointed mid-incident.
  3. A full restore throws away everything written since that moment.

A surgical repair restores the backup alongside production and copies back only what was damaged. Using the irreversible example above:

-- PostgreSQL. restored_applicant was copied in from a point-in-time restore
-- taken just before the migration ran.
ALTER TABLE applicant ADD COLUMN phone varchar(32);   -- bring the old column back

UPDATE applicant AS live
SET    phone = r.phone
FROM   restored_applicant AS r
WHERE  r.id = live.id;

-- Applicants created after the migration have no "before" value: check by hand.
SELECT id FROM applicant WHERE phone IS NULL AND mobile_e164 IS NOT NULL;

What a restore can’t undo

Emails already sent, cards already charged, records already passed on. The fix is to record every side effect in the database first.

Side effects a restore can't undo, and the outbox pattern A database restore can't undo emails already sent, cards already charged, SMS codes delivered, or records sent to other systems. The fix: the web app saves the application and an outbox entry in one database transaction. An outbox worker then sends emails and SMS once, and calls the payment provider with an idempotency key so retries never double-charge. After a restore, compare the outbox and the provider's records, then fix the gaps. A restore can't undo these: Emails already sent Cards already charged SMS codes delivered Records sent to others The fix: record every side effect in the database first Web app one database transaction Application saved Outbox: send email Outbox worker sends, marks done Email / SMS sent once, traceable Payments retry-safe: no double charge After a restore: compare the outbox with the provider's records Payments with no matching application are refunded or re-created, not guessed. Side effects a restore can't undo, and the outbox pattern A database restore can't undo emails already sent, cards already charged, SMS codes delivered, or records sent to other systems. The fix: the web app saves the application and an outbox entry in one database transaction. An outbox worker then sends emails and SMS once, and calls the payment provider with an idempotency key so retries never double-charge. After a restore, compare the outbox and the provider's records, then fix the gaps. A restore can't undo these: Emails already sent Cards already charged SMS codes delivered Records sent to others The fix: the outbox Web app one database transaction Application saved Outbox: send email Outbox worker sends, marks done Email / SMS sent once Payments retry-safe key After a restore: compare outbox and provider records, then fix the gaps.
Figure 7 — The outbox pattern and idempotency keys make side effects traceable and retry-safe.
Deep dive: outbox, idempotency keys and reconciliation SQL

Outbox. Save the business change and the intention to send in one transaction; a worker does the sending.

BEGIN;
UPDATE application SET status = 'submitted' WHERE id = 42;
INSERT INTO outbox (event_id, kind, payload)
VALUES (gen_random_uuid(), 'submission_email', '{"application_id": 42}');
COMMIT;  -- both rows are saved, or neither: no email for a submission that didn't happen

Idempotency keys. A timeout doesn’t tell you whether a payment went through. Send a unique key with each payment request; providers such as Stripe return the original result for a repeated key (within 24 hours) instead of charging again.

Reconcile the provider’s records against your restored database:

-- provider_payments: the payment provider's export for the incident window.
-- Charged at the provider, but missing from our database: refund or re-create.
SELECT p.provider_payment_id, p.amount, p.created_at
FROM   provider_payments AS p
LEFT JOIN payment AS d ON d.provider_payment_id = p.provider_payment_id
WHERE  d.provider_payment_id IS NULL;

Sessions and caches are the subtle case: if the new version saved session data in a new format, the old one may not read it. Put the release version in cache keys and force a fresh login after a rollback.

Roll back or fix forward?

This one decision often sets whether an outage lasts hours or days. The key question is the second one: can the old code still run against the new database?

Rollback decision tree Start: the deployment fails. Question 1: did the schema change? If no, roll back the code (low risk). If yes, question 2: is the new schema backward compatible? If yes, roll back the code (low risk). If no, question 3: can we fix forward safely? If yes, fix forward (medium risk). If no, restore from backup and reconcile (high risk). Deployment fails Did the schema change? No Roll back code Low risk Yes Backward compatible? Yes Roll back code Low risk No Can we fix forward safely? Yes Fix forward Medium risk No Restore backup + reconcile High risk: data loss Rollback decision tree Start: the deployment fails. Question 1: did the schema change? If no, roll back the code (low risk). If yes, question 2: is the new schema backward compatible? If yes, roll back the code (low risk). If no, question 3: can we fix forward safely? If yes, fix forward (medium risk). If no, restore from backup and reconcile (high risk). Deployment fails Did the schema change? No Roll back code Low risk Yes Backward compatible? Yes Roll back code Low risk No Can we fix forward safely? Yes Fix forward Medium risk No Restore backup + reconcile High risk: data loss
Figure 8 — Rollback decision tree. Green "✓" low risk, amber "!" medium risk, red "✕" data loss possible.

How a short window can become days (hypothetical)

No single step takes three days. The time comes from waiting, on a slow migration, a vendor, a decision or an approval, and each wait is reasonable on its own. This timeline is an illustrative example, not based on any real organisation.

A hypothetical three-day outage (illustrative, not based on any real organisation) An illustrative timeline only, not the real events at any organisation. Four swimlanes over three days. Day 1: the deploy finishes but the MFA smoke test fails; the migration runs four times longer than planned; an emergency change is raised; users see the holding page all evening. Day 2: engineering waits on the SMS vendor; data checks find bad rows and the team must choose restore or fix forward; each fix needs approval; users wait for updates. Day 3: a fix-forward is deployed, bad rows are reconciled, security signs off, and the site reopens in phases. The planned window was 3 hours; the actual outage was about 3 days. Illustrative example — not based on any real organisation's timeline. Day 1 Day 2 Day 3 Planned: 3 hours Actual: about 3 days Engineering Deploy finishes MFA smoke test fails Waiting on SMS vendor Fix-forward deployed Data Migration runs 4× long Checks find bad rows Restore or fix forward? Bad rows reconciled Security & approvals Emergency change raised Each fix needs approval Security sign-off Users Holding page all evening Users wait for updates Phased reopen A hypothetical three-day outage (illustrative, not based on any real organisation) An illustrative timeline only, not the real events at any organisation. Four swimlanes over three days. Day 1: the deploy finishes but the MFA smoke test fails; the migration runs four times longer than planned; an emergency change is raised; users see the holding page all evening. Day 2: engineering waits on the SMS vendor; data checks find bad rows and the team must choose restore or fix forward; each fix needs approval; users wait for updates. Day 3: a fix-forward is deployed, bad rows are reconciled, security signs off, and the site reopens in phases. The planned window was 3 hours; the actual outage was about 3 days. Illustrative example not a real timeline Planned: 3 hours Actual: about 3 days Day 1 Engineering Deploy finishes MFA smoke test fails Data Migration runs 4× long Approvals Emergency change raised Users Holding page all evening Day 2 Engineering Waiting on SMS vendor Data Checks find bad rows Restore or fix forward? Approvals Each fix needs approval Users Users wait for updates Day 3 Engineering Fix-forward deployed Data Bad rows reconciled Approvals Security sign-off Users Phased reopen
Figure 9 — A hypothetical three-day outage across four swimlanes. Illustrative only.

How teams keep maintenance short

The goal isn’t zero failures. It’s keeping each failure small and “undo” cheap. Two patterns do most of the work.

Blue-green deployment: run two environments and switch traffic between them. Rollback is flipping the switch back. (On Azure App Service, deployment slots provide this built in. I don’t know how the permits portal itself is deployed.)

Big-bang deployment versus blue-green deployment Left, big-bang: users reach one environment, which goes offline while the new version is deployed, so they see a holding page; rollback means redeploying the old version and possibly restoring data. Right, blue-green: users go through a router. Blue runs the live version, green runs the new version, and both share one database, so the schema must work for both. Going live means switching the router from blue to green; rollback means flipping the switch back, in seconds. Big-bang one environment Users Holding page users wait App: v1 → v2 offline during deploy Rollback: redeploy v1, maybe restore data (hours) Blue-green two environments Users Router Blue: v1 live now Green: v2 tested, ready Shared database Rollback: flip the switch back seconds, if the schema suits both Big-bang deployment versus blue-green deployment Left, big-bang: users reach one environment, which goes offline while the new version is deployed, so they see a holding page; rollback means redeploying the old version and possibly restoring data. Right, blue-green: users go through a router. Blue runs the live version, green runs the new version, and both share one database, so the schema must work for both. Going live means switching the router from blue to green; rollback means flipping the switch back, in seconds. Big-bang one environment Users Holding page users wait App: v1 → v2 offline during deploy Rollback: redeploy v1, maybe restore data (hours) Blue-green two environments Users Router Blue: v1 live now Green: v2 tested, ready Shared database Rollback: flip the switch back seconds, if the schema suits both
Figure 10 — Big-bang vs. blue-green. Both environments share one database, so the schema must suit both versions.

Expand/contract migrations: because blue and green share a database, split each risky schema change into steps the old code can live with.

Expand and contract: one risky change split into three safe releases Release 1, expand: add the new mobile_e164 column as nullable, write both columns, backfill in batches; rollback is a code rollback. Release 2, migrate: read mobile_e164, still write both, check there are zero mismatches; rollback is still a code rollback. Release 3, contract: take a fresh, tested backup, stop writing phone, then drop it; this is the only point of no return. Release 1 · Expand • Add mobile_e164 (nullable) • Write both columns • Backfill in batches phone mobile_e164 (new) Rollback: code only Release 2 · Migrate • Read mobile_e164 • Still write both • Check: 0 mismatches phone mobile_e164 Rollback: code only Release 3 · Contract • Fresh, tested backup • Stop writing phone • Drop phone phone mobile_e164 Point of no return Expand and contract: one risky change split into three safe releases Release 1, expand: add the new mobile_e164 column as nullable, write both columns, backfill in batches; rollback is a code rollback. Release 2, migrate: read mobile_e164, still write both, check there are zero mismatches; rollback is still a code rollback. Release 3, contract: take a fresh, tested backup, stop writing phone, then drop it; this is the only point of no return. Release 1 · Expand • Add mobile_e164 (nullable) • Write both columns • Backfill in batches phone mobile_e164 (new) Rollback: code only Release 2 · Migrate • Read mobile_e164 • Still write both • Check: 0 mismatches phone mobile_e164 Rollback: code only Release 3 · Contract • Fresh, tested backup • Stop writing phone • Drop phone phone mobile_e164 Point of no return
Figure 11 — Expand/contract. Until the last release, rollback is just a code rollback.
Deep dive: expand/contract in SQL 
-- Release 1: EXPAND. Add the new column, nullable, so old code is unaffected.
SET lock_timeout = '3s';
ALTER TABLE applicant ADD COLUMN mobile_e164 varchar(16);
-- The app now writes BOTH columns on every save.
-- Backfill old rows in small id ranges, outside peak hours. Repeat with the next range.
-- "AND mobile_e164 IS NULL" makes it safe to stop and re-run.
UPDATE applicant
SET mobile_e164 = normalise_phone(phone)
WHERE id BETWEEN 1 AND 5000
  AND mobile_e164 IS NULL;

-- Release 2: MIGRATE. The app READS mobile_e164 and still writes both.
-- Before moving on, prove the new column is complete and correct. Must return 0.
SELECT count(*) FROM applicant
WHERE  mobile_e164 IS DISTINCT FROM normalise_phone(phone);

-- Release 3: CONTRACT. Weeks later, once nothing reads "phone",
-- after a fresh, tested backup:
ALTER TABLE applicant DROP COLUMN phone;

Four more habits round it out:

  • Feature flags: switch a new code path off without deploying.
  • Rehearse the undo: run the down migration and a real restore on production-sized data, and time it. That’s your honest RTO.
  • Watch user journeys, not servers: can someone log in, get an MFA code, submit and pay?
  • Regular status updates: what’s affected, what still works, and when the next update is due.
Deep dive: a feature flag and a deep health check (Python) 
import zlib

# Percentage of users who get each new feature. Set it to 0 to "roll back" with no deployment.
FLAGS = {"new_mfa_provider": 5}

def enabled(flag: str, user_id: str) -> bool:
    # A stable hash, so each user stays in the same group on every request.
    bucket = zlib.crc32(f"{flag}:{user_id}".encode()) % 100
    return bucket < FLAGS.get(flag, 0)

A health check that tests each dependency a user journey needs:

import time

import psycopg
import requests
from flask import Flask, jsonify

app = Flask(__name__)
DB_URL = "postgresql://health_ro@db/permits"         # a read-only health-check user
SMS_STATUS = "https://sms-provider.example/status"   # your providers' real status endpoints
PAYMENTS_STATUS = "https://payments.example/health"


def database():
    with psycopg.connect(DB_URL, connect_timeout=2) as conn:
        conn.execute("SELECT 1")


def sms_mfa():
    requests.get(SMS_STATUS, timeout=2).raise_for_status()


def payments():
    requests.get(PAYMENTS_STATUS, timeout=2).raise_for_status()


def run(check):
    start = time.monotonic()
    try:
        check()
        status = "ok"
    except Exception as e:  # report the failure type, not the details: this endpoint may be public
        status = f"fail ({type(e).__name__})"
    return {"status": status, "ms": round((time.monotonic() - start) * 1000)}


@app.get("/health/deep")
def deep_health():
    checks = {"database": run(database), "sms_mfa": run(sms_mfa), "payments": run(payments)}
    healthy = all(c["status"] == "ok" for c in checks.values())
    return jsonify(status="ok" if healthy else "degraded", checks=checks), 200 if healthy else 503

Don’t point the load balancer’s liveness check (the probe that decides whether a server gets traffic) at this endpoint. A one-minute SMS blip should raise an alert, not pull every server out of service.

Why it matters for public services

Behind every maintenance page are people with start dates, renewals and appointments, and an engineering team usually working flat out to bring the service back safely. Downtime can’t always be avoided. What engineering can do is keep each change small, keep the undo cheap, and make an honest estimate easy to give.

I’ve spent about ten years building ETL pipelines (jobs that move and reshape data between systems) and moving production databases, including migrating SQL Server to Azure SQL. The hardest part was never the code. It was the data.

Sources

Update log

  • 27 October 2026: First published. The portal had been showing its “essential maintenance” page for about three days. The reason has not been shared publicly.

Frequently asked questions

Why is rolling back a database harder than rolling back code?

Code has no memory: you redeploy the old version and it behaves as before. A database keeps everything written since the change. Undoing it usually means restoring a backup, which loses the writes made after it, so someone has to reconcile them.

What are RPO and RTO?

RPO (recovery point objective) is how much data you can afford to lose, measured as time since the last good backup. RTO (recovery time objective) is how long you can afford to be down while you restore.

Why can one schema change freeze a whole website?

Most ALTER TABLE statements need an exclusive lock. If a long query is reading the table, the ALTER waits, and every new query queues behind it. Setting lock_timeout makes the ALTER give up after a few seconds so it can retry later.

What is an expand/contract migration?

A way to change a database schema in small, reversible steps across several releases: add the new structure, move the application and data over while both work, and only remove the old structure once nothing uses it.