/migration-release-check — what the migrations of a release can break on rollout

A report, not a gate. For every migration in the range it answers four questions and says what would make the answer "safe". Nothing is changed: no code, no migrations, no database. Prod numbers come from SQL the skill writes for the user — nobody on the team has prod SQL access from here.

Background and the incident behind it: tmp/s/2026-10-15-expand-contract-migrations.md (if it is not in your slot, the facts below are enough), issues #5313, #5327, #5331, #5350.

Why each check exists

Deploy facts the checks rely on — verify them on every run (step 5), they drift:

• Migrations run in a Job actual-chat-migration-<version> (Dockerfile target migrations-app, Flux HelmRelease actual-chat-migration, backoffLimit: 1). The app pods start with it, but their init container actual-chat-migration-wait (k8s-wait-for job actual-chat-migration-, NAME_SUFFIX=<version>) holds them until the Job completes — so a Job that reports success lets them in whatever happened inside (dev 2026-10-14: the Chat migration failed, the entrypoint went on, the last bundle's exit code made the Job Complete). With SKIP_WAIT_NOT_FOUND=true the wait is skipped if the Job does not exist yet, and Flux does not order the HelmRelease before the Deployment.
• App pods also migrate on start (DbInitializer.MigrateDb), serialized by EF's migration lock: whatever the Job left pending, or all of it when the wait was skipped, runs on the app path.
• A failed Job keeps the new pods in Init and the rollout stops. With maxUnavailable: 1 one old pod is already gone by then, so the app runs at reduced capacity until someone steps in.
• Each migration is its own transaction (EF Core 10+; EF 9 ran all pending migrations in one, EF 10 reverted that — "Breaking changes in EF Core 9", dotnet/efcore#34439; checked on EF 11 rc.1 on 2026-10-15: one Began/Committed pair per migration). Within a migration, DropColumn, AlterColumn, CreateIndex, DropIndex hold ACCESS EXCLUSIVE on the table until it commits: every read and write of that table waits, and queued waiters can exhaust the DB's pool. Locks of consecutive migrations do not add up — queries get through between them. A failed migration rolls back alone: the ones before it stay applied, so the database is left between two releases.
• Command timeouts: 30 s on the app path (DbInitializer.CommandTimeout), 300 s in the migration pod (Command Timeout=300 in entrypoint.sh).
• The rollout mixes versions. Old code meets the new schema from the moment the migration commits until the last old pod is gone (dev 2026-10-15: ~2 min, 324 column ... does not exist). New code meets the old schema if the migration failed and the app started anyway (dev 2026-10-14: 3 h of null value in column "duration"). Since #5331 a failed migration fails the Job (set -e) and stops an app pod (exit code 2) — step 5 checks that this still holds.
• EF selects every mapped column in every query, so a column the old model maps breaks all reads of that entity once it is dropped — not just the code that used the property.

The safe shape is expand/contract (parallel change): release N only adds or relaxes (nullable / default) and stops using the old thing; release N+1 drops it. With migrations generated from the model (CLAUDE.md → EF Core migrations), that means the property stays in the model, nullable, for one release before it is removed. Not a team rule — the report says where a migration departs from it and what that costs.

Arguments

• none — the range is merge-base(latest origin/release/v*, origin/dev)..origin/dev, i.e. what the next release will carry.
• #5339 / a PR number — the migrations that PR adds (gh pr diff <n> --name-only).
• <from>..<to> — an explicit git range.

Steps

1. Range and the previous release

git fetch -q origin
REL=$(git branch -r --list 'origin/release/v*' --sort=-version:refname | head -1 | tr -d ' ')
BASE=$(git merge-base "$REL" origin/dev)

$REL is the code that will still be running on the old pods during the rollout. For a PR, the old code is origin/dev at the PR's base, but say in the report that prod runs $REL, which may be older still.

2. The migrations

git diff --name-only --diff-filter=A "$BASE" origin/dev -- 'src/dotnet/*/Migrations/2*.cs' \
  | grep -v '\.Designer\.cs$'

A migration that was modified rather than added (--diff-filter=M) is a finding of its own: if it already ran anywhere — dev applies dev on every merge, prod applies release branches — the edit never runs there. Check with git branch -r --contains <the commit that added it> | grep release.

For each file, list the operations of Up():

git show origin/dev:<file> | awk '/override void Up/,/override void Down/' \
  | grep -oE 'migrationBuilder\.[A-Za-z]+' | sort | uniq -c

Then read Up() itself for every migration that has anything beyond CreateTable, nullable AddColumn or CreateIndex on a new table.

Look across the migrations too, not only inside each: an index or column created by one migration and dropped by a later one in the same range is built under a lock on prod for nothing (v2.22: Add_ChatCleanup builds ix_chat_entries_chat_id_kind_is_removed_removed_at on chat_entries, RemoveUnusedChatEntryColumns drops it). Dev already ran both, so the fix is usually just to count the extra lock time — but say it.

Candidates — a name that appears in Up() of more than one migration of the same service:

for f in <the migrations>; do
  git show origin/dev:$f | awk '/override void Up/,/override void Down/' \
    | grep -oE '^\s*name: "[^"]+"' | grep -oE '"[^"]+"' | sort -u \
    | sed "s|^|$(echo $f | cut -d/ -f3) |; s|\$| $(basename $f .cs)|"
done | awk '{k=$1" "$2; n[k]++; m[k]=m[k]" "$3} END{for(k in n) if(n[k]>1) print k":"m[k]}'

Expect noise: Users.Service.Migration holds two contexts (users, oauth), so every framework column shows up twice there, and the same column name can live in two tables (pace_data). Open each candidate and keep only a real create-then-drop of the same object.

3. Classify every operation

Operation	Old code on new schema	New code on old schema	Lock
CreateTable, nullable AddColumn	safe	safe (new code must tolerate the table/column missing only if the migration can fail — see #5331)	short
AddColumn NOT NULL with default	safe	new code writing it fails until migrated	rewrite on old PG, metadata-only on PG ≥ 11
AddColumn NOT NULL without default	old inserts fail	—	—
DropColumn / DropTable	every query of the entity fails unless the old model no longer maps it	safe if the new code has no property; inserts fail if the column is still NOT NULL in the old schema	ACCESS EXCLUSIVE, fast itself
RenameColumn / RenameTable	fails	fails	—
AlterColumn → nullable	safe	safe	short
AlterColumn → NOT NULL, type change	old writes of NULL / old type fail	—	may rewrite the table
CreateIndex on an existing table	safe	safe	holds the lock while the index builds; unique: true fails on duplicates
DropIndex	safe	safe	short; check the queries that used it
DropPrimaryKey / AddPrimaryKey	safe if columns unchanged	—	builds an index under the lock; fails on duplicates
Sql(...)	read it	read it	DELETE / UPDATE on a big table under the same transaction

How to settle the two old-code questions instead of guessing:

• Does the old model map the column? Look at the model snapshot on $REL: git show "$REL":src/dotnet/<Svc>.Service.Migration/Migrations/<Ctx>ModelSnapshot.cs | grep -n '"<column>"'. Mapped → old pods break on every query of that entity after the drop.
• Is it NOT NULL in the old schema? In the same snapshot, the property's .IsRequired() or a non-nullable CLR type. NOT NULL + the new code no longer writing it = the 2026-10-14 incident shape.

Expand/contract verdict per dropped column: follows when the snapshot on $REL already has it nullable and git grep -n '<Property>' "$REL" -- 'src/dotnet/' ':!*/Migrations/*' ':!*/Db/Db*.cs' finds no use; otherwise departs, and the report names the window it opens.

Framework-owned tables (_operations, _events, __EFMigrationsHistory) count too: old and new framework versions both write them on every command.

When the migration comes with a framework upgrade (Fusion, EF — the same migration name in every service, e.g. Fusion15_DeferredInvalidation), the schema is the smaller question. Ask first whether pods on the old and the new framework can run side by side at all (RPC protocol, operation log format, invalidation). If they cannot, no expand/contract makes the rollout safe — only a rollout that never mixes versions does. The report states this as an open question for whoever made the upgrade; it does not guess the answer.

4. Data and size

For every table touched by a lock-holding operation or a Sql(...):

• dev — run it yourself via mcp__voxt-robokitty-dev__run_sql_query (database: chat, users, media, contacts, invite, notification, flows, mlsearch, oauth — the ac_dev_ prefix is implied). If the dev schema already has the migration applied, say so: the numbers are post-migration.
• prod — write the queries for the user to run on ac_prod_<db>; never claim prod numbers you did not get back from them.

-- size
SELECT count(*), pg_size_pretty(pg_total_relation_size('<table>')) FROM <table>;
-- a new unique index: duplicates that would fail it (after any DELETE the migration does)
SELECT <cols>, count(*) FROM <table> [WHERE <rows the migration keeps>]
GROUP BY <cols> HAVING count(*) > 1 LIMIT 10;
-- a Sql DELETE/UPDATE: how many rows, and whether anything still references them
SELECT count(*) FROM <table> WHERE <its predicate>;

Lock estimate: dev time for the same migration, if dev applied it, scaled by the size ratio — and say it is an estimate. For a measurement on prod without a copy and without a lock: CREATE UNIQUE INDEX CONCURRENTLY tmp_measure ... ; then DROP INDEX CONCURRENTLY tmp_measure; (autocommit, outside a transaction). Over 30 s → the app path can time out and roll back.

Dev migration timing, when dev applied it:

timeout 120 gcloud logging read 'resource.labels.container_name="actual-chat-migration" AND textPayload:"<MigrationName>"' \
  --project=actual-chat-app-dev --freshness=30d --format='value(timestamp,resource.labels.pod_name,textPayload)' --order=asc | head

Keep the container filter: without it the query scans every pod's log for 30 days and does not finish in two minutes. Several names can go into one query with OR.

A migration that never shows up here was applied by an app pod, not by the migration pod — that is the case for any database whose bundle the image does not build (step 5). On prod it then runs under the app's 30 s command timeout; say so next to that migration.

That gives the start. For the end, read the same pod's log from that timestamp (resource.labels.pod_name="<pod>" AND timestamp>="<start>", add resource.labels.pod_name to --format) — its next Done. line. Several Applying lines for one migration mean earlier attempts failed and rolled back.

5. Deploy guards

Report each as OK / not OK; these decide whether a slow or failed migration stays a stopped deploy or becomes an outage.

# an initializer failure must reach Program (#5331): WorkerBase.Run() swallows exceptions
grep -nE ':\s*WorkerBase|WorkerBase\[\]' src/dotnet/App.Server/AppHost.cs src/dotnet/App.Server/Initializers/*.cs  # expect nothing
grep -n 'set -e' Dockerfile                                                   # entrypoint stops on the first failure
grep -n 'CommandTimeout' src/dotnet/Db/DbInitializer.cs
# every entrypoint call has a bundle and vice versa; any output line is a mismatch
comm -3 \
  <(grep -oE 'bundle (--context [A-Za-z]+ )?--output \./artifacts/[A-Za-z.]+' Dockerfile | sed 's|.*artifacts/|./|' | sort) \
  <(grep -oE '^\./[A-Za-z.]+\.Migration\.exe' Dockerfile | sort)

The wait for the Job lives in the cluster, not in this repo. Read it from the dev cluster when kubectl is set up (read-only; prod is not reachable from here, so say the prod setup is assumed to match):

kubectl get deploy actual-chat-app -o jsonpath='{.spec.strategy}{"\n"}{range .spec.template.spec.initContainers[*]}{.name} {.args} {.env}{"\n"}{end}' \
  | grep -E 'rollingUpdate|migration-wait'

Expect actual-chat-migration-wait with job actual-chat-migration-; report SKIP_WAIT_NOT_FOUND and maxUnavailable as they are.

6. Report

One table, then the prod SQL, then the guards:

Range    release/v2.21 (merge-base 4f3c…) .. origin/dev 380cf9fd4c — 24 migrations, 3 need attention

| Migration | Operation | Old code / new schema | New code / old schema | Lock | Verdict |
|---|---|---|---|---|---|
| Chat 20261014091148_RemoveUnusedChatEntryColumns | DropColumn kind, duration, audio_entry_id… | every chat_entries query fails (mapped on v2.21) | inserts fail: duration NOT NULL | 9 DropIndex + 8 CreateIndex + DELETE, one tx | departs |

Prod SQL (run on ac_prod_chat): …
Guards   #5331 OPEN · entrypoint without set -e · Flows.Service.Migration.exe called, not bundled · app timeout 30 s

Then, per departing migration, the cheapest way to make it safe — usually "keep the property nullable for one release, drop it in the next", or "do the DELETE ahead of the release in batches". Say what it costs (a release cycle, splitting an already merged PR, re-aligning dev). The team decides.

Common mistakes

Mistake	Fix
Judging a drop by git grep for the property alone	EF selects every mapped column; check the $REL snapshot
Quoting dev numbers as prod	Prod comes only from SQL the user ran
Treating a migration dev already applied as freely editable	Dev will never rerun it; say what dev needs
Skipping framework tables	_operations / _events are written by every command on every pod
Calling the report a verdict to block the release	It lists risks and costs; the team decides