Kortix Rollback

The inverse of kortix-release. A release moves prod FORWARD onto new code; a rollback re-points prod at an OLDER already-released vX.Y.Z, reusing that release's prebuilt artifacts (zero rebuild). Use it when a shipped version is breaking and the move is "go back now, fix forward later." The next forward release still comes from staging, not directly from dev/main.

A rollback is temporary, never sticky. It only flips image tags + re-promotes an old frontend build; it never touches VERSION or the trunk. The next promote supersedes it and returns prod to latest (see §6). So rolling back never "wedges" the release pipeline.

1. The one command

gh workflow run rollback-prod.yml --repo kortix-ai/suna --ref main \
  -f version=vX.Y.Z -f reason="<incident summary>" -f confirm="ROLLBACK PROD"

It rolls frontend AND backend and handles each surface independently:

• Frontend → promoted back to vX.Y.Z immediately (Vercel API, no rebuild).
• Backend (api, gateway) → re-tags the old :vX.Y.Z images as :latest and triggers an ECS Fargate rolling deploy via ecs-deploy.sh.

2. Per-surface mechanics

Backend (API + gateway) runs Docker images on ECS Fargate. Rollback = re-tag the old, immutable, prebuilt :vX.Y.Z image as :latest and trigger an ECS rolling update. The ECS deploy script (infra/scripts/ecs-deploy.sh) registers a new task-def revision pointing at the old image and waits for services-stable. Clean and exact — the image is frozen.

Frontend is built by Vercel from the prod branch SOURCE. Vercel is git-connected: every push to prod rebuilds the frontend from whatever apps/web source is on the branch at that moment and makes it live at kortix.com. There is no "image tag" to point at. A FE rollback is instead a promote: "take that old v0.9.68 build and make it the live one again." It does NOT change the branch source — the branch still holds the newer FE code.

3. The frontend "clobber" — and the skip that prevents it

Because any push to prod makes Vercel rebuild the FE from branch source, the backend rollback merge (which only changes infra/ YAML) would still drag the frontend forward again:

1. FE promoted to old build       → kortix.com = vX.Y.Z FE   ✅
2. merge the backend rollback PR   → push to prod
3. Vercel rebuilds FE from branch source (still the NEW code) → kortix.com = new FE ❌ CLOBBERED

The fix (shipped): a Vercel Ignored Build Step — apps/web/scripts/vercel-ignore.sh, wired via apps/web/vercel.json ignoreCommand. On every prod push it asks one question:

Did this push change anything outside infra/?

• No (only infra/ → it's a backend rollback) → skip the FE rebuild.
• Yes (apps/web, packages, lockfile… → a real release/promote) → build.

In one line: don't rebuild the FE for backend-only pushes. It defaults to BUILD on any uncertainty (no parent commit, empty diff, error) so it can never silently drop a real FE deploy. Verified at real commits: an infra-only rollback merge → SKIP; a promote merge (touches apps/web) → BUILD.

Transitional note: vercel.json only takes effect on prod once a push carries it there. The next promote does that automatically. Until then, a rollback's FE may still get clobbered by the backend merge — re-run the workflow with -f surfaces=frontend to re-promote the FE (the workflow prints this).

4. Safety check BEFORE you roll — DB / state drift is the real blocker

The image-reuse + Vercel-promote mechanics work for any released version. What actually makes a rollback unsafe is forward state the old code can't handle — overwhelmingly the database.

Migrations in this repo are forward-only and additive by convention (ADD COLUMN/VALUE, CREATE … IF NOT EXISTS). Against additive migrations old code is safe: the live schema is a strict superset of what the old version needs. Never reverse a migration — that's what breaks things, not rolling code back.

The danger is a destructive migration between the target and current — a DROP COLUMN, RENAME, ALTER … TYPE, or a new NOT NULL. Old code hitting that breaks. The workflow CANNOT detect this; a human must eyeball it. One command:

git diff vX.Y.Z..origin/prod -- packages/db/migrations

• All additive (ADD …, CREATE … IF NOT EXISTS) → DB-safe to roll there.
• Any DROP / RENAME / ALTER … TYPE / new NOT NULL → stop and think.

Lesser, non-DB couplings (rare long tail): forward-written persisted state old code can't parse (queue/cache formats), or external contract changes (Stripe API version, webhook shapes).

Kortix Apps: static hosting and retention do not roll back

Two Apps changes ship in one release and survive a code rollback:

1. Static Apps go offline on an older api. A static App is served by the api from storage (apps/api/src/apps/static-site.ts) and has no app_runtimes row. An api that predates that file requires a runtime, so every static App answers with a status page until the next promote. rollback-prod.yml refuses such an api target unless static_apps_ack is exactly STATIC APPS OFFLINE. Rolling only gateway,frontend is not affected.

2. Retired deployments stay retired. The first maintenance pass after the release retires every ready deployment beyond the active one and the newest KORTIX_APPS_RETAINED_DEPLOYMENTS (default 5), 50 Apps per pass. It tears down their runtimes and images; reclaimAppArtifacts deletes their archives 24 h later. A rollback cannot bring them back. Before the first release, count what it will retire:

   select app_id, count(*) - 6 as retired
   from kortix.app_deployments where status = 'ready'
   group by 1 having count(*) > 6;
   

   To defer it, set KORTIX_APPS_RETAINED_DEPLOYMENTS high on prod for that release and lower it on the next one.

5. Per-surface availability (the engine handles this)

You can't roll a surface BELOW where it existed. The gateway only entered prod at ~v0.9.72 (image exists from 0.9.69), so there is no kortix-gateway:0.9.68. The workflow checks each image (Docker Hub tags API) and skips any surface with no :vX.Y.Z image rather than wedging ECS on ImagePullBackOff.

6. Resuming — a later promote returns prod to LATEST automatically

You do NOT manually undo a rollback. To resume, get the fix onto staging, wait for staging deploy + QA, then Promote to Production (see kortix-release). The release PR carries the staging candidate into prod, Vercel builds the latest FE, and ECS rolls the latest API/gateway images. The rollback evaporates. No manual Vercel step to resume.

7. Gotchas

• The deploy-prod run on the backend merge goes partly red (its version-watch waits for the current VERSION while pods report the rolled version; release- create re-hits the existing release). Cosmetic — ECS does the real roll independently. Judge success by api.kortix.com/v1/health.
• Pushing vercel.json itself to prod rebuilds the FE (it's under apps/web, so the skip says "build") — which clobbers a FE rollback. Let it ride to prod on a normal promote instead of a standalone push.
• Verify after: api.kortix.com/v1/health (version), ECS service health, and the live Vercel production deployment (/v9/projects/{id} → targets.production).