High Availability skill

The single entry point for any HA work in Yuzu. Bundles three things:

1. The model — the two independent axes and the safe-to-scale gate (get this wrong and a second replica double-dispatches destructive commands).
2. The delivery matrix — every workstream WS-0…WS-14: what it delivers, current state, dependencies, whether it gates a second replica, routed reviewers, and status.
3. Gap/priority + workflow — what to build next and the standard procedure for landing a workstream slice.

This skill does NOT replace ADR-2002 or the routed docs. It tells you which to load, in which order, and what to confirm before you start cutting code. On any conflict, ADR-2002 governs the model and docs/ha-delivery-matrix.md governs delivery — this file is a pointer and loses to both.

---

Usage

/ha                       # default: print the matrix + the safe-to-scale gate, ask which WS to work on
/ha audit                 # snapshot of HA readiness — which WS are done, whether a 2nd replica is safe yet
/ha plan <WS>             # plan-only walk-through for a single workstream (e.g. WS-3, WS-7)
/ha implement <WS>        # full workflow: plan → governance → implement → test → docs for one WS slice

If invoked without a subcommand, default to printing **the matrix (Section 2)

• the safe-to-scale gate (Section 1)** and asking which workstream to work on.

---

1. The model — read before sequencing anything

Authoritative reference: docs/adr/2002-high-availability-architecture.md (the decisions) and docs/ha-delivery-matrix.md (the delivery spine). Read them before this skill claims anything is "done."

1a. Two independent axes

• Server-tier HA — N presentation + core replicas (WS-0…WS-6, WS-8, WS-10, WS-11, WS-13).
• Storage HA — HA Postgres (WS-7). A single Postgres is a valid, supported deployment; server-tier HA does not require it.

"HA" as a customer claim = both axes + WS-9 (validation) + WS-11/WS-12 (operability). Do not conflate the axes: WS-7 shipping does not make the server tier highly available, and a second replica is safe against a single Postgres.

1b. ⚠️ The safe-to-scale gate (the load-bearing constraint)

Do not equate "Phase A done" with "safe to run a second replica." The three-model adversarial review's strongest, unanimous finding:

A second server replica may be enabled only when WS-0, WS-1, WS-2(2a outbox), WS-3, WS-4, WS-5, WS-6, WS-10 are done, AND WS-8's per-tier /readyz exists.

Miss any one and the failure is not cosmetic:

• No WS-3 (fenced leader) → both replicas run the singleton workers → double-dispatch of destructive commands (software-deployment, quarantine, policy-remediation).
• No WS-4/WS-5 → commands can't reach agents / scope evaluation drops agents on the replica lacking the local stream → mis-targeting.
• No WS-6 → CRL numbering / enrollment diverge across replicas.

WS-7 (HA Postgres) is NOT in this gate set — it gates the storage axis (no SPOF / RPO=0), not the second server replica. The "Gates 2nd replica?" column in the matrix encodes this per row.

1c. Split ↔ HA — RATIFIED: decoupled (the split is NOT a prerequisite)

Is the presentation/core split (ADR-0031) a prerequisite for active-active? No — decoupled. Ratified by the ADR-1005 owner (Dave Rae), 2026-09-07. Monolith active-active is the target; the split is a parallel programme, not a gate for a 2nd replica. The one in-process blocker (direct agent Subscribe stream) is solved by gateway-fronting (WS-4), not the split.

• Kimi's dissent (recorded, rebutted): Kimi called the split a hard safety prerequisite. Every safety property it could mean — singleton double-dispatch (WS-3), in-memory sessions (WS-1a, done), stream routing (WS-4) — is closed in the monolith. What the split buys is presentation scale-out + separate readiness signals: capacity, not safety.
• One-way dependency: the split consumes HA (inherits WS-1 sessions, rides WS-2a's durable event_outbox for its event spine, takes MCP replay durability from WS-2b) — never the reverse. WS-2b spine and WS-8 tier-split /readyz are no-ops on the monolith and land with the split. Delivery lives in its own control plane: docs/presentation-core-split-delivery-matrix.md (the /split skill).

1d. Non-obvious model decisions (survived two+ adversarial reviews)

• Coordination = a narrow seam, Postgres-backed by default — fenced advisory-lock leader + LISTEN/NOTIFY-as-hint over a durable outbox + reconnect cursor-poll; pluggable for SaaS; ship one impl. No new SPOF (Postgres already bounds availability, ADR-0007 fail-closed).
• Effectively-once, NOT exactly-once — transactional outbox + claim-before-side-effect + receiver-side idempotency (agent/gateway dedup on command_id). A dup must replay the terminal outcome, never return a bare REJECTED (that is the "effect-once, result-maybe-lost" bug WS-0 closes).
• Intra-cluster gateway routing is net-new work — OTP pg is broadcast-only; per-agent lookup is node-local ETS. "pg finds the exact node" is false; per-agent distributed routing must be built (WS-4).
• The routing directory must be fenced — (agent_id, connection_epoch, cluster_id, session_id, lease_until) with CAS register/deregister, else a stale replay overwrites a newer re-home.
• Sessions → durable Postgres (DB wall-clock) reverses the as-built in-memory-monotonic design and owns the regression risk on JIT-elevation / MFA windows.
• Durability is selectable; default 3-node quorum (1 primary + 2 sync standbys, ANY 1) on distinct failure domains — unconditional 2-node RPO=0 was a footgun (single standby loss stalls all writes).
• Model: hybrid, general product capability (on-prem-first + SaaS-later), NOT one customer. Gateway-fronting is mandatory in HA (direct-connect kept for single-node). Don't assume HA Postgres.

---

2. The delivery matrix — current state, reviewers, status

Source of truth: docs/ha-delivery-matrix.md. Read it before this skill claims a status. WS-0…WS-10 come from ADR-2002 §Decomposition; WS-11…WS-14 are delivery/ops workstreams the three-model review surfaced as missing.

WS-6 re-stamped 2026-09-23 (PR #4833): slice 6.1 (cross-replica CRL publication, #4126) in review; 6.1b / 6.2 / 6.3 planned; the operator decisions (shared CA key custody, not SecretCodec; a one-time .cfg enrollment import) are recorded in ADR-2002 §8 "Update (2026-09-23)".

Verified 2026-09-21 (against origin/dev): DONE — WS-0 (#3662), WS-1 (1a+1b+1c), WS-2a (2a-1 + 2a-2 #3924), WS-3 (3.1–3.4 — #4011/#4134/#4169/#4194), WS-4 4.1 + 4.2a + 4.2b Tasks A–D (#4245/#4299/#4344/#4355, merged) + #4324 per-home stream-generation fence (PR #4492, merge commit c37306113, 2026-09-17T22:05:58Z, MERGED), WS-7 (#3627), WS-10 10.1/10.2. 4.3a DONE (intra-cluster pg-group agent→node lookup + the fanout_terminal cross-node fix + remote_dispatched telemetry — code-complete, empirically verified, no longer inert since #4555 merged). #4555 MERGED (2026-09-19, 495cf24c4, PR #4604): gateway cluster formation (yuzu_gw_cluster_discovery always-on redial loop, DNS-A-record + static-override discovery, dynamic address-only node identity, cookie length floor) implemented, Fable-adjudicated, empirically verified against a real OTP peer — see docs/ha-delivery-matrix.md's WS-4 row for the full mechanism. WS-4 4.4 DONE (2026-09-20, ADR-2002 §7c): gateway_node convergence + the #4246 #6 replay-session writeback, closed after a Fable pre-implementation review rejected the originally-proposed design and surfaced that ProxyRegister was unconditionally wiping placement on every replay-adopted registration (a live, single-replica bug, not just a multi-replica gap) — see the blockquote below and ADR-2002 §7c for the full mechanism. WS-4 rest of 4.3 DONE (2026-09-21, ADR-2002 §7d): cross-cluster gateway fan-out — AgentSession/GatewayPendingCmd now carry cluster_id end-to-end from the common send_to/send_to_all path (not just the 4.2b directory fallback), a new eager GatewayMgmtStubPool dials the OWNING cluster per command instead of the single legacy stub, --gateway-cluster-addr config, a response-agent forgery guard (closes ONE forgery shape — a cluster answering as a different agent — NOT a rogue gateway claiming an agent's own identity via ProxyRegister, which needs no per-agent secret; multi-cluster mode does NOT yet provide trust-zone isolation between clusters for a given agent, tracked as #4669, pr-rev-caught pre-merge), and one small required Erlang fix (stream_responses/3 threads the real command_id so a gateway-side "not connected on this cluster" error is no longer invisible). Closes WS-4's dial-selection half — the durable re-drive-on-failure half of §7's design remains open (#4672), and #4669 above is a separate, not-yet-closed item (non-blocking today: zero production multi-cluster deployments). The remaining safe-to-scale gate items are WS-5, WS-6, WS-8-readyz. See the blockquote below and ADR-2002 §7d for the full mechanism, including the 5 findings a pre-implementation Fable review caught before any code was written (the original resolution rule would have broken every upgrade), and the pr-rev (FortitudeEtc/Codex+Kimi) round that caught the response-forgery overclaim plus two real code bugs (terminal-outcome double-counting; an oversized cluster_id silently routing to the default cluster instead of being rejected) before merge. Next gate items: WS-5, WS-6, WS-8-readyz. Also open: #4669, #4672.

Re-verified 2026-09-24 (against origin/dev + gh): #4672 CLOSED; #4669 still OPEN (mitigation 1, agent↔cluster affinity, merged; mitigation 2, peer-identity binding, open). WS-5 slice 1 MERGED (PR #4745, 2026-09-22) — decommission cascade, plugin-capability visibility, fleet gauge and its WS-9 scenario remain. WS-6 6.1 MERGED (PR #4833, 2026-09-23) — 6.1b / 6.2 / 6.3 and WS-9 scenario #4832 remain. WS-8 readyz DONE for the monolith (this change): runtime pg_reachable probe + --shutdown-drain-seconds drain grace + WS-9 scenarios scripts/ha/ha-readyz-scenarios.sh; tier-split readyz lands with the ADR-1005 split; the BYO-LB doc (P2) is open and not in the gate. Remaining gate items: the rest of WS-5, the rest of WS-6, and WS-13 for gateway-fronted fleets.

WS-4 4.2a update (2026-09-13, PR #4299 round-5 review): #4246 item #4 (same-session late-DISCONNECTED tombstoning a newer re-home) is RE-SCOPED, not closed by 4.2a. The tombstone closed only the late-CONNECTED resurrection direction (#5). #4's mechanism is real but has no producer under the shipped gateway (exactly one CONNECTED(S)/one DISCONNECTED(S) per session), so it is unreachable today; the per-home stream-generation fence (a NotifyStreamStatus protocol change) is a precondition (#4324) of the first slice that re-CONNECTs under a reused session id. Invariant: session_id ≡ exactly one gateway stream placement.

WS-4 4.2b update (2026-09-14): the dispatch-wiring task landed as Tasks C (reader, fallback-only) + D (route_unreadable consumer + alert rules) — see docs/ha-delivery-matrix.md's WS-4 row and ADR-2002 §7's 4.2b status paragraph for the full narrative. 4.2b's review RE-VERIFIED the once-per-session property (#4324) before Task C landed (governance.d/ha-ws4-42b-4324-reverification.md) — the property held, no regression found; the per-home fence itself remained deferred at that point to the first slice that re-CONNECTs under a reused session id (4.3/4.4/#4246 #6).

WS-4 #4324 update (2026-09-17, MERGED — PR #4492, c37306113): the per-home stream-generation fence is CLOSED end-to-end — task 1 (46f1e72b6, StreamStatusNotification.stream_home_id wire field + Erlang producer), task 2 (4b248b504, GatewayRouteStore's nullable stream_home_id column + asymmetric tombstone predicate, migration v3), task 3 (feabccea7, NotifyStreamStatus's DISCONNECTED branch resolving the fence once, before any teardown effect, against AgentRegistry::gateway_stream_home_id). Re-verified the once-per-session property a second time before landing (governance.d/ha-ws4-4324-stream-fence-reverification.md) — no regression. Pre-merge PR-review fix (HIGH): external reviewer FortitudeEtc (Codex+Kimi, empirical reproduction) found task 2's original predicate — stream_home_id = $3 OR (stream_home_id IS NULL AND $3 = '') — wrongly required the incoming value to also be empty before a stored-NULL row admitted it; a stored-NULL home can also mean "this session's own announce_connected hasn't run yet" (CONNECTED/DISCONNECTED are two independently spawn_monitor'd RPCs with no ordering guarantee), so an ORDINARY (non-re-home) connect/disconnect could have its DISCONNECTED arrive first, get misclassified stale_home, and let a delayed CONNECTED publish a route for an already-dead stream — reachable with only the FIRST CONNECTED/DISCONNECTED pair, no second one needed. Fixed to stream_home_id IS NULL OR stream_home_id = $3 (f7f12bd59, mirrored in the in-memory fence); a Gate 8 re-review caught a second stale copy of the old formula in a doc comment (5160e0eeb). Both landed pre-merge in the same PR. Deliberately NOT closed: the pre-CONNECT-race ordering gap (a stale DISCONNECTED landing before its matching live CONNECTED still tombstones the row — see gateway_route_store.hpp's FORWARD NOTE), left for 4.3; and the duplicate_connected desync tripwire, deferred to its own follow-up, #4464.

WS-4 4.3a update (2026-09-18): the intra-cluster half of 4.3 (agent→node routing WITHIN one Erlang gateway cluster, distinct from cross-cluster fan-out) is DONE — yuzu_gw_registry.erl's lookup/1 gains a per-agent pg group ({agent, AgentId}) cross-node fallback on a local ETS miss; yuzu_gw_agent.erl fixes a latent bug the design review found (fanout_terminal was routed to the LOCAL node's router, stranding a cross-node fanout for the full 300s timeout — now routed to {yuzu_gw_router, node(ReplyTo)}); yuzu_gw_router.erl adds remote_dispatched telemetry. Design per ADR-2002 §7's own mechanism list (a per-agent pg group), NOT the dead hash_ring_vnodes config (#4556) or a server-supplied hint (both rejected by design review). Verified: rebar3 compile/dialyzer clean, full eunit suite green, a new OTP-peer-based 2-node test suite (4 tests) run 40+ times with zero flakes after two governance-round fixes (a false-green local-preference test, and a pg-propagation-delay race in the fanout-completion test). Also updates gateway_route_store.hpp's #4324 FORWARD NOTE: the pre-CONNECT-race ordering gap left open above is now resolved as unreachable BY INVARIANT (agent-driven reconnect + the gateway's NOT_FOUND refusal on an unknown session — no producer of a same-session different-home CONNECTED exists or is planned), replacing the previously-stated register_fresh-ordered-epoch/CAS resolution with a standing design constraint + a server-side tripwire folding into #4464. COMPONENT-COMPLETE-AND-INERT as of this note; #4555 (gateway cluster formation) MERGED 2026-09-19 (495cf24c4, PR #4604 — see the Verified line above), no longer inert. Remaining WS-4 work as of that merge: the rest of 4.3 (cross-cluster gateway fan-out — today one gw_mgmt_stub_), plus 4.4 (gateway_node convergence + replay-session writeback) — see the next blockquote for 4.4's closure.

WS-4 4.4 update (2026-09-20, #4246 #6, ADR-2002 §7c): DONE. A Fable pre-implementation review REJECTED the originally-proposed two-mechanism design (a periodic directory-sourced read-repair — would publish a route with no wire_capabilities, which dispatch denies outright; a gateway-side adoption of a server-minted session — explicitly prohibited by gateway_route_store.hpp's FORWARD NOTE, and unworkable regardless since agent heartbeats carry the AGENT's own session id, which the gateway cannot rewrite) and surfaced the actual dominant root cause: on EVERY replay-adopted ProxyRegister — even the ordinary same-replica case, no core restart needed — register_agent unconditionally installed a brand-new AgentSession, wiping gateway_node/wire_capabilities/ stream_home_id regardless of whether the session was being adopted or freshly minted. Fixed: ProxyRegister now decides ADOPT-vs-REFUSE before calling register_agent — whenever GatewayRouteStore is configured, the STORE governs this decision (never the in-memory gateway_sessions_ map on its own — a round-2 review fix, F1, closed a real bug where trusting memory alone let a tombstoned-but-still-gateway_sessions_-known session go unrepaired after a >270s gateway-uplink partition, and separately let a stale local session overwrite a genuinely-live one on a different node): ADOPT (response always carries the presented session, never a fresh mint) on a live directory row (renew_leases), or the new guarded CAS GatewayRouteStore::reclaim_tombstoned_session (re-arms a TOMBSTONED or absent row — epoch 0 on a brand-new row, untouched on a re-armed tombstone, either way no higher than register_fresh's nextval can mint, so a genuine register_fresh still always wins); REFUSE (FAILED_PRECONDITION, nothing installed) when the row is LIVE under a DIFFERENT session. gateway_sessions_ alone decides ONLY when no store is configured. Convergence comes from the GATEWAY: on ADOPT, yuzu_gw_upstream.erl tells the still-live yuzu_gw_agent process (reannounce/2) to re-send its OWN CONNECTED (same session, same home) through the ordinary NotifyStreamStatus path — now backed by drop telemetry and drip backpressure against the MAX_NOTIFY_INFLIGHT cap (F2); on REFUSE, it feeds its circuit breaker a SUCCESS (not a failure — the server answered authoritatively) and calls disconnect/1 to force the agent-driven reconnect the FORWARD NOTE requires. Also fixed a latent, pre-existing crash in do_rpc's error-clause pattern (never matched what grpcbox_client:unary/5 actually returns; unreachable before 4.4 since this RPC never previously returned a real non-OK, non-transport grpc status). Round-2 Fable review found 3 more real bugs before push: F1 (BLOCKING) — the adopt decision let an in-memory gateway_sessions_ hit override the store, missing a >270s gateway-uplink-partition tombstone and a cross-node zombie-session overwrite; fixed by making the store govern unconditionally whenever configured, memory only when no store exists. F2 (BLOCKING) — the reannounce CONNECTED shared ?MAX_NOTIFY_INFLIGHT with everything else and dropped silently at capacity (I6a); fixed with drop telemetry + drip backpressure. F3 (BLOCKING) — yuzu_gw_heartbeat_buffer.erl had the SAME impossible do_rpc-shaped error clause; fixed identically. A full /governance pass (round 3) then found 3 MORE blocking issues round 2 itself introduced or missed: sec-H1 — round 2's reannounce fix converges placement only if its notify is actually delivered, and a dropped one left cluster_id permanently NULL while BatchHeartbeat's renew_leases kept extending the row's lease AND updated_at forever regardless — fixed by making renew_leases freeze BOTH lease_until AND updated_at for a cluster_id-NULL row (freezing lease_until alone, the round-3 fix as first shipped, does nothing for a register_fresh row whose lease_until is already NULL from creation — only the updated_at-keyed tombstone/ never-announced purge sweep can ever reach it; caught by a targeted post-build Fable review before push), so it reaches its own ordinary TTL+grace/purge age and gets deleted, giving the next replay another reclaim shot — not an instant self-heal, since convergence still needs that next replay or reconnect. NEW-1/NEW-2 — round 2's own claim that both drop reasons "now emit telemetry" was FALSE as shipped: the event was never wired into yuzu_gw_telemetry.erl's ?EVENTS/handle_event/ declare_metrics, so it was a pure no-op — two Gate-6 reviewers independently caught the same false ADR claim in the same wave; fixed by wiring it for real, with a new test file that fires the REAL event through the real handler (every other module's tests mock telemetry itself, which can't catch this class of gap). UP-4/COMP-6 — the decision's own renew_leases/reclaim_tombstoned_session calls were fail-OPEN on a degraded Postgres read, so a degraded-store window bypassed the entire stale/zombie-refusal mechanism outright; fixed to fail-CLOSED, mirroring register_fresh's own precedent for exactly this class of decision. Deliberately deferred as the one remaining follow-up, not WS-4-gate-blocking: the reap-vs-replay-drip race after a >270s outage at fleet scale — filed as #4627, with related hardening filed alongside it (#4628-#4632). The yuzu_gw_cluster gossip/adjacency gen_server #4555's design notes named as "remaining 4.4 scope" is judged OUT of WS-4 entirely (capacity/rebalancing, not reachability — the WS-4 gate exists because "commands can't reach agents").

⚠️ Standing instruction — update on close. Every PR that closes or materially changes the status of a workstream here MUST update its row and re-stamp the Verified line in the SAME PR (source of truth: docs/ha-delivery-matrix.md; this table mirrors it). A matrix from a stale checkout is worse than none — treat status drift as a review-blocking defect, exactly like a stale doc comment. (This skill was itself found 557 commits stale once; hence the mandate.)

Gates? = does the safe-to-scale gate depend on this row (distinct from Prio).

WS	Delivers	Depends on	Gates?	Routed reviewers	Prio	Status
WS-0	Durable agent-side command idempotency + terminal-outcome replay (dup replays stored result, not bare REJECTED)	—	Y	cpp-safety+cpp-expert	P0	done (PR #3662 merged)
WS-1	Server-plane state → Postgres: (1a) sessions DB-time; (1b) execution_tracker+command-correlation atomic counters; (1c) HA-critical store subset	migration ladder (serializes at the migration-version counter)	Y	authdb+security-guardian (1a); architect+sre+cpp-safety (1b/1c)	P0	done — 1a+1b+1c
WS-2	(2a) durable event outbox + NOTIFY fan-out [monolith-OK]; (2b) core→presentation event spine [defers to ADR-1005]; MCP session/replay durability	2a: WS-1(1b); 2b: ADR-1005 split	Y (2a)	architect+sre+security-guardian(MCP)+docs-writer	P1	2a done (2a-1 + 2a-2 #3924); 2b/MCP outstanding
WS-3	Coordination seam: fenced LeaderElector (monotonic epoch in claim txn) + leader/transactional-outbox/receiver-idempotency worker refactor incl. policy remediation	WS-0, WS-1, WS-2(2a)	Y	architect+cpp-safety+security-guardian	P1	3.1 done (#4011); 3.2 done (#4134); 3.3 done (#4169); 3.4 done (#4194)
WS-4	Gateway routing + multi-cluster: fenced agent→cluster directory, net-new distributed intra-cluster agent→node routing, gateway_node convergence	WS-1, WS-3, WS-0	Y	gateway-erlang+security-guardian+architect+cpp-safety	P1	4.1 + 4.2a + 4.2b Tasks A–D + #4324 per-home stream-generation fence + #4555 cluster formation + 4.3a intra-cluster lookup + 4.4 (gateway_node convergence + #4246 #6 writeback) all MERGED to origin/dev; rest of 4.3 (cross-cluster fan-out) / WS-5 cross-replica lookup remain — see docs/ha-delivery-matrix.md
WS-5	Shared agent presence / health / scope-eval population across core replicas	WS-4, WS-1, WS-3, WS-10	Y	security-guardian+architect+sre+docs-writer	P1	slice 1 MERGED (#4745); decommission cascade / capability visibility / fleet gauge / WS-9 scenario remain
WS-6	PKI/CA HA: shared CA key custody + node admission (NOT a SecretCodec blob — ADR-2002 §8 Update 2026-09-23, keeps ADR-0010 Decision 6), CaStore → PG (already done, ADR-0053), durable CRL numbering + publication state machine, KEK versioning/rollout/rollback, enrollment → PG (one-time .cfg import)	WS-3 (freshness pass); ca_store migration done	Y	security-guardian+cpp-safety+docs-writer (+authdb on 6.2)	P1	6.1 CRL publication (#4126, PR #4833); 6.1b / 6.2 / 6.3 planned — see docs/ha-delivery-matrix.md
WS-7	HA-PG delivery: Patroni+etcd+HAProxy Compose profile, selectable durability (3-node quorum default, distinct failure domains), operator-plane LB profile	— (storage axis; parallel)	N	release-deploy+build-ci+sre	P1	done (PR #3627 merged to dev)
WS-8	Per-tier health contract (/livez vs /readyz; presentation→operator LB, core→presentation routing). BYO-LB doc + LB/session semantics	conceptual on WS-1/WS-2; readyz before LB fronts replicas	Y (readyz)	docs-writer+release-deploy+sre	P0 readyz / P2 doc	readyz DONE (monolith, 2026-09-24): pg_reachable probe + drain grace; tier-split with the split; BYO-LB doc (P2) planned
WS-9	Failover test harness — continuous, incremental scenarios added as each feature lands (not a final gate): session survival, no double-dispatch, effectively-once, cursor-poll no-loss, re-home races, quorum-degrade, standby loss	scenarios track WS-0…WS-7 as they land	N	build-ci+release-deploy; scenarios by chaos-injector	P1 (continuous)	planned
WS-10	Background-job replica-safety classification — checked-in, CI-auditable table (job → fenced-leader-only / replica-safe / disabled-until-fixed); bring #2508 wall-clock passes to clock-guard	audit+disable need nothing; fenced-leader-only enforcement needs WS-3	Y	cpp-safety+sre+compliance-officer	P0	10.1/10.2 done (#4092); 10.3 enforcement wired by WS-3 3.2
WS-11	HA-state observability (NEW): leader identity/epoch, replica lag, quorum state, outbox backlog, failover duration, routing re-homes, split-brain alerts — Prometheus metrics + rules	WS-3, WS-7	N (ship with the 2nd replica)	sre+docs-writer (+ alert-rule gate)	P1	planned
WS-12	Single→HA cutover + DR (NEW): runbook to move an existing single deployment to HA-PG + gateway-front rollout; backup/PITR/WAL under HA-PG (replaces superseded disaster-recovery.md)	WS-7	N	release-deploy+sre+docs-writer	P1	planned
WS-13	Agent-side gateway failover rollout (NEW): agent endpoint discovery/failover + fleet rollout flipping direct-connect fleets to gateway-front	WS-4	Y (gateway-fronted fleets)	cross-platform+security-guardian	P1	planned
WS-14	Coordination-substrate security & capacity review (NEW): threat review of advisory locks / etcd / Patroni / HAProxy / fencing tokens; capacity + failure-domain + sync-quorum latency validation	WS-3, WS-7	N	security-guardian+sre	P2	planned

Hard invariants that must NOT regress when landing any WS

Pulled from ADR-2002 and the matrix's Definition-of-Done. Every WS PR checks:

• Effectively-once, not exactly-once — one effect and the original outcome across an agent bounce + a primary failover. A dup never loses the first result.
• Fenced leadership — a monotonic epoch is checked inside the claim transaction; a stale leader's writes are rejected. No singleton worker runs without holding the fence.
• No committed event lost across failover — events are durable (outbox) and replayable from a cursor on a different replica.
• Routing directory is fenced with CAS register/deregister — no stale replay overwrites a newer re-home.
• RPO=0 across a host failure with 3-node quorum on distinct failure domains; quorum loss blocks writes (fail-closed) — unless the operator has explicitly opted into degrade-to-async (WS-7's YUZU_PG_SYNC_STRICT).
• A new background job is classified (fenced-leader-only / replica-safe / disabled) in the WS-10 table before it can run under active-active.
• Every WS is done only when it has a passing WS-9 scenario (WS-9 is continuous, so this is not circular).

---

3. Gap / priority — what to build next

Prio in the matrix is build-effort; the safe-to-scale gate is the real constraint. Delivery phases (dependency-ordered):

1. Phase A — Foundation (build now, parallel where the ladder allows): WS-1 (state→PG; 1a/1b/1c pace on the migration ladder, not fully parallel), WS-0 (agent idempotency+replay), WS-2a (durable outbox), WS-10 (job-safety audit), and WS-7 (HA-PG — storage axis, fully parallel; DONE, PR #3627 merged, unblocks WS-9's stack).
2. Phase B — Multi-instance safety (ALL required before a 2nd replica): WS-3 (fenced leader), WS-4 (gateway routing), WS-5 (presence), WS-6 (PKI), WS-8-readyz, WS-13 (agent gateway-front rollout). The split question (§1c) is RESOLVED — decoupled, ratified by the ADR-1005 owner (Dave Rae, 2026-09-07); the split does not gate a 2nd replica.
3. Phase C — Enable, validate, operate: flip on the second replica; WS-9 runs continuously throughout B/C; WS-11 ships with the second replica; then WS-12 (cutover/DR), WS-14 (security/capacity). WS-2b (spine) and WS-8 tier-split readyz land with the ADR-1005 split.

Suggested next slices (as of 2026-09-24 — docs/ha-delivery-matrix.md row cites are authoritative):

• Done (dial-selection, the safe-to-scale gate's WS-4 item): WS-0 (#3662), WS-1 (1a+1b+1c), WS-2a (2a-1 + 2a-2 #3924), WS-3 (3.1–3.4, #4011/#4134/#4169/#4194), WS-4's dial-selection half (4.1+4.2a+4.2b+#4324+#4555 (gateway multi-node cluster formation, 495cf24c4)+4.3a (intra-cluster pg-group agent→node lookup)+4.4 (gateway_node convergence + #4246 #6 writeback, ADR-2002 §7c)+rest-of-4.3 (cross-cluster gateway fan-out, ADR-2002 §7d)), WS-7 (#3627), WS-10 10.1/10.2. The storage axis is complete. WS-4 is NOT entirely done — two named items remain open and are tracked, not disclosed-only: #4669 (multi-cluster mode has no agent↔cluster affinity/peer-identity binding, so it does not yet provide trust-zone isolation — caught by pr-rev, FortitudeEtc/Codex+Kimi, pre-merge on the rest-of-4.3 PR) and #4672 (§7's durable outbox re-drive was never wired into forward_gateway_pending's terminal-failure branches). Neither blocks the safe-to-scale gate today (multi-cluster mode has zero production deployments; the re-drive gap is inherited from before 4.3, not introduced by it), but do not read "WS-4 done" as "WS-4's every design goal closed."
• The highest-leverage next is now the remaining gate set directly: the rest of WS-5 (slice 1 merged, #4745 — decommission cascade, capability visibility, fleet gauge, WS-9 scenario) and the rest of WS-6 (6.1 merged, #4833 — 6.1b / 6.2 / 6.3, WS-9 scenario #4832). WS-8-readyz is DONE for the monolith (2026-09-24). A loss-free cross-replica reconnect (durable outbox replay) is the open 2a-2 follow-up.

Shared with ADR-1005 (do not duplicate): engine-tier HA (incl. NVD/CVE sync, withdrawn from WS-1), the MCP Decision-15 pre-commitments WS-2 inherits, and the presentation/core split (WS-2b + WS-8 tier-split readyz land with it).

---

4. Standard workflow for a workstream slice

For every WS in the matrix:

1. Read first, in order:

   • This skill (gap framing + the safe-to-scale gate).
   • docs/ha-delivery-matrix.md for the row's current-state cites + dependencies + reviewers.
   • docs/adr/2002-high-availability-architecture.md for the decision the slice must satisfy (the ADR governs on any conflict).
   • docs/postgres-migration-ladder.md for any WS-1/store-migration slice (it is the authoritative per-store status).
   • docs/adr-1005-execution-plan.md if the slice touches a shared seam (presentation/core split, engine tier, MCP Decision-15).

2. Plan. Produce a short plan covering: the ADR-2002 guarantee it delivers; whether it is in the safe-to-scale gate set; schema changes (via MigrationRunner / the PG store playbook); the fence/epoch or idempotency/outbox mechanism if applicable; and the WS-9 scenario it must add (a WS is not done without one).

3. The split question (§1c) is RESOLVED — decoupled, ratified by the ADR-1005 owner (Dave Rae, 2026-09-07). Phase-B proceeds on the monolith track; no per-slice re-confirmation needed.

4. Implement as a focused PR per slice, in an isolated worktree off dev. Drive schema through MigrationRunner / docs/postgres-store-playbook.md, fencing through the LeaderElector epoch contract, and receiver dedup through the agent/gateway command_id path.

5. Test. Run /test before commit; add the WS-9 failover scenario and run the failover harness (scripts/ha/ for the storage axis). Note WS-7-class deploy-only changes are a /test no-op — CI's build legs + the harness are the real gate there.

6. Governance. Run /governance dev..HEAD before pushing — the routed reviewers in the WS row are unconditional (e.g. cpp-safety on any fencing /ownership change, security-guardian on the coordination substrate, gateway-erlang on WS-4). CRITICAL/HIGH block merge. Self-adversarial review (Sol / Kimi) before pushing review-fixes on HA work — it has repeatedly caught blocking findings a reviewer would hit next round.

7. Docs + evidence. docs-writer picks up user-manual / ops-runbook updates in Gate 2; ship them in the same PR. For the coordination substrate, WS-14's threat/capacity notes and a docs/security-reviews/ record apply.

---

5. Cross-references

• The decisions: docs/adr/2002-high-availability-architecture.md
• The delivery spine (authoritative status): docs/ha-delivery-matrix.md
• Per-store migration status: docs/postgres-migration-ladder.md
• PG store authoring recipe: docs/postgres-store-playbook.md
• Shared decomposition / engine tier / MCP Decision-15: docs/adr-1005-execution-plan.md + docs/adr/003{1,2,3}-*.md
• WS-7 shipped surface: deploy/docker/docker-compose.ha-postgres.yml, deploy/docker/ha-postgres/, scripts/ha/, docs/user-manual/ha-postgres.md, docs/ha-postgres-ws7-plan.md
• Routed reviewers are per-row in the matrix; the coordination-substrate concern routes security-guardian + architect.