/deploy — pick the route by TARGET, then verify what is actually running
🚨 The cluster is not the surface — the control instance is (maintainer, 2026-09-08: "all the
operations through the memex api", "no direct access of aks"). On AKS, an instance is a
Deployments/<name> record on the CONTROL INSTANCE (reached through that instance's own MCP server;
any other portal's Deployments/* copy is a lagging second sync, not the record) and every operation is a
Hosting/InstanceAction node — Roll (the record's image pin, or an explicit imageTag, then WAIT
for the rollout), Restart, Suspend/Reactivate, Audit, Reconcile, HelmRelease — run by
the in-cluster operator and reported on the same node. Every az aks command invoke / kubectl
line below is what the operator runs FOR you, kept here as the bootstrap form and as break-glass;
a break-glass write is half an operation (the Hosting manual's "Break glass" section lists the
other half). Policy page, with what the API does NOT answer yet:
OperatingFromThePortal.md.
Two deploy routes, different targets — neither deprecated. Don't mix them.
Index: Deployment.md.
🚨 Before any AKS deploy, read DeploymentAKS.md end-to-end — it is the source of truth for
build → roll-out → verify. 🚨 It no longer describes an auto-baked mesh-local #r package feed: the
BakeMeshLocalFeed target was REMOVED (#395), dist/packages is gone, and a legacy
#r "nuget:MeshWeaver.BusinessRules.Generator" now hard-fails on a deployed image ("The local source '/app/dist/packages' doesn't exist" — the prod BalanceSheet failure). The generator ships
built-in and GeneratorPipeline strips that #r instead. Nothing bakes at BUILD time on any
machine, dev Macs included: --bake-output exists only in CI scripts, never in a
.targets/.props/.csproj. The commands below are a quick reference, not a substitute for the
doc.
🚨 The ladder: which image and which plugins, and when a roll is HELD
Policy platform-backwards-compatibility: a portal rolls to the newest promoted image its update
policy admits and on which no installed plugin is held — and on a normal build nothing is held, with
NOTHING sealed for the new build. It keeps its plugin bytes across the roll, and adopts a newer plugin
publication later, independently, when that publication's floor is ≤ the running platform. The one
roll hold is a DECLARED break (another compatibility key, or a ceiling an installed build declares)
with no replacement sealed for the target: the self-updater's candidate is declined as
PlatformRangeExceeded, naming the plugin and both versions, and the portal lands on the newest
release its plugins still cover. A routed Roll runs unattended under a Continuous record whose
pattern admits the tag; it carries the target's migration Job first. 🚨 Never pin an image tag in a
record or a chart value — not to unblock a roll, not ever (PlatformDeliveryNeverWaitsOnPluginsGuard
refuses one under deploy/). Full procedure and decision table: Deploying Across Platform Versions.
🚨 First: the memex API, not the cluster
Maintainer directive, 2026-09-08: every operation goes through the memex API — no direct az /
kubectl access. The control instance (e.g. https://control.example.com, reached through its own MCP server, module
Hosting from MeshWeaver.Plugins) exposes each of the recipes below as a Hosting/InstanceAction node whose
answer is a node; the in-cluster operator and the monitoring stack hold the only credentials.
Reach for the commands further down ONLY when the control plane itself is what is broken (the
"break glass" section of the fleet guide, get @Hosting/Guide).
| you want | create (MCP create, or POST the node) | read the answer at |
|---|
roll <id> onto a tag | pin pinnedImageTag on Deployments/<id>, then { "requestedAction": "Reconcile", "confirmation": "<id>" } — or { "requestedAction": "Roll", "imageTag": "<tag>", "confirmation": "<id>" } for the portal image alone | the action node's state/log; then a Sample |
what is it running, per replica (image, ready, restarts, started, its own /health + detail) | { "requestedAction": "Sample" } | Ops/Status/<id> (replicas[], warnings[]), the Deployment page's status strip |
| the last hour's error lines | { "requestedAction": "Logs", "query": "fail:|crit:|Exception", "sinceMinutes": 60, "limit": 300 } | the action node's logQl / entryCount / truncated; lines under Ops/Logs, @Deployments/<id>/area/Logs |
| what lives only on the cluster | { "requestedAction": "Audit" } | Ops/Audit/<id> |
| re-read ONE address no person may recycle (a NodeType and its dependency network, or any node) | { "requestedAction": "Recycle", "recycleTarget": "<path>", "reason": "…" } + one approval | the run's log. Manual: Hosting/RecycleAction (Plugins) |
| delete a space no person may delete (owner gone, stranded partition) — the sanctioned break-glass for DATA, since a global admin is not a data superuser | { "requestedAction": "DeleteSpace", … }; it parks WITH the full plan | the plan on the node, the audit record. Manual: Hosting/DeleteSpaceAction (Plugins) |
| upgrade the cluster's Kubernetes version (control plane + every pool) | plan: { "requestedAction": "UpgradeCluster", "dryRun": true }; then { "requestedAction": "UpgradeCluster", "upgradePlan": "<plan node path>", "confirmation": "<id>" } + ONE mesh approval — never az aks upgrade (policy cluster-upgrade-governed) | the plan on the dry-run node; the run's log, aks_versions; a stuck portal roll refuses it by name. Manual: Hosting/ClusterUpgrade (Plugins) · DeploymentAKS.md "Kubernetes version upgrades" |
All nodes are nodeType: Hosting/InstanceAction, content.$type: InstanceActionContent,
content.deployment: "Deployments/<id>". The full table — including what each action does and
which credential runs it — is the fleet guide's "Roll, restart, observe — the ops actions".
🚨 A gated action: you approve a PLAN, and the approval is bound to it
Every mutating kind parks at awaiting approval WITH the plan it will run (the steps and commands,
in the order they run). The approval binds the request, the action, the deployment, the image tag,
the kind's own binding (a Recycle's target and force flag, a DeleteSpace's space) and the DIGEST of
that plan. A plan that would come out differently re-parks; it never runs unapproved. What else is
true, and bites:
- Every step runs as system, and the plan PRE-FLIGHTS rights and scale (policy
governed-action-preflight, proposed; implementation in flight). An action refuses at park
when a step cannot complete, never partway through a destructive run. The evidence is a
DeleteSpace that failed at step 9 after it had already removed the space's grant and GitSync. It
hit 31,138 per-node delete validations against a 25 s bound. Until the pre-flight ships, read a
destructive plan's scale (the rows per table) yourself before you approve it.
- Plans are deterministic. A clock-derived value (a dump's object name) comes from
planSeedAt,
stamped at the first park of an approval round. A node that re-parks every few seconds with a
moving parkedAt is a plan that is not deterministic: a defect to file, not an approval to retry.
- The requester is recorded once, at the first park. A re-park never rewrites it, so the approver
is never mistaken for the requester.
- Approve with the button. It attests the AUTHENTICATED clicker (an HMAC over the approval, the
request, the park and the node revision). An approval typed by someone else is not an approval.
- Only the configured maintainer (
Hosting:Operator:Maintainer) may approve their own request.
From an agent session that is ONE MCP patch, written as the maintainer, carrying requestedBy,
approvedBy, approvedAt and every binding field as parked (approvedRequest, approvedAction,
approvedDeployment, approvedImageTag, plus approvedRecycle / approvedPlan where the kind
binds them). Create the action and WAIT FOR THE PARK first: an approval in the create write,
or one dated before parkedAt, is refused and re-parks. approvedPlan is not stored on the node,
so recompute ActionPlanSnapshot.Digest() from the parked actionPlan (sha256: lower-hex).
Split the approval into several writes, or let a binding field drift, and the gate re-parks. This
is a write in the maintainer's name, so do it only when the maintainer asked for it.
- The digest binds the COMMANDS, not the values file or the operator environment rendered at
dispatch. After a record change that follows an approval, re-request the action rather than
trusting the binding to catch it.
- Approvable is not yet runnable. An approved plan can still stop on an unbound
$VAR or a
declared vault object that is absent. Check both before you approve.
- A
Running action survives a restart of the control instance. The watcher re-opens every
orphaned Running action on start and resumes following it. Do not re-file it.
- One rollout budget. How long a roll may take is the chart's own progress deadline
(
ChartRolloutBudget), read by the action's wait and by the stuck-roll alarm alike.
- A
Reconcile that ADDS a Key Vault key needs a Restart after it (open finding,
Systemorph/MeshWeaver.Feedback#48): the CSI driver syncs the new object after the pod has resolved
its environment.
- A
Reconcile can refuse on the Key Vault values half before helm runs, if the half carries
structure. Run a HelmRelease action with helmAction: capture first, then the Reconcile.
- A secret is never set or read with
az by hand (policy secrets-write-only-entry). Paste or
generate it on the control instance: Deployments/<id> → Set Key Vault secrets…. That files a
SetSecrets action whose Job writes as the WRITER identity (it can list, set and delete, but never
read a value) and stamps the mw-fp fingerprint tag. Compare the two ends of a pairing by that
fingerprint, never by reading the values. CI refuses a new hand-run vault command
(check-manual-keyvault.py). Full model:
SecretsWriteOnlyEntry.md.
The rules, the page layout and the threat model: Hosting/AksOperationsViaActions (Plugins,
"Approval in the mesh" and "What operating it taught") · DeploymentAKS.md
"Diagnostics — through the memex API first".
🚨 Authoring or reading a governed action: system credentials from the first step, pre-flight at park
Policy governed-action-preflight. Every step of a governed action runs under the system
identity from its first step, and at PLAN time — before it parks for approval — the action
verifies each step is feasible: the rights the step needs as system, and the scale bound of the
mechanism it uses. A step the system identity cannot perform, or one known to exceed its bound, is
refused at park with the reason — never discovered partway through a destructive run, where it
leaves the target half-removed. The plan names, per step, the identity, the mechanism and its bound.
The incident that set it: a governed DeleteSpace of UWDeepfield (31,138 descendants) ran its
grant and GitSync removals as system, then failed in Delete content — a per-node recursive
DeleteNodeRequest whose ValidateDeleteRequest pre-flight fan-out could not settle in its 25 s
bound — leaving the space with no grant and no sync but all its content. A whole-space deletion now
drops the partition as ONE teardown (PartitionTeardown.TearDownPartition, which itself refuses any
identity but system), and its plan refuses at park when a per-node leg exceeds its bound. Manual:
Partition Teardown →
The direct teardown; the action's own manual is Hosting/DeleteSpaceAction (MeshWeaver.Plugins).
🚨 Probes: the startup budget covers a cold boot, never a bake
The bake gate and required_modules are ROLL GATES (policies bake-gate-readiness-only,
required-modules-readiness-only): their verdict is read by /ready alone, so a refusal stalls the
roll with the previous image serving and kills nothing. The startup probe proves only that the
process booted, and the fleet's portal instances run it at failureThreshold: 60 (600 s). Never raise
probes.startup to "cover a bake" — that is the configuration that killed every container of both
images every 3 h and took the control instance down. A container that dies at exactly
periodSeconds × failureThreshold after boot is a startup-probe kill, not a crash: read Sample's
lastTerminationReason / lastExitCode per replica. Full account and incident history:
TheBakeGateOnlyStallsARoll.md.
🚨 A schema change is planned, and every roll migrates first
Policy db-migration-planned: migrations are expand-only, a DbVersion bump declares
Db-migration: V<N> — … in its PR, every release publishes ExpectedDbVersion, and every roll path
runs the migration Job before the image moves or refuses (the operator's run.sh interlock does it
whatever Hosting generation planned the roll). The one exception is the break-glass HelmRelease
dispatch; use Reconcile. Manual:
PlanningADatabaseMigration.md.
The AKS route (break glass — the control plane is what is broken)
🚨 Pass --subscription <subscription> on EVERY az call, as every example below does.
<subscription> is the fleet's own subscription; its id is in the estate's private deployments repository
(see "Where an estate's instances are declared" in
Instances.md), never in this public one. The signed-in account's default subscription may
belong to another tenant's installation, which holds another cluster with the SAME name
(<aks-cluster>); without the flag you read the wrong system and nothing says so.
az role assignment delete --ids … needs it too, or it fails with InvalidAuthenticationTokenTenant.
A portal instance runs on the shared AKS cluster <aks-cluster> (RG <aks-resource-group>,
swedencentral) — namespace <namespace> — against the Postgres Flexible Server, images in ACR
meshweaver.azurecr.io. Private cluster: kubectl ONLY via
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command "…" — and only when
the API above cannot answer.
On AKS a code update = the record's image pin + a Roll action. What the operator then runs is
build image → set image → restart (the AKS route does NOT use tools/deploy.sh or aspire deploy
— those are the Container Apps route). The commands, for the bootstrap / break-glass case only:
az acr login --subscription <subscription> -n meshweaver
# Portal (custom base) AND migration (the migration is what creates schema + the matview):
dotnet publish ../MeshWeaver.Plugins/src/Memex.Portal.Distributed/Memex.Portal.Distributed.csproj -c Release \
-t:PublishContainer -p:ContainerRegistry=meshweaver.azurecr.io \
-p:ContainerRepository=memex-portal-ai -p:ContainerImageTag=<tag> \
-p:ContainerBaseImage=meshweaver.azurecr.io/memex-portal-ai-base:latest
dotnet publish memex/aspire/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release \
-t:PublishContainer -p:ContainerRegistry=meshweaver.azurecr.io \
-p:ContainerRepository=memex-migration -p:ContainerImageTag=<tag>
# Roll out (NS = the instance's namespace). 🚨 The MIGRATION is a Job, not a Deployment — see below.
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command "\
kubectl -n <NS> set image deployment/memex-portal-deployment memex-portal=meshweaver.azurecr.io/memex-portal-ai:<tag>; \
kubectl -n <NS> rollout restart deployment/memex-portal-deployment; \
kubectl -n <NS> rollout status deployment/memex-portal-deployment --timeout=300s"
- An env's
deploy.sh is first-time ENV SETUP only (helm install + PVCs + KV
SecretProviderClass + ingress + connection-string patch). Do NOT use it for a code update — it
re-runs the whole chart and can reset live config. 🚨 Env folders live in the estate's PRIVATE
deployments repository, not deploy/aks/envs/<env>/ — they moved out 2026-08-08/09
(a69959165) because their directory names are tenant identities; deployments/aks/example/ (Systemorph/Memex, moved from this repo's deploy/aks/envs/example/) in
this repo is the reference template only.
- Don't run
tools/deploy.sh or aspire deploy against the AKS cluster — those are the
Container Apps route (a different target), not a code-update path for AKS.
🚨 The migration is a run-once Job, NOT a Deployment
deploy/helm/templates/memex-migration/job.yaml: kind: Job, restartPolicy: Never, the name
embeds .Release.Revision, ttlSecondsAfterFinished. It runs on helm upgrade, not on an
image-tag roll — so a kubectl set image / rollout restart aimed at a
memex-migration-deployment is a command with no object: the chart does not define one, and any
that exists in a live namespace is a cluster-only orphan of a chart revision that predates the Job
(#1788). Do not resurrect it. A MigrationWorkloadModelGuard
(test/MeshWeaver.Documentation.Test) fails the build if such a command returns to AGENTS.md, a
skill under .claude/skills/, a doc, or a deploy script.
A crash-looping migration pod is a FAILURE, not noise. It used to be modelled as a Deployment,
which restarted it forever after each clean exit — three prod namespaces sat at 50/53/38 restarts,
each rebuilding public.top_level_index across every partition schema. Documenting that as
"benign" is what made a real migration failure unreadable. With the Job form the pod runs once and
stops, so CrashLoopBackOff on it now means exactly what it says.
Before declaring a deploy successful, confirm the Job's log shows
Database migration completed. Version: N AND the portal serves (HTTP 200):
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command \
"kubectl -n <NS> get jobs -l app.kubernetes.io/component=memex-migration"
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command \
"kubectl -n <NS> logs job/memex-migration-<revision>"
The schema has a hard gate. DbVersionGate (a hosted service in Memex.Portal.Distributed)
reads admin.mesh_nodes.db_version at startup and StopApplication()s with a LogCritical when it
is below DbVersion.Latest. So a portal rolled ahead of its schema refuses to serve rather than
serving a half-migrated database — loud, and the reason the ordering between the two workloads
cannot silently invert.
Verify the IMAGE, never the green tick
A merged, green, CD-published change is not a deployed change until the running pod says so. The
delivery chain, the publish batching window and the reconciler are in
/release; the assertions are:
az acr repository show-tags --subscription <subscription> -n meshweaver --repository memex-portal-ai --orderby time_desc --top 5 -o tsv
az aks command invoke --subscription <subscription> -g <aks-resource-group> -n <aks-cluster> --command \
"kubectl get deploy -A -o custom-columns=NS:.metadata.namespace,IMAGE:.spec.template.spec.containers[0].image --no-headers | grep memex-portal-ai"
.github/scripts/check-image-set.sh <short-sha> # the exact assertion CD itself makes
🚨 …and the RIGHT IMAGE is still not the right ANSWER — the grain serves what it activated with
"The grain keeps serving old state until we send a dispose request" (maintainer, 2026-09-17).
A per-node hub binds its configuration ONCE while activating and is then pinned by address — routing
never resolves that path again. A Roll or a Restart does end the activations on the pods it
replaces, so those addresses re-read; the gap is everything a portal absorbs while it keeps
running (a package installed or updated, a NodeType recompiled in place, a sync landing new
source), and what the fresh activation then chooses. The rolled tag is necessary and not
sufficient.
Concretely, after a roll or a package publish:
- 🚨 Read the EFFECTIVE
Modules:AutoRecycleOnStaleBuild — the code default and the fleet
disagree. The runtime default is off, where the stale-build watcher publishes only a banner
offering a recycle and a portal stays a mixture of old and new assemblies for as long as viewers
do not click (the 2026-08-25 Store outage is what that costs). But deploy/aks/values.aks.yaml
sets it true fleet-wide, so the portals this repo deploys converge by themselves. On neither
setting does it fire for a same-path byte change.
- Verify by EXERCISING the feature at the running address, not by the tag, the install record or
the green tick. Then, if it still answers the old way, recycle that address — the
recycle verb
(MCP tool, the node's Recycle menu entry, mw recycle <path>) or hub.RecycleNode(path, reason: "…") in code — and exercise it again.
- The two surfaces differ.
hub.RecycleNode is dispose-only. The operator recycle verb also
stamps RequestedReleaseAt + RequestedReleaseForce, but only when the target is a NodeType
node — and a FORCED release is precisely the one that skips on-demand prebuilt adoption and
compiles the live source (#2818), so it is the operator's rebuild-from-source remedy. Recycling an
ordinary page recompiles nothing.
- 🚨 A recycle makes the activation RE-READ; it does not decide what the re-read FINDS. A third
recycle finds nothing a second did not: a dispose-only recycle re-resolves the same store key
(nodeTypePath, version) and re-binds the same local copy (#2471 — measured over 30+ minutes and
six recycles), and an UNFORCED trigger, or a Modules:RequirePrebuilt mesh, re-adopts the same
bundle — remedy there is a rebake and republish for THIS framework identity. And no recycle fixes
a package that never reached the pod: read the [ModuleLoad] <Assembly> ← … (written=…) line
before recycling anything.
Full reference:
StaleStateUntilRecycle.md.
Running the portal locally (not a deploy route)
dotnet watch --project ../MeshWeaver.Plugins/src/Memex.AppHost restarts only the affected Aspire
resource on save; the dashboard's Resources → ⋯ → Restart is the fallback; a process kill is the
last resort. Don't kill the whole aspire / Memex.AppHost process unless you changed AppHost
wiring itself — a full restart costs 30–60 s and loses the dashboard auth token. Full reference:
LocalDevWorkflow.md.
Checklist