/new-deployment — a new instance is a record, provisioned by the control instance
A deployment is one Hosting/Deployment record — mesh/Deployments/<id>.json in the estate's PRIVATE
deployments repository — that the control instance (e.g. https://control.example.com, its own record carrying
operator.enabled: true) turns into a namespace, a database, DNS, a Helm release and a certificate
when a global admin files a Provision action against it. Everything else — cluster, ingress,
Postgres server, Key Vault, the registry, observability — already exists and is shared.
🔒 The inventory is private. Hosts, namespaces and database names live in the estate's private
deployments repository (mesh/Deployments/*.json, its inventory, the Deployments tab) — see "Where an
estate's instances are declared" in Instances.md. The public MeshWeaver repo
carries the mechanism (chart, operator scripts, the Hosting plugin's sources) and must never
carry who runs what.
🚨 The maintained runbook is docs/new-deployment.md in the estate's private deployments repository. It is authoritative
wherever it and this page disagree; this page is the map, not a copy of the commands.
Where the work happens — and where it does not
| who | how |
|---|
| the record | a human, by PR | mesh/Deployments/<id>.json (GitSynced to the portals; a mesh-side edit is written back as a PR by DeploymentGitSync) |
| secrets an instance cannot mint for itself | a human, once | Key Vault <vault>, names <keyVaultSecretPrefix><Section>-<Key> |
| namespace, database, identity, DNS, pull secret, release, TLS | the operator, from the Provision plan | a Hosting/InstanceAction node on the control instance |
| the cluster | nobody by hand | kubectl is not part of this procedure; a step that needs it is a defect in the lane, file it |
1. What a human does first
Every one of these is a prerequisite the Provision REFUSES without, so do them before filing it.
<prefix> is the record's keyVaultSecretPrefix.
- The sign-in app (multi-tenant; invitation-only is the gate, not the audience):
az ad app create --display-name "<Name> Portal (<host>)" --sign-in-audience AzureADMultipleOrgs --web-redirect-uris https://<host>/signin-microsoft,
then az ad sp create --id <appId>, then the client secret into the vault as
<prefix>Authentication-Microsoft-ClientSecret — through the record page's Set Key Vault
secrets… dialog on the control instance (write-only; the governed writer identity stores it,
policy secrets-write-only-entry, Doc/Architecture/SecretsWriteOnlyEntry), never a vault
command of your own. The appId goes on the record as signIn.microsoftClientId. 🚨 az … credential reset PRINTS the secret on stderr: --query password -o tsv > "$S/x" 2>/dev/null,
paste it from that file into the dialog, then rm. Never cat a stderr capture from a
minting verb.
- The registry instance key — ONE key, two jobs (plugin catalog AND image pull):
POST https://<plugin-registry-instance>/api/instances/register with
{"bootstrapKey":"","instanceId":"<id>","displayName":"…","homeUrl":"https://<host>"} (open
registration = the free plan; an admin raises it under Admin ▸ Instance grants). Paste
.instanceKey into the same Set Key Vault secrets… dialog as <prefix>PluginCatalog-RegistryToken.
hosting-kv-ensure lists this object as REQUIRED and refuses the whole Provision without it —
"minting a random value here would produce an instance that boots, registers nothing, and shows
an empty Store with no error".
- The database connection string —
<prefix>db-connection
(InstanceSpec.DatabaseSecretName), composed from the shared admin password in vault object
memex-postgres-password: Host=<pg-server>.postgres.database.azure.com;Port=5432;Username=<pg-admin>;Password=…;Database=<db>;SslMode=Require;Trust Server Certificate=true.
FQDN, never an IP — Azure moves the backing instance and DNS follows it. Mapped on the
record's keyVaultSecrets.secrets as ConnectionStrings__memex.
- An image tag that EXISTS in
cr.meshweaver.cloud. imageRepository: cr.meshweaver.cloud/memex-portal-ai, pinnedImageTag: <tag> — a first install with no pin is
REFUSED by hosting-deploy ("Pin an image tag on the Deployment record for the first install").
If the tag is not in cr yet (CD pushes to ACR), copy it there as the registry's publisher account:
crane copy meshweaver.azurecr.io/memex-portal-ai:<tag> cr.meshweaver.cloud/memex-portal-ai:<tag>
and the same for memex-migration (the migration image is DERIVED by replacing
memex-portal-ai → memex-migration; both must be present).
- The record itself, by PR, with (a sibling record is the worked example):
imagePullSecret: registry-pull; volumes[] for data/content/attachments/users with claimName,
mountPath, size, storageClass: azurefile-memex, accessMode: ReadWriteMany and
create: true — the chart creates the claims; without create a new instance gets
emptyDir and every restart wipes it; keyVaultSecrets naming the
three vault objects above plus <prefix>Ai-KeyProtection-MasterKey (GENERATED by
hosting-kv-ensure, never regenerated — it seals every stored enc: provider key);
pluginRepos[0] = { name: "Plugins", url: "https://<plugin-registry-instance>" } (the name is the
REGISTRY's name, never the instance's — a bare preInstall id is qualified against it and the
catalog FAILS CLOSED on a mismatch); preInstall: ["Essentials"]; updatePolicy: Stable for a
customer; extraPortalConfig with Hosting__Deployment: <id> and Hosting__ReportTo: https://control.example.com (the control instance).
Gates that read it: check-record-renders-overlay.py (record ↔ values.<release>.public.yaml,
extraPortalConfig one way, keyVaultSecrets both ways), check-image-pins.py (resolves every
pin — cr.meshweaver.cloud pins need MW_REGISTRY_KEY), config-key-coverage.
2. What the operator does — the Provision plan, in order
Composed by InstanceActionPlan.ProvisionSteps (MeshWeaver.Plugins, in-mesh — it compiles in the
portal, no CI type-checks it), run by MeshWeaver.Plugins/deploy/hosting-operator/bin/run.sh in a Job in memex-ops,
fail-fast, the failing STEP named on the node:
az postgres flexible-server db create — control plane, workload identity. (Memex#132 —
PGPASSWORD has no secret channel — blocks Backup/Suspend/Teardown/Restore, not this.)hosting-kv-ensure — generates the master key, REQUIRES the registry token.hosting-federate — federated credential, subject exactly
system:serviceaccount:<ns>:memex-portal-sa; a mismatch does not error, it silently stops
self-update tag discovery.hosting-dns upsert — the A record at the shared ingress IP. DNS BEFORE TLS: HTTP-01 needs it.hosting-pull-secret — ensures the namespace, turns the vault's registry token into the
kubernetes.io/dockerconfigjson Secret named by imagePullSecret, reads it back, never prints
the key.hosting-deploy --image <imageRepository>:<pinnedImageTag> with values rendered from the
record (HelmValues) — the chart is BAKED INTO THE OPERATOR IMAGE, so the control record's
operator.image (meshweaver.azurecr.io/hosting-operator:<short sha>, published on every push
to MeshWeaver main) decides which chart deploys. A stale operator image deploys a stale chart.- rollout wait → 8.
hosting-tls (waits for the Secret to carry a key pair) → 9.
hosting-verify → 10. hosting-verify-catalog (the expected mounts and packages, so an empty
Store cannot read as healthy).
3. Filing it
On the control instance (through its own MCP server), create under Deployments/:
{ "id": "<id>-provision", "namespace": "Deployments", "nodeType": "Hosting/InstanceAction",
"name": "Provision <id>", "content": { "$type": "InstanceActionContent",
"deployment": "Deployments/<id>", "requestedAction": "Provision",
"confirmation": "<id>", "dryRun": true, "reason": "…" } }
Dry run first — Authorize → Read record → Plan → "Dry run — nothing changed" prints the composed
step list on the node's log; a refusal here (state: Refused, error naming the field) is the
ANSWER, not an error to retry. Then the real one (dryRun: false). Read state/phase/message
and the log on the node; a failed step names itself and the plan is idempotent from the top, so
fix the cause and file again.
4. After it answers
- A cold instance recompiles every dynamic NodeType. The startup budget on the record
(
startupProbe.budgetSeconds, 3 h on the siblings) exists for this; a pod 0/1 during the bake
is the gate doing its job. Do not cycle pods while it warms.
- Register the GitHub Environment and a deployments entry in the estate's private deployments
repository, add the inventory row (see "Where an estate's instances are declared" in
Instances.md), set
Admin/UpdatePolicy (Stable for a customer). A deployment nobody
recorded is a deployment nobody will remember to patch.
🚨 Traps that read as success
- HTTP 200 proves nothing. The Blazor shell answers 200 for an error page or a paywall redirect;
hosting-verify checks rendered content — so should you.
- Nobody can sign in is a perfectly healthy instance: every
Authentication__*__ClientId
empty and dev login off means it boots, passes probes, serves, and has nothing to authenticate
against. Step 1 is not optional.
- The chart ConfigMap emits only the keys it templates; a record
extraPortalConfig key the
chart has no line for reaches NO container, with no error. config-key-coverage is the guard.
kubectl set env is not a deployment. Helm's three-way merge never removes an inline entry it
did not own; the record's inlineEnv exists to make such drift visible, not to bless it.- Never emit
"" for an int/bool key — it fails binding, DI throws, the landing page dies.
- A refused Provision with
hosting-kv-ensure naming a missing vault object wants the object
ISSUED (step 1.2), never a random value.
az prints minted secrets on stderr — see step 1.1; a leaked value is rotated in the same
turn.
Related
/plugins — the registry the instance consumes packages from/release — how images get built; a tag missing from cr is copied there by hand (step 1.4)/deployment · /delete-user · /storm · /debug — operating a live instance- Design: ContainerRegistryInMemex.md
(the registry as a separate service) · Memex#132 (the operator's PGPASSWORD gap)