CI
Drive CI through the odu MCP server — the single front door. The runner is
odu (npins-pinned by ci/flake.nix, nix run ./ci#odu): mcp__odu__run
spawns the coordinator, mcp__odu__wait_for_settle blocks until settle or first
red node (its verdict is stamped sha7#seq; pass expected_sha to hard-check),
the surface://collections/logs/{id} resource drills into a failure,
mcp__odu__node_rerun closes a red check. Runner mechanics: /odu skill.
Banned flags: never --no-post, --no-strict, --no-snapshot. CI here is
always strict and always posts commit statuses — a run that doesn't post isn't
CI, it's a private dry-run that leaves the PR unverified.
Push before CI — remote lanes git fetch the pinned SHA from origin; an
unpushed commit can't run.
Sync master before CI, every run, whoever is driving. Fetch and merge
origin/<default> unless it's already an ancestor
(git merge-base --is-ancestor). The changelog is merge=union and never
conflicts; a real conflict is yours to resolve now. Never merge while a
gauntlet round is mid-commit — it races the index.
Every run is x86_64-linux ONLY. Pin it explicitly —
platforms=["x86_64-linux"] — never trust a machine-local default: a platform
you don't name silently drops, and an unpinned run picks up whatever
hosts.json happens to list (today that includes a darwin box). A linux green
IS the green this repo gates on; master's required-status set is linux-only.
Never run aarch64-darwin unless the user asks for it in THIS
conversation. Not "to be safe", not because a change looks platform-shaped,
not because an older note called two-platform coverage mandatory. Darwin's Nix
closure still gets built by .github/workflows/nix-cache.yml on every push —
that lane is GitHub's to drive, not yours.
<details>
<summary>Asked for a darwin lane explicitly? Then, and only then:</summary>
Add "aarch64-darwin" to platforms and --platform aarch64-darwin to
just ci::protect. A named platform with no pool entry is refused loudly
(juspay/odu#46).
Read hosts.json, never this skill, for the box. The darwin fleet is
volatile (this file hard-named the live box three times and was stale each
time) — ~/.config/odu/hosts.json (nix run ./ci#odu -- hosts) is the only
source of truth; copy entries verbatim when pinning (user@ and tailnet suffixes
differ). Durable rules: under a coordinator, ask it before every darwin
dispatch (single-tenant); treat fs-watch-class reds on darwin as suspect box
environment and report before chasing; a dead pool entry silently stalls the
whole darwin lane — report it for removal rather than waiting it out.
A companion repo's darwin lane uses --host aarch64-darwin=<box>, never
inline $ODU_HOSTS — odu reads $ODU_HOSTS as a file path, so inline JSON
is silently ignored and burns a run on a dead host.
</details>
Linux lane: odu leases natively from the warm Incus pool
(kolu-ci-1..8 in hosts.json) — picks a free box, holds it for the run,
releases on settle. No manual leasing. Saturated pool waits in line
(no_wait: true to fail fast instead); nix run ./ci#odu -- hosts shows
free/busy. To keep the same hot box across fix→rerun cycles, take an
agent-held lease with mcp__odu__lease first and mcp__odu__release when
done. Pool upkeep: just ci::pool-ensure / just ci::pool-status; warm idle
slots by running the linux lane against master with a hosts= pin (strict and
posting, like every run).
Fail fast on the MCP; don't drain the pipeline. wait_for_settle returns
on the first red node with {failed[], errored[]} — read the red node's log
(the log resource, or .ci/<sha7>/<platform>/<recipe>.log) and start the
fix → fmt → commit → retry loop immediately. Waiting IS wait_for_settle
re-invoked: an unsettled return at timeout means call it again — never park
the wait on a background shell watcher over .ci/<sha7>/runs/ or a
gh pr checks poll. A hand-guessed ledger predicate hangs silently, and a
watcher that only detects settle forfeits first-red. errored (vs failed)
means infrastructure death — node_rerun it rather than hunting a test bug.
pu misbehaves → log it on
juspay/kolu#1204 via
.apm/skills/ci/pu/diagnose.sh <stage> <host> (best-effort, never blocks the
run). Capture the stage's stderr:
pu create "$host" 2> >(tee /tmp/pu-$host.err >&2).
After a green settle, post e2e metrics to the PR. Source of
truth: the run ledger .ci/<sha7>/runs/<seq>.json for each ci::e2e@<platform>
node's recipe-wall durationMs and host (never Cucumber's internal timer), and
the LAST ^e2e: workers= line in that node's own log (an unsharded lane's is
.ci/<sha7>/<platform>/ci::e2e.log) for resolved parallelism. Maintain
one comment keyed by the
<!-- kolu-ci-e2e-metrics --> marker (edit if present, create if absent) with
the run identity <sha7>#<seq> and a table of
platform · host · workers · cores · cap · shards · e2e duration. Fail loud
rather than comment if any ingredient is missing. Post only after the full run
settles green, before reporting the gate.
ci::e2e is sharded, so the ledger carries shard nodes and the comment has
rows for them. [metadata("odu:shard=4")] lets odu borrow up to four free
pool slots on Linux; the slices are ledger nodes ci::e2e[k-of-N]@<platform>
(their burst dependency copies are ci::e2e[k-of-N]::ci::install and
…::ci::nix), and they fold into the ONE logical ci::e2e@<platform> node
whose duration is the slowest slice. So:
- the
shards column is obtained/ceiling (3/4) — read the obtained count
from the shard nodes present in the ledger, never assume four;
- add one row per shard (
host · workers · cores · duration) under the
logical row. A sharded ci::e2e.log holds only odu's fold bracket
([odu] N shards started…), so each shard's ^e2e: workers= and
e2e: shard=<k>/<N> lines come from its OWN
.ci/<sha7>/<platform>/ci::e2e[k-of-N].log;
- a darwin lane (only ever present when explicitly requested) has a
single-host pool:
1/4 and exactly one row;
node_rerun targets ONE shard node (ci::e2e[2-of-3]@x86_64-linux), not the
logical context: re-running the fold would re-run every slice.
The green-gate: every required status check green on the PR's current
HEAD. Source the required list from just ci::protect --dry-run — the
script pins x86_64-linux itself, so no --platform flag is needed (pass
--platform aarch64-darwin only on a run you were explicitly asked to take to
darwin). Verify with gh pr checks (a green from a retry counts).
Code-scanning alerts block merge too, and gh pr checks doesn't show them.
Query by PR number, never by branch ref — code scanning here is GitHub's
default setup, which analyses only the default branch and refs/pull/<n>/*, so
a ref=refs/heads/<branch> query answers "no alerts" about a ref nobody
scanned (measured on #2017: branch ref → 0, pr=2017 → an open HIGH):
gh api "repos/{owner}/{repo}/code-scanning/alerts?state=open&pr=<n>" \
--jq '.[] | "\(.rule.security_severity_level) \(.rule.id) \(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line)"'
Fix every alert on code this PR introduced; name (don't silently ignore) any
pre-existing alert on untouched code.