Tooling & analysis — recipes (loop tier)

On-demand recipes (26). Trigger→action heuristics; pull the relevant one when its situation arises.

1. Bypass the coworld/cogames CLI when it lags the server (pydantic ValidationError)

loop · negative result · ⚠ session-derived, unverified · tool: scripts/fetch_episodes.py

When the installed coworld/cogames CLI is older than the server, its episode/replay/membership commands crash with a pydantic ValidationError (Field required) before emitting any JSON (--json does not help: validation runs first, prints zero bytes, exits non-zero). Schema drift cause: client model V2EpisodeRequestRow still requires assignments/policy_version_ids/participants/scores but live rows omit assignments, and LeaguePolicyMembershipPublic requires is_active/is_champion while the server now returns a status field (e.g. 'qualifying'). Workaround: call a rooted raw REST endpoint such as /v2/policy-versions/{id}/episode-requests or /v2/rounds/{id}/episode-requests with a plain httpx client and parse dicts yourself, or use scripts/fetch_episodes.py which hits skew-immune raw JSON routes. _{sources: claude-code:0661ae70-2949-4d2e-8910-676f5a940d53, claude-code:0a2634a1-19b7-402e-841e-084ce80c015f, claude-code:25488cd9-61d5-4231-bd87-e2744b6db043}

2. Observatory REST chain to find one policy's episodes, with host/param gotchas

loop · ⚠ session-derived, unverified · tool: scripts/fetch_episodes.py

Current chain: resolve the policy version, then page /v2/policy-versions/{id}/episode-requests; for a tournament round, page /v2/rounds/{id}/episode-requests. These rooted endpoints return thin summaries. Load /v2/episode-requests/{id} only when full participants, scores, or game config are needed, then use /v2/episode-requests/{ereq}/{pv}/policy-logs/{slot} for one participant's stderr. The broad episode-request collection was removed; do not reconstruct policy or round history by scanning unrelated recent requests. Two hosts share the login token at different paths: softmax.auth.get_api_server() returns https://softmax.com/api, which serves the data API under /observatory, while https://api.observatory.softmax-research.net serves the same endpoints at its root. Web search will not find an internal game by name; use coworld download <name> and the game's GitHub repo. A reporter run (rrun_...) survives killing the local watcher; poll /v2/reporter-runs/{id}. Check that coworld bundle, make-policy, and episode-bundle exist in your CLI before depending on them. _{sources: claude-code:0661ae70-2949-4d2e-8910-676f5a940d53, claude-code:0a2634a1-19b7-402e-841e-084ce80c015f, claude-code:b952ba45-d32f-4ce0-a227-c691de8c588f, codex:019eaa2f-3bfd-73b3-aaec-4d709e098c39 (+6)}

3. The coworld CLI (not the web UI) is ground truth; key off the episode request ID, and beware scripting hazards

loop · negative result · tool: coworld

The Coworld CLI is the ground truth for debugging a hosted player. Use the public coworld CLI through uv run coworld and inspect local --help. Key off the episode request ID (ereq_...) for stats, results, logs, replays, and replay-open. Inspect a round through rounds, results, episodes --round ROUND_ID, then the selected request's logs and results. coworld episodes and coworld replays require exactly one --round or --policy. The server therefore applies the root before cursor pagination. Use full UUIDs in scripts. episode-logs --agent N names the WebSocket game slot, not the participant-list index. A wrong slot can return another policy's logs; use --mine or confirm the content. Downloaded logs can contain only a Kubernetes collection failure, so check their content before analysis. _{sources: players_checkouts/players/docs/coworld-integration-guide.md, players/users/james/personal_cogs/among_them/guided_bot/coworld/DC_DEB, archive/cogames_playground/BULBACOG_LEADERBOARD_GOAL.md, players/users/james/personal_cogs/among_them/guided_bot/coworld/INSPEC (+5)}

4. coworld/cogames CLI cheatsheet, verified against editable source not PyPI docs

loop · tool: coworld

Do NOT trust published/PyPI docs for the coworld/cogames CLI -- install it as a local editable install from the metta source and verify every claim, because published docs lag. Verified cheatsheet: episode-stats returns an EMPTY table (zero rows, Steps: -) for league episodes (the league pipeline does not populate EpisodeStatsResponse; can be empty even for COMPLETED episodes -> {game_stats:{},policy_stats:[],steps:null}) -- fall back to episode-results plus replays --download-dir index.json (participants, scores, episode/job IDs, seed, S3 replay URL). League replay URLs are public S3 (https://softmax-public.s3.amazonaws.com/replays/<job-id>.json.z; only the Mettascope viewer needs auth; treat any replay URL as world-readable); league episodes route in-cluster via k8s DNS (ws://job-<jobid>-game:8080/...). pools --round returns empty for completed rounds -- fish a pool_id out of any episode JSON and call pools <pool_id>. events is ~99/100 round_completed, so filter --event-type league_policy_membership_placed; submissions --player <ply_id> is the cleanest rejected-submission history. replay-open --hosted is disabled (HTTP 409 from POST /v2/coworlds/replays/session; COWORLD_REPLAY_SESSIONS_ENABLED defaults false) -- use local replay-open. To run against real-tournament rules without a full submission, use cogames assay submit <policy> (--source mission set, --episodes, --max-steps, -w/--watch) tracked by assay status/results/list -- distinct from local-only cogames scrimmage/run/pickup. Flag trap: -s means --steps (max steps), NOT seed; seed is the long-form --seed only. Attributing results to a policy is costly: episode-results reports opaque PlayerN slot numbers; recover the mapping from the episode-detail JSON's assignments array (participant-index -> slot, positions non-contiguous e.g. 0,1,2,3,6,7,8,9) plus log-filename correlation -- there is no per-policy aggregate-stats command, so listing episodes + downloading logs + per-episode results + manual aggregation is ~100 API calls per round; script it. _{sources: claude-code:67ff1fe3-48c0-4738-b7fb-8b994e567434, archive/cogames_playground/docs/coworld_cli.md, claude-code:cd2e0b52-b541-4e03-9fd3-798db4591603, claude-code:b1fb3772-05cf-4de7-b1d7-a0732e2a61e3 (+3)}

5. Discover league episodes and resolve replay IDs across disjoint Coworld populations

loop · tool: coworld

Discover tournament episodes through their stable root: coworld episodes --round <round_id> or coworld episodes --policy <name>:vN. coworld replay-open takes one episode request ID (ereq_...), not an experience-request or batch ID. The --download-dir short alias differs across commands (replays uses -o; episode-logs uses -d), so prefer the long form in scripts. When docs, commands, runtime behavior, logs, or replays disagree, preserve the evidence and file a GitHub issue with the command, IDs, logs or replay links, and smallest repro. Semantic recall tools require indexed embeddings; if empty, run bunx tsx scripts/backfill-embeddings.ts and treat recall as best-effort. _{sources: personal_labs/crewrift_lab/best_practices.md, coworlds/coworld-crewrift/README.md, personal_labs/crewrift_lab/lessons_archive/TENTATIVE_LESSONS-20260612-, personal_labs/crewrift_lab/docs/crewrift-replays.md (+3)}

6. Choose the right Coworld run mode: download, run-episode, play, certify

loop · tool: coworld

Use 'coworld download <slug-or-cow_id> --output-dir <dir>' (flag is --output-dir/-o NOT --output; default ./coworld, override COAGENT_COWORLD_DIR; --refresh re-pulls) -- it writes the manifest, images json, and AGENTS.md and re-tags baseline images runnable; first download pulls the game's Docker image (hundreds of MB, cached/idempotent, not stuck) then runs offline (the downloader resumes after a read-timeout when rerun with the same -n N). Pass the bare coworld-id to 'coworld play' or 'coworld run-episode'. run-episode is headless and writes artifacts but always runs only the manifest's certification.game_config smoke test (e.g. a 3-step cogsguard fixture), NOT the league mission, and has no --variant; 'coworld play' runs the actual league variant (e.g. CvC machina-1-daily, 10000 steps) for the live browser viewer; to smoke a fuller game headlessly pass an episode_request.json whose game_config is the variant. In run-episode the positional image is reused for ALL slots (self-play) -- pass one image per slot to mix; there is no single-slot override and reference players are used only with zero images. coworld certify does NOT accept player_images (cannot substitute a policy); use run-episode for the strictest pre-flight (asserts client/WS reachability, token rejection, non-empty first global message). 'coworld play' produces only an uncompressed replay.json while run-episode/certify produce BOTH replay.json and replay.json.z (account for it when a downstream tool expects compressed). Prefer stable cow_/cow_id ids over names for reproducibility. Expect version drift (coworld download crewrift may pull 0.1.45 vs a doc's 0.1.40). _{sources: coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/downloa, player_labs/.claude/skills/coworld-local-run/SKILL.md, archive/cogames_playground/docs/coworld_cli.md, players_checkouts/players/users/james/personal_cogs/among_them/README. (+1)}

7. Coworld auth lives in its own credential store; probe with a read call

loop · tool: coworld

Authenticate to the Coworld Observatory backend once with 'uv run softmax login'; there is no whoami, so probe auth with any auth-requiring read like 'coworld leagues'. A sudden failure on the SECOND API message (package clearly installed) means the token expired (401/403), not a missing package. The 'coworld' CLI authenticates from its OWN store (~/.softmax/credentials.yaml) and does NOT read the standalone 'softmax' CLI's token store, so 'softmax set-token'/'status' do not affect coworld; if using 'softmax login' use the softmax binary bundled with coworld, not a separate install. To act as a specific player, mint a player-scoped token via POST /observatory/players/{player_id}/login (requires a USER token, no body) and save it as coworld's active player session; load_current_token returns the player_session if present else the user token, so deleting the session reverts to the default-player user token. 'coworld download' by canonical name always pays an auth round-trip even on cache hit (name->ID resolves before the cache check); download/play by cow_<id> is auth-free on cache miss. _{sources: archive/cogames_playground/docs/coworld_cli.md, players_checkouts/players_1/docs/coworld-player-swap.md}

8. Hosted-game sessions leave no trail; capture docker logs before docker rm

loop · negative result

coworld hosted-game sessions leave no discoverable trail: after a hosted run, episodes --mine and replays --mine return nothing and final scores never reach the player container. Capture per-slot 'docker logs' BEFORE 'docker rm', or use 'coworld play' (which writes a re-watchable replay.json) for records. _{sources: claude-code:67ff1fe3-48c0-4738-b7fb-8b994e567434, claude-code:cd2e0b52-b541-4e03-9fd3-798db4591603, archive/cogames_playground/docs/coworld_cli.md}

9. Read persisted episode artifacts (.optimizer-artifacts), and know coworld play vs run-episode layout

loop · ⚠ session-derived, unverified

The live Coworld episode replay lives in a system temp dir (e.g. /private/var/folders/.../T/coworld-ep-<id>-<rand>/replay/) OUTSIDE the optimizer working dir, so ls/file is blocked by the sandbox -- read the persisted artifacts under .optimizer-artifacts/episodes/ep-<id>/ (results.json, config.json, replay.json, run_parameters.json, coworld_manifest.run.json) by full path instead of listing the temp dir. coworld play injects COGAMES_ENGINE_WS_URL=ws://host.docker.internal:<port>/player?slot=<i>&token=<t> per container, exposes /healthz, and writes results.json/replay.json/per-slot logs into an EpisodeArtifacts temp workspace. coworld run-episode is the headless sibling (no browser; --output-dir defaults to ./coworld-episode-results, default --timeout-seconds 3600 vs 60) and writes directly into that dir with NO per-run subdirectory (the prefix='coworld-run-' arg is dead code), producing BOTH replay.json (large, uncompressed) and replay.json.z (small, zlib) -- do not assume only the compressed one exists. _{sources: claude-code:38ddef93-3214-489f-a178-33dddf030d9e, claude-code:4c0f63b0-f7eb-408e-ac3f-00feb1dcc193, claude-code:8ec27261-9517-4365-944b-2880873bda47, claude-code:394cc2af-72e4-41bf-861b-9b245d819229 (+1)}

10. Read game results from saved replays/traces, not stdout; verify format by magic bytes and enable -v talk

loop · ⚠ session-derived, unverified

Never judge a live coworld match from stdout: once a match ticks the console goes quiet by design and the real signal (LLM decisions, chat, votes) lives in JSONL/replay files -- always pass --save-replay-file/--save-replay-dir, wait for a clean exit, then parse. Saved replays may be zlib-compressed (.json.z) so decompress before grep; files can carry a misleading extension (e.g. coworld 'replays' get .json but are .bitreplay starting with ASCII magic, here CREWRIFT) so hexdump the magic bytes rather than trusting the extension. Some text channels are off by default (for crewrift's machina_1.clips add -v talk to record role/target/heart claims). coworld replay footguns: pass --no-open-browser in headless shells; 'coworld replay' writes a tmp/coworld-replay-* dir under the CURRENT repo even for /tmp inputs (clean it up). _{sources: codex:019e14de-cdca-7482-88fa-81716ee830ae, codex:019eaa2f-3bfd-73b3-aaec-4d709e098c39, codex:019e1591-004e-7040-b768-062bd8e6acd4, codex:019e1579-a737-7b70-bf0e-172ff04307cf (+2)}

11. Histogram trace events first, and harden jq against non-JSON lines and role-gated absence

loop

Before querying a game-policy JSONL trace for whether an event happened, run an event-type HISTOGRAM first (jq select kind=='trace', project .event, sort | uniq -c) so an empty later query reads as 'that event did not occur' rather than 'wrong query/file/not captured.' Trace defaults drift (per-tick decision_snapshot/metrics move off by default) and role/phase-gating makes 'missing' normal (e.g. crewrift: a crewmate episode has no kill_* events, an imposter has no suspicion_snapshot, role may read 'dead' after death). When piping a hosted log to jq, prefilter with grep '^{' (final line is plain text, collector-error lines interleave) and guard select(.kind=='trace') / (.event//'') before test() because metric lines carry metric_kind/name/value/tags with no event field. Engines write PREFIXED event names (crewborg: domain.meeting_*, not bare meeting_llm_decision) -- grep the names ACTUALLY emitted, not names read in mode code. _{sources: personal_labs/crewrift_lab/crewrift/crewborg/docs/trace-logs.md, personal_labs/crewrift_lab/best_practices.md, personal_labs/crewrift_lab/docs/crewrift-replays.md}

12. Emit merged stdout+stderr per tick; the per-slot log is the runtime source of truth

loop

Per-agent logs (logs/policy_agent_{N}.log / policy_agent_{slot}.txt) are the richest behavioral record of a Coworld episode and the runtime source of truth when debugging, while results.json and the episode row drive scoring. Emit diagnostics to stdout/stderr (not stderr-only) and flush each tick or set PYTHONUNBUFFERED=1, because the runner captures BOTH streams merged into the per-slot log -- the only channel by which player-side traces survive for retrieval via 'coworld episode-logs ereq_... --agent <slot>'. This merged log is marked diagnostic-only with NO header/prefix/separator -- do not assume a line format, and do NOT filter out 'INFO -' lines as noise (they are usually the agent's real Python logging); the CLI's own bracketed console.print messages can be mistaken for captured lines. For guided_bot, structured JSONL trace streams surface in hosted logs only when GUIDED_BOT_TRACE_DIR=stderr and GUIDED_BOT_TRACE_LEVEL is non-off (off/events/decisions/full); absent trace lines means env vars are unset, not a connection failure. Never dump raw per-frame perception into hosted logs -- use summarized diagnostics. Make artifact downloads best-effort and idempotent: record a missing replay/log rather than aborting, and skip already-complete episode dirs unless forced. Treat OFFLINE developer tools (asset bakers, nav baker, waypoint/frame editors, frame-to-text dumper) as exempt from any Coworld-only run restriction because they only produce or inspect checked-in baked artifacts and recorded traces; never conflate them with a live raw-frame-capture run path. _{sources: player_labs/.claude/skills/coworld-episode-artifacts/SKILL.md, players_checkouts/players/users/james/personal_cogs/MISSION.md, players_checkouts/players/users/james/personal_cogs/AGENTS.md, players_checkouts/players/docs/coworld-integration-guide.md (+5)}

13. Design player tracing as tiered structured JSONL with a lean hosted-cap default; emit domain events post-step

loop

Design a player's tracing as tiered structured JSONL (OFF < EVENTS < DECISIONS < VERBOSE/FULL, each a superset, emitted as category+data+level so analysis is machine-parseable not free-text scraping), with a lean DEFAULT that fits under hosted log caps. By default emit only durable low-volume DOMAIN events (meeting context, vote cast, chat sent/received, phase/role/task changes, high-level actions) and drop per-tick payloads (decision snapshots, belief/suspicion ticks, framework-boundary traces, viewer frames, metrics) behind an opt-in debug/viewer flag -- a decision_snapshot per-tick object is usually the single biggest volume contributor (one session produced ~380k trace lines across three games; raw frame dumps reach ~63 MB / 4000 frames; emit those only at 'full'). Gate ALL instrumentation behind env vars (output dir + verbosity) with a single early if trace == nil: return guard so every log call is near-zero cost when off, and gate expensive instrumentation behind the verbosity check BEFORE computing it. Emit game-level domain events from an on-step-complete hook rather than inside agent 'modes', because attempt events key on the produced wire command (modes never see it) and outcomes like task-completion conclude only after the mode runs; post-step emission keeps the strategy layer pure and reads off the finalized belief. Route all traces through ONE sink (full for local/tests; an env-aware factory returning the filtered/metrics-off sink by default in the hosted bridge) plus a bridge-level test that fails if the hosted path starts sending metrics again. Provide env knobs (CREWBORG_TRACE_GROUPS=decision|voting|action, CREWBORG_TRACE_INCLUDE/EXCLUDE globs, CREWBORG_TRACE_DECISION_FIELDS, CREWBORG_METRICS=1) to capture one event family without full volume. EXCEPTION: promote the COMPACT decision geometry (mode, target color, target distance/position) into the always-on lean stream (crewborg's decision_snapshot in events.py) so thrash/flee bugs are diagnosable without resimulating; keep the FULL per-tick payload debug-only. Keep policy logic clean by splitting the decision into a pure core (decideNextMaskCore) plus a thin trace-owning wrapper (decideNextMask); never export an internal field just so a call site can derive a value. Decouple analysis from behavior: emit analysis-only annotations (vote_bandwagon_detected, goal-tier) into decisions.jsonl read by NO policy code. Log the FINAL emitted action mask, not only upstream intent (a trace logging only intent blocked a navigation diagnosis; '535 identical decisions, 2 transitions, 0 events, 0 LLM calls' localized it in minutes), treat perception inputs (e.g. chat OCR) as first-class trace events, and log every LLM round trip with raw_response, prompt/response token counts, latency, validation status, published directive. For a deduction agent the single highest-value forensic record is one 'suspicion snapshot' at the start of every meeting carrying ranked posteriors, each suspect's event log, the would-be vote, and the decision bar. Make it purely additive over a green suite (keep the old Callable[[str],None] interface and exact legacy strings; emit structured entries only when a real Logger is passed; inject clock+sink for deterministic tests), each controlled slot writes its OWN trace file, and read the existing struct/enum defs and trace writer FIRST so field names and phase/role/mode stringifies match. _{sources: claude-code:3c867a71-3057-4e87-a149-4185e82ee728, auggie:ac491a46-e7f7-4271-a1c0-f0e69987d424, codex:019e0383-1db4-7f90-a085-04e6f27b26e5, codex:019ea9af-7020-7550-b427-d3478fed2057 (+13)}

14. Prove an optional feature engaged before tuning it; absence of success is ambiguous

loop · negative result · ⚠ session-derived, unverified

When an LLM-augmented (or any optional) feature underperforms, FIRST prove it actually fired before tuning prompts or strategy -- absence of success events is ambiguous between 'feature off', 'tried and failed', and 'precondition never met'. Concrete diagnosis: (1) check the trace CHANNEL exists at all (a [trace:guidance] channel carrying snapshot_sent/llm_response/directive_published absent, with every decision showing directive_source 'reflex'/'default' and never 'llm', means the path never engaged); (2) grep the ACTUALLY emitted prefixed event names, not names read in mode code (a scan for meeting_llm_decision found zero while logs used domain.meeting_* prefixes); (3) confirm the precondition occurred (a meeting actually happened) before blaming the feature. A single llm_call_failed reason=no_key meant the guidance layer never ran the whole match and the bot ran purely on its symbolic fallback. Distinguish decision-layer from execution-layer failure: count LLM responses/directives separately from meeting-actions-received and actual button presses/vote_attempted -- a high response count with few meeting actions (or ~39% noop, most 'skip' decisions never reaching vote_attempted) means the model decided right but the action never landed, so chase the action loop not the model. Make engagement greppable with a one-shot startup trace (e.g. llm_init once per match) recording provider selected and credential-presence signals, self-contained so it survives ring-buffer eviction; and verify the full prompt/snapshot payload is actually persisted (a design doc referenced snapshot_sent but live runs emitted none), not just the response. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, claude-code:d48f9347-7884-4ea2-8659-230051d86410, codex:019ea93a-43b0-7800-b34e-7b3053c46177, codex:019e1593-c00d-7a32-9dfa-1cf6b16b01a4 (+1)}

15. Debug the trace, not the outcome or the on-screen moment; re-run at the right log level; trust server-confirmed over agent-side menu completion

loop

You cannot debug an outcome, only a trace: pivot immediately from a win/loss to the player's internal reasoning stream -- the gap between what was objectively true and what the policy believed/chose is where the mechanism lives. Visible behavior is an effect with a long causal delay, so trust the trace tick over the on-screen moment: a visible stop-and-turn at tick ~390 was actually caused by a target-commitment timer expiring at tick 319, the body bleeding momentum over ~70 ticks. The default log level often omits the decision telemetry you need (early analysis had only events-level logs and lacked meta_decide_reason, emitted only at decisions level), so re-run at the verbose/decisions level and reproduce on a fixed seed before forming root-cause hypotheses. A server/global log of 'X happened' is NOT proof the agent EXPERIENCED X: server 'opened whisper' events looked like success but the agent's own frames never showed the whisper view (events belonged to other players or a player-index mismatch), and in persephone a whisper_exchange_outcome with server_confirmed=false only means the agent completed/timed-out a menu sequence, NOT that a win-relevant exchange happened -- only server_confirmed=true or the authoritative server full.log ('joined','offered role','shared roles','winner') proves a real exchange; when agent-side perception and server state disagree (client still seeing PLAYING/overworld pixels while the server's whisper state was active), debug the transport/timing path rather than trusting that menu completion implies a server effect. Read game progress from the right channel: a player's spatial-memory tick= lines stop updating once local territory stops being newly observed and make a run look stalled -- read the game tick from declarative-memory lines instead. _{sources: claude-code:3c867a71-3057-4e87-a149-4185e82ee728, claude-code:3c13aeb0-5617-452d-a8d7-aff376e3d4ff, player_labs/best_practices.md, personal_labs/crewrift_lab/.claude/skills/crewrift-diagnose/SKILL.md (+3)}

16. Use per-agent action histograms and mode churn as cheap stuck-detectors, decoded against tick rate

loop · ⚠ session-derived, unverified

A healthy match shows many distinct action types (~18) with movement/interaction spread; a distribution collapsed onto one action -- especially noop (a bot pinned on a wall shows huge noop plus one directional button) -- flags stuck/idle/crashed behavior. CAVEAT: subtract legitimately-idle frames first; raw NOOP looked ~20% but interstitial/voting frames are noop by design and true gameplay NOOPs were ~0.4%. Compare per-bot counts before/after a fix to confirm the bot actually moves. A flat modes log (one mode at setup, never transitioning) means transition triggers are unwired. A regular fixed-period idle cycle (a repeating ~72-tick = 3s @ 24Hz DisciplineNoOp loop) is a scan/cooldown timer firing with no actionable condition -- convert tick spans to seconds and match against game constants (the 72-tick span did NOT match the 1200-tick kill cooldown, ruling it out). But interpret in light of role speed: low-speed agents rarely trigger a COAST branch because a path-following deadband catches arrivals first -- few coast events is not a bug. Diagnose a stuck crewmate from action traces by a long single-task dwell dominated by ONE repeated d-pad direction: a slot spent 984 consecutive ticks (~41s) on one complete_task, 837 monotonic RIGHT and 72 ticks of A -- a wall-wedge against impassable geometry (the task only 'completed' ~200 ticks later because ghosts slide through walls). Add stuck-detection/replan guards triggered by such monotonic-direction dwell. _{sources: opencode:ses_1fb1d9989ffeYXiew23WUqo2hw, opencode:ses_201637e3bffeRa6Y0x3DOs8hh1, opencode:ses_21b273d4fffee0T9oMpvlktzkg, opencode:ses_20422c08ffferRLvWnNE5DeB4w (+3)}

17. Distinguish a kicked slot from a finished one via game_over and a known-good opponent

loop · ⚠ session-derived, unverified

To tell whether a player slot ended naturally or was cut mid-game, diff it against a known-good full-game opponent slot in the SAME episode on frame/tick count and end-event: a healthy slot processed ~4841 frames over ~200s and saw game_over, while a cut player closed at ~880 frames having never seen game-over -- 'did this slot receive game_over?' is a clean kicked-vs-finished test. A player 'disappearing early' in a replay has at least three viewer-identical causes (network disconnect, freeze-in-place, meeting/body teleport), so cross-check the server-side result (stayed alive? won?) against the bot's perception trace before concluding elimination or disconnect. _{sources: claude-code:5f9e8287-da92-4dff-bf39-2511bcaf0700}

18. Triage episodes by failure CLASS and a quantitative fingerprint before reading events

loop · ⚠ session-derived, unverified

Triage low-score episodes by failure CLASS before coding, not by eyeballing logs. Recurring score buckets map to root causes: zero-score = never left the interstitial/unknown-role screen or died early as crew; flat-low = tasks done but no win (a strategic-conversion problem, not a patchable bug); stuck-movement = alive but world position frozen. Only mechanical failures (stuck navigation, missed votes, slow first task) are cheap root-cause patches; do not ship a code fix for a strategic-conversion or one-off failure. Get a fast quantitative fingerprint first: (1) group and count trace event kinds (task_started/completed/abandoned, vent_suspected, mode transitions) plus the gameplay-vs-interstitial phase split; (2) read the end-of-episode action-mix histogram (interact + diagonal moves reads as normal task-running; mostly noop or one stuck direction flags a navigation/decision bug); for LLM/meeting matches compute a scripted rollup (roles, outcome, meeting/chat counts, vote targets, LLM validation failures, winning side) first -- counting events tells you whether the run is worth a deep read. Reduce raw episode JSON dumps to your-player-only scores plus low-score request IDs before analysis, then download logs only for the worst/representative cases; raw CLI dumps of thousands of JSON lines are too large to reason about and not actionable. _{sources: codex:019e4c28-d516-7ca0-b145-153c152b4f43, claude-code:dfa4502e-c404-40e8-bf86-8d15c72ba5af, codex:019e1591-004e-7040-b768-062bd8e6acd4}

19. Pull competitors' logs from the same episode; keep opponent scouting separate from self-analysis

loop

Pull and inspect OTHER competitors' agent logs from the same episode to learn opponent behavior and calibrate debugging: cross-agent log SIZES alone are diagnostic (one opponent emitted 7000+ lines while others emitted near-zero; grepping all logs for chat markers showed only one bot ever chatted) -- a cheap way to see which opponents are sophisticated vs inert and whether the chat channel is used in the league. Keep opponent scouting separate from self-analysis: an opponent-modeling pass studies THEIR tells, biases, predictable openings, target selection, voting behavior, and panic conditions and must never record our own policy's mistakes; self-failure analysis is a different pass. Distinguish subject from opponent slots via per-player attribution (e.g. lib/episodes/players.ts: each slot carries role 'subject' vs 'opponent', slot index, name, policyLabel, policyId/libraryId, source), filtering to opponent-role slots first. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/explo}

20. Autonomously escalate tracing when it is a config knob; emit JSON+Markdown

loop

If a policy's logs are too thin to find the mechanism, autonomously escalate tracing when it is just a config knob (turn up the trace level, re-run the experience request, re-pull artifacts, re-report, resume) WITHOUT a human gate; only flag the human when getting the signal requires a code change to the player's tracing -- which is itself a finding. Emit analysis as both human-readable Markdown and a machine-readable JSON block (e.g. {"score":6,"rolling_avg_5":5.4,"trend":"improving","bottleneck":"...","escalation":"none"}) so downstream automation consumes results without parsing prose. _{sources: personal_labs/crewrift_lab/.claude/skills/crewrift-diagnose/SKILL.md, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/analy}

21. Run evaluation headless with explicit seed; fail loud on unparseable scores

loop

Pass -r none on every headless cogames play eval run: rendering costs ~60s of a 62s machina_1 500-step run (~2.5s off), roughly a 25x speedup (~1 min vs ~25 min for a 12-seed A/B); reserve GUI for debugging one episode. Always pass an explicit --seed since omitting it picks a non-default seed and hand-repro will not match harness scores. Because the A/B harness scrapes scores from the end-of-episode table via regex (no structured CLI output), it MUST emit a 'could not parse' error rather than silently producing bad numbers, and re-verify the regex on every cogames major bump. _{sources: alpha_cog/experiments/README.md, archive/cogames_playground/alpha_cog/experiments/notebook/protocol.md}

22. Diagnose scripted cogsguard programmatically with DebugHarness, not log greps

loop · tool: DebugHarness

Diagnose a scripted cogsguard policy with DebugHarness rather than only grepping logs: build from a recipe (DebugHarness.from_recipe(num_agents, seed)), step the sim, then call diagnose_stuck_agents(threshold), diagnose_coordinate_system(), and get_agent_state(id) to inspect role/phase/position -- faster root-cause access than greps. To run headless: ./tools/run.py recipes.experiment.cogsguard.play policy_uri=role render=log max_steps=100, piping stderr+stdout through grep on the per-agent prefix (e.g. ^\[A2\]) to trace one agent. Verify cogs_vs_clips outcomes from the end-of-episode game state: a rising cogs aligned.hub count confirms aligners, miner deposits show resource flow, a decreasing enemy junction-alignment count confirms the scrambler. _{sources: players/cogsguard/role/AGENTS.md, players_checkouts/players/players/cogsguard/role/CLAUDE.md}

23. Kill subprocesses on every failure path, match the player roster, and reap children robustly

loop · ⚠ session-derived, unverified

A multi-player game won't start until the required player count is present, so a single agent against a full-roster server hangs waiting -- match the spawned-bot count and use the project's multi-bot launcher (which traps Ctrl-C and reaps children) rather than backgrounding bots by hand. Kill temporary server/opponent/agent subprocesses on EVERY failure path or a broken run leaves orphans blocking later runs; a one-off smoke episode driven via the backend function (node --import tsx) has no TTY so Ctrl-C can't reach it -- kill node/tsx and its Coworld child by PID and docker stop the orphaned Crewrift container. In shell harnesses: use exec in the server subshell so $! is the real pid, add a re-entrancy guard on the shutdown trap (SIGINT and EXIT both fire teardown), and do TERM -> poll kill -0 to a timeout -> KILL -> verify -> wait (a flat sleep 1 between TERM and KILL is unreliable). For experience/episode requests with no cancel endpoint, delete the underlying k8s jobs AND the per-player child pods by job-id label (deleting parents doesn't remove children). Scan launch stdout for 'error'/'unknown option' -- a '--seed' error meant an unsupported CLI arg. _{sources: codex:019e94d1-e329-7590-b6b2-bfac0eef9950, codex:019e1940-5a75-7f51-b057-63f894ddb061, codex:019ea9af-7020-7550-b427-d3478fed2057, opencode:ses_21a0d30b6ffewSKOVSUyO4lZkU (+2)}

24. Use Chebyshev distance and ownership-flip windows for grid-game metrics; for cogs-vs-clips score on junctions HELD not resources

loop · ⚠ session-derived, unverified

Domain metric definitions for grid games (paint_arena): use Chebyshev (king-move) distance, not Euclidean, for grid-tile proximity events ('close proximity' = Chebyshev distance <= 2); and operationalize a 'highlight' as contested territory -- a tile changing owner >= K times within a W-tick sliding window (tune K and W per game) -- because the meaningful signal is repeated ownership flips on the same tile, not single captures. When evaluating a cogs-versus-clips player, do NOT trust per-resource withdrawn counts as success: an episode can show cogs/heart.withdrawn ~5 while per_episode_per_policy_avg_rewards is 0.0 and cogs/aligned.junction.held is 0.0 -- withdrawing hearts is not holding junctions, which is what scores. A near-zero cogs/aligned.junction.held while clips/aligned.junction.held is large and reward is 0.0 is the dominated-team signature and the primary failure alarm. Distinguish CAPTURE from RETENTION: on a full 10k-tick seed-42 run Cogs gained 111 junctions but the opponent held far more junction-ticks -- 'junctions gained' can look healthy while 'junction-ticks held' is losing, so measure held-ticks to surface late-game defense failure. _{sources: claude-code:2e0ca80b-d747-4358-a5dc-650397742760, claude-code:36cee047-63f0-4c81-b5e6-8c24ec57afba, codex:019e19a7-8605-7992-9b28-cb4d2ca9132e}

25. Self-evaluate against the public Cue-n-Woo judge worker; batch and cache

loop · ⚠ session-derived, unverified

The Cue-n-Woo judge worker (https://cue-n-woo-worker.softmax-research.net, Gemma-2-9b-it on an L4) is publicly callable unsigned with no VPN/auth, so you can self-evaluate against the real scorer locally with no league. It is slow and shared (~1 generation per 10s at 128 tokens), so batch requests and cache generations rather than calling it per-candidate at runtime. _{sources: claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b}

26. Enumerate coworlds org-wide by combining local manifests with a GitHub code search, excluding schema/template files

loop · negative result

To enumerate ALL coworlds org-wide, don't rely only on local clones (a biased subset): combine a local manifest scan with a GitHub org-wide code search for coworld_manifest*.json, then shallow-clone the missing repos into /private/tmp and parse their manifests rather than trusting search snippets. A broad glob also matches the Metta schema/template file, producing a false 'missing grader' hit, so narrow validation to actual world manifests and exclude schema/template files. A diffable baseline source tree is resolved automatically during the games sync by trying candidates in order: (1) the Metta-AI/coworld-<gameSlug> GitHub repo under players/<playerId> (default branch via the GitHub API), (2) manifest player/game-source doc pages, (3) other manifest doc URLs pointing at a coworld-* repo, and (4) a structured artifact fallback (baseline.json + README.md) that is always non-empty; private repos authenticate via GITHUB_TOKEN/GH_TOKEN or a logged-in gh CLI. _{sources: codex:019e746d-062c-78d3-a0ed-ef2fcb892baf, coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/downloa}