Tooling & analysis — recipes

On-demand recipes (39). Each is a trigger→action heuristic; pull the relevant one when its situation arises. Tier tags and ⚠ (unverified) as in AGENTS.md.

1. Bypass the coworld/cogames CLI when it lags the server (pydantic ValidationError)

loop · negative result · ⚠ session-derived, unverified · tool: scripts/fetch_episodes.py

When the installed coworld/cogames CLI is older than the server, its episode/replay/membership commands crash with a pydantic ValidationError (Field required) before emitting any JSON (--json does not help: validation runs first, prints zero bytes, exits non-zero). Schema drift cause: client model V2EpisodeRequestRow still requires assignments/policy_version_ids/participants/scores but live rows omit assignments, and LeaguePolicyMembershipPublic requires is_active/is_champion while the server now returns a status field (e.g. 'qualifying'). Workaround: call a rooted raw REST endpoint such as /v2/policy-versions/{id}/episode-requests or /v2/rounds/{id}/episode-requests with a plain httpx client and parse dicts yourself, or use scripts/fetch_episodes.py which hits skew-immune raw JSON routes. _{sources: claude-code:0661ae70-2949-4d2e-8910-676f5a940d53, claude-code:0a2634a1-19b7-402e-841e-084ce80c015f, claude-code:25488cd9-61d5-4231-bd87-e2744b6db043}

2. Observatory REST chain to find one policy's episodes, with host/param gotchas

loop · ⚠ session-derived, unverified · tool: scripts/fetch_episodes.py

Current chain: resolve the policy version, then page /v2/policy-versions/{id}/episode-requests; for a tournament round, page /v2/rounds/{id}/episode-requests. These rooted endpoints return thin summaries. Load /v2/episode-requests/{id} only when full participants, scores, or game config are needed, then use /v2/episode-requests/{ereq}/{pv}/policy-logs/{slot} for one participant's stderr. The broad episode-request collection was removed; do not reconstruct policy or round history by scanning unrelated recent requests. Two hosts share the login token at different paths: softmax.auth.get_api_server() returns https://softmax.com/api, which serves the data API under /observatory, while https://api.observatory.softmax-research.net serves the same endpoints at its root. Web search will not find an internal game by name; use coworld download <name> and the game's GitHub repo. A reporter run (rrun_...) survives killing the local watcher; poll /v2/reporter-runs/{id}. Check that coworld bundle, make-policy, and episode-bundle exist in your CLI before depending on them. _{sources: claude-code:0661ae70-2949-4d2e-8910-676f5a940d53, claude-code:0a2634a1-19b7-402e-841e-084ce80c015f, claude-code:b952ba45-d32f-4ce0-a227-c691de8c588f, codex:019eaa2f-3bfd-73b3-aaec-4d709e098c39 (+6)}

3. The coworld CLI (not the web UI) is ground truth; key off the episode request ID, and beware scripting hazards

loop · negative result · tool: coworld

The Coworld CLI is the ground truth for debugging a hosted player. Use the public coworld CLI through uv run coworld and inspect local --help. Key off the episode request ID (ereq_...) for stats, results, logs, replays, and replay-open. Inspect a round through rounds, results, episodes --round ROUND_ID, then the selected request's logs and results. coworld episodes and coworld replays require exactly one --round or --policy. The server therefore applies the root before cursor pagination. Use full UUIDs in scripts. episode-logs --agent N names the WebSocket game slot, not the participant-list index. A wrong slot can return another policy's logs; use --mine or confirm the content. Downloaded logs can contain only a Kubernetes collection failure, so check their content before analysis. _{sources: players_checkouts/players/docs/coworld-integration-guide.md, players/users/james/personal_cogs/among_them/guided_bot/coworld/DC_DEB, archive/cogames_playground/BULBACOG_LEADERBOARD_GOAL.md, players/users/james/personal_cogs/among_them/guided_bot/coworld/INSPEC (+5)}

4. coworld/cogames CLI cheatsheet, verified against editable source not PyPI docs

loop · tool: coworld

Do NOT trust published/PyPI docs for the coworld/cogames CLI -- install it as a local editable install from the metta source and verify every claim, because published docs lag. Verified cheatsheet: episode-stats returns an EMPTY table (zero rows, Steps: -) for league episodes (the league pipeline does not populate EpisodeStatsResponse; can be empty even for COMPLETED episodes -> {game_stats:{},policy_stats:[],steps:null}) -- fall back to episode-results plus replays --download-dir index.json (participants, scores, episode/job IDs, seed, S3 replay URL). League replay URLs are public S3 (https://softmax-public.s3.amazonaws.com/replays/<job-id>.json.z; only the Mettascope viewer needs auth; treat any replay URL as world-readable); league episodes route in-cluster via k8s DNS (ws://job-<jobid>-game:8080/...). pools --round returns empty for completed rounds -- fish a pool_id out of any episode JSON and call pools <pool_id>. events is ~99/100 round_completed, so filter --event-type league_policy_membership_placed; submissions --player <ply_id> is the cleanest rejected-submission history. replay-open --hosted is disabled (HTTP 409 from POST /v2/coworlds/replays/session; COWORLD_REPLAY_SESSIONS_ENABLED defaults false) -- use local replay-open. To run against real-tournament rules without a full submission, use cogames assay submit <policy> (--source mission set, --episodes, --max-steps, -w/--watch) tracked by assay status/results/list -- distinct from local-only cogames scrimmage/run/pickup. Flag trap: -s means --steps (max steps), NOT seed; seed is the long-form --seed only. Attributing results to a policy is costly: episode-results reports opaque PlayerN slot numbers; recover the mapping from the episode-detail JSON's assignments array (participant-index -> slot, positions non-contiguous e.g. 0,1,2,3,6,7,8,9) plus log-filename correlation -- there is no per-policy aggregate-stats command, so listing episodes + downloading logs + per-episode results + manual aggregation is ~100 API calls per round; script it. _{sources: claude-code:67ff1fe3-48c0-4738-b7fb-8b994e567434, archive/cogames_playground/docs/coworld_cli.md, claude-code:cd2e0b52-b541-4e03-9fd3-798db4591603, claude-code:b1fb3772-05cf-4de7-b1d7-a0732e2a61e3 (+3)}

5. Discover league episodes and resolve replay IDs across disjoint Coworld populations

crewrift · tool: coworld

Discover tournament episodes through their stable root: coworld episodes --round <round_id> or coworld episodes --policy <name>:vN. coworld replay-open takes one episode request ID (ereq_...), not an experience-request or batch ID. The --download-dir short alias differs across commands (replays uses -o; episode-logs uses -d), so prefer the long form in scripts. When docs, commands, runtime behavior, logs, or replays disagree, preserve the evidence and file a GitHub issue with the command, IDs, logs or replay links, and smallest repro. Semantic recall tools require indexed embeddings; if empty, run bunx tsx scripts/backfill-embeddings.ts and treat recall as best-effort. _{sources: personal_labs/crewrift_lab/best_practices.md, coworlds/coworld-crewrift/README.md, personal_labs/crewrift_lab/lessons_archive/TENTATIVE_LESSONS-20260612-, personal_labs/crewrift_lab/docs/crewrift-replays.md (+3)}

6. Choose the right Coworld run mode: download, run-episode, play, certify

loop · tool: coworld

Use 'coworld download <slug-or-cow_id> --output-dir <dir>' (flag is --output-dir/-o NOT --output; default ./coworld, override COAGENT_COWORLD_DIR; --refresh re-pulls) -- it writes the manifest, images json, and AGENTS.md and re-tags baseline images runnable; first download pulls the game's Docker image (hundreds of MB, cached/idempotent, not stuck) then runs offline (the downloader resumes after a read-timeout when rerun with the same -n N). Pass the bare coworld-id to 'coworld play' or 'coworld run-episode'. run-episode is headless and writes artifacts but always runs only the manifest's certification.game_config smoke test (e.g. a 3-step cogsguard fixture), NOT the league mission, and has no --variant; 'coworld play' runs the actual league variant (e.g. CvC machina-1-daily, 10000 steps) for the live browser viewer; to smoke a fuller game headlessly pass an episode_request.json whose game_config is the variant. In run-episode the positional image is reused for ALL slots (self-play) -- pass one image per slot to mix; there is no single-slot override and reference players are used only with zero images. coworld certify does NOT accept player_images (cannot substitute a policy); use run-episode for the strictest pre-flight (asserts client/WS reachability, token rejection, non-empty first global message). 'coworld play' produces only an uncompressed replay.json while run-episode/certify produce BOTH replay.json and replay.json.z (account for it when a downstream tool expects compressed). Prefer stable cow_/cow_id ids over names for reproducibility. Expect version drift (coworld download crewrift may pull 0.1.45 vs a doc's 0.1.40). _{sources: coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/downloa, player_labs/.claude/skills/coworld-local-run/SKILL.md, archive/cogames_playground/docs/coworld_cli.md, players_checkouts/players/users/james/personal_cogs/among_them/README. (+1)}

7. Coworld auth lives in its own credential store; probe with a read call

loop · tool: coworld

Authenticate to the Coworld Observatory backend once with 'uv run softmax login'; there is no whoami, so probe auth with any auth-requiring read like 'coworld leagues'. A sudden failure on the SECOND API message (package clearly installed) means the token expired (401/403), not a missing package. The 'coworld' CLI authenticates from its OWN store (~/.softmax/credentials.yaml) and does NOT read the standalone 'softmax' CLI's token store, so 'softmax set-token'/'status' do not affect coworld; if using 'softmax login' use the softmax binary bundled with coworld, not a separate install. To act as a specific player, mint a player-scoped token via POST /observatory/players/{player_id}/login (requires a USER token, no body) and save it as coworld's active player session; load_current_token returns the player_session if present else the user token, so deleting the session reverts to the default-player user token. 'coworld download' by canonical name always pays an auth round-trip even on cache hit (name->ID resolves before the cache check); download/play by cow_<id> is auth-free on cache miss. _{sources: archive/cogames_playground/docs/coworld_cli.md, players_checkouts/players_1/docs/coworld-player-swap.md}

8. Hosted-game sessions leave no trail; capture docker logs before docker rm

loop · negative result

coworld hosted-game sessions leave no discoverable trail: after a hosted run, episodes --mine and replays --mine return nothing and final scores never reach the player container. Capture per-slot 'docker logs' BEFORE 'docker rm', or use 'coworld play' (which writes a re-watchable replay.json) for records. _{sources: claude-code:67ff1fe3-48c0-4738-b7fb-8b994e567434, claude-code:cd2e0b52-b541-4e03-9fd3-798db4591603, archive/cogames_playground/docs/coworld_cli.md}

9. Read persisted episode artifacts (.optimizer-artifacts), and know coworld play vs run-episode layout

loop · ⚠ session-derived, unverified

The live Coworld episode replay lives in a system temp dir (e.g. /private/var/folders/.../T/coworld-ep-<id>-<rand>/replay/) OUTSIDE the optimizer working dir, so ls/file is blocked by the sandbox -- read the persisted artifacts under .optimizer-artifacts/episodes/ep-<id>/ (results.json, config.json, replay.json, run_parameters.json, coworld_manifest.run.json) by full path instead of listing the temp dir. coworld play injects COGAMES_ENGINE_WS_URL=ws://host.docker.internal:<port>/player?slot=<i>&token=<t> per container, exposes /healthz, and writes results.json/replay.json/per-slot logs into an EpisodeArtifacts temp workspace. coworld run-episode is the headless sibling (no browser; --output-dir defaults to ./coworld-episode-results, default --timeout-seconds 3600 vs 60) and writes directly into that dir with NO per-run subdirectory (the prefix='coworld-run-' arg is dead code), producing BOTH replay.json (large, uncompressed) and replay.json.z (small, zlib) -- do not assume only the compressed one exists. _{sources: claude-code:38ddef93-3214-489f-a178-33dddf030d9e, claude-code:4c0f63b0-f7eb-408e-ac3f-00feb1dcc193, claude-code:8ec27261-9517-4365-944b-2880873bda47, claude-code:394cc2af-72e4-41bf-861b-9b245d819229 (+1)}

10. Crewrift local replay viewing: env var, singular /client/ path, image parity, confirm playback advanced

crewrift · negative result

To view a Crewrift .bitreplay locally, launch the game image with COGAME_LOAD_REPLAY_URI=file:///coworld-replay/replay.json (path inside the container) and open the SINGULAR /client/replay page -- this is the only mechanism the image honors. Do NOT use 'coworld replay <manifest> <replay.json>': it only sets COGAME_REPLAY_SERVER=1 (which Crewrift never reads; replayServerMode is set at boot from replayLoaded), so the uri is ignored and the container streams a live game stuck at 'waiting for players: 0/8'. URLs are singular /client/ (no .html): live=/client/global, replay=/client/replay?uri=...; the CLI's advertised PLURAL /clients/replay is dead and reports 'disconnected'. When mounting a replay output dir, do NOT mount over /workspace/crewrift (it hides bundled assets like data/tiny5.aseprite and the server fails before players connect) -- mount only config.json there and the replay dir separately. Playback requires producer and viewer image versions to MATCH or it aborts on a replay-hash mismatch; refresh both together (uv tool upgrade coworld + pull current crewrift image, then regenerate). CONFIRM playback actually LOADED before trusting visual checks: health and 'the map rendered' do not prove it. The /replay stream carries a 'replay tick <N>' sprite -- confirm N advances ~24/sec (resets to 0 each loop, plays once start-to-finish then holds the final frame; looping defaults false). During playback the logs/viewer show the NORMAL lifecycle ('waiting for players 3/8','game starting: players=8, imposters=2') because the replay feeds Join records -- that is the re-simulated recorded game and the expected sign it works; ticks 0-120 lobby and 120-240 role reveal genuinely show 'waiting for players', so account for that recorded phase before diagnosing 'stuck'. NEVER set COGAME_SAVE_REPLAY_URI together with COGAME_LOAD_REPLAY_URI (server refuses: 'Cannot save and load a replay together'); local server runs save no replay by default -- set COGAME_SAVE_REPLAY_URI to capture a .bitreplay. Episodes do not auto-end (run to the tick cap, ~2100 ticks ~= 3-3.5 min), so set a background timer to kill the container. Playback is real-time ~24fps with no scrubber (~587KB initial frame, ~1.2KB deltas; playback time = tick/24). The MettaScope viewer has macOS/Nim gotchas (the bundled 'cogames replay' wrapper fails; invoke MettaScope directly with --path to ~/.nimby/pkgs/*/src incl opengl, launch from a temp cwd pointing data at the matching MettaScope bundle's real cached assets NOT the metta checkout whose atlas is Git LFS pointer text, run in a foreground Terminal); for pure metric extraction parse the compressed replay JSON directly. _{sources: claude-code:962c8ab4-4ea3-4aaa-9f6d-20ea3cfce003, codex:claude-code:ff9f693c-b88a-419d-be7a-efc46750dc23, codex:019e8fc7-ebad-7b00-bc95-df1727641ce0, players_checkouts/players/players/crewrift/crewborg/docs/crewrift-repl (+5)}

11. A crewrift replay is INPUTS not state; recover events by re-running the real sim via expand_replay

crewrift · tool: expand_replay.nim

A crewrift .bitreplay/CREWRIFT replay is a byte stream of per-tick player INPUT masks, not state, so recover events by replaying inputs through the real simulator (tools/expand_replay.nim) and reusing the sim's exported public procs (e.g. tryReport for body-report attribution, the sim's ordered side effects for kill attribution) -- never reimplement game rules in Python; the CLI text just renders the typed API expandReplayTimeline(data: ReplayData): ReplayTimeline. Read a finished game from two aligned sources: the objective expand_replay event timeline (every player, true roles, all kills/bodies/votes/tasks/chat -- policy-independent ground truth) for WHAT happened, and each player's own per-slot subjective trace log for WHY, aligned by tick. You CAN read format CONSTANTS from game source to decode without the SDK (the older .rep format was a BITWORLD magic header plus little-endian join/leave/input/hash records with a JSON config blob; a standalone decoder extracted per-slot policy names, per-player input counts, last-active tick, button-press timings), and you can extract the CREWRIFT header/gameConfig (timing params, player names), but you canNOT decode per-tick events without the native decoder -- any tick derived only from gameConfig timing is an ESTIMATE and must be flagged. Chat is NOT stored in the replay stream -- agent logs are the only source for whether anyone chatted. Per-player input counts plus last-active tick make a fast liveness check (one opponent logged only 4 inputs and went silent; another left mid-match). Keep the replay parser PURE (no feature logic): produce a typed Game (players/roles, sampled positions, visibility intervals, kills/bodies/ejections, meetings with votes, chats, task completions, map geometry) shared by every downstream consumer, with the event vocabulary and correctness rules in exactly one place. Build position/visibility lookups queried by bisect: state_at(slot,tick) returns the latest sample at or before the tick, sees(observer,target,tick) is true when any visibility interval covers the tick; sort samples/intervals by tick in a post-pass before querying. suspicion_lab's replay_parse.parse_game expects a pathlib.Path, not a str (a str crashes with AttributeError on .suffix). A one-off offline tool can reuse this corpus/parsing infra without being wired into the nightly pipeline -- keep exploratory tools descriptive (no pass/fail gate), emitting a text report plus optional JSON. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, claude-code:4c0f63b0-f7eb-408e-ac3f-00feb1dcc193, claude-code:10f74020-e0e6-4515-8d0c-f071d41766ad, coworlds/coworld-crewrift/README.md (+6)}

12. Version-match expand_replay to the recording build; hash-validation catches skew; trust results.json for kill COUNTS not replay ATTRIBUTION

crewrift · negative result · tool: expand_replay

expand_replay validates a per-tick state HASH and stops on the first mismatch, so it correctly re-simulates ONLY replays recorded by the same crewrift build (e.g. build 3ea899eb matched game v0.1.51); a binary from a different version diverges within a few hundred ticks, prints 'hash failed', and emits almost nothing. Keep one expander per version (expand_replay-<sha>), tag each scraped episode with the version that expanded it, and bump the ref when the league redeploys and fresh replays start hash-failing. Build a version-matched host-native binary with crewrift_lab/tools/build_expand_replay.sh (fetches public crewrift source as a tarball -- no credentials, just the host Nim toolchain) tracking the pinned CREWRIFT_REF; the coarse embedded gameVersion is the constant '1' and does NOT catch a build mismatch, so match the git ref. When you cannot supply that ref (older season, someone else's build), fall back to the visual replay viewer (loads into the episode's own game image, always re-simulates correctly) plus the version-independent player logs. Do NOT trust the bundled tests/replays/notsus.bitreplay fixture as an oracle (stale, hash-fails ~tick 36) nor a tools/expand_replay binary already in a checkout (matches whatever it was last built from). Even version-matched, trust results.json for kill COUNTS (server-authoritative kills[i]) but NOT replay-derived kill ATTRIBUTION: the re-sim attributes a kill to whichever imposter is near the body when it appears, mis-assigning at simultaneous-body ticks (it showed crewborg killing two players in two rooms on the same tick under cooldown) -- use the replay for kill TIMING and positions/rooms, results.json for counts. --snapshot-every N thins ONLY periodic player/body/visibility state sampling; every discrete event (kills, votes, room transitions, task completions) still emits on its exact tick and state rows are force-emitted on event ticks, so thinning never drops events (verified: kills land at ticks not divisible by the cadence). An observability-only addition (e.g. a JSONL visibility-interval emitter) does not affect the hash and can be cherry-picked onto an older ref. Compile replay-analysis helpers as optimized/release builds for bulk passes (a debug build managed ~37/560 replays in minutes while an optimized rebuild finished all 560 quickly). _{sources: claude-code:63658f66-83c7-482e-a805-d58527e10975, codex:019eaea8-4f78-7072-b4b0-539df963b7df, codex:019eaa2f-3bfd-73b3-aaec-4d709e098c39, personal_labs/crewrift_lab/crewrift/crewborg/docs/designs/suspicion-le (+5)}

13. Use results.json scores as primary evidence; apply inference rules and score fingerprints, but do not invent state the replay lacks

crewrift · ⚠ session-derived, unverified

When the replay is opaque/binary, results.json is the most reliable evidence and a crewrift episode-report is assemblable from results.json plus the replay header alone (no raw logs). It exposes per-slot arrays: scores, win, tasks, kills, imposter, crew, vote_players, vote_skip, vote_timeout. Inference rules: (a) a player with vote_skip=0 while all other alive players have vote_skip=1 was almost certainly dead at that meeting (cross-check the kills array for the killer's slot); (b) dead crew still complete tasks (ghost completion), so do NOT infer a player is alive from high task/score. Score-fingerprint triage flags anomalous episodes cheaply before pulling replays: a 'successful' crewmate score is 8 (all tasks, lost) or 108 (all tasks, won); a 'successful' imposter is 20/30/120/130/140 (2+ kills) -- auto-flag any score off those values. But do not infer state the replay does not record: among_them 'likely_dead' and estimate_meetings heuristics were REMOVED because there is no clean way to know deaths or meetings from the artifact -- surface only what is directly present and drop derived fields whose ground truth the format cannot supply. Note: optimizer run-*-stdout.log files are agent/session metadata (thinking tokens), NOT decoded game events -- do not mine them for action.kill/vote.cast. _{sources: claude-code:4c0f63b0-f7eb-408e-ac3f-00feb1dcc193, claude-code:5ba30745-2180-4e0c-9cb5-13846d2aa6a6, claude-code:f70c9801-7ee7-499b-97f9-4fd0848a6b8e, claude-code:e32290d2-800b-4c28-8f9d-ac6f65e9ae76}

14. Map slot->policy by NAME AND VERSION from episode metadata; crewborg logs do not self-identify version or role

crewrift

Map slot->policy by NAME AND VERSION from episode metadata before reading per-slot logs, never by eyeballing which logs are JSON: the 'first line is JSON' heuristic only proves a log is from the crewborg JSON-logging family, not which version, and ONE league episode can run several versions of the SAME policy at once (slot 0 = v15, slot 5 = v6) emitting identical trace schema, so a head/jq sniff proves 'a player log' not which version. The slot map lives in episode.json/the episode record, NOT results.json (which has outcome arrays but no slot map): league episodes expose policy_results[]=[{position,policy:{name,version}}]; experience-request episodes expose participants[]=[{position,policy_name,version}]. crewborg's per-tick stderr trace (perception/belief/mode decisions plus metric lines) is the richest behavioral record and comes ONLY from the Observatory policy-logs endpoint indexed by SLOT, so resolve crewborg's slot from participants first, then request policy-logs/{slot}. An EMPTY result from a vote/kill/suspicion jq recipe is AMBIGUOUS -- crewmate-role logs and no-meeting episodes legitimately emit nothing on imposter-only kill paths, so count domain.phase_change and check domain.decision_snapshot role before concluding the recipe is broken. meeting_context_serialized is multi-kilobyte; extract only the fields you need (e.g. per-slot {slot,color,alive,self}). For mechanic audits, grep the installed package for EVERY handler touching the mechanic (hp/death/kill); finding exactly ONE death handler is itself a load-bearing negative result. _{sources: claude-code:63658f66-83c7-482e-a805-d58527e10975, personal_labs/crewrift_lab/crewrift/crewborg/docs/trace-logs.md, personal_labs/crewrift_lab/best_practices.md, personal_labs/crewrift_lab/docs/crewrift-replays.md (+3)}

15. Read game results from saved replays/traces, not stdout; verify format by magic bytes and enable -v talk

crewrift · ⚠ session-derived, unverified

Never judge a live match from stdout: once a match ticks the console goes quiet by design and the real signal (LLM decisions, chat, votes) lives in the JSONL/replay files -- always pass --save-replay-file/--save-replay-dir (a GUI run without one leaves only the terminal summary), wait for a clean exit, then parse. Saved replays are zlib-compressed JSON (.json.z), so decompress before grep/ripgrep; files saved by 'coworld replays' get a .json extension but are .bitreplay format beginning with ASCII magic CREWRIFT, so hexdump the magic bytes rather than trusting the extension. The talk/text channel is OFF by default for machina_1.clips, hiding role-negotiation -- add -v talk to record role/target/direction/heart claims (e.g. cogames play -m machina_1.clips -v talk -c 8 -p class=...BulbacogPolicy --seed 44 -s 1000 -r gui --autostart --save-replay-file <p>.json.z). coworld replay footguns: pass --no-open-browser in headless shells; 'coworld replay' writes a tmp/coworld-replay-* dir under the CURRENT repo even with /tmp inputs (clean it up). _{sources: codex:019e14de-cdca-7482-88fa-81716ee830ae, codex:019eaa2f-3bfd-73b3-aaec-4d709e098c39, codex:019e1591-004e-7040-b768-062bd8e6acd4, codex:019e1579-a737-7b70-bf0e-172ff04307cf (+2)}

16. Histogram trace events first, and harden jq against non-JSON lines and role-gated absence

crewrift

Before querying a JSONL trace for whether an event happened, run an event-type HISTOGRAM first (jq, select kind=='trace', project .event, sort | uniq -c) so an empty later query reads as 'that event did not occur' rather than 'wrong query/file/not captured.' Trace defaults drift over time (per-tick decision_snapshot/metrics moved off by default) and role-gating makes 'missing' normal: a crewmate episode has no kill_* events, an imposter has no suspicion_snapshot and null visible_players[].suspicion, role may read 'dead' after death. When piping a hosted log to jq, prefilter with grep '^{' (the final line is plain text 'game over...' and k8s collector-error lines can be interleaved -- some whole per-slot 'logs' are collector errors), and guard with select(.kind=='trace') or (.event//'') before test() on .event because crewborg metric lines carry metric_kind/name/value/tags with NO tick/event field. crewborg writes prefixed event names (domain.meeting_*, not bare meeting_llm_decision) -- grep the ACTUALLY emitted names, not names read in mode code. _{sources: personal_labs/crewrift_lab/crewrift/crewborg/docs/trace-logs.md, personal_labs/crewrift_lab/best_practices.md, personal_labs/crewrift_lab/docs/crewrift-replays.md}

17. Emit merged stdout+stderr per tick; the per-slot log is the runtime source of truth

loop

Per-agent logs (logs/policy_agent_{N}.log / policy_agent_{slot}.txt) are the richest behavioral record of a Coworld episode and the runtime source of truth when debugging, while results.json and the episode row drive scoring. Emit diagnostics to stdout/stderr (not stderr-only) and flush each tick or set PYTHONUNBUFFERED=1, because the runner captures BOTH streams merged into the per-slot log -- the only channel by which player-side traces survive for retrieval via 'coworld episode-logs ereq_... --agent <slot>'. This merged log is marked diagnostic-only with NO header/prefix/separator -- do not assume a line format, and do NOT filter out 'INFO -' lines as noise (they are usually the agent's real Python logging); the CLI's own bracketed console.print messages can be mistaken for captured lines. For guided_bot, structured JSONL trace streams surface in hosted logs only when GUIDED_BOT_TRACE_DIR=stderr and GUIDED_BOT_TRACE_LEVEL is non-off (off/events/decisions/full); absent trace lines means env vars are unset, not a connection failure. Never dump raw per-frame perception into hosted logs -- use summarized diagnostics. Make artifact downloads best-effort and idempotent: record a missing replay/log rather than aborting, and skip already-complete episode dirs unless forced. Treat OFFLINE developer tools (asset bakers, nav baker, waypoint/frame editors, frame-to-text dumper) as exempt from any Coworld-only run restriction because they only produce or inspect checked-in baked artifacts and recorded traces; never conflate them with a live raw-frame-capture run path. _{sources: player_labs/.claude/skills/coworld-episode-artifacts/SKILL.md, players_checkouts/players/users/james/personal_cogs/MISSION.md, players_checkouts/players/users/james/personal_cogs/AGENTS.md, players_checkouts/players/docs/coworld-integration-guide.md (+5)}

18. Design player tracing as tiered structured JSONL with a lean hosted-cap default; emit domain events post-step

loop

Design a player's tracing as tiered structured JSONL (OFF < EVENTS < DECISIONS < VERBOSE/FULL, each a superset, emitted as category+data+level so analysis is machine-parseable not free-text scraping), with a lean DEFAULT that fits under hosted log caps. By default emit only durable low-volume DOMAIN events (meeting context, vote cast, chat sent/received, phase/role/task changes, high-level actions) and drop per-tick payloads (decision snapshots, belief/suspicion ticks, framework-boundary traces, viewer frames, metrics) behind an opt-in debug/viewer flag -- a decision_snapshot per-tick object is usually the single biggest volume contributor (one session produced ~380k trace lines across three games; raw frame dumps reach ~63 MB / 4000 frames; emit those only at 'full'). Gate ALL instrumentation behind env vars (output dir + verbosity) with a single early if trace == nil: return guard so every log call is near-zero cost when off, and gate expensive instrumentation behind the verbosity check BEFORE computing it. Emit game-level domain events from an on-step-complete hook rather than inside agent 'modes', because attempt events key on the produced wire command (modes never see it) and outcomes like task-completion conclude only after the mode runs; post-step emission keeps the strategy layer pure and reads off the finalized belief. Route all traces through ONE sink (full for local/tests; an env-aware factory returning the filtered/metrics-off sink by default in the hosted bridge) plus a bridge-level test that fails if the hosted path starts sending metrics again. Provide env knobs (CREWBORG_TRACE_GROUPS=decision|voting|action, CREWBORG_TRACE_INCLUDE/EXCLUDE globs, CREWBORG_TRACE_DECISION_FIELDS, CREWBORG_METRICS=1) to capture one event family without full volume. EXCEPTION: promote the COMPACT decision geometry (mode, target color, target distance/position) into the always-on lean stream (crewborg's decision_snapshot in events.py) so thrash/flee bugs are diagnosable without resimulating; keep the FULL per-tick payload debug-only. Keep policy logic clean by splitting the decision into a pure core (decideNextMaskCore) plus a thin trace-owning wrapper (decideNextMask); never export an internal field just so a call site can derive a value. Decouple analysis from behavior: emit analysis-only annotations (vote_bandwagon_detected, goal-tier) into decisions.jsonl read by NO policy code. Log the FINAL emitted action mask, not only upstream intent (a trace logging only intent blocked a navigation diagnosis; '535 identical decisions, 2 transitions, 0 events, 0 LLM calls' localized it in minutes), treat perception inputs (e.g. chat OCR) as first-class trace events, and log every LLM round trip with raw_response, prompt/response token counts, latency, validation status, published directive. For a deduction agent the single highest-value forensic record is one 'suspicion snapshot' at the start of every meeting carrying ranked posteriors, each suspect's event log, the would-be vote, and the decision bar. Make it purely additive over a green suite (keep the old Callable[[str],None] interface and exact legacy strings; emit structured entries only when a real Logger is passed; inject clock+sink for deterministic tests), each controlled slot writes its OWN trace file, and read the existing struct/enum defs and trace writer FIRST so field names and phase/role/mode stringifies match. _{sources: claude-code:3c867a71-3057-4e87-a149-4185e82ee728, auggie:ac491a46-e7f7-4271-a1c0-f0e69987d424, codex:019e0383-1db4-7f90-a085-04e6f27b26e5, codex:019ea9af-7020-7550-b427-d3478fed2057 (+13)}

19. Render traces human-readably, store them in a stable repo dir, and confirm the specific bot is wired

generic · ⚠ session-derived, unverified

Ship a small formatter/viewer alongside raw JSONL traces -- raw JSONL is hard to scan and a trace is only useful if a human or agent can read it. Store traces in a stable, documented repo-relative directory (e.g. traces/ under the player package), never /tmp, where temp traces get wiped and the next session's tooling won't look. Observability is per-bot, not a framework property: in Among Them, evidencebot_v2 wrote no trace files (only a stdout 'frames dropped' line) while siblings mod_talks and modulabot each had a dedicated trace.nim -- before trying to 'read the traces,' confirm the specific player has tracing wired, since empty trace files despite a visibly-acting bot means broken plumbing, not behavior; verify which directory actually receives traces. _{sources: opencode:ses_21a708a4dffeBXpeHY15fnGABP, opencode:ses_1fecece4dffe779FmeajgdwhMD, opencode:ses_200be1a72ffevDDc1PULNNG9UF}

20. Prove an optional feature engaged before tuning it; absence of success is ambiguous

loop · negative result · ⚠ session-derived, unverified

When an LLM-augmented (or any optional) feature underperforms, FIRST prove it actually fired before tuning prompts or strategy -- absence of success events is ambiguous between 'feature off', 'tried and failed', and 'precondition never met'. Concrete diagnosis: (1) check the trace CHANNEL exists at all (a [trace:guidance] channel carrying snapshot_sent/llm_response/directive_published absent, with every decision showing directive_source 'reflex'/'default' and never 'llm', means the path never engaged); (2) grep the ACTUALLY emitted prefixed event names, not names read in mode code (a scan for meeting_llm_decision found zero while logs used domain.meeting_* prefixes); (3) confirm the precondition occurred (a meeting actually happened) before blaming the feature. A single llm_call_failed reason=no_key meant the guidance layer never ran the whole match and the bot ran purely on its symbolic fallback. Distinguish decision-layer from execution-layer failure: count LLM responses/directives separately from meeting-actions-received and actual button presses/vote_attempted -- a high response count with few meeting actions (or ~39% noop, most 'skip' decisions never reaching vote_attempted) means the model decided right but the action never landed, so chase the action loop not the model. Make engagement greppable with a one-shot startup trace (e.g. llm_init once per match) recording provider selected and credential-presence signals, self-contained so it survives ring-buffer eviction; and verify the full prompt/snapshot payload is actually persisted (a design doc referenced snapshot_sent but live runs emitted none), not just the response. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, claude-code:d48f9347-7884-4ea2-8659-230051d86410, codex:019ea93a-43b0-7800-b34e-7b3053c46177, codex:019e1593-c00d-7a32-9dfa-1cf6b16b01a4 (+1)}

21. Debug the trace, not the outcome or the on-screen moment; re-run at the right log level; trust server-confirmed over agent-side menu completion

loop

You cannot debug an outcome, only a trace: pivot immediately from a win/loss to the player's internal reasoning stream -- the gap between what was objectively true and what the policy believed/chose is where the mechanism lives. Visible behavior is an effect with a long causal delay, so trust the trace tick over the on-screen moment: a visible stop-and-turn at tick ~390 was actually caused by a target-commitment timer expiring at tick 319, the body bleeding momentum over ~70 ticks. The default log level often omits the decision telemetry you need (early analysis had only events-level logs and lacked meta_decide_reason, emitted only at decisions level), so re-run at the verbose/decisions level and reproduce on a fixed seed before forming root-cause hypotheses. A server/global log of 'X happened' is NOT proof the agent EXPERIENCED X: server 'opened whisper' events looked like success but the agent's own frames never showed the whisper view (events belonged to other players or a player-index mismatch), and in persephone a whisper_exchange_outcome with server_confirmed=false only means the agent completed/timed-out a menu sequence, NOT that a win-relevant exchange happened -- only server_confirmed=true or the authoritative server full.log ('joined','offered role','shared roles','winner') proves a real exchange; when agent-side perception and server state disagree (client still seeing PLAYING/overworld pixels while the server's whisper state was active), debug the transport/timing path rather than trusting that menu completion implies a server effect. Read game progress from the right channel: a player's spatial-memory tick= lines stop updating once local territory stops being newly observed and make a run look stalled -- read the game tick from declarative-memory lines instead. _{sources: claude-code:3c867a71-3057-4e87-a149-4185e82ee728, claude-code:3c13aeb0-5617-452d-a8d7-aff376e3d4ff, player_labs/best_practices.md, personal_labs/crewrift_lab/.claude/skills/crewrift-diagnose/SKILL.md (+3)}

22. Use per-agent action histograms and mode churn as cheap stuck-detectors, decoded against tick rate

loop · ⚠ session-derived, unverified

A healthy match shows many distinct action types (~18) with movement/interaction spread; a distribution collapsed onto one action -- especially noop (a bot pinned on a wall shows huge noop plus one directional button) -- flags stuck/idle/crashed behavior. CAVEAT: subtract legitimately-idle frames first; raw NOOP looked ~20% but interstitial/voting frames are noop by design and true gameplay NOOPs were ~0.4%. Compare per-bot counts before/after a fix to confirm the bot actually moves. A flat modes log (one mode at setup, never transitioning) means transition triggers are unwired. A regular fixed-period idle cycle (a repeating ~72-tick = 3s @ 24Hz DisciplineNoOp loop) is a scan/cooldown timer firing with no actionable condition -- convert tick spans to seconds and match against game constants (the 72-tick span did NOT match the 1200-tick kill cooldown, ruling it out). But interpret in light of role speed: low-speed agents rarely trigger a COAST branch because a path-following deadband catches arrivals first -- few coast events is not a bug. Diagnose a stuck crewmate from action traces by a long single-task dwell dominated by ONE repeated d-pad direction: a slot spent 984 consecutive ticks (~41s) on one complete_task, 837 monotonic RIGHT and 72 ticks of A -- a wall-wedge against impassable geometry (the task only 'completed' ~200 ticks later because ghosts slide through walls). Add stuck-detection/replan guards triggered by such monotonic-direction dwell. _{sources: opencode:ses_1fb1d9989ffeYXiew23WUqo2hw, opencode:ses_201637e3bffeRa6Y0x3DOs8hh1, opencode:ses_21b273d4fffee0T9oMpvlktzkg, opencode:ses_20422c08ffferRLvWnNE5DeB4w (+3)}

23. Distinguish a kicked slot from a finished one via game_over and a known-good opponent

loop · ⚠ session-derived, unverified

To tell whether a player slot ended naturally or was cut mid-game, diff it against a known-good full-game opponent slot in the SAME episode on frame/tick count and end-event: a healthy slot processed ~4841 frames over ~200s and saw game_over, while a cut player closed at ~880 frames having never seen game-over -- 'did this slot receive game_over?' is a clean kicked-vs-finished test. A player 'disappearing early' in a replay has at least three viewer-identical causes (network disconnect, freeze-in-place, meeting/body teleport), so cross-check the server-side result (stayed alive? won?) against the bot's perception trace before concluding elimination or disconnect. _{sources: claude-code:5f9e8287-da92-4dff-bf39-2511bcaf0700}

24. Anchor underperformance to a quantitative baseline, split by role, and surface behavioral gaps

crewrift · ⚠ session-derived, unverified

When diagnosing a role's underperformance, anchor to a concrete quantitative baseline before reading traces: framing a vague 'imposters are inefficient' (0-1 kills per 3-min match) against 'a competent imposter averages a kill every 30-45s when cooldown allows' converts it into measurable lost-time accounting (e.g. '13% of the match completely idle') attributable to specific phases. Split a dominant opponent's win-rate by ROLE first, because the edge is usually concentrated in one: league leader RowDaBoat's edge was crew discipline (+14pp crew win from near-zero complicity in crew ejections, burning the emergency button every game to reset imposter kill cooldowns, always 8/8 tasks), not kills or task speed -- the crew-win gap accounted for essentially the whole leaderboard distance while imposter kills/game were identical. A run summary should surface behavioral GAPS, not just win/loss: a scripted bot logged vent_suspected twice but never reached a meeting to report it -- the actionable finding is the missing detect-to-report linkage, which a bare score hides. _{sources: claude-code:0919285d-1374-4827-b596-6e7107d95526, claude-code:0e7b14ca-ffd5-4137-9456-47485d3c6f87, claude-code:962c8ab4-4ea3-4aaa-9f6d-20ea3cfce003, claude-code:dfa4502e-c404-40e8-bf86-8d15c72ba5af}

25. Triage episodes by failure CLASS and a quantitative fingerprint before reading events

loop · ⚠ session-derived, unverified

Triage low-score episodes by failure CLASS before coding, not by eyeballing logs. Recurring score buckets map to root causes: zero-score = never left the interstitial/unknown-role screen or died early as crew; flat-low = tasks done but no win (a strategic-conversion problem, not a patchable bug); stuck-movement = alive but world position frozen. Only mechanical failures (stuck navigation, missed votes, slow first task) are cheap root-cause patches; do not ship a code fix for a strategic-conversion or one-off failure. Get a fast quantitative fingerprint first: (1) group and count trace event kinds (task_started/completed/abandoned, vent_suspected, mode transitions) plus the gameplay-vs-interstitial phase split; (2) read the end-of-episode action-mix histogram (interact + diagonal moves reads as normal task-running; mostly noop or one stuck direction flags a navigation/decision bug); for LLM/meeting matches compute a scripted rollup (roles, outcome, meeting/chat counts, vote targets, LLM validation failures, winning side) first -- counting events tells you whether the run is worth a deep read. Reduce raw episode JSON dumps to your-player-only scores plus low-score request IDs before analysis, then download logs only for the worst/representative cases; raw CLI dumps of thousands of JSON lines are too large to reason about and not actionable. _{sources: codex:019e4c28-d516-7ca0-b145-153c152b4f43, claude-code:dfa4502e-c404-40e8-bf86-8d15c72ba5af, codex:019e1591-004e-7040-b768-062bd8e6acd4}

26. Decompose Crewrift episode analysis by role, cheap-to-deep, flagging outliers

crewrift

Always decompose Crewrift episode analysis by role -- crewmate and imposter are effectively two policies, so an aggregate distribution hides one role being broken; keep the split in every distribution. Connect/disconnect timeouts (scored -100) are crashes, not strategy flaws -- give them their own category and exclude from behavioral distributions. Analyze a batch cheap-to-deep across three tiers: Tier 1 reads results.json + episode.json for role flags/scores/tasks/kills/votes/timeouts without parsing replays (scales to hundreds); Tier 2 runs version-matched expand_replay on flagged episodes to recover killed-by-imposter vs ejected vs survived, itemized score breakdown, and vote correctness; Tier 3 reads the policy's own logs for the subjective 'why' at the tick of interest. Flag an episode as 'unusual' when it is an outlier on some feature via robust z-score or rank, and report the specific deviating feature (e.g. 'alive only 180 ticks vs median 3000', '0 chats vs median 4') rather than a generic anomaly flag. _{sources: personal_labs/crewrift_lab/.claude/skills/crewrift-report/SKILL.md}

27. Pull competitors' logs from the same episode; keep opponent scouting separate from self-analysis

loop

Pull and inspect OTHER competitors' agent logs from the same episode to learn opponent behavior and calibrate debugging: cross-agent log SIZES alone are diagnostic (one opponent emitted 7000+ lines while others emitted near-zero; grepping all logs for chat markers showed only one bot ever chatted) -- a cheap way to see which opponents are sophisticated vs inert and whether the chat channel is used in the league. Keep opponent scouting separate from self-analysis: an opponent-modeling pass studies THEIR tells, biases, predictable openings, target selection, voting behavior, and panic conditions and must never record our own policy's mistakes; self-failure analysis is a different pass. Distinguish subject from opponent slots via per-player attribution (e.g. lib/episodes/players.ts: each slot carries role 'subject' vs 'opponent', slot index, name, policyLabel, policyId/libraryId, source), filtering to opponent-role slots first. _{sources: claude-code:00f61c5a-0c3b-41ca-a586-969b3ce1ce83, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/explo}

28. Autonomously escalate tracing when it is a config knob; emit JSON+Markdown

loop

If a policy's logs are too thin to find the mechanism, autonomously escalate tracing when it is just a config knob (turn up the trace level, re-run the experience request, re-pull artifacts, re-report, resume) WITHOUT a human gate; only flag the human when getting the signal requires a code change to the player's tracing -- which is itself a finding. Emit analysis as both human-readable Markdown and a machine-readable JSON block (e.g. {"score":6,"rolling_avg_5":5.4,"trend":"improving","bottleneck":"...","escalation":"none"}) so downstream automation consumes results without parsing prose. _{sources: personal_labs/crewrift_lab/.claude/skills/crewrift-diagnose/SKILL.md, coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/analy}

29. Run evaluation headless with explicit seed; fail loud on unparseable scores

loop

Pass -r none on every headless cogames play eval run: rendering costs ~60s of a 62s machina_1 500-step run (~2.5s off), roughly a 25x speedup (~1 min vs ~25 min for a 12-seed A/B); reserve GUI for debugging one episode. Always pass an explicit --seed since omitting it picks a non-default seed and hand-repro will not match harness scores. Because the A/B harness scrapes scores from the end-of-episode table via regex (no structured CLI output), it MUST emit a 'could not parse' error rather than silently producing bad numbers, and re-verify the regex on every cogames major bump. _{sources: alpha_cog/experiments/README.md, archive/cogames_playground/alpha_cog/experiments/notebook/protocol.md}

30. Commit cheap provenance, gitignore regenerable artifacts in a replay cache

generic

When building a replay-cache/analysis tool, extract a small per-episode metadata file (seed, map size, scores, split assignments, game/agent stats) at download time and store it separately from the multi-MB compressed replay so list/analyze commands run without decompressing. Commit the small index.json and metadata/ (they document what has been reviewed) but gitignore the large reproducibly-downloadable raw replays (~2-5 MB each compressed) -- commit cheap provenance, not bulky regenerable artifacts. Match project CLI conventions (argparse add_subparsers, sys.exit(main()), a _REPO_ROOT import shim, ThreadPoolExecutor for I/O-bound downloads, as in ab.py). _{sources: archive/cogames_playground/alpha_cog/docs/league-replay-tool-design.md}

31. Local debug launchers for the Crewrift/Among Them bots and their lifecycle footguns

crewrift · tool: run_debug_game.sh

Run the among_them Python smart bot via among_them/bot-policies && ./run_debug_game.sh (launches server, Nim opponents, the LLM-brained bot, and a debugger UI), or scripted/no-LLM with python3 -m sidecar.bot --name smartbot --debug; configure native inference through COWORLD_LLM_ENDPOINT in hosted runs or OPENROUTER_API_KEY locally. Run the notsus Nim bot headless by setting COWORLD_PLAYER_WS_URL to the ws://127.0.0.1:8080/player slot+token URL, or with the visual debugger via -d:notsusGui (renders walkability mask, viewport, A* path, current/roam goal, input mask, velocity, stuck state). Know launcher footguns: quick_player prepends the standard players dir to bare names (a nested bot needs its full path), propagates --gui to EVERY spawned instance, and kills ALL child bots when any one exits -- to watch one bot, spawn N-1 headless via the launcher and one GUI'd directly. _{sources: bitworld/among_them/players/how_to_make_a_bot.md, coworlds/coworld-crewrift/players/notsus/README.md, opencode:ses_224322057ffexj7SYlcfVA0zgy}

32. Diagnose scripted cogsguard programmatically with DebugHarness, not log greps

crewrift · tool: DebugHarness

Diagnose a scripted cogsguard policy with DebugHarness rather than only grepping logs: build from a recipe (DebugHarness.from_recipe(num_agents, seed)), step the sim, then call diagnose_stuck_agents(threshold), diagnose_coordinate_system(), and get_agent_state(id) to inspect role/phase/position -- faster root-cause access than greps. To run headless: ./tools/run.py recipes.experiment.cogsguard.play policy_uri=role render=log max_steps=100, piping stderr+stdout through grep on the per-agent prefix (e.g. ^\[A2\]) to trace one agent. Verify cogs_vs_clips outcomes from the end-of-episode game state: a rising cogs aligned.hub count confirms aligners, miner deposits show resource flow, a decreasing enemy junction-alignment count confirms the scrambler. _{sources: players/cogsguard/role/AGENTS.md, players_checkouts/players/players/cogsguard/role/CLAUDE.md}

33. Kill subprocesses on every failure path, match the player roster, and reap children robustly

loop · ⚠ session-derived, unverified

A multi-player game won't start until the required player count is present, so a single agent against a full-roster server hangs waiting -- match the spawned-bot count and use the project's multi-bot launcher (which traps Ctrl-C and reaps children) rather than backgrounding bots by hand. Kill temporary server/opponent/agent subprocesses on EVERY failure path or a broken run leaves orphans blocking later runs; a one-off smoke episode driven via the backend function (node --import tsx) has no TTY so Ctrl-C can't reach it -- kill node/tsx and its Coworld child by PID and docker stop the orphaned Crewrift container. In shell harnesses: use exec in the server subshell so $! is the real pid, add a re-entrancy guard on the shutdown trap (SIGINT and EXIT both fire teardown), and do TERM -> poll kill -0 to a timeout -> KILL -> verify -> wait (a flat sleep 1 between TERM and KILL is unreliable). For experience/episode requests with no cancel endpoint, delete the underlying k8s jobs AND the per-player child pods by job-id label (deleting parents doesn't remove children). Scan launch stdout for 'error'/'unknown option' -- a '--seed' error meant an unsupported CLI arg. _{sources: codex:019e94d1-e329-7590-b6b2-bfac0eef9950, codex:019e1940-5a75-7f51-b057-63f894ddb061, codex:019ea9af-7020-7550-b427-d3478fed2057, opencode:ses_21a0d30b6ffewSKOVSUyO4lZkU (+2)}

34. Scope a regression with git first; watch for shadowed functions, unresolved diff paths, hidden test collection, and over-broad edits

generic · ⚠ session-derived, unverified

When a bot misbehaves or you suspect upstream broke tooling, scope the window with git BEFORE reading game logic: list commits since a known-good date and classify each diff (decision-logic vs packaging; policy-API / env-config / tournament-runner / CLI). Eliminate behavior-neutral commits fast -- changes entirely inside build-flag-guarded blocks (when defined(...)) or FFI-only export renames don't affect the default run path, and most monorepo commits are server-side. A locally-defined function can silently shadow an imported one of the same name (a compile-time currentSourcePath()-based gameDir() shadowing an imported getCurrentDir()-based one) -- grep the file for a same-named local def before tracing the import. An empty/surprising git diff may mean the path arg did not resolve to a real tree (running git diff HEAD -- among_them/guided_bot from inside among_them silently diffs a nonexistent path) -- verify cwd and path resolution. To find the PR that introduced a contract, search stable code identifiers (env-var/constant names) in the code, not PR title/body words; a public name can be renamed within the same PR, so read current main. A broken import path can silently hide a test suite by preventing COLLECTION (three files using bare from perception import vs from orpheus.perception import dropped ~1200 lines / 151 tests) -- check collected-vs-passing counts, not just pass counts. A search-and-replace whose oldString recurs can match the LAST occurrence and delete a much larger span (file shrank 2237->1401 lines, ~836 lost) -- uniquely anchor oldString and verify line-count deltas after large edits. _{sources: opencode:ses_200be1a72ffevDDc1PULNNG9UF, opencode:ses_201bc0c24ffe4uYJuw52Gs42LG, opencode:ses_201972733ffexPIP9mq6yEDpqJ, opencode:ses_200deb284ffe7RFSHqkkWQqwAx (+3)}

35. Use Chebyshev distance and ownership-flip windows for grid-game metrics; for cogs-vs-clips score on junctions HELD not resources

loop · ⚠ session-derived, unverified

Domain metric definitions for grid games (paint_arena): use Chebyshev (king-move) distance, not Euclidean, for grid-tile proximity events ('close proximity' = Chebyshev distance <= 2); and operationalize a 'highlight' as contested territory -- a tile changing owner >= K times within a W-tick sliding window (tune K and W per game) -- because the meaningful signal is repeated ownership flips on the same tile, not single captures. When evaluating a cogs-versus-clips player, do NOT trust per-resource withdrawn counts as success: an episode can show cogs/heart.withdrawn ~5 while per_episode_per_policy_avg_rewards is 0.0 and cogs/aligned.junction.held is 0.0 -- withdrawing hearts is not holding junctions, which is what scores. A near-zero cogs/aligned.junction.held while clips/aligned.junction.held is large and reward is 0.0 is the dominated-team signature and the primary failure alarm. Distinguish CAPTURE from RETENTION: on a full 10k-tick seed-42 run Cogs gained 111 junctions but the opponent held far more junction-ticks -- 'junctions gained' can look healthy while 'junction-ticks held' is losing, so measure held-ticks to surface late-game defense failure. _{sources: claude-code:2e0ca80b-d747-4358-a5dc-650397742760, claude-code:36cee047-63f0-4c81-b5e6-8c24ec57afba, codex:019e19a7-8605-7992-9b28-cb4d2ca9132e}

36. Self-evaluate against the public Cue-n-Woo judge worker; batch and cache

loop · ⚠ session-derived, unverified

The Cue-n-Woo judge worker (https://cue-n-woo-worker.softmax-research.net, Gemma-2-9b-it on an L4) is publicly callable unsigned with no VPN/auth, so you can self-evaluate against the real scorer locally with no league. It is slow and shared (~1 generation per 10s at 128 tokens), so batch requests and cache generations rather than calling it per-candidate at runtime. _{sources: claude-code:77e158b0-bd16-41f8-8473-d1b1522d451b}

37. Recover interfaces from bytecode/git history, fix import cycles without copied constants, clean generator debris, and rebase stacks

generic

If a session times out mid-rewrite and leaves a Python module deleted, the surviving pycache/<name>.cpython-3XX.pyc (from an earlier import) is the best recovery source -- it preserves function names, imports, constants, and line numbers, so reconstruct from it rather than stubbing. When tooling targets a .gitignore'd framework dir, read the API from GIT HISTORY, not the working tree (the ignored dir held only cached bytecode while the real source lived in history); and beware implicit layout conventions in path resolution -- a resolver computed package_source_root = framework_dir.parent/'src', so --agent-framework-dir /x/coborg_framework silently implied /x/src/cogames_agents/..., invisible until it resolves to a stale checkout, so make adjacent-layout assumptions explicit in errors and docs. Avoid copying numeric constants to dodge an import cycle: put shared constants in a tiny dependency-free module and re-export through the package's constants module. Clean engine-checkout debris immediately -- a temporary generator with a misparsed arg wrote a stray file literally named -- into the engine root; detect/remove such debris and harden arg parsing (read the last arg so both invocation forms are safe). Graphite non-interactive submit aborts when a stacked branch's PR is already merged but trunk lacks it; gt sync to update main, gt move the branch onto fresh main, and resubmit. To recover deleted files with exact prior content, restore from the commit BEFORE the deletion: verify the file exists at <delete-commit>^ then run git checkout <delete-commit>^ -- <path>, confirm fidelity by matching restored line counts against the deletion commit's diff stats (note git checkout <commit> -- <path> auto-stages; git reset HEAD <path> to unstage). _{sources: codex:019e05ca-d8e8-7d02-adf9-43354a64543b, codex:019e1940-5a75-7f51-b057-63f894ddb061, codex:019e147f-2fcd-7e83-93ab-fab57ce504b4, codex:019eaee5-b54e-79f2-bf0f-ac59a6a1eae2 (+2)}

38. Coerce forwarded kwargs to declared types (bool before int); deep-merge config tweaks

loop

When a runner forwards CLI kwargs as kw.<name>=<value> strings, coerce each string to the declared dataclass field type before applying it, and check bool BEFORE int (bool is an int subclass, so 'True' coerces wrong otherwise); route nested config via a prefix like nav__<field> and WARN (don't error) on unknown knobs, since sweep harnesses share one kwarg set across multiple policies. Prefer a launcher that deep-merges an inline JSON config tweak with the defaults (e.g. launch_server.py --config-json '{...}') so you can change one field like obstacleCount without re-specifying the full roles/rounds config; named-preset, full-file, and inline-JSON config inputs are mutually exclusive. _{sources: optimizers/.cursor/skills/scripted-navigation/SKILL.md, optimizers/.cursor/skills/scripted-navigation/reference.md, players_checkouts/players/users/james/personal_cogs/persephone/GAME_AP, players_checkouts/players/users/james/personal_cogs/persephone/README.}

39. Enumerate coworlds org-wide by combining local manifests with a GitHub code search, excluding schema/template files

loop · negative result

To enumerate ALL coworlds org-wide, don't rely only on local clones (a biased subset): combine a local manifest scan with a GitHub org-wide code search for coworld_manifest*.json, then shallow-clone the missing repos into /private/tmp and parse their manifests rather than trusting search snippets. A broad glob also matches the Metta schema/template file, producing a false 'missing grader' hit, so narrow validation to actual world manifests and exclude schema/template files. A diffable baseline source tree is resolved automatically during the games sync by trying candidates in order: (1) the Metta-AI/coworld-<gameSlug> GitHub repo under players/<playerId> (default branch via the GitHub API), (2) manifest player/game-source doc pages, (3) other manifest doc URLs pointing at a coworld-* repo, and (4) a structured artifact fallback (baseline.json + README.md) that is always non-empty; private repos authenticate via GITHUB_TOKEN/GH_TOKEN or a logged-in gh CLI. _{sources: codex:019e746d-062c-78d3-a0ed-ef2fcb892baf, coworld-source-repos.crewrift-upload/optimizers/.cursor/skills/downloa}