Perception & state decoding — recipes

On-demand recipes (57). Each is a trigger→action heuristic; pull the relevant one when its situation arises. Tier tags and ⚠ (unverified) as in AGENTS.md.

1. Split a scripted game player into perception / belief / policy layers with a stateless perception core

loop

Structure a scripted policy as thread-safe layers and enforce the invariant that raw frame bytes/pixels NEVER leave perception: perception is a pure stateless parser parse_frame(raw)->FramePerception (one frame in, typed symbolic facts out, no memory, reports OBSERVATIONS not interpretations -- emit 'a color-3 dot on the minimap', not 'Hades is nearby'); belief is a separate step merging facts into persistent state (fast loop writes, slow loop reads under a lock); selection/policy reads belief and emits action mask + chat, consuming typed intents only. Keep raw frame buffers, token streams, and pixel atlases OUT of belief. Keep the raw scan layer (actors, bodies, ghosts, task icons, radar dots, OCR) SEPARATE from the task/belief state machine: raw scans emit only match records while icon-to-task assignment, radar projection, checkout latching, and target commitment live in the policy. Percepts must be typed, bounded, ADDITIVE (new detector fields add without changing existing meanings; to structure new info like whisper messages add a typed belief field plus parsing, NOT logic inside perception; probabilistic detectors preserve uncertainty and hard-vs-weak provenance). Build perception->belief as one pure mutate-in-place apply(belief, perception, previous_view) that threads the prior view explicitly so transition-triggered logic (non-Lobby->Lobby reset) is testable with one canned frame; add new perception capability inside that hook. Honor hard ordering dependencies (actor and task scans need camera localization first; task-icon scan after actor scan; ignore-mask exclusions stamped in a fixed documented order so later scans don't re-detect earlier sprites). Pick ONE canonical perception channel and enumerate documented exceptions rather than mixing sources ad hoc. This split is what lets perception be unit-tested against captured fixtures independently of the agent. Distinguish a per-frame View from a high-level GamePhase and map many views to one phase (WAITING_ENTRY/WHISPER/GLOBAL_CHAT/INFO_SCREEN -> PLAYING); split a view into its own type only when policy semantics differ. Treat the belief->LLM snapshot as its own versionable module that converts screen coords to WORLD/room space and derives non-first-class events (body seen, meeting started, role revealed) by edge-detection across frames. For an LLM-driven player keep moment-to-moment action selection on the deterministic symbolic reflex+mode path and gate the LLM to social/meeting/voting phases (toggled e.g. by GUIDED_BOT_LLM_DISABLE), emitting a guidance_suppressed trace so the symbolic-vs-LLM boundary is observable. CAVEAT: high perception/localization quality does NOT imply good behavior -- localization can be 96-97% accurate while the policy is useless. _{sources: players_checkouts/players/archive/players/among_them/coborg/DESIGN.md, players_checkouts/players/users/james/personal_cogs/persephone/README., DESIGN_perception.md (Persephone perception module design), auggie:e251085a-b35d-4171-8dbd-d006b8bc2bd9 (+17)}

2. Order memory subsystems spatial->declarative->temporal with clear write-ownership

crewrift · ⚠ session-derived, unverified

Structure a multi-subsystem agent memory layer (spatial, declarative, temporal) with clear write-ownership and update order. Each tick update spatial first (it establishes the agent's absolute position via localization), declarative second, temporal last, since everything keys off the spatial position estimate. Let ONLY the decision layer write declarative memory while spatial and temporal are populated from observations, so perception layers are not polluted by belief/strategy state. Implement temporal memory as keyed ring buffers with per-key capacity and a configurable sampling policy (every_tick / every_n / on_change) rather than logging every signal every tick; on_change sampling is what makes cheap stuck-detection and trend queries possible. Decide the natural ownership scope of each belief field before adding it: place cross-cutting transient timers (a grace-period counter, a previous-pip-count describing the bot's overall view transition) on the per-bot state object, not the per-station/per-slot object, or one entity's timer resets another's. _{sources: codex:019e05eb-dbd6-76f0-8bf4-3ef821cb7cc6, codex:019e0850-2833-7773-a5e4-80ee662745c0}

3. Decode the crewrift Sprite v1 scene from retained tables, labels and object-id ranges, not pixels; re-derive the vocabulary from game source

crewrift

Crewrift Sprite v1 is stateful and incremental with NO whole-frame message: maintain three retained tables -- Layers (u8 id), Sprites (u16 id with width/height/UTF-8 human-readable label/RGBA), Objects (u16 id with x/y/z/layer/sprite_id) -- and apply each server update (define/delete-object, define-sprite/layer, set-viewport, clear-objects). The first message is an init burst (clear-objects, define-layer 0, set-viewport 128x128, static sprite defs); thereafter each 24Hz tick carries only changed objects, so a bridge must TOLERATE partial startup frames (don't assume the map object or walkability sprite arrived). Reconstruct state from the triple (object-id range, sprite label, xy); only the walkability map and shadow/vision overlay need pixel decoding. Identify entities by sprite LABEL plus object-id RANGE (no CV): players 'player <color> right/left' at ids 1000+joinOrder; bodies 'body <color>' at ids 2000+i (link a body to its player by COLOR, not id); task bubbles ids 3000+idx; the color vocabulary is the 16 PlayerColorNames. Object-id ranges are stable and disjoint, so the same label means different things across ranges: the voting candidate grid (~9300+) gives an authoritative per-color alive/dead census every meeting ('player <color>' alive vs 'body <color>' dead) and the role-reveal icon range (~9500+) shows an imposter its teammates. Read game time from the invisible 1x1 'tick <N>' marker sprite (id 5016) re-sent every frame, not a local message counter. These label strings and id bases ARE the perception contract but are NOT in sprite_v1.md and are version-sensitive -- when perception breaks after a version bump, suspect object-id ranges, sprite labels, or camera offsets drifted and re-derive them from coworld-crewrift src/crewrift/{sim,global}.nim (including the button-to-action contract in sim.nim applyInput, since gamepad buttons map to low-level attack/vent/move, not named actions). Over Sprite v1 the LABEL path is authoritative (notsus's spriteDetectionsReady), so ignore legacy CV/OCR/patch-hash machinery. _{sources: coworlds/coworld-crewrift/players/notsus/README.md, metta/agent-plugins/kitchensink/skills/ks.observe-protocol/SKILL.md, personal_labs/crewrift_lab/best_practices.md, personal_labs/crewrift_lab/crewrift/crewborg/AGENTS.md (+2)}

4. Crewrift Sprite-v1 coords are camera-relative; self is the implicit camera center -- compute self-position from constants and all motion in world coords

crewrift

Crewrift's Sprite v1 /player stream is structured scene data, not a framebuffer: the server streams object placements with exact coordinates plus sprites carrying text labels precisely so agents read state from structured data with NO computer vision or OCR. The only image decodes needed are two alpha masks (static walkability, dynamic LOS shadow); the whole-screen pixel sprite-decomposition approach was abandoned as too CPU-bound. The single MOST IMPORTANT fact (not in sprite_v1.md): object x/y are CAMERA-RELATIVE (0-128 screen range), not world space. Recover the camera from the world-map object (object id 1, sprite id 1, at (-cameraX,-cameraY)) so worldX = obj.x + cameraX. Your OWN avatar is NOT a scene object -- it is the implicit camera center; derive self world position by inverting the camera relation (cameraX = (player.x - SpriteDrawOffX + SpriteSize/2) - ScreenWidth/2; with SpriteSize=12, SpriteDrawOff (2,8), screen 128 -> self_world = (camera_x+60, camera_y+66), valid from tick 1; in Among-Them screen ~(60,66), cameraX=player.x-60, cameraY=player.y-66, full 128x128 window, no view radius) and read your own color/role/state from HUD sprite labels (imposter/ghost icon). Get this transform exact and confirm it round-trips before building nav -- every movement target depends on it. Do NOT compute velocity from on-screen self position -- it reads the same center forever and reports zero velocity even while sprinting, silently breaking anti-stuck jiggle and hunt/flee checks; compute velocity and all distances/commitment from WORLD/camera coordinates (camera delta). World coords are unavailable until the map object arrives (degrade gracefully on first ticks), and camera state resets on clear/delete of the map object -- handle that reset inside the decoder. Two pitfalls: a perception adapter that paints a 'visible' stub for every off-screen entity clamps their screen coords to the same edge pixel giving near-identical distances so the pick flips on noise -- treat any always-visible stub as a candidate-oscillation source; and sprite/draw coordinates and collision coordinates have DIFFERENT origins (player.x is the collision corner, sprite is centered, ~4px gap), so reuse the existing kill/report-range conversion before adding CollisionW/H/2 rather than reinventing it. Verify protocol facts against the notsus reference (protocols.nim, sim.nim, global.nim); rewrite any AGENTS.md/design section that implies CV for Crewrift. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/lively_lecun/ROADMAP.md, coworlds/coworld-crewrift/players/notsus/README.md, personal_labs/crewrift_lab/crewrift/crewborg/AGENTS.md (+10)}

5. Decode 4-bpp packed PICO-8 framebuffers carefully; use exact palette matching and asymmetric fixtures to catch nibble/threshold bugs

loop · negative result · ⚠ session-derived, unverified

Among Them / Coworld / Persephone pixel framebuffers cross the wire as a 4-bits-per-pixel packed byte array (8192 bytes = 128128/2) of PICO-8 palette indices, with no structured state API -- all information (phase, roles, positions, chat, offers) is extracted visually. For pixel (x,y) the byte index is (y128+x)//2 and the value is the LOW nibble (bits 0-3) when x is even/left, the HIGH nibble (bits 4-7) when odd/right, stored left-to-right top-to-bottom. Wrong nibble order silently mirrors/garbles all downstream vision, so decode one saved frame to PNG and eyeball it against the known 16-color palette before building any parser. Tests MUST use asymmetric, distinguishable fixture values (distinct high/low nibbles, ascending sequences), never uniform fills -- a decoder unpacked the high nibble first when spec was low-first but a 0x0F fill made order irrelevant so the test passed and the bug shipped. For 16-color exact-palette-index pixel-art matching with a shadow-equivalence LUT, direct EXACT pixel matching beats every standard technique because data is exactly quantized not noisy (FFT cross-correlation assumes L2 not categorical equality; integral images need separability; downsampling can't average palette indices; SIFT/ORB are poor on sparse-gradient pixel art; full-frame hashing is useless as every frame is unique). Template/shape matching can pass on busy backgrounds yet silently fail on uniform ones: sprite outlines drawn as color 0 cleared a 70% threshold on a colorful overworld but were indistinguishable from a black HUD background, so add a context flag (outline_is_black=True) accepting the bg color as valid outline and verify the matched-pixel fraction in EACH distinct background context. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, players_checkouts/players/users/james/personal_cogs/persephone/GAME_AP, players_checkouts/players/users/james/personal_cogs/persephone/README., players_checkouts/players/users/james/personal_cogs/persephone/RULEBOO (+5)}

6. Do all pixel perception in palette-index space, and know the ShadowMap collisions

crewrift

For a fixed 16-color indexed renderer (Among-Them/crewrift, indices 0-15), parse in PALETTE-INDEX space, not RGB, because tinting, shadows, and UI are all index transforms; index 255 is a transparent 'don't draw' sentinel that appears only in sprite source data, never in a received frame. Shadowed floor is recolored through ShadowMap[idx & 0xF], and ShadowMap collapses indices 1/13/14 down to 12 (void), so a shadowed dark-navy/light-blue tile is indistinguishable from out-of-bounds void; when scoring frame pixels against the map you MUST accept both the normal map color and ShadowMap[mapColor] (without this, localization fails in dark rooms like Electrical), and decide ambiguous tiles from the wall mask / shadow geometry, not raw color. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/how_to_make_a_bot.md, opencode:ses_2051bd225ffefNc9y2aOEEFjHW, opencode:ses_21eeac2baffegQWJK4QL6GuTEI}

7. Compute line-of-sight geometrically by mirroring the renderer's raycast, not from pixel color

crewrift

Shadow/out-of-LOS is a palette remapping (ShadowMap) that collapses several blues onto the void index, so you cannot read visibility from raw color; compute it geometrically from the wall mask and known camera origin. Among-Them uses an integer Bresenham raycast (steps = max(|dx|,|dy|)) from the player to each pixel: floor behind a wall is shadowed but walls themselves stay visible, living viewers are shadowed, ghosts see unshadowed, and there is NO circular view radius (LOS reaches the full screen edge bounded only by walls). Make the parser's entity-visibility test MIRROR the renderer exactly (entity drawn only if its center tile is unshadowed and on-screen) or you get phantom/missed entities. In Crewrift the per-player shadow overlay (object 13000, sprite 5010, transparent alpha = visible) gives this raycast result directly, resent on every camera move with no staleness, absent for ghosts and meetings. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, personal_labs/crewrift_lab/crewrift/crewborg/design.md, opencode:ses_21eeac2baffegQWJK4QL6GuTEI}

8. Determine PHASE first via an ordered cheapest-most-discriminating cascade that enumerates every render path; read the renderer source for ground truth

crewrift · ⚠ session-derived, unverified

A pixel/sprite parser must determine game PHASE before interpreting anything else, then dispatch to a phase-specific reader -- only the play phase has real world vision, the rest are fixed-coordinate menu/grid layouts, so one universal layout parser is a dead end. In Among Them (pixel) use tiny-font OCR for phase labels (WAITING, IMPS, CREWMATE, SKIP, CREW WINS, IMPS WIN, DRAW, NO ONE); if none match but bottom-right shows digits with no icon grid it is Playing. In Crewrift Sprite-v1 there is no phase field: read phase from which interstitial TEXT sprites are present, and learn your own role mid-game from imposter/ghost HUD overlays. A single logical phase can have multiple RENDER PATHS (leader-in-chatroom vs non-leader-in-overworld), each drawing different text/colors, so phase-detection must enumerate every render path -- one phase appeared once as a round clock in color 2 (detected as generic 'playing') and once as a colored phase label, and a single rule missed half the cases. When two phases share gross visual features (both a double border plus black interior), a new phase gets swallowed by an existing branch, so place the minimal most-discriminating distinguisher BEFORE the generic fallback (specific phase-label reads at color 8 / color 1 at x=42 ahead of the round-clock 'PLAYING' check, since the round clock is present in many phases and shadows the specific signal), order cheapest-most-discriminating first, and document why each ordering exists. Phases that look structurally identical are often disambiguated by a single visual signature such as BORDER COLOR -- classify on the discriminating pixel (a neutral palette index for roster/summary panels vs the player's team color for the role-reveal panel) rather than OCR-ing/template-matching panel contents. To split one class into finer sub-views, classify by unique anchor signatures plus elimination (a 'YOU ARE' anchor for one panel, a 'ROUND' header for another; assign the remainder only by elimination, leave None when nothing confidently matches). Read the game server's RENDERING code, not just the player's perception code, for ground-truth pixel layout (a draw routine gave cell 16x18, max 8 columns, startX=(128-cols*16)/2, startY=42); HUD text-detection bugs hide as 'phase X never gets detected' with root cause a wrong (x,y) read position or wrong color filter (glyph OCR reads left-to-right and stops at the first non-matching glyph, so a wrong x or palette color silently returns an empty string). When restructuring a detector that returned early after parsing one case, move the early-return so classification runs first and field parsing stays gated to its panel. Ground heuristics in real recorded golden frame fixtures (128x128), not doc coordinates -- when doc and pixels disagree the pixels win -- and assert classification of all golden phase fixtures as a per-phase test gate (an aggregate accuracy number hides per-phase regressions). A phase the renderer produces but perception never detects is a latent player bug only cross-checking surfaces. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, personal_labs/crewrift_lab/crewrift/crewborg/AGENTS.md, personal_labs/crewrift_lab/docs/crewrift-player.md, claude-code:fa645b7b-4fa0-4b55-992e-b273ee703391 (+9)}

9. Detect view/phase as an ordered priority cascade; gate interstitials by global black-pixel percentage and clean state on transitions

crewrift

Multi-phase games render radically different screens (lobby, role-reveal, play, voting, vote-result, game-over), so classify the PHASE first then dispatch to a phase-specific reader. Use an ORDERED disambiguating cascade (black-interior check, room-name prefix check) because views share pixel features (role-reveal vs info screen, chatroom vs global chat), and handle text VARIANTS explicitly (global-chat SHOUT variant, info-screen KNOWN variant, centered vs left-aligned game-over). Detect interstitials by GLOBAL black-pixel percentage >= 30% (the sim renders off-map gameplay space as MapVoidColor, not black; the older 'four corners black' test broke when chat/borders touched the corners) and gate them before any localization. On ENTERING an interstitial clear gameplay state and parse result/voting text; on LEAVING clear voting state, reseed localization from the remembered home, and clear stale path/goal/task-hold/velocity/jiggle because the sim teleports players home after voting. An agent can HALLUCINATE a view that never existed (a 'CHATROOM' view) -- validate the real view list against the reference implementation, not earlier agent assumptions, and scrap fabricated parsers entirely. _{sources: bitworld/among_them/players/SMART_BOT_GUIDE.md, bitworld/among_them/players/how_to_make_a_bot.md, opencode:ses_20b179f2fffeJe4vpbx2D4tWuR, opencode:ses_20c066070ffeBjXMnfQULai0Lw (+2)}

10. Enumerate and classify every game phase; interstitials are phases, not skips, and are the one-time channel for hidden setup info

loop · ⚠ session-derived, unverified

Interstitial/intro screens are distinct game PHASES a pixel-perception player must classify, not skip: a player that only knows 'lobby' and 'playing' will misread enumerated pre-play phases (roster panel, role panels, schedule panel), and a missed special screen (voting/meeting) passes through as gameplay, manifesting as silent paralysis (no-ops for the whole phase), not an error. Enumerate EVERY phase the server can render and decide per phase what the policy does (usually: detect, extract one-time info, then idle), and validate any pixel-based classifier gate (e.g. a single >=30%-black-pixels threshold) against every special screen, not just the obvious ones. Reveal/intro screens are the one-time channel for hidden setup info (room assignment, color, sprite/shape, alliance/team, own role, full role list, schedule) shown exactly once and never re-shown -- a policy needing it must PARSE and PERSIST it during the reveal phase. Actively MANAGE the intro sequence: hold a minimum dwell per panel so perception can parse, emit idle/no-op to avoid accidentally advancing, and gate advancement on a concrete belief-state condition (all room assignments populated, my_role set) rather than a blind timer. Beware optimizing by short-circuiting gameplay sprite/actor scanners on OCR-classified interstitial frames: information visible ONLY during the interstitial (teammate identities on a role-reveal screen) is on the pixels but then never extracted, so before assuming the player knows something shown at game start, verify the pipeline actually scans that frame. _{sources: opencode:ses_1fb2002bcffeK2Ldmjf34wdpM3, opencode:ses_20101b42dffeJU331wyDQFEn9W, opencode:ses_2010d071bffeCAzYxkexP7hIQ8, opencode:ses_201713d2dffe7pZ6AcZMDZXvNZ (+1)}

11. Detect the Persephone view from ordered fixed pixel probes, and give pending-entry its own enum

crewrift

Detect the current Persephone view from fixed pixel probes in a fixed ordered sequence (not a state field), because views share visual elements and order resolves ambiguity: check the double-border first (matching non-zero double border at (0,0) and (2,2) with black at (4,4) = role/roster/info reveal), then OCR header text at fixed offsets in specific palette colors ('R'...':' = playing, digit/digit = lobby, 'WHISP' = whisper, 'REVEAL' = reveal). Caveats: HUD titles are not always left-aligned -- the hostage-exchange screen renders its title centered at y=14, so detect it with a multi-x-offset scan; and the global-chat header suffix varies across versions (older 'CHAT', current 'SHOUT'), so accept both. Treat a waiting/pending-entry state as its own view enum even though it is mechanically the overworld with a special bottom bar, so the agent knows not to press buttons that would cancel the pending request, while still populating the still-extractable overworld spatial data. _{sources: players_checkouts/players/users/james/personal_cogs/persephone/GAME_AP, DESIGN_perception.md (Persephone perception module design)}

12. Read your own role/color from the fixed HUD corner sprite and center-spawn; make role/phase derivation STATEFUL and never latch a default on frame one

crewrift · negative result · ⚠ session-derived, unverified

Determine the viewer's role from the single corner HUD icon, not the player avatar: in Among-Them at screen (1,115) = SCREEN_HEIGHT - SPRITE_SIZE - 1, ghostIconSprite=ghost, killButtonSprite=living imposter (drawn shadowed when kill cooldown > 0), no icon=crewmate. Exploit deterministic game-start invariants for cheap self-identification: the server spawns the controlled player centered in the camera on init, so read your own color from the center sprite at game start. Cross-check every hardcoded HUD coordinate against the server render code AND a working reference bot (a comment claiming (109,110) 'must match' another bot was false; a wrong anchor makes the feature silently never fire). Phase and role derivation must be STATEFUL because downstream behavior modes branch on them and never activate if unset: a purely stateless derive_phase returns 'unknown' during ordinary play and sticks at RoleReveal, leaving crewmate Normal mode inert because it never sees Playing. Infer Playing once a reveal/meeting clears (or via the task counter on a mid-game join), and set self_role explicitly via the DEFAULT branch (derive 'crewmate' once in Playing with no imposter/ghost HUD marker present) -- omitting the alive/default branch means alive crewmates never get a role. Do NOT latch RoleCrewmate on frame one when no positive evidence is seen, because a broken detector or a not-yet-rendered HUD then permanently locks the wrong role; default only after N frames or derive role from the role-reveal interstitial classification, and reuse an existing debounce (e.g. the ghost-icon 2-frame threshold) rather than a 'not found -> assume X' branch. Enumerate every phase/role the deriver can output, test it transitions out of the initial/reveal state, and note role is not binary (a snapshot can carry role:'dead'). _{sources: bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/how_to_make_a_bot.md, personal_labs/crewrift_lab/crewrift/crewborg/AGENTS.md, claude-code:4430a077-5411-426b-a87a-75512ec4300f (+7)}

13. Don't latch role/identity on one sprite frame; require K frames, cross-check, gate the WRITE, and provide a logical fallback

crewrift · negative result · ⚠ session-derived, unverified

For role/identity classification driven by a single sprite or pixel match, never let one positive frame flip a latched state -- one sprite collision at a fixed HUD position can misclassify the agent's own role. Require K consecutive positive frames before promoting, use a one-way latch for irreversible transitions (crewmate->imposter happens mid-game; imposter->crewmate cannot), and cross-check cheap independent channels (if interstitial OCR already asserted 'CREWMATE', require a stricter threshold before a HUD sprite override). But a stability gate is mathematically unreachable if the triggering screen lasts fewer than K frames, so verify the trigger window exceeds the requirement -- and first compare to existing working baselines: upstream bots simply scanned reveal-screen sprites and immediately stamped colors with no histogram/24-frame gate, and that simpler proven path was more correct. Gate the WRITE to the belief field, not just the detector: a state field unconditionally overwritten every frame has no stable value (belief.self.role was reassigned whenever roleUpdated was true with no guard, so any false positive flipped it permanently; the fix changed the promotion guard from 'prevRole != Imposter' to 'prevRole == Unknown' -- only promote from Unknown). State derived from a single brittle sprite match also needs a logical fallback: ghost detection keyed only on a ghost-icon sprite left a dead bot unaware it was a ghost, wasting 160+ seconds in its alive-only hunting loop -- infer such state from orthogonal signals (e.g. 'kill_ready false indefinitely since a meeting'). Symmetrically, an N-frame confirmation threshold permanently LOSES high-stakes cues visible for fewer than N gameplay frames (a ghost HUD icon cleared by a meeting a few ticks after a kill), so lower the threshold for high-stakes cues and re-arm/persist-and-recheck on the first gameplay frame after the interstitial ends. _{sources: players_checkouts/players/users/james/personal_cogs/among_them/guided_, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56, claude-code:91f456a7-41e8-4a89-98e9-44e379160b15, codex:019e08db-c4f1-7ec1-8f76-108fbbe21008 (+4)}

14. On the role-reveal screen, stamp durable facts and know the crewmate/imposter sprite asymmetry

crewrift

Parse the role-reveal interstitial text and stamp durable per-game facts: CREWMATE sets the crewmate role; IMPS marks self imposter AND records the teammate colors shown so they are never later treated as kill targets or suspects; reset ALL per-round state on CREW WINS / IMPS WIN. Critically, the same sprite-grid render means different things by role: the crewrift server draws ALL players for crewmates but only the imposter TEAM for imposters, so the role-reveal scan directly reveals teammates when you are the imposter but is a trap if you assume the sprite set is symmetric across roles. _{sources: bitworld/among_them/players/evidencebot_strategy.md, bitworld/among_them/players/how_to_make_a_bot.md, opencode:ses_204eb0641ffeIAD293yQHlSd61}

15. Localize the camera with a tiered cost ladder against a map cache, masking ALL dynamic pixels, with a tighter threshold for the local search; wander to lock

crewrift

Localize a screen-reading player's camera against a known static map by comparing the live frame, pre-hashed into 8x8 patch fingerprints, over a tiered cost ladder: (1) cheap sticky LOCAL search within ~8px of the last lock (~1ms) with a TIGHTER mismatch threshold; (2) patch-hash GLOBAL search where frame patches vote for the best (cameraX,cameraY) and top candidates are fully scored (~10-170ms, robust to occlusion/HUD overlay); (3) spiral full-frame fallback. Before scoring ANY candidate, mask out all dynamic pixels (centered local-player sprite, other crewmates, bodies, ghosts, task icons, radar pixels, kill-button icon, ghost status icon) or matching fails. Gate acceptance on errors<=maxErrors AND comparedPixels>=a minimum, and use a tighter threshold for the hinted local search than for brute-force: clean frames score ~5 mismatches but cluttered mid-game frames hit 50-90, and a slightly-drifted lock can still pass a loose fixed 100-mismatch gate, so a near-miss should be rejected not accepted as drift. A pixel-localizer needs fresh diverse map pixels to lock, so before lock drive the bot with raw cardinal-direction wandering (cycle up/right/down/left every ~36 ticks) rather than waypoint routing -- a stationary bot may see only a static crop that never matches uniquely; once locked but still bootstrapping, steer toward map centre to feed diverse pixels and move off the spawn edge. Fuse independent cues for a robust fix (e.g. the colored dot on the 20x20 minimap plus floor-grid dot alignment). Run sprite scans against the PREVIOUS frame's camera, then localize, and only re-scan against the new camera when the lock jumps beyond a teleport threshold (~32px). Keep landmark-based localization as the absolute anchor and use observed local-position displacement only as a correction layer when last-action timing is stale or no movement was observed -- do not replace landmark localization with displacement integration. _{sources: archive/cogames_playground/docs/plans/bop-it-localization-equip.md, bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/evidencebot_strategy.md, bitworld/among_them/players/how_to_make_a_bot.md (+5)}

16. Localize from the highest-signal source; treat minimap as a fog-immune fallback only

crewrift · ⚠ session-derived, unverified

Localize the agent's current position from the HIGHEST-signal observation source (direct camera/viewport, designed to be 1-2 tick and accurate) and NEVER estimate current position from a low-signal aggregate like the minimap; enforce 'current position is never estimated from minimap' on the perception side and treat coarse sources as best-guess fallback only. When fog-of-war limits direct view, the minimap is a PARTIALLY fog-immune source for persistent location estimates, but verify its limits in the renderer source first because coverage is COMPLEMENTARY, not full -- out-of-viewport players bypass the minimap shadow check entirely (the shadow buffer is only viewport-sized and the bounds check short-circuits before the shadow lookup). Minimap dots typically encode only COLOR, not identity, so identity must be inferred: when a known player is co-located with a dot, bind that player to the dot and track its movement to maintain identity across frames; model proximity (sharing a private channel) as a LIKELIHOOD of co-presence, not a guarantee. _{sources: opencode:ses_1fb148d00ffe5ob6k0yKT3aHke, opencode:ses_1fb48dd11fferLGXS1vWdHa0wS, opencode:ses_200bab964ffe5MjqDy3HjInDmF}

17. Throttle expensive fallback parses with a tick-interval cooldown

loop · ⚠ session-derived, unverified

Add a tick-interval COOLDOWN (e.g. probe every ~12 ticks / ~0.5s) when running an expensive fallback parse triggered by failed localization. Probing every tick during any perception drop (kill animation, spawn) starves the localizer's re-acquisition, while a periodic probe still catches the target screen within ~1s of it appearing. Relatedly, make perception tunables discoverable and overridable: define a match threshold as a named module constant in one obvious place AND expose it as a function kwarg defaulting to that constant, so callers that know they have a clean/unoccluded sprite can pass a stricter value. _{sources: opencode:ses_20101b42dffeJU331wyDQFEn9W, opencode:ses_20463f5e6ffeWi1ooj1agqH80a}

18. Suspect perception before strategy and trace pixels-upward in stages; silent perception no-ops are worse than crashes -- trace consumers and alarm when a gate never fires

loop · negative result · ⚠ session-derived, unverified

When a vision-based player behaves mysteriously, an LLM reports it 'can't see' state, or a whole game state is ignored, suspect the perception/detection pipeline BEFORE strategy, and trace every layer frame -> OCR/decode -> keyword/region match -> belief update -> decision snapshot, instrumenting each stage independently: (1) OCR/decode never extracted the value, (2) extraction happened but belief/event code dropped it, (3) belief had it but the LLM/decision snapshot omitted it. A dedicated perception.jsonl recording raw OCR separately from belief localizes the break (outbound chat_sent worked but inbound chat_observed was empty -> downstream of OCR). 'Perception recognized the phase' and 'the policy acted on it' are SEPARATE failures: cross-reference the engine/server log against the policy's own trace (a server log 'vote called: lime called body' proved a vote screen existed while the policy trace showed voting:null, pinning the bug to parsing). 'X is detected but behavior never changes' almost always means the detection RESULT is computed but never wired into state -- grep for CONSUMERS of a signal, not just producers (an interstitial classifier returning InterstitialRoleReveal is useless if nothing maps it to RoleImposter; a radar-dot scanner can fill a radar_dots field task filtering never reads). A perception subsystem that silently fails to fire is more dangerous than one that crashes: meeting/vote parsing returning valid=false for every bot meant the policy never entered meeting mode, never voted, and lost to a server default, invisibly; a failed parseVotingScreen left the bot reading interstitial garbage and the voting state machine, gated on a voting.active flag the parser never set, silently never advanced. So when a perception gate guards a whole behavior mode, add an alarm/log that fires when the gate never passes during a window it should, gate state machines on POSITIVELY-CONFIRMED perception, emit a distinct diagnostic when a parser fails, and make the idle/unknown state do something observable (gentle exploratory movement) so 'warming up' is distinguishable from 'stuck'. Concrete chained failure modes seen: messages stacked from the bottom (y=104,111) while the parser scanned top-down (y=10,17); a leading sender sprite shifting text start x=2->x=13 past a space-skip peek; matching the engine log string ('shared roles') instead of the rendered string ('ROLE XCHG:'); a silently-dropped None lookup (decode_player_index returned None so pending_entry stayed None and entry was never granted). When OCR reads a value correctly but the parsed field is None, the bug is DOWNSTREAM in the parse/offset step. Before debugging any detection bug, confirm the expected signal even exists in the stream (dump a raw frame at the moment the server log says X happened and confirm the pixels are present AND parsed), add graceful fallbacks (color-only resolution), and guard every sprite/region lookup that can return None before slicing the frame. Often the real fix is exposing the right structured fields (alive_players, dead_players, self_can_vote), not prompt wording. _{sources: claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56, claude-code:91f456a7-41e8-4a89-98e9-44e379160b15, claude-code:aae25940-ade3-41bc-b817-517dc86ccebf (+15)}

19. Triage perception regressions with git and replay artifacts, not edits

loop · ⚠ session-derived, unverified

When perception 'worked recently' but now fails, separate 'the game changed' from 'our code regressed' BEFORE touching code: run git log -G on the parser and confirm the file is not dirty. A green local suite can be false confidence when the test uses a stale fixture (a color-ordered fixture passed while the real change was switching to the public coworld package/image flow). Reconstruct the exact failing frame from the replay artifact and run the real parser gates against it from a DISPOSABLE probe OUTSIDE the repo (a throwaway Nim probe under /private/tmp that replays the killed episode to the failing tick) -- replaying replay.json to the specific vote tick yielded a precise 'first_mismatch slot=0 expected=0 got=2' pinpointing the failing gate in one shot, while keeping the working diff clean. Keep replay artifacts as ground truth for perception debugging. _{sources: codex:019e2886-d576-7770-be1b-6d92ca97761e}

20. Diagnose a perception bot via the event trace first, raw frames second

crewrift · ⚠ session-derived, unverified

When behavior drifts after a fork, suspect game-graphics changes (sprite atlas, ASCII font, room coordinates, voting layout) before bot logic. The signature of stale perception is correct preconditions firing (body_reported) but a screen transition never detected (meeting_started/interstitial_entered never fire) -- missing high-level events localize the break to perception/state-detection, while spurious high-frequency events (kill-cooldown firing 984x, excessive self_color_changed) indicate a stuck/oscillating state. Only after the trace narrows it, dump the raw framebuffer (16384 bytes/frame here) around the suspect tick and render it as an image so a human can visually confirm the HUD layout, mapping report time to frame index via fps -- but remember the bot's internal frameTick can diverge from the capture index when frames are dropped. _{sources: opencode:ses_20b1744aeffettq8pxHNVgjKNC, opencode:ses_20b179f2fffeJe4vpbx2D4tWuR, opencode:ses_21f67d3f5ffeBF92agcd0cbOV0}

21. Build a frame recorder first; test against real captured frames not synthetic/noise, and validate at fixture AND live-match levels

loop · ⚠ session-derived, unverified

Build a frame/observation recorder BEFORE debugging perception so you capture exactly what the agent saw, not what you assume: dump every tick's raw frame to disk (e.g. concatenated 16384-byte uint8 frames in frames.bin plus per-frame decisions.jsonl, events.jsonl, reflexes.jsonl, manifest.json). Build perception tests on REAL sim-rendered frames (a capture client recording a live episode's frame stream plus a timeline viewer), because synthetic frames don't reproduce calibration/palette drift, and build the harness with known-good expected outputs BEFORE fixing bugs (e.g. ~42 tests over ~10 live fixtures) or you cannot tell whether a fix regressed another view. Feeding RANDOM-NOISE frames is a weak signal: random pixels neither localize to the real map nor reliably trip the same code paths as real frames, so 'it returns noop on noise' proves almost nothing. Validate a perception change at TWO levels: fixture/unit tests on real captured frames prove the parser matches/rejects, and a full live match proves detection actually triggers the mode transition end-to-end -- passing fixtures alone does NOT prove behavior changed. Also live-verify against a real server with a fixed seed: connect a WebSocket client, call parse_frame() per frame, and print the view PLUS every extracted FIELD (run both a 1-player lobby and a full 6-player game to trigger transitions), because per-field printing exposes partial failures a view-label-only check misses (a 'role=None, team=Shades, room=None' read surfaced a centered-text extraction gap). Make diagnostics ASSERT, not print: require real positive fixtures to parse valid AND representative negative fixtures (gameplay, interstitial) to be rejected, so the threshold cannot drift to always-true/false. Commit each fixture as a frame (.npy/.bin) paired with a .json of expected assertions; strengthen integration tests beyond single-frame parsing with temporal stability (same view across N repeated frames), state-transition cleanup (alternating playing/whisper frames clearing stale chatroom state), and an exhaustive synthetic matrix (every shape x color x {normal, shadow, partial-shadow}); use explicit skip markers tied to fixture availability so a test announces a coverage gap and auto-activates once the .npy fixture is captured. Land small isolated offset/flag fixes first (one constant each with a focused unit test asserting corrected coordinates) before a larger structural change so a regression bisects to one edit, and prefer a standalone per-region sub-parser module over burying region logic in a general parser. Before writing assertions, probe canned fixtures through the real parse_frame to learn which fields actually populate ('raw frame has X but belief lacks X' is expected layering, not a bug). Caveat: fixtures captured DURING a bug encode the OLD buggy expected outputs, so after a fix update assertions to the corrected truth (RoleCrewmate -> RoleImposter) rather than reverting, and a downstream mode transition (ModeTaskCompleting -> ModeHunting) confirms the fix flipped behavior. Your opponent/filler-bot policy shapes which sub-states you can observe, so some fixtures must be choreographed (a waiting_entry view couldn't be captured because aggressive fillers always initiated whispers first; fix with less-aggressive fillers or two-bot choreography). Pin concrete numeric edge cases for capture tooling: change-detection threshold boundaries (at vs just-under), all-black vs all-white max-diff frames, identical consecutive frames (zero diff), and the first frame (no prior to diff). _{sources: claude-code:fa645b7b-4fa0-4b55-992e-b273ee703391, codex:019dffc3-a08f-76b1-8b07-8a16f6e5ae38, codex:019e004f-1647-77a1-9e58-fa29c8f8ef89, codex:019e0691-0971-7623-9fb2-5b1aed778856 (+10)}

22. Diagnose frame delivery/latency before blaming a parser; prefer the server-authoritative signal over local estimators

loop · negative result · ⚠ session-derived, unverified

Frame delivery lags the game-state tick and can stall, so diagnose delivery before blaming a parser or threshold. HUD/visual signals lag server state by one+ frames (a kill cooldown set on tick N is visible as the shadowed sprite only at N+1+), so don't treat a freshly-changed pixel as instantaneous truth -- allow slack or confirm against an independent signal. A server may resend an identical frame for hundreds of ticks (observed 360+ voting ticks holding ~2 unique frames), so count UNIQUE frames over the window before concluding a parser failed; detect frame-TYPE switches by diffing a cheap per-frame statistic (black-pixel count jumping ~7%->~93% in one tick signals a server-side switch, not a gradual transition). Prefer the server-authoritative rendered signal over a shadowing local estimator and delete the duplicate: a killCooldownRemaining field never updated (always 0) shipped to the LLM as a constant lie while lit-button perception was the real kill_ready gate -- when a binary perception flag never flips, check the server's draw code for the exact ready condition before adding state. Before adding new protocol, check whether the renderer's info is already decoded and thrown away: a vote-timer was collapsed to timer_present:bool, discarding SpriteDef width the decoder already keeps, even though a shrinking-width bar is a drift/latency/early-close-proof server-authoritative fraction-remaining strictly better than a local frame counter. _{sources: claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40, claude-code:3837ef5e-3e9c-41bc-b290-df5865581698, claude-code:4ec865f5-bcbc-44f1-b109-a27d69f55506, claude-code:73fa0a3d-37fb-4408-bae8-245db6e62c56}

23. Treat the renderer source as the spec for every pixel offset; invert algebraically and accept multiple known scan positions

loop · negative result · ⚠ session-derived, unverified

Treat the game's renderer source (border colors, title position, column offsets, per-row stride, label format, alignment anchors) as the authoritative spec for every pixel offset a parser reads -- not parser constants, docstrings, or screenshots. Derive the inverse algebraically: if the renderer draws a feature at (sprite_x - dx, sprite_y - dy), the parser must recover sprite = (feature_x + dx, feature_y + dy); an off-by-one (+3 vs +4) silently misreads state with no crash. In Crewrift Sprite-v1 specifically the DRAWN object position is NOT the collision/interaction point (entities drawn at (x - SpriteDrawOffX - 1, y - SpriteDrawOffY - 1) while report/kill/collision logic uses raw x/y, a (+3,+9) offset), so add the collision offset uniformly in the resolve step or every distance check is silently off. Right-aligned/width-dependent HUD text means start-x depends on rendered width, so compute scan bounds from the alignment anchor minus max text width (start = anchor - max_chars*char_width) and scan multiple x-offsets for centered titles. Fixed-coordinate parsers silently break when a client UI moves between versions (a 'WHISP' header moved (2,2)->(42,2) so the agent read the round clock and misclassified the view as PLAYING): ADD the new scan position while keeping the old one (accepting multiple known positions kept 475 tests green and tolerated drift). You can encode and test a phase against the renderer with synthetic frames reproducing the exact layout instead of waiting for a live session to hit a rare phase. The same 'port the server's exact algorithm' rule applies to off-screen target projection: a naive clamp-to-screen-rect is wrong for diagonally off-screen targets, so port the server's reference ray-clip from the player's screen position toward the target (handle the on-screen case separately). A clipped radar/minimap pip encodes only DIRECTION not distance, so reconstruct a world-space ray with origin at the agent's world position and direction = (pip_screen - agent_screen) (camera translation is pure), and add a t>0 forward check to the ray-AABB intersection or the ray 'hits' regions behind the agent. _{sources: personal_labs/crewrift_lab/docs/crewrift-player.md, claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, claude-code:4430a077-5411-426b-a87a-75512ec4300f, codex:019dfef8-dc40-7273-b85a-04aa3a51ff07 (+4)}

24. Derive on-screen positions from ONE shared server-matching formula and an exact noise budget, not margin-expanded boxes

crewrift · negative result · ⚠ session-derived, unverified

Use ONE canonical world-to-screen anchor formula per object and match the scanner's detected icon in that SAME coordinate space; audit every conversion path so they agree, because when one code path gates visibility and another does the actual match, the gate disagrees with the matcher and causes premature/false triggers (taskIconExpectedScreenPos did not match the anchor findIconForStation used, a ~14px premature trigger; a helper used a station's geometric center while the icon renders ~14px above the rect -- expected screen = obj.x + obj.w/2 - 6 - camX, obj.y - 14 - camY; a parser used sprite_y+3 while the renderer drew at (sx-3, sy-4)). Match by the server's exact render offset, NOT a margin-expanded bounding box: expanding each target rect by 16px overlaps rects in dense layouts where targets sit 20-30px apart, so one icon matches multiple targets and selection picks wrong; since the icon sits at a fixed offset, the rect is recoverable EXACTLY. Derive tolerances from the actual noise budget, not round numbers: the radar pip points at the task ICON center (~8px above the rect top, +-1px bob), not the rect center, so size the test rect from real jitter (12x12 icon + ~1px bob + ~1px screen quantization + ~1px player-position quantization -> ~14px half-extent / 28x28 box) and document each term. A single-exact-pixel probe silently disables the whole feature if off by a few px (self-color returned -1 for thousands of frames probing exact anchor (57,54) when the rendered top-left was (58,58)), so derive the anchor from the rendering source and probe a small tolerance window. Validate with a two-line manual arithmetic check on the exact failing case. A policy thrashing on a target it can never satisfy is usually ambiguous belief object-to-target attribution, not the action/nav policy. _{sources: codex:019e0105-2b01-71a3-8986-eb1e5269ea83, codex:019e04d1-5052-7b42-907f-c0a2e3d539a8, codex:019e05eb-dbd6-76f0-8bf4-3ef821cb7cc6, codex:019e132d-8773-7503-86cd-a77b0a8d21eb (+5)}

25. Separate OCR coordinate-calibration bugs from glyph bugs; render the frame and read off positions

generic · negative result · ⚠ session-derived, unverified

When EVERY text field on a view is misread by the same offset, suspect a GLOBAL screen-coordinate calibration error, not per-field glyph bugs: the dominant OCR failure was text y-positions systematically ~10px off (role name at y=28 not y=18, room name at y=56 not y=46). Layout/offset bugs are a different root cause from glyph-level bugs (wrong glyphs in the font table, false ambiguous-pair entries, a wrong binarization threshold), and treating one as the other wastes effort. Before touching parser constants, render the frame to an image and read off the actual (y, x, color) of each text block. Watch scan-window edges: scanning chat from chatY+2 instead of chatY+1 dropped the oldest message by one pixel of the glyph top row, so verify the window covers the FULL glyph height (ascenders live in the top row). _{sources: opencode:ses_20b179f2fffeJe4vpbx2D4tWuR, opencode:ses_21f8ce4f6ffe6Uh3krQPdNOwZe}

26. Tiny bitmap-font OCR: read coords/metrics from the renderer, use two-sided scoring, and handle ambiguous pairs, trailing space, odd-x centering

generic · ⚠ session-derived, unverified

Read the server's RENDER routine for ground-truth glyph metrics and start positions (an authoritative cheaper source than reverse-engineering from captures), and understand a fuzzy-matcher's error metric before blaming it: an OCR maxErrors=2 looked loose ('2 of 4 chars') but meant 2 mismatched PIXELS across the whole ~60-pixel string, and on a near-black screen every opaque 'IMPS' pixel mismatches, making accidental matches far harder than they appear. Tiny bitmap-font OCR has recurring failure modes: (1) S/5 and O/0 are ambiguous -- return raw text plus as_text()/as_numeric() helpers and let the caller disambiguate by context. (2) Naive 'fraction of glyph on-pixels present' scoring lets a subset glyph (A's pixels all lie within 8's) falsely score 1.0 -- use two-sided/false-positive-aware scoring that ALSO penalizes target-colored pixels the glyph says should be OFF (missing foreground + unexpected extras, tie-break on opaque-pixel count), the model the up-to-date sibling bot used. (3) A '4 empty columns = space' rule misreads the void past text end as endless spaces up to max_chars -- stop once no glyph content follows a detected space, since the renderer emits no trailing spaces. (4) Centered text can start on an odd x that a step-2 scanner skips entirely ('Hades' at x=54, 'Hello' returns None) -- scan centered text in steps of 1, and verify each field because working view-detection does NOT imply working field extraction. A trailing marker can carry semantic state -- a role name's trailing '*' denotes leader, so strip it from the name AND set an is_leader flag. _{sources: opencode:ses_1ffe75d89ffeHvW4p2ARluHh51, opencode:ses_200ab2780ffe2Zt927TZY4Y8Yf, opencode:ses_204eb0641ffeIAD293yQHlSd61, opencode:ses_20c066070ffeBjXMnfQULai0Lw}

27. OCR fails live after a font migration: source glyph metrics from the sim and reuse its font engine

loop

When OCR works in tests but fails LIVE, suspect a font/rendering mismatch, not the parsing logic. The classic breakage: a stale hardcoded glyph stride (a fixed 7-pixel stride while the engine migrated to a variable-width font with dynamic advances and a font-file-derived background color), silently garbling reads; modulabot's chat OCR mis-read every line until ported to the shared variable-width tiny5 engine (and its panel scan was off by one row, chatY+2 vs chatY+1, dropping first lines). The font path is often a compile-time constant while the font file loads at runtime, so check git log for font/OCR commits and compare binary build timestamps against them. Fix by REUSING the renderer's own font decode helpers and sourcing constants (text X offset, glyph stride, char count, panel width/start row) FROM the simulator, not per-bot copies. Beware glyphs composited by the NATIVE CLIENT (a chat input box from a separate ascii.png sheet drawn atop the framebuffer): they never appear in the server-sent frame, so a wire-frame parser must not expect them. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/modulabot/TODO.md, opencode:ses_21b008642ffeCM6Q48G44aM6k0, opencode:ses_21eeac2baffegQWJK4QL6GuTEI (+2)}

28. Match the Persephone 3x5 pixel font by filled/empty cells and outline color; re-verify glyph-identity claims

crewrift

Match a fixed pixel-font glyph by requiring target-color pixels to fall on the glyph's filled cells and non-target-color pixels on its empty cells, accepting at ~90% to absorb rendering artifacts. The 3x5 font supports only A-Z, 0-9, and :!?'.,-/*()<> and silently skips unsupported characters (#@%_+=[] and lowercase), so do not emit those in chat. Outline color is context-dependent -- Persephone renders outlines as color 0 (black) in HUD contexts (speech bubbles, pending-entry sprites, whisper header) rather than color 1, so a shape classifier needs an explicit outline_is_black flag to work in both overworld and HUD; reliable HUD detection (outline_is_black=True gave ~100% accuracy) can make a color-only id fallback unnecessary, but remove fallbacks only after verifying no regression on shared-color edge cases. NEGATIVE RESULT: do not trust a stale doc claim that distinct glyphs are pixel-identical -- the claim that S/5 and O/0 draw identically was wrong (the glyphs are distinct and OCR matches them unambiguously), so the parser's digit-normalization helpers (norm()) were demoted to no-ops; re-verify against the actual renderer bitmaps. _{sources: players_checkouts/players/users/james/personal_cogs/persephone/GAME_AP, players_checkouts/players/users/james/personal_cogs/persephone/TODO.md, DESIGN_perception.md (Persephone perception module design)}

29. Keep a documented legacy coordinate fallback so renderer moves do not break the parser

crewrift

Pixel/HUD parsers tied to renderer-specific coordinates break when the renderer moves elements; keep a primary coordinate plus a documented legacy fallback (e.g. chatroom header sprite at x=66 for the current renderer, falling back to x=22 for legacy fixtures) so both old fixtures and new frames parse. _{sources: players_checkouts/players/users/james/personal_cogs/persephone/TODO.md}

30. Detect actors by anchored sprite matching with tint-only color voting, structural-pixel anchors, and shadow tolerance

crewrift · negative result · ⚠ session-derived, unverified

For detecting on-screen ACTORS, prefer template-anchored brute-force sprite matching (for every (x,y) anchor try the template with flips=[false,true], since sprites flipH when moving left and a one-sided matcher misses half the actors) over whole-frame palette-index histograms: histograms are fragile because UI/text pixels leak into counts (white menu text read as a 'white player'), and anchored matching needs no N-identical-frame stability gate (an interstitial can appear and vanish faster than the required streak). Every upstream Among Them bot (nottoodumb, evidencebot_v2, ivotewell, italkalot, mod_talks, modulabot) uses this identical scan, so adopt the convergent pattern. Among-Them sprites use wildcard tint pixels: color 3 (TintColor) is replaced by the player's color and color 9 (ShadeTintColor) by ShadowMap[color]; detect a player by matching the FIXED non-tinted structural pixels (outline, visor, backpack) then read the dominant tint-pixel color for identity. Let ONLY the template's tint/body pixels vote on color, never stable structural pixels (outline, visor, eyes), to sidestep palette collisions; and ensure each template has enough distinctive non-background stable pixels (~18 visor+eye) to anchor even when the body color equals the background. Distinguish chrome from identity by PIXEL COUNT, not presence: visor renders as palette 14 regardless of player but 14 is also a real body color, so a real tinted body (~40 px) vs visor (~10 px) needs a FIXED absolute margin (count > stableContribution + 20, ~half a body) not a proportional one that collapses at 8 sprites; calibrate any pixels-per-entity divisor to the true ~63 non-transparent pixels per sprite (an /100 estimate under-counted 2 as 1). Make matching shadow-tolerant by accepting each pixel if it equals the normal OR shadowed value (handles fully-lit, fully-shadowed, mixed sprites in one path), but when shadow_fill == shadow_outline the sprite collapses to a solid block -- restrict matching to normal colors then and return None honestly when genuinely ambiguous. Histogram-with-baseline is only for the narrower PRESENCE (not location) question, subtracting each sprite's known stable contribution. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, opencode:ses_1f6f76557ffeCfmMTG7BPJwGRE, opencode:ses_1f7b6dc30ffeDmQtlUZI25HAjh, opencode:ses_20463f5e6ffeWi1ooj1agqH80a (+1)}

31. Decode identity from the unique (color, shape) pair, thread the new channel everywhere, and fuse complementary minimap/viewport coverage; report None honestly

crewrift · negative result

When an observable attribute has FEWER distinct values than the number of entities (8 colors but up to 24 players), identity from that attribute alone is ambiguous -- find a second independent attribute whose COMBINATION is unique (color x shape): lcm(8 colors, 12 shapes)=24 equals max player count, so if the product/LCM of value-space sizes is at least the max entity count and pairs are assigned by independent moduli, every (color, shape) pair decodes deterministically to one canonical index. Score shape templates against a 7x7 sprite with vectorized NumPy matching (matching/non-transparent pixels), accepting the best above a tunable threshold (default 0.70). When adding the new identity attribute, thread it through ALL downstream perception dataclasses in parallel with the existing one (add a shape field everywhere a color field exists: bubbles, chatroom occupants, exchange players, known players, message senders); missing one channel reintroduces ambiguity for that path. The minimap and viewport give complementary INVERSE coverage: the viewport gives full color+shape identity for nearby unobstructed players but cannot see far/behind obstacles, while the minimap reliably shows far-away players as color-only dots (they bypass the fog/shadow check since the 128x128 shadow buffer has no data for them) but hides nearby players behind walls. Minimap dots are color-only and up to 3 players share a color (index mod 8), so disambiguating a dot requires co-locating a viewport-identified player; the viewer's own dot is drawn last and overwrites others at its cell, so a closely-pursued target 'disappears' -- track lastSawTargetTick. Role indicators (pixel bars below sprites) reveal team and key-vs-grunt class, distinguishing the two roles within a key class only by dot position (center vs split), not color. Cap detected actors at the game's known maximum and reject scans that exceed it as noise (RoleRevealMaxDetectedColors = 3 discards any role-reveal scan returning more than 3 colors). A single-pixel sample is fragile (some shape templates have an outline/transparent center pixel) -- validate PER-TEMPLATE that any sampled pixel carries the expected signal and prefer scanning a region for the dominant signal. Make perception best-effort and honest: set fields to None on failure (fog, animation) and require agents to tolerate it -- when a player's shadow-fill color equals the shadow-outline color (blue and purple both map to palette 12), a fully-shadowed sprite's shape is genuinely unrecoverable, so report shape=None and let agents fall back to positional continuity. _{sources: players_checkouts/players/users/james/personal_cogs/persephone/GAME_AP, DESIGN_perception.md (Persephone perception module design), opencode:ses_1f6f76557ffeCfmMTG7BPJwGRE, opencode:ses_20463f5e6ffeWi1ooj1agqH80a (+2)}

32. Match task icons by bob phase and disambiguate the palette-9 / player-color collision

crewrift · negative result

Among-Them on-screen task icons (taskIconSprite) are palette index 9 and bob above the station on a 10-element cycle [0,0,-1,-1,-1,0,0,1,1,1] indexed by (tickCount//3) mod 10, so test all bob phases when matching at the 40 known task positions; off-screen tasks are a single yellow palette-8 pixel on the screen border (when showTaskArrows is on) -- reverse the player-to-pixel line for direction, matching radar within ~2px tolerance. NEGATIVE RESULTS: steering toward palette 10 chases map decoration, and a bare palette-9-cluster heuristic false-positives because blitSpriteOutlined tints players by replacing wildcard pixels with the player color, so a player whose color is 9 renders pixels indistinguishable from a task icon (and the walkability filter doesn't help, since players stand on walkable floor). Disambiguate real icons by multi-frame world-position stability (icons don't drift, players do), the +/-1px 3-tick bob, or sprite-template correlation -- never a global scan. _{sources: bitworld/among_them/GRAPHICS_REPORT.md, bitworld/among_them/players/lively_lecun/ROADMAP.md}

33. Gate visual detectors on positive confirmation of the right screen; rank sources by confidence with hard-over-soft and a behavioral fallback

crewrift · negative result · ⚠ session-derived, unverified

Gate a heuristic perception detector on POSITIVE confirmation of the right screen context (via OCR of a title/banner) before letting it run or write state. Color-presence heuristics for role/identity false-positive on screens that legitimately show all entities: a color-pixel scan ran on every interstitial including the lobby, and because the lobby and CREWMATE reveal screen both show all player sprites, once all 4 colors were stable for 24 frames the scan wrongly concluded all 4 bots were imposters at tick 41. The fix ran the color scan only once OCR confirmed the imposter-reveal screen (cachedInterstitialKind == InterstitialRoleRevealImposter), after which detection was correct (3 crewmates + 1 imposter). Before believing visual evidence for X, confirm you are on the screen where X-evidence is meaningful; localize premature-firing bugs by dumping per-frame interstitial/gameplay classification and comparing when state CHANGED vs when the triggering frame actually appears (counting near-black frames showed gameplay started at frame 137, yet the bot assigned its role at tick 41, before the real role-reveal at frame 50). Establish a confidence hierarchy among perception sources, with a HARD positive signal overriding a SOFT one, and never let a soft exclusion block a confirmation the agent can directly observe: treat OCR-read literal text as authoritative over single-position sprite/HUD matches (a crewmate was falsely flipped to imposter because a HUD kill-button sprite matched at one pixel while standing at a task station -- once OCR confirms 'CREWMATE', a noisy single-point sprite match must never override it). Gate soft signals behind 'is the hard signal currently unavailable?' (radar-pip gating applies only when the task icon is off-screen, since the pip disappears once the icon is visible). Build role/state detection with redundant layers keyed to different phases: a primary that fires early (OCR banner on the role-reveal interstitial) plus a backup during gameplay (HUD icon presence, kill-button vs ghost icon at a fixed HUD slot), defaulting to the common role when neither triggers. When pixel-perception of teammate identity is fragile, add a BEHAVIORAL fallback driven by action outcomes -- after 2 failed strikes against the same color, infer that color is a teammate and stop targeting it. _{sources: opencode:ses_1f6f76557ffeCfmMTG7BPJwGRE, opencode:ses_1f7b6dc30ffeDmQtlUZI25HAjh, opencode:ses_1fc4cec1bffe16bZw4eHP3SDBL, opencode:ses_1ffe75d89ffeHvW4p2ARluHh51 (+2)}

34. Build an explicit observability catalog phase-by-phase and make last-seen memory mandatory under partial observability

crewrift · negative result · ⚠ session-derived, unverified

Before writing a policy for a partially-observable game, build an explicit observability catalog classifying every state element per phase as always-visible, conditionally-observable (LOS/viewport/one-time-reveal gated), or never-directly-visible (with a 'when revealed' note); use an explicit gap table of {view enum? detection? parser? key gaps} -- this table drives the whole perception+memory architecture and a missing-phase extraction silently destroys foundational belief state (a roster-reveal phase with no view enum/detection/parser lost all player->room mappings; a reveal screen kept only winner text while discarding the full no-fog world state of all players' roles and positions). Anything conditionally visible MUST be persisted into last-known-sighting memory the instant it is observed (other players' colors/positions visible only on viewport+raycast LOS; teammate identities only during reveal), because a policy that re-reads the live view each tick loses it once the entity leaves sight -- memory is mandatory, not an optimization. Deduplicate raw per-tick sightings before storing, keyed on BOTH time and space (collapse same-entity sightings within N ticks and M world-pixels, separate dedup for static artifacts like bodies); these dedup constants control how trigger-happy downstream inference is, so pick them deliberately. Distinguish 'the game does not expose X' from 'our code does not parse X yet' before discarding a feature: the Crewrift meeting-caller was wrongly called unobservable because perception predates the MeetingCall interstitial and simply does not parse it, even though the game renders the caller icon (button_calls_made was the strongest offline-only feature at -1.59). Separate logic you can write now from work blocked on data the perception layer cannot yet extract (round schedule, own sprite identity, cross-room leaders); the latter is an upstream perception DEPENDENCY to implement first, documented in the design doc so behavior pauses to collect that data, instead of permanently assuming it unavailable. Ground any claim that an entity/objective belongs to the bot in a server-rendered perceptible cue, not inferred behavior (a trace field showing the bot literally saw the own-tasks-only task-icon sprite proved ownership and redirected a false 'wrong task' diagnosis to the real positional bug). _{sources: claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40, claude-code:0e7b14ca-ffd5-4137-9456-47485d3c6f87, claude-code:d4cde1e2-f4b5-4bd6-bf05-dd48002edf58, opencode:ses_1fb1787afffemhwtF09UMzPc5V (+3)}

35. Fuse multi-fidelity perception channels with per-cell provenance; never overwrite confirmed data with low-confidence hints

loop · negative result · ⚠ session-derived, unverified

Perception channels deliver the same entity at different fidelity (a precise nearby sighting vs a low-res far minimap dot). Feed all position-bearing channels into one unified known-positions view downstream modes consume (reading only the high-fidelity channel makes the agent blind outside immediate range), but do NOT pollute the authoritative entity registry with low-confidence observations: keep speculative sightings (minimap dots) in a separate clearly-labeled last-seen lane. When building an occupancy/belief grid from sources of differing reliability, track provenance per cell (e.g. a viewport_confirmed boolean): high-confidence direct observation may overwrite anything, low-confidence hints may only fill cells still UNKNOWN, and shield immutable cell types (WALL, hub, stations, extractors) from floor overwrites. Write a cell only on positive evidence (its position appears in grid_tags/grid_inventory/grid_agent_ids); absence of tokens means 'unobserved', NOT 'empty floor', and since the engine emits tokens only inside a ~circular ~6-cell vision radius while the buffer is a full 13x13 square, iterate only over positions actually present in the parsed dicts or the corners overwrite real data with bogus floor. Track stationarity with a movement threshold (>2px resets the counter) and per-entity tick stamps (cleaner than a separate visibility flag); treat reachability discs only as a NEGATIVE signal (where an agent cannot be), inferring heading from observed motion over dozens of ticks. Don't let code needing a fully-resolved handle silently block acting on a visible-but-unidentified target -- add a fallback that interacts on reaching its last-known position, and prefer a blind state-free action when one exists (a 'blind offer after entry request' completed exchanges that perception-gated WAITING_ENTRY pixel parsing never could). _{sources: codex:019e031e-f7d3-7342-a7d9-96993bc91097, codex:019e067b-643a-7100-abdb-8d6a0c79d242, codex:019e08cf-4f50-7ef1-bc11-82b8d92354b8, codex:019e08e5-547c-7da1-b1de-44d0b4133972 (+3)}

36. Use absence of a cue to prune hypotheses, but tag evidence absolute vs positional, gate on prior confirmation and grace, recompute each frame, and debounce flicker

loop · negative result · ⚠ session-derived, unverified

Use the ABSENCE of an expected perception cue as a first-class signal to PRUNE belief hypotheses, not just presence to confirm: if an off-screen task's projected radar-dot position stays far from every detected dot, that task is probably not assigned. But explicitly tag each negative-evidence source as ABSOLUTE vs POSITIONAL/directional and never write a positional signal into a permanent/terminal belief state: absolute ('I stood there and saw no icon') is permanent; directional ('from here no radar pip points at it') is transient and reversible because its value depends on where the agent is standing -- conflating them latched directional pip-exclusions into the same one-way terminal state as absolute icon-miss evidence, so tasks excluded from one position stayed dead forever even after moving, leaving the bot idle. An affordance DISAPPEARING can mean SUCCESS, not absence: the 'missing icon = wrong task, prune fast' rule wrongly pruned a task whose icon vanished because the action SUCCEEDED (the server accepted the task while the bot recorded zero task_completed events), so gate negative-evidence pruning on prior positive confirmation and shield positively-confirmed targets (state==confirmed) from fast miss-pruning while leaving speculative targets unshielded. Guard the inference: run it only when >=1 indicator is detected this frame (with zero signal you cannot distinguish 'far from all dots' from 'no perception'); gate on N consecutive confirming frames with a separate tunable distance threshold (one 'no-match' frame is unreliable; 2 frames was too low, ~6 frames/~0.25s absorbed noise); reset the counter both when localization is lost and the moment contradicting positive evidence reappears. Across a channel handoff (far radar pip -> high-detail icon) expect a frame gap: the pip vanished the instant the player got close but the full icon took 2-3 frames to scroll in, so add a grace period (PipDisappearGraceTicks=5) keyed to pip count dropping to zero. Debounce perception against single-frame flicker rather than reacting to it and recognize PARTIAL targets: a task icon detected one frame and missing the next (bob, clipping) should not immediately resolve gone -- raise the miss-resolve threshold (TaskIconMissResolveFrames 2->6), and if ~70% of an icon shows, recognize it (full-icon-required creates a window where the target is dropped between radar-pip loss and full-icon registration). For a flag meant to reflect 'true right now', recompute it from scratch each frame rather than accumulating in a saturating counter (a counter that saturated after ~12 frames latched and stopped tracking reality), and clear any transient per-entity flag at the TOP of the per-entity loop before any 'continue'. 'For each task, is there a pip near its projected dot?' and 'for each pip, which tasks lie along its ray?' are geometrically equivalent under exact projection, but the per-pip framing naturally yields a TRANSIENT exclusion set rather than a per-task accumulating counter -- the framing is a real design lever. Prefer fixing the upstream setter of a sticky/permanent flag over making it forgiving/expiring (resolvedNotMine was made accurate by fixing its setter, not adding expiry; a sticky flag is acceptable only if set accurately). When negative evidence IS reliable and you know the exact deterministic offset where evidence should appear, check that point with a couple-pixel tolerance and treat absence as INSTANT resolution (~24-consecutive-empty-frame threshold dropped to ~2). Tightening a resolution threshold yields nothing if a blanket shield still exempts entities (dropping missing-frames 24->2 did nothing); audit every caller that exempts entities from a decay, and beware the race where the policy holds N ticks but the server confirms at K<N (held 84, accepted at 72, then 24 confirmation frames ran with the signal gone) -- trim hold time to just past the acceptance threshold. _{sources: codex:019e0140-e554-78e3-b66f-80634cc63f5d, codex:019e03bc-5520-7e72-aa5c-a572ead62b91, codex:019e03bd-5f9b-7482-a03a-24cb99c6c7ed, codex:019e0436-7b17-7433-b3d9-5c78fad74312 (+8)}

37. Resolve task ownership with authoritative signals and observable NEGATIVE evidence, latched against re-flagging

crewrift · negative result

Identify which signals are authoritative for ownership before trusting them: an on-screen task-icon match is authoritative (the server renders the icon only for YOUR tasks), an off-screen arrow/radar-dot is authoritative only if gated on a real radar match, and raw rectangle intersection (standing on a station) carries ZERO ownership info -- setting the arrow flag for every off-screen task or the active flag on any overlap destroys filtering. Infer assignment from graded evidence: a visible task icon is definite; a matched off-screen radar dot is probable (high-confidence single sighting); and accumulate negative evidence to mark a station resolved-not-mine after its icon is absent for several consecutive frames. Presence-only perception leaves most candidates unresolved (32 of 40 stations), so observing ABSENCE rules out the bulk -- model per-task state NotDoing -> Maybe(radar) -> Mandatory(icon visible) -> Completed, where seeing an icon always wakes a task and radar is weak evidence never proof. Crucially, only count absence you can observe: increment the task-icon miss counter ONLY while the icon area is fully on-screen with a small clear margin, never for off-screen/near-edge stations. The recurring bug is clearing a task by checking the standing RECTANGLE instead of the ICON area drawn ABOVE it; require the full icon rect visible with margin AND repeated absence for N frames. Exclude off-screen candidates with radar-ray geometry (cast a ray from the player through each screen-edge radar dot; if none intersects a task's padded icon box, exclude it this frame) but treat that exclusion as per-frame and reversible since movement changes alignment, unlike the durable resolved-not-mine flag. Sticky candidate flags cause loitering: DECOUPLE the clear condition from the set condition (don't let a stale/ambiguous radar dot veto a clean direct observation), add a per-target 'resolved' latch with exactly one override (a real icon re-rendering) reset each round, and only begin task-checking AFTER localization (which follows round-start assignment) so the latch doesn't fire before icons render. When matching a sprite to an expected object, compute the expected RAW screen position from the server's fixed render offsets plus ~2px tolerance and do not apply extra draw offsets the scan does not report. Reset all per-round belief state (slot states, checkout latches, resolved-not-mine flags, miss counters) on the role-reveal interstitial and skip belief updates entirely on interstitial frames. _{sources: bitworld/among_them/players/SMART_BOT_GUIDE.md, bitworld/among_them/players/evidencebot_strategy.md, bitworld/among_them/players/how_to_make_a_bot.md, personal_labs/crewrift_lab/crewrift/crewborg/design.md (+3)}

38. Key edge-triggered reflexes on stable identity, not visible-entity counts; require true rising edges

crewrift · negative result · ⚠ session-derived, unverified

Edge-triggered reflexes keyed on a raw count of currently-visible entities have no identity memory and re-fire on the same stimulus: a bot fleeing its own kill victim re-triggered each time the body left and re-entered the viewport because the reflex fired on visible-body-count increasing. Key reflexes on stable identity (remembered body positions within a radius) and clear the memory set at round/meeting boundaries. Perception gaps also create false hard evidence: a 'newly detected at location X' accusation rule fires on scanner dropouts (a player briefly missed then re-detected), so require 'newly appeared from off/elsewhere', gate on the previous frame showing the full sprite footprint at the candidate position, and suppress reappearances when the same player was recently seen nearby. With rising-edge (0->1) input detection a held or re-pressed button can trigger a DIFFERENT action once game state changes underneath you (an A press/release cadence that created a whisper kept firing A and, once inside the whisper, A mapped to a menu action): make the press pattern hold long enough for the server's 0->1 edge (2+ frames) and gate the task (valid_views) so it stops emitting once the target view is reached. When a single control is overloaded by context, observe the relevant world state first and pick the correct button once rather than blindly cycling A and B. _{sources: claude-code:067e7439-cbd1-4b26-9b3f-d054bde3aa40, claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, codex:019e1591-004e-7040-b768-062bd8e6acd4, codex:019e1593-c00d-7a32-9dfa-1cf6b16b01a4}

39. Drive the body-report reflex from position-based body identity, not a visible-body count

crewrift

Trigger the crewmate body-report reflex from position-based body identity (screen coords + camera offset), NOT a visible-body count: count-edge detection misses same-count corpse swaps (one body leaves view as another appears), whereas remembered-position unknown-body detection fires on a newly visible corpse even when the total count is stable. Match a visible body to a target within a generous radius (~30 world px) to absorb camera jitter and sprite-anchor offset, debounce single-frame misses by resetting the miss counter on any matching frame, and only give up after a sustained run of misses (~36 frames / 1.5s) indicating real despawn. Add a reflex cooldown (~96 ticks / 4s) plus known-body memory so the reflex does not re-fire on the same corpse while still allowing a different newly-seen body to trigger. _{sources: players_checkouts/players/users/james/personal_cogs/among_them/guided_, players_checkouts/players/users/james/personal_cogs/among_them/guided_, players/users/james/personal_cogs/among_them/guided_bot/coworld/README}

40. Drive the edge-triggered voting cursor through the parser's slot map, re-parsing every interstitial frame

crewrift

The voting UI is edge-triggered: it advances exactly one slot per fresh left/right keydown, so holding a direction moves only one step -- drive the cursor with a pulse-then-release state machine (hold a few ticks, emit a no-op release tick, recompute). Map a vote color target to its live slot through the perception parser's slot-to-color array, not slot==color, because the game may draw slots in live join order (fall back to slot==color only when no parser slot map exists). Re-run the voting parse on every interstitial frame, including after the phase is established, to keep cursor position and per-slot alive/dead state fresh; and when the parse fails and cursor index < 0, degrade gracefully to blind single-direction (right) navigation rather than stalling. _{sources: personal_cogs/among_them/guided_bot/MEETING_DESIGN.md}

41. Detect a real watched-task completion via the global HUD counter decrement plus a long dwell

crewrift

Derive a watched-task-completion (exculpatory) signal from the global remaining-tasks HUD counter: count a completion only when the counter decrements by exactly 1 WHILE exactly one visible living player ends a task dwell of >=56 ticks. Fake 'pretend' holds never decrement the global counter, so they cannot trigger the signal. _{sources: personal_labs/crewrift_lab/WORKING_CONTEXT.md}

42. Port a perception layer behind a byte-for-byte parity rig against the original as ground-truth oracle, module-by-module in dependency order

crewrift · ⚠ session-derived, unverified

Port a perception layer to a new language behind a byte-for-byte parity rig that uses the original implementation as the ground-truth oracle: nothing merges into the new perception tree until the parity suite is green on a fixed fixture set. When a player must run completely in Python (no Nim toolchain in the runtime image), parse the raw packed pixel frame directly (pixel-first) rather than relying on the structured state vector, which is lossy for many perceptual facts -- the one allowed exception is a documented 'state-vector tap' for a field lossless only there (e.g. exact task-progress percentage). Decide the oracle mechanism up front (a small CLI reusing the original perception vs instrumenting the original bot), capture 50-100 fixtures spanning every game phase (lobby, role-reveal, playing, body-sighted, meeting, voting, results, interstitial), and port the hottest smallest dependency-free module first, in strict dependency order (interstitial and ignore are independent and cheap; localize gates ocr, voting, and task-icon scanning -- a working order is interstitial -> ignore -> localize -> task-icon -> ocr -> voting), rolling the sidecar schema version forward as each field lands and gating each module with parity tests against the Nim ground-truth oracle (e.g. 160 parity checks across 10 fixtures). Do NOT re-derive behavior from prose docs (lossy and slow); assert equality against captured reference output (reshape/frombuffer the source .bin and assert equality can substitute for visual inspection of a ported asset). Commit each fixture as a locked triple in one commit -- raw input frame (.npy or palette-indexed .bin), capture metadata (.meta.json), reference ground truth (.nim.json computed by the original) -- and stamp each with the sprite-atlas/asset hash so the harness rejects fixtures whose truth was computed against a different asset version. Treat the original as a PARITY REFERENCE and SPEC: consume its shared vectorized kernels (sprite matching, task-icon scan, glyph/text match, localization) so behavior stays bit-aligned, reimplementing a scan only when none exists and it is trivial (radar dots = yellow pixels in the border ring); pin the port against the original's exact outputs on shared fixtures (same camera lock, e.g. (504,54)) before adding behavior; verify EVERY signal the reference consumes survives your simplification (a port dropped the reference's radar-dot/checkout bookkeeping assuming it was redundant, but the pixel adapter only surfaced the mandatory icon signal, so the dropped logic was load-bearing); and diff against a SIBLING bot that already did the migration (italkalot/ivotewell for modulabot). When a parity test fails on an unrepresented phase, capture a fresh fixture and confirm the bug reproduces BEFORE fixing. Use the canonical UPSTREAM repo for assets (map JSON, spritesheet, palette, aseprite), not a derived copy that silently drifts, and ground 'what is rendered' in the LOAD PATH (what the sim init loads and render() blits), not the asset directory -- several sprite/aseprite files were never loaded, so reading them as live would mislead. Optimize correctness first and defer JIT/numba/vectorization until a measured budget forces it (numba cold-start can blow a tight ~5s player-init deadline). The archived Among Them 'Coborg' player preserves the full v5 pipeline (frame, sprite_match, actors, geometry, ignore, interstitial, localize, ocr, tasks) plus the parity harness and a Coworld policy bridge -- use it as the reference when porting or comparing. _{sources: players_checkouts/players/archive/README.md, players_checkouts/players/archive/players/among_them/coborg/DESIGN.md, players_checkouts/players/archive/players/among_them/coborg/PLAN.md, players_checkouts/players/archive/players/among_them/coborg/README.md (+10)}

43. Verify perception field schemas against live code, never the brief; re-derive game-flow phases/timers/limits from source

loop · negative result · ⚠ session-derived, unverified

Before writing inference code, verify perception field schemas against the live framework code that produces the field, not the task prompt's description: grep the actual belief struct to confirm every field exists. Briefs are often wrong (a brief promised memory.players[color].lastSeenRoom but the code only stored last-seen world coordinates; a spec described a position tuple as (tick,x,y) but the updater wrote (x,y,tick)), and getting layout wrong silently corrupts every derived feature because the values still typecheck. When a field has an ambiguous or drifting tuple/dict layout (e.g. (tick,x,y) in synthetic fixtures vs (x,y,tick) in live frames), parse it defensively to accept both forms from one code path so unit tests and live episodes both stay green. Treat any field named in a prompt as a claim to confirm, not a given. Likewise re-derive every documented game-flow phase, timer, and limit from the actual game code -- one audit found missing phases (RosterReveal, LeaderSummit), a '5-second countdown' that was 1s (24 ticks), a role-reveal '5 seconds' that was a 15s 4-panel intro, and a '36-char' chat limit that was actually 58 chars. _{sources: codex:019e05e2-eadc-70a1-a58f-add2c62f1046, codex:019e08bb-a4ea-7c83-a8d6-2652eaa1a3b8, opencode:ses_1ffc701efffeS5c5ueXd2vWIkD}

44. Detect game events from genuinely-consecutive frame diffs with robust structural cues

crewrift · ⚠ session-derived, unverified

Detect events from consecutive-frame diffs, requiring the two compared frames to be genuinely consecutive and appending tape frames only on camera-ready frames (so a meeting's tick gap is self-protecting -- a detector cannot compare across it). Among-Them signals: not-interstitial -> interstitial = meeting or game-over; 0 -> >=1 bodies = body discovered; kill icon invisible -> visible = kill-cooldown ready; N -> N-1 visible crewmates = player went missing; ejection text appearing = ejection; chat naming a color = accusation. Prefer robust STRUCTURAL cues (region pixel counts, color-class ratios, blob detection) with wide threshold margins over exact-template or per-pixel comparisons, so cosmetic rendering changes don't break the agent; full OCR is overkill for most cues. Use physically-impossible-otherwise observations as cheap phase ground-truth: seeing seven players in one snapshot (the camera normally sees 1-2) reliably signals a meeting room, and a phase classifier that misses it skips all voting and casts zero votes despite seeing 7 players. A perception-driven player only knows the game ended if it detects the game-over screen (game_over is emitted solely on a PhaseGameOver transition in the rendered frame), so a slot whose connection was severed mid-task never emits game_over and ends on an ordinary task_started -- 'no game_over in the trace' reliably means the connection was cut before the game ended for that slot. _{sources: bitworld/among_them/players/SMART_BOT_GUIDE.md, bitworld/among_them/players/lively_lecun/ROADMAP.md, personal_labs/crewrift_lab/crewrift/crewborg/design.md, claude-code:5f9e8287-da92-4dff-bf39-2511bcaf0700}

45. Recover chat from pixels with a structured parser and color-pip speaker attribution

crewrift

When the game gives only the rendered framebuffer (all players see the OCR-readable voting screen), build a STRUCTURED frame parser (parseVoteFrame returning {colorIndex, lines} plus a derived 'who is being called sus' color) and feed concatenated text to the LLM, preferring it over ad-hoc pixel OCR once it exists. Attribute speakers by sampling the per-message color PIP at a fixed x-column immediately left of the chat text (VoteChatIconX=1), taking the dominant color with a 'prefer-above' tie-break, because naive nearest-pip mis-credits WRAPPED multi-line messages to the next speaker (wrapping is the edge case that breaks the obvious approach); return unknown/-1 on low confidence. This perception unlock has the widest downstream blast radius -- once speaker attribution worked, imposter-chat detection and vote-bandwagon correlation became feasible -- so prioritize it. The meeting/voting UI is the AUTHORITATIVE alive/dead census and vote record (Crewrift voting grid ids 9300+seq show 'player <color>' alive vs 'body <color>' dead, vote-result id 9600 names the ejected, 'NO ONE DIED' is a distinct sentinel); attribute votes via per-target vote dots, but the REPORTER needs a dedicated meeting-intro pass and must stay a -1 sentinel (not a fabricated value) until that pass exists. _{sources: bitworld/among_them/players/mod_talks/LLM_SPRINTS.md, bitworld/among_them/players/mod_talks/TODO.md, bitworld/among_them/players/modulabot/DESIGN.md, bitworld/among_them/players/modulabot/TODO.md (+4)}

46. Keep perception schemas additive, treat fallbacks as code smells to revisit, and log a single converged JSONL trace

loop · ⚠ session-derived, unverified

Extend a perception dataclass ADDITIVELY when upstream exposes more than it carries (keep existing fields, add new ones like sprite shape/position alongside color) so downstream policy reading old fields does not break -- backward-compatible schemas decouple parser upgrades from policy rewrites. When a shared sprite reader has blind spots on specific shapes (read_sprite() misses centerless shapes like a ring), layer a fallback (try the shared reader, then explicit shape detection plus a dominant-fill-color read in the bounding box) rather than modifying shared sprite behavior and risking regressions. But treat any perception fallback as a code smell to revisit, not a permanent fix: resolving a player 'by color' was flagged 'almost always not a great sign,' and once the underlying outline-color-0 shape detection was fixed the color fallback became dead code -- file the root cause, fix the real detector, verify the fallback is now redundant. Log the FULL per-frame perception output to one JSONL trace (camera localization+score, self position/color/role/ghost, detected actors/bodies/ghosts, task icons, radar dots, ignore-mask pixel count, interstitial flag, voting parse) emitted at the SINGLE convergence point where all phases merge, so downstream tooling reconstructs exactly what the policy saw without re-running perception and the record reflects fully-resolved (not mid-pipeline) state. Scope perception work to perception assets only: grep which modules actually import an asset before assigning it to a phase (nav_paths.bin/nav_graph.json were imported only by navigation/planner modules; defer an OCR-only font blob to the phase that first needs it). Capture a 'visual bootstrap' of live frames during a generated player's first run (dump per-frame packed .bin under visual_bootstrap/live_run/frames/) to give the downstream decoder-generation agent ground-truth data. _{sources: auggie:5d1ff8d5-2505-4dca-af45-56d3070363e2, claude-code:3b14ac5a-1d66-4524-94c0-59a48466d223, claude-code:7b66497b-650c-4bba-b410-6c854f59163d, codex:019dfef8-dc40-7273-b85a-04aa3a51ff07 (+2)}

47. Reset belief fully (including ad-hoc attrs), reject malformed readings, and accumulate sticky channels rather than overwrite

loop · ⚠ session-derived, unverified

When a belief/world-model object permits ad-hoc attributes (setting obj.foo outside the declared schema), reset() must clear them too: after re-defaulting declared dataclass fields, iterate self.dict and delattr any name not in the declared field set, or stale state silently survives a game restart -- state that must persist belongs in an explicit declared container, not an ad-hoc attribute. Treat invalid perceived inputs as invalid, not real data, when perception feeds geometry into navigation: a role-reveal frame reporting a non-positive room_size was accepted as a real zero-size room, so add a production guard rejecting malformed readings before they produce a degenerate grid. Model runtime-variable settings as belief, not constants: whether Crewrift task arrows show is a per-map setting (set true on load if arrow sprites present, false if absent) -- when present use arrows for off-screen tracking, when absent fall back to checking every room. Do NOT mark a task complete merely because it left the visible set (it can leave by going off-screen when arrows are disabled); active_task_progress_pct ramps 1->98 then jumps to null on completion (never reaches 99/100), so don't gate completion on ==100 (never fires) nor >=98 (stops the hold early) -- the authoritative signal is the task bubble disappearing while you are inside the task rect, and completion detection belongs in the mode that knows which task you stand on. For sticky channels like chat where the same line stays visible across frames, key structured updates off newly-APPENDED messages not currently-visible ones (a lingering system line otherwise re-writes its event tick and corrupts recency), accumulate-and-dedup rather than fully replace the field each frame (a dropped OCR line loses history), and match semi-structured system messages with case-insensitive substring patterns ('offered role', 'swapped colors') not brittle exact-match. _{sources: claude-code:4430a077-5411-426b-a87a-75512ec4300f, claude-code:48add98e-a83d-4348-97bf-0079c48c42d6, codex:019e03bf-982c-7fc0-8ba9-d4c640feb779, codex:019e04e9-c357-7c71-a255-929f3a97fda1 (+3)}

48. Disambiguate overloaded Crewrift object-id bases by stream and value range; loop the whole packet and special-case skip-vote dots

crewrift · negative result · ⚠ session-derived, unverified

Crewrift object-id bases are overloaded across streams and value ranges, so assigning meaning by id-base alone misclassifies objects. (1) id base 7000 = player-name labels on the global/spectator stream (gated on showPlayerLabels) but = task arrows on /player, so confirm which stream the player reads first. (2) Vote dots split by range: normal vote dots are 10100 + target*16 + voter (bounded < 10100 + MAX_PLAYERS^2) while skip votes use a high base (observed 10400 + voter) sharing the same sprite, so decoding everything >=10100 as a normal dot yields bogus target ids -- split by range and represent skip as a distinct sentinel (e.g. target=-2). When decoding a binary websocket message, loop an offset across the WHOLE packet: one message can carry multiple concatenated sub-messages and reading only the first silently drops state. _{sources: claude-code:4430a077-5411-426b-a87a-75512ec4300f, codex:019e2886-d576-7770-be1b-6d92ca97761e, codex:019e66c3-638e-7e70-afdf-8f6df9299f6a, codex:019e7174-3079-7351-b8ea-b1bfcb917ff4}

49. Bake Crewrift static geometry from the server resource file at startup; track last-seen; treat task arrows as tri-state

crewrift · ⚠ session-derived, unverified

Bake the Crewrift static map (vent, emergency-button, room, task coordinates) from the server's resource file at startup, because those coordinates are NOT recoverable at runtime from the Sprite-v1 stream -- they live only in rendered map-sprite pixels and the walkability mask, and no message or HTTP endpoint serves them (server exposes only /healthz, /admin, /control/restart, /control/kick, replay/global client files, and the /player websocket). The resource format is line-based CSS-flavored (not brace-delimited): each rect block starts with a /* name / comment plus px-suffixed width/height/left(=x)/top(=y) and background: rgba(...); the parser flushes at the next / name / and commits only rects with name+x+y+w+h and w,h>0 (so a leading un-colored / vents */ bounding block is dropped). Classify at startup: 'task' rects -> task list in file order (matching the 3000/7000+idx stream index), 'ventN' rects -> grouped by trailing digit, named rects -> rooms, emergency button is a derived 28x34 rect centered on 'bridge' and clamped. Store the initial server packet's full contents (all task/event locations, room names) for later, since room names are needed to describe body locations during voting ('Body found around hydroponics'). Validate the vendored bake against the live walkability mask with a one-time fail-loud check the first time the alpha arrives (catches stale-bake mismatch); the walkability alpha is raw-snappy compressed, decode via cramjam. An entity's absence from a per-tick percept array (visible_players/visible_bodies) means 'not currently in my vision', NOT 'gone' -- track last-seen state and observability separately. Two per-incomplete-task finding signals exist: a 'task bubble' object (id 3000+idx, exact world position, on/near screen) and a 'task arrow' object (id 7000+idx, bearing only, off-screen, only if showTaskArrows is enabled); map the stream index to a world rect via the file-order task list. Detect whether arrows are enabled by OBSERVATION not assumption (the arrow sprite is always defined at init but arrow OBJECTS may be gated off): treat the flag as tri-state -- True on the first arrow object for a known off-screen task, False if several ticks pass with off-screen tasks and no arrow -- and when off, fall back to a room-by-room map sweep. When deriving evidence from positions, reuse this static map data (e.g. vent waypoints already in the nav graph) instead of adding a new perception pass. _{sources: coworlds/coworld-crewrift/players/notsus/README.md, personal_labs/crewrift_lab/crewrift/crewborg/AGENTS.md, personal_labs/crewrift_lab/crewrift/crewborg/design.md, claude-code:4430a077-5411-426b-a87a-75512ec4300f (+3)}

50. Decode the input/observation protocol from the canonical shared module, including sentinels, the clock, and set-vs-increment semantics

crewrift · ⚠ session-derived, unverified

Decode the input/observation protocol exactly from the canonical shared module before writing a policy, including reserved sentinel values, and do not infer bit layout from usage sites: Crewrift input is a bitfield (one bit per button) where mask 255 (0xFF) is reserved as a game-reset signal, so a bot emitting arbitrary masks can trigger reserved behavior. Watch for path drift -- authoritative protocol/constant definitions (input bits, screen dims) lived in a shared module outside the game dir and a Go port redefined the button bits independently, so treat the original shared module as source of truth and verify other ports have not drifted. Before driving the agent to any target screen, map the simulator state machine from source: the phase enum, every transition with its trigger (timer expiry vs player action vs server event vs player-count threshold), the tick rate, and each timer's value in ticks. Audit whether the player has any authoritative clock: a Crewrift policy's only time sense was a local scene.tick incremented per received frame, with nothing on the stream carrying a server tick/timestamp, so every tick-based threshold (vote budget, kill cooldown, flee hysteresis) silently drifts undetectably unless the stream is exactly one-message-per-tick -- correctness reduces to verifying there is no server-side frame coalescing/dropping, which you must check, not assume (note: the invisible 'tick <N>' marker sprite id 5016 does carry a server tick when present, so prefer it). Likewise read the engine's mutation/handler source for set-versus-increment semantics before baking economy/survival math: energy regen ADDS the solar value as a per-tick delta rather than SETTING energy to solar (16 energy + solar 3 -> 19, not 3), and a wrong assumption silently corrupts strategy. _{sources: claude-code:27984094-3b9e-4e67-92c5-da2a61ceb32a, claude-code:3837ef5e-3e9c-41bc-b290-df5865581698, claude-code:ebd6cd36-8956-4a1e-b039-841683ff7792}

51. Order crewrift events by (real tick, seq, key) and mark every estimated tick explicitly

crewrift

Sort crewrift events by the canonical key (ts, seq, key) where ts is the real game tick (the 0x01 frame tick id, a monotonic u32 with 0 = pre-spawn/lobby and gameplay starting at gameStartTick) and value.seq tie-breaks events sharing one tick (votes, multi-kill ticks). For two-party events like kills, put the actor (killer) in the player column and the other party in value.subject_player (victim), so 'how did slot N die' is action.kill rows where subject_player == N. Stdout lines are not individually tick-stamped, but during discussion/voting the map freezes while the replay advances as dense consecutive ticks (>=60) -- detect meetings that way and stamp vote/chat events inside [meeting.start, meeting.end], or at meeting.start ordered by seq if only the meeting number is known. The current reporter reverse-engineers ticks by regex-parsing game.stdout.log and interpolating between replay anchors, so most events carry tickEstimated:true and actor/action columns are null; always flag estimated ticks and never present one as real -- a true events.parquet would let the reporter set tickEstimated:false and populate actor/action. _{sources: coworld-source-repos.crewrift-upload/optimizers/instructions/mvp/repor, optimizers/docs/events-parquet-spec.md}

52. Parse crewrift meeting chat into (speaker,stance,target) triples, separate kills from ejections, and mirror the training transform at runtime

crewrift

Reduce templated crewrift meeting chat to (speaker, stance, target) triples: match color names with a per-game regex ordered longest-color-first (so 'pale blue' beats 'blue'), use keyword hints to classify accuse vs defend, ignore lines naming only the speaker, and drop unparseable lines rather than guessing. Distinguish kills from ejections by recording each kill's (tick, victim) so a later bare 'died' event with no matching kill is attributed to the open meeting's ejected slot rather than miscounted as a murder. At inference, the runtime suspicion scorer must mirror the training transform contract exactly -- dot the fit coefficients with features, add the intercept, apply a sigmoid, and apply the bin_spec boundary indicators (e.g. follow_death_samples bins [2,6], observed_samples bins [10,40]) so runtime binning matches how the model was trained. _{sources: personal_labs/crewrift_lab/crewborg/data/suspicion_weights.json, personal_labs/crewrift_lab/suspicion_lab/tools/features.py, personal_labs/crewrift_lab/suspicion_lab/tools/replay_parse.py}

53. Use crewrift's fixed slot-to-color palette and pin self color from the supplied slot

crewrift

Crewrift's slot-to-color palette is fixed and validated (0=red,1=orange,2=yellow,3=light blue,4=pink,5=lime,6=blue,7=pale blue,8=gray,9=white,10=dark brown,11=brown,12=dark teal,13=green,14=dark navy,15=black); cross-check stdout imposter colors against results.imposter[] (pink+pale blue -> slots 4,7) to confirm all identity namespaces agree. When Coworld supplies a slot (slot= in COWORLD_PLAYER_WS_URL or --slot), pin self color from that slot rather than reading it from noisy vote-screen markers, which makes meeting context and self-vote filtering unstable. _{sources: players_checkouts/players_2/players/crewrift/suspectra/README.md, optimizers/docs/events-parquet-spec.md}

54. Dead-reckon from the engine-observed last_action, not the action you emitted

loop · negative result

For dead-reckoning position in an egocentric-observation policy, use the engine-observed last_action token as the PRIMARY movement source, NOT the action you emitted. Root cause of a corruption bug: in an async live runner that observes-then-applies the latest queued action with no per-step wait, the emitted action LAGS the engine-applied one, so dead-reckoning from the emitted move pairs a movement result with the wrong action and writes immutable landmarks at shifted coordinates, creating duplicate hubs/stations/walls that corrupt pathing. EXCEPTION: when the engine reports noop for a blocked/invalid emitted move, use the EMITTED move only to mark the blocked cell, never for successful dead-reckoning. Treat type:wall as a permanent blocker but type:agent as temporary traffic, and store timestamps with observed tags because junction control mutates over time. _{sources: archive/cogames_playground/alpha_cog/docs/cogs-v-clips-strategy.md, archive/cogames_playground/bulbacog/designs/INNER_LOOP_IMPL.md, archive/cogames_playground/bulbacog/designs/LOCALIZATION_ROOT_CAUSE_RE}

55. Infer spatially-gated quantities from globally-available tokens, and keep your own id in the teammate set

loop

When the engine exposes global team-level inventory tokens (team:<element> counts) for some quantities (hub element counts oxygen/carbon/germanium/silicon) but surfaces others (hub hearts) only when spatially adjacent, INFER the gated quantity between visits from the global counts plus the known crafting cost (heart craftable when >=7 each) rather than treating it as unobservable. Relatedly, keep an agent's own id in its own observed-teammate set so an agent that sees no teammates by the assignment tick falls back safely (assumes it is the highest-id visible and takes the lead role) instead of failing on an empty set. _{sources: archive/cogames_playground/bulbacog/designs/MEMORY.md, archive/cogames_playground/bulbacog/designs/STRATEGY_V2.md}

56. Detect CvC interactive cells via a precomputed tag set; read teammate hearts from inventory

crewrift · ⚠ session-derived, unverified

Detect interactive cells cheaply during CvC perception by precomputing a frozenset of interactive tag IDs once (extractor, hub, junction, gear-station tags) and intersecting it with each observed cell's tag IDs; resolve the tag IDs once in the policy and pass them down rather than re-deriving the name-to-id mapping every tick. The interactive-cell set grows monotonically because interactive objects do not move. For a visible teammate, read heart count from the inventory observation (inv:* tokens parsed into a per-position inventory, stored as last_known_hearts), NOT from text. Use a talk field like h=<count> only for commitments that outlive visibility: once an agent leaves view last_known_hearts goes stale, so the h= carried in a role claim keeps the commitment's heart context for a TTL. _{sources: codex:019e14de-cdca-7482-88fa-81716ee830ae, opencode:ses_1ffa96717ffeq1J1GeKsUpSp3k}

57. Gate stale observations and resource claims on a TTL, but keep ownership-change detection sticky

loop

Gate ghost/stale-POI chasing and resource claims on a staleness TTL: is_stale is (current_step - last_seen_step) > ttl (strict, so the boundary is not stale), and published claims auto-expire after claim_ttl unless renewed so a stuck or dead agent cannot lock a resource forever. Separately, keep ownership/scramble detection sticky across intermediate transitions -- track that a junction was once friendly so a cogs->clips->neutral sequence still fires a scramble alert, suppress alerts whose prior observation is older than the freshness window, and leave alerts un-removed on consumption so multiple racing agents can respond. _{sources: optimizers/.cursor/skills/scripted-navigation/reference.md}