Qualify Tutti Agent Session Replay Cassettes

Work from the Tutti checkout. Keep product implementation and the generic runner in Tutti; keep Case metadata, scenario scripts, fixtures, qualified Cassettes, and evidence in the external case repository (tutti-os/tutti-replay).

Mental model (script-first, not UI recording):

1. Humans/agents write a deterministic scenarios/*.mjs (prepare / drive / assert) in the case repository — that script is the recording plan.
2. Record means: Tutti runner launches Desktop, CDP-executes that script against a live Provider, and captures the Cassette. There is no separate click-to-record UI workflow for Session Replay.
3. Day-to-day Record/Replay is usually triggered from the case repository QA console; this skill is for Tutti-side CLI qualification, diagnosis, runner or Replay product defects, and new Provider capture support.

Prove qualification in this order:

existing scenario script -> live Record (script + Provider) -> structural audit -> fresh Replay -> optional publication

Never call a Cassette qualified until Record, audit, and a fresh isolated Replay have all passed. If the scenario script itself is missing or wrong, stop and use the case repository write-replay-case skill — do not invent Cases inside Tutti.

Qualification is assertion-specific, not command-specific. A replay passed exit proves transport and semantic playback, but does not prove that every Case action ran during Replay. Build a four-column matrix for each core assertion: Record proof | Cassette proof | Replay proof | Evidence. Stop and repair the scenario when any required Replay cell is empty.

Start the QA console (case repository)

Browsing Cases, Test Plans, and one-click Record/Replay (which run the same scenario scripts) live in the case repository (sibling checkout, commonly ../tutti-replay; GitHub: tutti-os/tutti-replay).

From the case repository root:

pnpm install
pnpm dev

Open only http://127.0.0.1:3333 (never the API port :3334). In the UI, set the Tutti checkout absolute path, create a Test Plan, then Record or Replay. First-time machine setup: that repository's SETUP.md. Authoring or mirroring scenario scripts: .agents/skills/write-replay-case/.

For a long-lived LAN service on macOS use pnpm replay:service install (port 2333); do not run pnpm dev and the stable service at the same time.

Establish scope

1. Read the Tutti root and closest AGENTS.md files.
2. Read docs/architecture/agent-session-replay.md (Provider support: developer recording currently accepts local:codex and local:claude-code only).
3. For AgentGUI behavior, also read docs/architecture/agent-gui-node.md and packages/agent/gui/AGENTS.md.
4. For Session, Turn, Goal, or runtime-operation lifecycle behavior, read packages/agent/host/README.md; lifecycle semantics remain in Host.
5. Inspect git status --short in both repositories and preserve pre-existing work.
6. Resolve the case repository from the user-provided path, the configured cases path, or the sibling ../tutti-replay checkout. Do not guess another location if none exists.

Read the selected Case before planning:

• cases/<case-id>/case.json
• every cases/<case-id>/scenarios/*.mjs except *.impl.mjs
• referenced shared scenario helpers and runtime-fixtures/
• existing cassettes/, evidence/, and relevant Run artifacts
• the case repository's README.md and CONTEXT.md when publication or Case lifecycle is involved

If case.json declares executionKind: "ui", stop this Session Replay workflow. Pure UI Cases are still script-driven (defineUiScenario + CDP), but they use ui-drive and publish ui/ screenshots — they do not Record Provider Cassettes. Author and run them via the case repository write-replay-case skill and the QA console; do not use this skill's --record / Cassette audit path for them.

Use CDP through Tutti's repository runner. Do not use Computer Use unless the user explicitly requests it.

Maintain the ownership boundary

• Add or update Case scenarios only under cases/<case-id>/scenarios/*.mjs in the case repository.
• Put reusable scenario helpers in that repository's scenario-runtime/.
• Do not add Case registries, Case-specific scenarios, fixtures, or qualified Cassettes to Tutti.
• Change Tutti only for generic product, runner, protocol, or Replay defects.
• Fix root causes. Do not relax transport matching, semantic verification, terminal assertions, or checkpoint requirements to accept a broken Case.

Add a new Provider (product vs case repository)

Today Replay recording targets are local:codex and local:claude-code. The shared Session Replay core is provider-neutral, but each new Agent Target still needs Tutti capture + fail-closed playback before any Case work is useful.

Split work explicitly:

Layer	Where	Who	Scope
Product	Tutti	experienced / mentored	Adapter capture, projected tape, portability, structural audit, outbound verification, input-unit barriers, isolated Provider home, deterministic fail-closed Replay for local:<provider>
Cases	case repository	can hand to intern after product gate	providerProfiles, KNOWN_PROVIDERS, defineMirroredRecordScenario mirrors, Record via console, publish cassettes

Do not start by writing Cases for an unsupported Provider. Product must accept --agent-target-id local:<provider> for Record and Replay first.

Tutti checklist (this repository)

Copy and tick:

- [ ] 1. Provider adapter can Record real traffic into a Cassette
- [ ] 2. Projected tape + portability (paths/homes) match session-replay contract
- [ ] 3. Structural audit passes (manifest, frames, activity causality)
- [ ] 4. Fresh isolated Replay is fail-closed (no live Provider fallback)
- [ ] 5. Runner accepts --agent-target-id local:<provider>
- [ ] 6. docs/architecture/agent-session-replay.md Provider support updated
- [ ] 7. Hand off to case repository write-replay-case for mirrors + console Record

Prove with one smoke scenario from the case repository (or a temporary scenario file) using the Tutti runner Record → audit → fresh Replay loop below. Keep account secrets out of logs and reports.

Case repository handoff

After the product gate is green, follow that repository's write-replay-case skill section on multi-provider mirrors. Typical touch points there (not in Tutti): scenario-runtime/shared.mjs providerProfiles, src/shared/agent-target.ts KNOWN_PROVIDERS, mirrored *.impl.mjs + variant Case dirs, then console Record/Replay publication.

Scenario scripts (owned by the case repository)

Session Replay drive logic is authored as scripts, not captured from manual UI interaction. Prefer the case repository's write-replay-case skill and defineMirroredRecordScenario / defineRecordScenario helpers.

When diagnosing or qualifying from Tutti, still require that the loaded scenario:

• exports prepare, drive, and assert;
• sets Provider via profile/helper (not a hard-coded Codex-only identity);
• sets every behavior-affecting composer default explicitly;
• uses one stable prompt per intended Turn, with exact markers / final tokens;
• uses accessible labels, test IDs, or semantic DOM state instead of coordinates;
• waits before each interaction; answers each approval/question/plan once;
• asserts a terminal state with no enabled stale controls;
• declares expectedRecordingMode when continuing an existing Session.

Know the runner hook boundary before accepting the scenario:

• Record calls prepare, drive, assert(phase="terminal"), and assert(phase="recorded").
• Fresh Replay does not call scenario.assert; it replays Cassette traffic and calls settleForScreenshot at eligible checkpoints.
• captureEvidence proves a Record state only. captureFrame inside settleForScreenshot proves a Replay checkpoint state.

For each action under test, require one Replay mechanism:

1. Cassette-native: the action appears in activity-events.jsonl as the expected intent/effect or direct stimulus and is consumed by a checkpoint.
2. Replay driver: when the action is not recorded (for example a local UI action after Turn completion), settleForScreenshot performs it through the real product UI/API at one exact checkpoint, asserts before/after state, and captures transient evidence.

If neither exists, classify the assertion as Record-only. Do not call the Case Replay-qualified or infer coverage from the final UI looking correct.

For question cards, the script must trigger the Provider's real user-input request. For plan Cases, wait for the completed plan and implementation decision before driving the real action.

If the script needs to change, edit it in the case repository — then re-Record.

Record = execute the scenario script with a live Provider

From the Tutti root, run the repository runner so it CDP-drives the scenario file and captures a Cassette. Derive scenario ID, Cassette name, and Agent Target from the loaded scenario:

pnpm e2e:agent-gui -- \
  --record .tmp/cassettes/<cassette-name> \
  --scenario <scenario-id> \
  --scenario-file <case-repository>/cases/<case-id>/scenarios/<scenario-id>.mjs \
  --agent-target-id <agent-target-id> \
  --keep-runtime \
  --timeout-ms 300000 \
  --stall-timeout-ms 60000

Prefer the case console Test Plan「录制」for routine work; use this CLI when debugging runner/Replay behavior or when the console is unavailable.

Omit --headless while debugging; add it for unattended execution. Keep the runtime only long enough to inspect or collect its artifacts.

Inspect failures from the smallest relevant evidence set:

• record screenshots and checkpoint screenshots;
• logs/desktop.log;
• state/logs/tuttid.log;
• state/tuttid.db through targeted queries;
• the incomplete Cassette and a small decoded Provider-frame window.

Do not dump an entire Provider stream or expose account data.

Audit each Cassette

Run the bundled structural audit from the Tutti root:

node .codex/skills/tutti-record-agent-session-replay/scripts/audit-cassette.mjs \
  .tmp/cassettes/<cassette-name>

Require:

• Cassette inventory, hashes, and size policy verify;
• the Provider manifest is complete;
• global and per-connection frame sequences are continuous;
• Activity sequences are continuous;
• intent-to-effect causality satisfies packages/agent/session-replay/activity-contract.json;
• expected interactions, plan decisions, tools, exits, terminal Turns, and final response state match case.json and the scenario assertions.
• every Case-critical action expected to replay is present in activity-events.jsonl / checkpoint-plan.json, or is explicitly owned by a checkpoint Replay driver.

The bundled audit proves structural invariants and emits a semantic summary. It does not replace Case-specific assertions or a fresh Replay.

Before Replay, compare the audit output and raw checkpoint plan with the Case's coverage matrix. Counts alone are insufficient: for example, an Activity stream containing only activation/requested + session/activate cannot prove a later Undo/Reapply click.

Run a fresh Replay

Replay from a fresh isolated Tutti runtime and pass the scenario so checkpoint screenshot settling runs. Remember that scenario.assert does not run here:

pnpm e2e:agent-gui -- \
  --replay .tmp/cassettes/<cassette-name> \
  --scenario <scenario-id> \
  --scenario-file <case-repository>/cases/<case-id>/scenarios/<scenario-id>.mjs \
  --screenshot-checkpoints \
  --keep-runtime \
  --timeout-ms 300000 \
  --stall-timeout-ms 60000

Require the runner's replay passed result, all planned checkpoints, the expected AgentGUI terminal state, and a fully drained Provider transport. Provider transport remains fail-closed; only repository-declared observer-only probes may yield to causal traffic.

Then prove each target assertion through its selected Replay mechanism:

• Cassette-native: show the matching Activity/direct stimulus and resulting checkpoint state.
• Replay driver: show its before/after assertions completed and inspect the named captureFrame screenshots, including transient states rather than only the final stable UI.
• External filesystem/process side effects: prove how the fresh Replay project receives the minimal deterministic pre-action state. Cassette semantic state does not by itself recreate arbitrary Provider filesystem mutations.

Never report “Record & Replay cover the behavior” when only Record executed the Case action. Report transport/semantic Replay separately from assertion-level behavior coverage.

When one Case owns multiple scenarios, Record and audit every resulting Cassette, then qualify them together through one Replay Workspace. Let the case repository workflow generate the workspace manifest; do not invent a second Case registry in Tutti.

Publish only qualified artifacts

Prefer the case repository's publication workflow because it records the Run, archives prior artifacts, and publishes cassettes/ plus evidence/ only after qualification.

If the user explicitly requests manual publication:

1. Stage all new Cassettes and evidence without touching the qualified copies.
2. Verify every staged Cassette completed Record, audit, and fresh Replay.
3. Replace cases/<case-id>/cassettes/ and evidence/ as one Case operation, preserving the previous qualified artifacts in the case repository's archive convention.
4. Re-read every published manifest and evidence directory.

Do not infer Case lifecycle from a successful command. Run status, manual acceptance, and case.json lifecycle status are distinct. Do not set status: "confirmed" or acceptedAt unless the case repository's acceptance requirements have been satisfied.

Finish

• If Tutti implementation changed, run the validation selected by docs/conventions/testing.md plus any closest-area checks.
• If case metadata or scenarios changed, run the case repository's metadata, scenario, and type checks.
• Recheck both worktrees and separate pre-existing changes from this task.
• Perform the Tutti documentation-impact check.

Report:

• Case and Cassette names;
• Record, audit, and fresh Replay results;
• Provider-frame, Activity, interaction, tool, Turn, and final-state summaries;
• the assertion coverage matrix and exact Replay mechanism for each core action;
• Tutti implementation changes and case repository artifact changes;
• changed-line distribution by functional area, excluding pre-existing work;
• documentation impact;
• failed gates and unimplemented scope.