Use when a change adds or changes a host decision (a stop, park, decline, refusal, retry, deadline or cap growth), when the bench loop exports a round's traces, or when replay-corpus.test fails. Keeps the bench-trace replay corpus complete and current.
The English summary is being prepared. Showing the original description.
Run score Not tested
See how to installReview the instructions the agent will receive before you install.
BAKED's host decisions are replayed on recorded bench rounds before a change merges (#1649, #1715). Four fixes in three days each broke the next bench round (#1643, #1627, #1639, #1640), because a scenario test only covers the run that exposed the bug. The corpus is the set of real runs every change must still decide correctly. It works only when every decision point has a replay hook and the corpus stays readable by the current code.
Traces: tests/fixtures/bench-traces/RNN/<app>.sqlite.gz. Each is an anonymised copy of a bench app's .baked/workflows.sqlite, written by bun scripts/bench-trace.ts export <workspace> <out>. The export (exportTrace in src/core/workflows/trace.ts) does the following:
keeps only the artifact kinds in KEPT_ARTIFACTS;
drops the tables in DROPPED_TABLES;
replaces every string under a key in PROSE, and every other string that contains whitespace, with a stand-in KEY-N (StandIns). The same text gets the same stand-in across the whole trace, so equality checks between artifacts and columns still hold;
writes a request text (requestText, prompt) as its stand-in followed by the file names and paths the request names to change, one per line (requestedPathTokens in auto-approval.ts), so the approval replay decides request-named manifest edits as the run did; traces exported before that keep the stand-in alone;
keeps the run's state artifact with the content in STATE_CONTENT (spec, plan, research, reports) stood in, and rewrites every contract hash to the hash of the scrubbed spec, so a replay that reopens the run through openTrace sees the same features, runs and shares the round saw.
keeps each finding's body (findings.body_json) with its prose stood in, so repeatOf, file, symbol and repro stay readable for the finding-chain replays (#1981); traces exported before that have {} there, and their review findings come only from the run state's last two reviews.
Policies and workflow definitions stay whole, because they hold settings and enums, not prose. Traces exported before #1707 have no state artifact; re-export them with the current code before a hook needs the run state. .gitignore ignores *.sqlite* everywhere except under this folder.
Replay: replayTrace in trace.ts. It runs each hook over every run of a trace and reports where the current code decides differently from the recording.
Gate: tests/forecast-replay-corpus.test.ts is part of bun run test, so it runs in the required check job. A decision the recording did not make fails the suite. The only exception is a run listed in tests/fixtures/bench-traces/accepted.json as "RNN/app.sqlite.gz#RUN_ID": { "<counter>": n, "reason": "..." }.
CLI: bun scripts/bench-trace.ts replay <trace>... [--json] prints each run's differing decisions.
Recordings: tests/fixtures/bench-recordings/RNN/<app>/RUN_ID.json.gz and its manifest.json, written by bun scripts/bench-trace.ts recordings <workspace> <out-dir> --run RUN_ID. Every bench run and initiative records its model calls (--record-model-calls); a recording holds the prompts and the model's answers, so only the runs that stopped are kept, and only from public bench apps. They are the input of the replay to the first divergent request (#1715 step 2), which compares the decisions that need the model's answers.
| Decision | Hook | Counter in accepted.json |
|---|---|---|
Run cost-forecast stop (assertForecastFits) | replayRun: rewinds the run to each priced attempt | newStops |
| Cap growth to the forecast | replayRun | newGrowth |
A model call's cost-cap check (costCapDecision, from reserve and expandReservation) | replayRun: checks each recorded attempt's largest reservation on its stage and model where the run checked it: at the last raise while the attempt ran (a budget.updated event, which expandReservation emits with the run's open reservation total), rewound there with the other open attempts scaled down to that total, or at its start when it was never raised. Every recorded attempt passed, so a replayed refusal is new. Rewound to the start with its largest reservation, R70 evalpin suite-runner's two reviews were refused on a cap the initiative grew from $5.37 to $5.69 before the raise | newCostStops |
A model call's first-response deadline (providerFirstByteMs, and the adaptive one of FirstByteHistory.deadline, #1766, #2041) | replayFirstByte / replayFirstBytes: reads each call's calls timing in attempts/*/usage.json. A call that recorded an adaptive deadline (limitMs) is decided again by the adaptive rule over a history of the trace's earlier calls, in the order their attempts started, with each call's prompt size (promptChars, from #2041); replayTrace(path, { adaptiveFirstByte }) takes an older rule, null for the fixed deadline alone. Older traces fall back to the attempt length of a one-step buffered CLI attempt | newFirstByteStops |
A planned test task's split (exceedsTestCapacity, from sizePlan) | replayTestSplits: reads each sizing/plan-N.json report and decides again from its recorded inputs (each test task's cases and estimate, testTaskLimit, maxAttemptTokens). Slices and reports without case counts (before #1562) are left out. Takes testSplit to replay an older rule (#1789). From #1930 a report marks a rows task (rowsTask), which the rule decides by its cases alone; older reports did not, and replay as recorded | newTestSplits |
An initiative feature's start on its budget share (shareShortfall) | replayShareGate: reads each feature.share_gate initiative event, which records the gate's inputs (remaining budget, dollar cost view, the waiting features' shares and estimates) and its verdict. Keyed by initiative: "RNN/app.sqlite.gz#INITIATIVE_ID". Traces before #1707 record none. From #1890 each feature also records shareUsd and needUsd, so the dollar start gate ("The shared dollar budget is short") replays too; older records have neither and replay without it | newShareStops |
Whether the host would extend an initiative's token pool inside the dollar cap at a park (poolGrowthOf, poolExtensionDecision, shadow, #2042) | replayPoolGrowth: reads each parked feature.share_gate event; from #2042 on it records the decision (poolGrowth) and the pool's inputs (limits, used, pool), and a would-extend also records a decision.would_extend event with its inputs. A park recorded before #2042 is decided on the limit of the trace's next budget.extended event (its previous), else the initiative's settings or approved revision, without a token forecast or a measured dollar rate, and counts as hold. Takes poolGrowthRule. replayInitiativeStarts(path, { budgetAtStart: true, moments }) rewinds the budget and each run's used tokens too and computes the pool's forecast at further moments; the corpus gate keeps the final budget | newPoolGrowth |
The same start gate on recomputed inputs, the initiative's dollar forecast included (InitiativeService.view, then shareShortfall) | replayInitiativeStarts: from the last feature start back to the first, rewinds a copy of the trace to the moment the start was linked (later runs, links, events and requests removed, runs still going rewound like a run replay) and asks the gate again. The policy's sizing.historyStores are replaced by their corpus traces (corpusHistoryTrace: .../RNN/app is RNN/app.sqlite.gz), trimmed to the runs that existed then; stores older than the corpus are listed as missingHistory. Features without a run are routed with the runner's model routing, read from the latest routing artifact before the start (each route names the modelRouting class, role or profile it came from); on R52 srcwhere this replays the runner's recorded $46.80 advisory exactly. Takes priceSamples to replay an older pricing. Each start also records every feature's token range in the estimate the current code gives there (estimate); the corpus test compares the first start's ranges with what the features used (#1806). Starts made under an earlier split revision are not replayed | newStartStops |
The plan approval's price of its tasks (planCostOf, #1792) | replayPlanPrices: rewinds a copy of the trace to each accepted plan approval (runs made later removed, runs still going not counted as finished, the run rewound like a run replay), prices the tasks again with the given history stores (other corpus traces opened by the caller, or null for no policy as before #1792) and returns the recorded and replayed dollar ranges next to what the run's tasks then cost. A price shown to a person, not a stop: the corpus test compares its error, it adds no counter to accepted.json | none |
A stalled call's next rung on the recovery ladder (nextRung, #1877) | replayStallLadder: walks each stage's recorded attempts in order; an attempt whose last call in attempts/*/usage.json got no first response is a stall, as is one whose usage carries lostAnswer (a turn with no usable answer, #1957); a completed one ends the ladder. The route comes from the run's routing record, the effort from the attempt's usage; attempt.recovery events say which rung the run took. The effects table is not in a trace, so an attempt with a pre_image tool phase (the tool policy emits one as it prepares each file effect) is one that changed a file: the scheduler keeps the provider wait for it, so it replays no rung and the next attempt starts the ladder again (#2196). Takes stallRule to replay an older rule | newRecoveries |
A provider's usage-policy refusal of an attempt (policyRefusalDecision, #1895) | replayPolicyRefusals: reads each provider/policy-refusals/*.json record (kind policy-refusal, kept in traces; the provider's message is a stand-in) and decides again from the refusals of the same task or stage profile and model before it. Traces before #1895 record none | newPolicyStops |
A corrective's rewrite of a finding chain's symbol (chainRewriteDecision), the battery's reopen before a review (batteryDecision, #1959) and whether a sibling probe's case joins the battery (familyDecision, #1969) | replayFindingChains: reads each chains/tasks/*.json record (the chain's length, threshold, policy and decision), each chains/battery/*.json record (each case's outcome and the decision) and each chains/family/*.json record (each probed case's outcome and decision, one boundary per case), kind finding-chain, and decides again. Takes chainRule, batteryRule and familyRule. Traces before #1959 record none. Two readers beside it gate nothing: replayChainEscalation decides each recorded finding's chain from the findings themselves under a given policy (where the rewrite would have come on a run with it off, and its battery, with each step's family; families replays a run with the labels a reviewer would have declared, #2048), and chainCosts gives each recorded chain's length, battery results and corrective cost; bench-trace.ts chains prints both (#1981) | newChainDecisions |
Whether a probed edge input of a planned source task is a finding (edgeDecision, #1808) | replayEdgeProbes: reads each probes/edge/*.json record (kind edge-probe, kept in traces: each case's criterion, outcome and decision) and decides again from the outcome. Takes edgeRule. Traces before #1808 record none | newEdgeDecisions |
A corrective's required checks failing exactly as the corrective's before it (repeatedFailureDecision, stop reason repeated_failure in shadow, #1996) | replayRepeatedFailures: reads each repeats/tasks/*.json record (kind repeated-failure, kept in traces; each check's normalized output is a stand-in, so equal outputs stay equal) and decides again against the record of the corrective it corrects. Takes repeatRule. A run recorded before #1996 is decided from its check findings instead (findingRounds: the findings repairPredecessors gives each corrective, without a review's basis, by stood-in impact and file; no bodies before #1981). Counted once per chain, as the run records it; R64 cfgquill lint a843cc11 is the accepted one (T006) | newRepeatStops |
A spec conflict settled for the request (requestConflictDecision, #2016) | replayRequestConflicts: reads each conflicts/request/*.json record (kind request-conflict, kept in traces; it holds each side's kind, ref and whether the host found its quote, never the quote) and decides again from the sides, the tests the run did not write, the cited criteria and the re-audit's verdicts. A record whose re-audit ran is decided on its verdicts. Takes requestConflictRule. Traces before #2016 record none | newConflictDecisions |
Whether the host would apply a planned task's verbatim content itself (hostApplyDecision, shadow, #2023) | replayHostApply: reads each host-apply/tasks/*.json record (kind host-apply, kept in traces; per write its supplied block counts, whether a fenced file is whole, whether the path exists and whether its diff passed git apply --check, the refusal a stand-in) and decides again. Takes hostApplyRule. Traces before #2023 record none | newApplyDecisions |
Whether a task-level corrective can write the files its failed check names in the failing task's files (missingDiagnosed, stop reason corrective_scope in shadow, #2094) | replayCorrectiveScopes: reads each scope/tasks/*.json record (kind corrective-scope, kept in traces; the failing tasks, checks, diagnosed files, writes and decision, written when the corrective is created and before each session) and decides again. Takes scopeRule. Traces before #2094 record none | newScopeStops |
Whether the drive runs the ready tasks a blocked corrective does not hold before the run stops (correctiveDeferDecision, policy field correctiveDefer, shadow by default, #2109) | replayCorrectiveDefers: reads each blocks/correctives/*.json record (kind corrective-defer, kept in traces; the stop's code and reason, the ready task ids, the earlier parked correctives, the writers and the decision) and decides again. Takes correctiveDeferRule. Traces before #2109 record none | newDeferDecisions |
Whether a planned test task's failure whose expected values are all literals of its criteria goes to the finished sibling that wrote the imported source (sourceDefectDecision, policy field sourceDefectRoute, shadow by default, #2220) | replaySourceDefects: reads each source-defect/tasks/*.json record (kind source-defect, kept in traces; counts of expected and matched values, owner task ids, source paths, the decision; no literals) and decides again. Takes sourceDefectRule. Traces before #2220 record none | newSourceDefectRoutes |
What the host does about a file a corrective's evidence names outside its writes (unownedRule, hostDecision: route, widen, park, wait or refuse; stop reason corrective_unowned_file in shadow, #2133) | replayCorrectiveScopes: reads the same scope/tasks/*.json records (written also on a blocked or failed result, at: "result"), each named path's recorded owner, unfinished writers, feature and approval, and decides again. Takes unownedRule. Records before #2133 have no named paths | newUnownedDecisions |
Whether a required check that fails before any change on files no task writes is repaired by the host's fixer, recorded as a would-stop or left alone (baselineFailureDecision, policy field baselineFix, stop reason baseline_unrepairable in shadow, #2215) | replayBaselineFailures: reads each baseline-fix/<check>.json record (kind baseline-fix, kept in traces; the located finding and correctable counts, whether they account for the tool's summary, the named files and how many a task writes or git does not track, the check's fix, the flag and the decision) and decides again. Takes baselineFailureRule. Traces before #2215 record none | newBaselineDecisions |
The plan auto-approval (planApprovalDecision, which the scheduler calls) | replayPlanApprovals: at each plan approval of a run with auto approval, the run state of that moment, the run's earlier requests and its artifacts until then go to planApprovalDecision again (or a decide that emulates an older rule). Deliverable coverage is not replayed (the request has no text left), and a path exists when the plan modifies rather than creates it. A package the approval reviewer approved counts as approved while the replay leaves it reviewable. Gated from R52 on: older states counted the host's own test splits in hostSplitTasks, which sizing slices now say (#1711) | newApprovalDeclines (approved then, needs a person now), newAutoApprovals (a person then, approved now) |
The test-integrity guard's findings (integrityDecision, after each task and at feature scope) | replayIntegrity: reads each integrity/decisions/ record (kind test-integrity), written by every scan whether or not it found anything: per changed file, the counts and flags measureTestIntegrity read (tests and assertions before and after, matched and requested ones, left-out protected tests, skips, hooks, test-input literals, suppressions, bypasses) and the exemptions that applied (a reasoned write, a requested removal, a rewrite or strengthen finding), without names or code. integrityDecision decides them again; takes integrityRule. How a rule counts reads code a trace does not keep, so a counting change is covered by the scan's own tests, not the corpus. Traces before #1715 record none | newIntegrityFindings |
The task-end case walk's refusals (caseWalkDecision, caseCheck) | replayCaseWalks: reads each case-walk/tasks/ record (kind case-walk), written by every walk: per case, its criterion, whether it is a host row cell, whether the task's own test files hold it and, for a case they do not, whether a sibling task's files do, and whether the host's rows blocks are intact, without literals or code. caseWalkDecision decides them again; takes caseWalkRule. Whether a file holds a case reads code and spec literals a trace does not keep, so a matching change is covered by the case-walk scenario tests, not the corpus. Traces before #1715 record none | newCaseRefusals |
Which review findings start a correction (reviewGateDecision: severity floor, inferred and confidence notes, sibling notes, the repro gate) | replayReviewGates: reads each review-gate/ record (kind review-gate), written by every review: the reviewer, whether it is required, the severities it blocks on and the run repairs that round, the confidence floor, whether a repro can run, and per finding its id, severity, kind, basis, confidence, sibling note, whether it carries a repro and the repro's outcome, without claims, files or repro text. reviewGateDecision decides them again; takes reviewGateRule. A repro ran only for the recorded gate's candidates, so a rule that makes another finding a candidate replays it without an outcome. Traces before #1715 record none | newGateRepairs |
Which convergence findings repair, become known limitations (notes rule, repro gate) or stay notes, and which criteria stay failing (convergenceGateDecision, with notesPass) | replayConvergenceGates: reads each convergence-gate/ record (kind convergence-gate), written by every converge stage: the review gate's finding labels plus whether each finding is on a test or documentation file and the criteria it names, each criterion's verdict and evidence count, the failing criteria before the gate, whether the required checks pass and whether the walker could run repros, without claims or paths. convergenceGateDecision decides them again; takes convergenceGateRule. As for the review gate, a repro ran only for the recorded candidates. Traces before #1715 record none | newConvergenceVerdicts |
The provider fault matrix (tests/provider-fault-matrix.test.ts) reads the same traces through corpusCallProfile: each recorded call shape's p99 first byte and the recorded retry-after waits become its fault profiles (#1729). A new field a profile needs goes into the calls timing, so the next round's traces carry it.
A decision computed from the whole run or initiative state, like the initiative forecast, needs no new record: its hook rewinds a copy of the trace and calls the current code. Keep what that code reads in the export (the state artifact, the history stores' own traces), and seam only the part a proof test swaps, as InitiativeService.priceSamples is.
Every host decision #1715 listed is replayed: each records the counts and flags it decides from (never the model's text), and a hook decides them again. A replay of --record-model-calls recordings up to the first divergent request was set aside for this: recordings keep no seed repository and only hashes of the prompts, so any prompt change diverges at the first call. A new host decision gets the same treatment (step list below).
A host decision is anything that stops, parks, declines, refuses, retries, times out or grows a limit on its own. In the same pull request:
calls was added to usage.json for the first-response deadline. If it lives in an artifact kind that is not kept, add the kind to KEPT_ARTIFACTS and check that scrubTrace removes its prose.trace.ts. It computes the decision from the recorded inputs with the current code. It reports recorded and replayed per boundary and a counter on RunReplay for decisions that are new against the recording. Take the decision function as an optional parameter, as replayTrace(path, { firstByteLimit }) does, so a test can show that the hook catches the old behavior.forecast-replay-corpus.test.ts is the pattern.CONTRIBUTING.md (the replay paragraph) and in docs/workflows/runtime.md when the decision is user-visible.When your change means to decide differently on a recorded run, add that run to accepted.json. Use the exact counter and a reason that names the issue. Find the run with bench-trace.ts replay. Bisect when it is unclear which change moved a decision: copy trace.ts and scripts/bench-trace.ts into a worktree of the older commit and replay there. Never widen an entry to make an unrelated difference pass. A difference you did not intend is a regression; file it with the file-issue skill.
C:\Project\baked-bench\RNN\<app> folders.WorkflowStore migrates a trace when it opens it, so run the corpus test after every migration.decisionOf in src/core/workflows/trace.ts): a run checks its forecast again on later calls of an attempt, after a parallel attempt may have started, and a token growth raises the cap with the tokens (R60 wheeltags, #1937).pre_image tool phases (R69 md-site, #2196). A missing guard shows up as an accepted.json entry whose reason argues the guard did not apply.previous values in policy revisions. Apply each revision's next forward from the starting policy: a revision written from a stale policy undid an earlier growth (#1704).feature/bench-traces-rNN pull request with the traces. Add the recording of every run that stopped (a forecast, budget, provider or gate stop, or an intervention) to the same pull request. A trace above 2 MB is mentioned in the round report. Run git status before committing to confirm the .sqlite.gz files are staged.accepted.json with the fix's issue as the reason. Each new stop the round shows makes that run part of the corpus automatically, once its trace merges..baked/ goes into the repository.No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Use when adding a language to BAKED (a new entry of SUPPORTED_LANGUAGES) or filling the `none` rows of a supported one (gate detector, test runner, formatter, reviewer, the per-language issues
The English summary is being prepared. Showing the original description.
Use when adding, changing or graduating a task in bench/suite/ (the nightly multi-language mini-suite), when adding a supported language or toolchain, or when a bench round stop can be reproduced by a small task.
The English summary is being prepared. Showing the original description.
Use every time you open or update a GitHub issue for a BAKED bug or finding. Finds the root cause, sweeps the code for every other place with the same or a neighbouring defect, including every supported language when the bug is bound to one, and files issues that cover all of them.
The English summary is being prepared. Showing the original description.