<!-- desc:full -->

Full description

Shared bounded-loop runner for the Karpathy autoresearch pattern applied to code: ONE mutable artifact + a FIXED evaluation harness emitting ONE comparable composite score, looped as snapshot -> one focused edit -> score -> keep-if-strictly-better-else-revert -> log, bounded by per-pass max_iterations / max_consecutive_reverts and the cost brake. Invoked by the pipeline-orchestrator for the AR_DESIGN / AR_FUNCTION / AR_ACCEPTANCE stages, or runnable standalone. Validation only; NEVER auto-commits and NEVER runs destructive git.

Autoresearch — bounded keep/revert runner

This skill is the runner. It does not define what to optimize or how to score — those live in the three per-pass prompt files. The runner owns the loop spine: setup, one-edit-at-a-time iteration, snapshot/keep/revert, the results.tsv log, and the bounded stop conditions. It is adapted directly from the AMS PRD autoresearch program.md, generalized from "edit one PRD to maximize a reflection score" to "edit the production source touched by a task to maximize a per-pass composite score."

Invariant (Karpathy autoresearch): ONE mutable artifact + a FIXED harness emitting ONE comparable score, then loop { pick one idea -> edit -> score -> keep-if-better-else-revert -> log } -> stop on threshold / plateau / cap. You read this file and follow it.

---

How it is invoked

This runner has three modes, one per pipeline pass. The active mode selects exactly one per-pass prompt file, which is the authoritative source for that pass's mutable surface, fixed harness, rubric/KPIs, composite formula, and hard constraints:

Mode	Pipeline stage	Per-pass prompt file (READ THIS for the active mode)
design	AR_DESIGN (AR-1, pre-TDD)	program.ar-design.md
function	AR_FUNCTION (AR-2, post-TDD)	program.ar-function.md
acceptance	AR_ACCEPTANCE (AR-3, post-QA)	program.ar-acceptance.md

Two entry paths:

1. Pipeline-orchestrated (normal). The /pipeline state machine invokes this runner at AR_DESIGN (after CODE_REVIEW), AR_FUNCTION (after TDD), and AR_ACCEPTANCE (after PHASE_GATE). The orchestrator passes the mode and fills all placeholders from .aid/context.json + .aid/pipeline/config.json. On stop the runner FIRST restores the editable set to the best-kept version, THEN logs final scores to step_history and writes .aid/pipeline/state.json unconditionally as the FINAL action — after the last source mutation of ANY kind (kept edit, revert/restore cp, or crash restore), so the Stop hook (dev-pipeline-gate.sh) freshness check still passes (newest source mtime <= state mtime).

2. Standalone. A human runs it directly on a single mode. Resolve placeholders yourself from .aid/context.json + .aid/pipeline/config.json; if a value is absent, use the defaults noted in the per-pass file and state the assumption in chat.

Order matters and is enforced upstream: function mode requires a green test suite to exist (TDD produced it); acceptance mode requires QA to have run. Do not invent a harness a pass does not yet have — that is precisely why the passes are split.

---

Placeholders (filled at runtime)

Resolved from .aid/context.json and .aid/pipeline/config.json. The per-pass file declares which it uses; the runner only needs these for the loop spine:

Placeholder	Source	Meaning
{{WORKSPACE}}	context.json	Repo root path
{{TASK_ID}}	context.json	Current task id; names the results dir
{{CRITICALITY_PROFILE}}	context.json / config.json	Risk profile gating ASK-before-edit behavior
{{KPI_TARGET}}	config.autoresearch.kpi_target (default 9.5)	Quality-score target (0–10)
{{TEST_COMMAND}}	config.test_commands.unit (+coverage)	Harness command (function mode)
{{MIN_COVERAGE}}	config.thresholds.min_coverage_percent	Coverage gate (function mode)
{{SAFETY_CRITICAL_FLOWS}}	context.json	Flows requiring ASK-before-edit
{{RECORD_FILE}}	derived	.aid/pipeline/autoresearch/{{TASK_ID}}/results.tsv
pass-specific	config.autoresearch.<pass>	max_iterations, max_consecutive_reverts, internal rounds, and (AR-3) max_acceptance_rounds, pass_rate_target

Per-pass caps come from config.autoresearch: { kpi_target, ar_design:{max_iterations,max_consecutive_reverts}, ar_function:{max_iterations,max_consecutive_reverts,internal_rounds}, ar_acceptance:{max_acceptance_rounds, pass_rate_target} }.

ar_design has NO internal_rounds. The bounded internal refine-rounds knob belongs to ar_function ONLY (AR-2 allows a few internal apply/refine rounds before official scoring); AR-1 design does a single apply per iteration. The internal_rounds key was therefore removed from the ar_design block in config.default.json — it is a dead key under ar_design and must not be reintroduced there.

Placeholder → config bindings

The per-pass prompt files may keep their {{...}} placeholder names; this table is the authoritative binding of each to its config.autoresearch key:

Placeholder	Config key
{{AR1_MAX_ITERS}}	config.autoresearch.ar_design.max_iterations
{{AR1_MAX_REVERTS}}	config.autoresearch.ar_design.max_consecutive_reverts
{{AR2_MAX_ITERS}}	config.autoresearch.ar_function.max_iterations
{{AR2_MAX_REVERTS}}	config.autoresearch.ar_function.max_consecutive_reverts
{{AR2_INTERNAL_ROUNDS}}	config.autoresearch.ar_function.internal_rounds (NEW key, default 2)

---

The loop SKELETON (shared by all three modes)

The mode's per-pass file supplies the harness (how to produce the composite score), the rubric/dimensions, the composite formula, and the hard constraints. Everything below is identical across modes.

Setup phase (run ONCE per pass, then never again)

1. Read the active per-pass prompt file for this mode (table above). It is authoritative for mutable surface, harness, rubric, composite, constraints, and stop conditions specific to the pass. The runner's stop conditions below are layered on top of those.
2. Resolve the mutable surface. The artifact under optimization is the production source changed by this task:

   git -C "{{WORKSPACE}}" diff --name-only HEAD          # tracked changes
   git -C "{{WORKSPACE}}" ls-files --others --exclude-standard   # untracked (new) files
   

   The union (filtered to source per the per-pass file) is your mutable surface. Everything else — the harness, tests (in function/acceptance), criteria YAMLs, .aid/*.json, config — is read-only. (git here is read-only inspection; no commits, no resets, no checkout/restore of tracked files. Rollback is done via snapshot copies, below.)
3. Create the results directory and snapshots dir:

   mkdir -p ".aid/pipeline/autoresearch/{{TASK_ID}}/snapshots"
   

4. Init results.tsv ({{RECORD_FILE}}) if absent or header-only, with this header line (tab-separated):

   iter	composite	delta	<one column per rubric dimension>	status	snapshot	utc
   

   The per-pass dimension columns sit between delta and status and are named by the active per-pass file using the EXACT criterion ids (e.g. AR-1: structural_health loudness_of_failures naming_clarity docs_traceability; AR-2: quality pass_rate coverage). Each per-pass program file prints its OWN exact header. Single-letter abbreviations (S/L/N/D) are NEVER allowed as TSV column names — only in the human one-line chat output. If results.tsv already has data rows, do NOT re-baseline — resume from the next iter.
5. Score the baseline using the per-pass harness on the current on-disk source. Record the composite as best_score. Snapshot the baseline of every mutable file:

   TS="$(date -u +%Y%m%dT%H%M%SZ)"
   # Directory form — AUTHORITATIVE (matches each per-pass program file; per line 111 the
   # per-pass file wins): ONE baseline directory with the mutable files copied in preserving
   # their relative paths, so verify-run.sh's per-file revert check can diff against it.
   # Do NOT use a flat 000-baseline-<safe-F> file — that splits the baseline across files and
   # makes verify-run.sh skip the revert check (WARN, not FAIL).
   DST=".aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/000-baseline-$TS"; mkdir -p "$DST"
   # for each mutable file F: copy it into "$DST" preserving its relative path
   

6. Append row 0 to results.tsv with iter=0, composite=<best_score>, delta=0.00, the baseline dimension scores, status=baseline, the baseline snapshot tag, and UTC.
7. Record the rubric fingerprint (harness-tamper detection). At PASS START — once, here in setup — compute a SHA-256 fingerprint over the EXACT bytes of the harness inputs that define the score: the active mode's criteria YAML plus the reflection-agent's {{WORKSPACE}}/.claude/agents/reflection-agent.md (concatenate the files in a fixed order, then hash). Record it in state.json under state.autoresearch.<mode>.rubric_sha256 and note it in the stop summary. The whole point of the keep/revert loop is comparing scores across iterations against a FIXED harness — if the criteria YAML or the agent prompt change mid-pass, every score before and after the change is incomparable and the loop's keep decisions are meaningless. Therefore verify-run.sh recomputes the same hash from the on-disk inputs and FAILS LOUDLY on any mismatch with the recorded rubric_sha256, rather than silently trusting a tampered run. (Spec-level: this runner is prose-driven — the runner computes/records the hash in prose and verify-run.sh owns the recompute-and-compare gate.)

Experimentation loop

Track an in-memory counter N (next iter from results.tsv) and consecutive_reverts. For each iteration:

1. Read state. Re-read the on-disk mutable files. Read results.tsv; set best_score = max(composite where status='kept'), fallback baseline. Skim recent reverted ideas — do not repeat them.

2. Pick ONE focused edit. A single, small, reviewable, single-purpose change permitted by the active per-pass file's constraints. One idea per iteration — never "while I'm here, also fix X" (multi-variable edits give uninterpretable scores).

3. Snapshot PRE-edit. Before touching anything, copy every mutable file you are about to change to a snapshot, so it is a clean rollback target:

   SLUG="<short-kebab-of-the-idea>"; TS="$(date -u +%Y%m%dT%H%M%SZ)"
   cp "$F" ".aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/$(printf '%03d' $N)-pre-$SLUG-<safe-F>-$TS"
   

4. Apply the edit. Edit only the mutable source. Some per-pass files allow a bounded set of internal refine rounds before official scoring — follow the file; default is one apply.

5. Score via the pass's harness. First, re-verify the harness fingerprint (mid-pass tamper guard): recompute the rubric SHA-256 (criteria YAML + reflection-agent AGENT-PROMPT.md, same recipe as setup step 7) and compare it to the recorded state.autoresearch.<mode>.rubric_sha256; on mismatch, HALT the pass loudly with status rubric-tampered — every score across the change is incomparable — instead of waiting for the out-of-band verify-run.sh. Then run the per-pass harness exactly as that file specifies (e.g. design: reflection-agent only; function: {{TEST_COMMAND}} + coverage as Tier-1 gates × reflection quality; acceptance: validators + reflection). Compute the composite per the file's formula. This is the official iteration score.

   reflection-agent harness: spawn the subagent named reflection-agent and load its prompt from {{WORKSPACE}}/.claude/agents/reflection-agent.md. NEVER hardcode a user-level path (e.g. ~/.claude/skills/reflection/...) — it does not exist.

6. Keep or revert (STRICT improvement).

   • If composite > best_score (strictly greater): status = kept; set best_score = composite; leave the edited files on disk; reset consecutive_reverts = 0.
   • Else: status = reverted; restore each touched file from its step-3 snapshot (cp -p <snapshot> "$F" — preserve mtime so a restored file never reads newer than state.json); best_score unchanged; consecutive_reverts += 1.
   • If scoring crashed / harness errored (e.g. could not run tests): status = crash; restore from snapshot (cp -p); consecutive_reverts += 1.
   • Restore robustness (every restore cp -p): check the exit status of EVERY restore cp -p. If ANY restore fails, do NOT advance on a mixed/partial tree: set the distinct loud status restore-failed, append the row, and HALT the pass immediately (then jump to the stop sequence so state.json is still written last).
   • Function mode only: a test going from green to red is an automatic revert regardless of the quality sub-score (the per-pass composite already zeroes out via the Tier-1 hard gate); log it as status = reverted-redtest, restore from snapshot, consecutive_reverts += 1.

   The consecutive_reverts counter increments on reverted, reverted-redtest, and crash alike (all three count toward the pass's max_consecutive_reverts); only a kept iteration resets it to 0.

7. Append a results.tsv row:

   <N>	<composite>	<composite - best_score_before_this_iter>	<dim scores...>	<kept|reverted|crash>	<snapshot>	<UTC-iso>
   

8. Print ONE chat line (see Output), increment N, check stop conditions, loop.

results.tsv columns (authoritative)

This SKILL.md is authoritative for the COMMON columns, in this exact order:

iter	composite	delta	<one column per rubric dimension>	status	snapshot	utc

• iter — iteration index; AR-1/AR-2 use an integer, AR-3 may use a string iter id (e.g. ar3-r<R>).
• composite — the pass's composite score.
• delta — composite - best_score_before_this_iter.
• <one column per rubric dimension> — the per-pass dimension columns, inserted between delta and status. Each per-pass program file defines its OWN dimension columns and MUST print its exact header, using the EXACT criterion ids as column names:
  • AR-1: structural_health · loudness_of_failures · naming_clarity · docs_traceability
  • AR-2: quality · pass_rate · coverage
  • AR-3: per program.ar-acceptance.md. Single-letter abbreviations (S/L/N/D) are allowed ONLY in the human one-line chat output — NEVER as TSV headers.
• status — kept | reverted | reverted-redtest | crash | restore-failed.
• snapshot — the pre-edit snapshot filename for this iter; AR-3 uses -.
• utc — ISO-8601 UTC.

Right-anchored positional parse rule (authoritative). The renderer/parser tolerates per-pass dimension column sets by reading the common columns positionally from BOTH ends: the FIRST column is always iter and the LAST three columns are always status snapshot utc, in that order, for EVERY pass — AR-1, AR-2, AND AR-3 alike. Everything between iter and status is that pass's own columns: AR-1 and AR-2 lead with composite delta then their rubric dimensions, while AR-3 has no composite/delta — its in-between columns are exactly its two dimensions pass_rate why_alignment. A parser MUST key dimension columns off the per-pass header row, NOT assume composite/delta at fixed positions 2–3. For this right-anchored rule to hold, a row must NOT carry any trailing free-text field after utc: AR-3 in particular does NOT append an inline one-line gap-summary field to its results.tsv row — every AR-3 pass ends, exactly like AR-2, with status then snapshot (-) then utc as its final three columns. The human one-line gap summary lives in chat/summary output ONLY, never in the TSV row. (program.ar-acceptance.md's row spec MUST agree with this: trailing status, snapshot = -, utc.)

---

Stop conditions (bounded — the loop ALWAYS terminates)

Stop the loop when ANY fires. The first three caps come from config.autoresearch.<pass>; all are enforced by the runner regardless of mode:

• Target met: the per-pass success condition holds (e.g. all KPI dimensions ≥ {{KPI_TARGET}}; for function the composite ≥ target with all Tier-1 gates green; for acceptance the per-pass DONE condition — pass_rate ≥ pass_rate_target, no blockers, why_alignment ≥ target).
• Plateau: consecutive_reverts reaches the pass's max_consecutive_reverts. This counter increments on reverted, reverted-redtest, and crash iterations; only a kept iteration resets it to 0.
• Iteration cap: N reaches the pass's max_iterations.
• Cost brake: the pipeline-wide cost_limits.max_per_run_usd is hit — stop immediately.
• Human interrupt: any new prompt / interrupt.
• AR-3 outer loop: acceptance mode additionally bounds its DEVELOP re-entry by max_acceptance_rounds; on the cap with open blockers it ESCALATES to the user rather than looping the pipeline. (Details in program.ar-acceptance.md.)

On stop, the runner MUST, in this order: (1) restore the editable set to the best-kept version (the highest-scoring kept snapshot, or the baseline if nothing was kept) — restore each file with cp -p <snapshot> "$F" (preserve mtime so a restored file never reads newer than state.json), and check the exit status of EVERY restore cp -p; if ANY restore fails, set the distinct loud status restore-failed and HALT (do not advance on a mixed tree) — but STILL proceed to write state.json last (step 3). Before the baseline fallback, test -r the target snapshot first and surface an explicit error if it is absent or unreadable, rather than cp-ing a missing path. (2) summarize: starting composite, final best_score, count of kept vs reverted vs crash, the top kept changes by delta, and which stop condition fired; (3) in pipeline mode, write the final scores to step_history and write .aid/pipeline/state.json UNCONDITIONALLY as the FINAL action — after the last source mutation of ANY kind (kept edit, revert/restore cp -p, or crash restore), so the Stop hook does not re-block. Because a plateau stop often fires right after a run of reverts (whose restore cp -p sets the source mtime to the snapshot's preserved mtime), the state.json write MUST be the last disk write of the pass — never conditioned on the last write being a kept edit. Verify the write symmetrically with the restore checks: check the exit status of the state.json write (prefer write-to-temp then atomic rename) and re-read it to confirm it parses and its mtime ≥ the newest source mtime. If the write fails, set the distinct loud status state-write-failed and surface it — do NOT rely on the downstream Stop gate, whose "re-run /pipeline" message would otherwise mask a genuinely failed write.

---

Safety brakes & hard constraints (non-negotiable — org + pipeline policy)

• NEVER auto-commit. NEVER run destructive git (no commit, reset, checkout/restore of tracked files, clean, force-push, branch deletion). git diff/ls-files/status for inspection only. Rollback is always via snapshot cp. The human commits after review; the runner only validates and leaves the existing "validated — your call" nudge.
• Touch only the mutable surface resolved in setup. Never edit the harness, tests (in function/acceptance), criteria YAMLs, .aid/*.json, config.json, .mcp.json, or any pipeline/skill file. Modifying the harness makes the score meaningless.
• Snapshots only under .aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/, only via the cp commands above.
• No new dependencies, no framework migration, no plaintext secrets. If a fix seems to need a new dep or a secret, stop and surface it — do not add it.
• No scope reduction to game the score (don't delete behavior/criteria/tests to raise a number).
• One idea per iteration. Strict-improvement keep. Equal scores revert.
• Respect {{CRITICALITY_PROFILE}}: ASK before edits touching {{SAFETY_CRITICAL_FLOWS}}.
• Bounded by construction: per-pass max_iterations + max_consecutive_reverts + the cost brake (and AR-3's max_acceptance_rounds) guarantee termination.

---

Output expectations (be quiet)

After each iteration, print exactly ONE line to chat:

iter <N> | <kept|reverted> | <composite> (Δ <delta>) | <one-line desc of the change>

Do not narrate reasoning or dump the reflection-agent's full breakdown. results.tsv is the source of truth; the user can ask for detail. Emit the stop summary only when the loop ends.