Skill Eval

Answer one named maintenance decision: retain, revise, remove, or insufficient evidence. Choose the measurement that can answer that decision, use the caller's accepted cases and resource envelope, make one scoped recommendation, and stop. A completed evaluation does not require a positive difference.

This is an optional specialist. The selected runner owns execution and bounds; native results own measurements; BD and Git retain their authority. Do not add a core skill, AO evaluation command, scheduler, dashboard, second tracker, or mandatory review merely to run an experiment.

Rules that decide the answer

• Count every attempt. Keep failed, crashed, interrupted, blocked, abandoned, missing and infrastructure-invalid attempts in the all-attempt accounting. A rerun adds an attempt; it never overwrites the one that failed. A run that shared its host with other work still counts for completion and tokens; keep its duration, labeled as measured on a shared host, rather than dropping it. Its timeout is not by itself a skill failure; when timeouts fall unevenly between arms, report them as a possible skill effect, not infrastructure.
• Vary one thing. Equalize instructions, tools, environment, model and effort across arms apart from the intended variable. If one arm's task prompt repeats the skill's direction, attribute the result to the combined instructions, not the skill alone. When you control launch order, alternate arms run by run, swap which goes first in each pair and rerun a dead run in its own slot; one whole arm and then the other turns load, rate limits and time of day into an apparent skill effect.
• Confirm the skill loaded. Before reading a zero or small delta as no benefit, check each treatment run for the skill actually being loaded or injected. A run where it never loaded measures routing, not content. When the act depends on a rule kept in a reference, also check that the run opened that file; a run that did not still counts; read its miss as the body's pointer to the reference failing, not the reference's rule, and report how many treatment runs opened the file. The behavioral probe injects only SKILL.md or a prelude, so it cannot answer this.
• Calibrate the grader. Before trusting scores, confirm the judge or discriminator passes a response that plainly meets each criterion and fails one that plainly does not. A weak judge can fail correct responses wholesale.
• Small samples are directional. Report uncertainty with every difference. A difference without it shows neither benefit nor equivalence, and a zero-crossing interval is not equivalence. For two unpaired success rates, give each arm a Wilson interval, the difference with its own interval and Fisher's exact test, not a normal approximation, which breaks at 0/n, n/n and a handful of runs. Beside an inconclusive result, state the runs per arm needed to detect the smallest effect that would change the decision, and return insufficient evidence; running them is a new measurement with its own fixed stop and the caller's authorization, never an extension of this one.
• Fix the stop before running. Do not add trials until the result turns positive, remove losing observations or relax acceptance.
• Know the noise before cutting. Before trimming or retuning skill text for a new model, measure how much identical arms differ on the same cases. A cut must clear a bar fixed before the run, above that spread. A static read of the text is not a behavioral result. Saving a result means keeping the completed run; rerunning to save it adds another sample and more spend.

Choose the question

Caller decision	Measurement	What it can establish
Does a natural request load this skill?	An authorized available routing probe that observes loading; optionally claude plugin eval with a with-only tool_used: Skill grader when Claude is explicitly authorized	Natural-request loading on the observed host/catalog; not whether loading helps
Does supplied skill text change a specific observable act?	Behavioral probe with the available authorized Codex adapter in scripts/probe-skill.sh	Behavior change under forced source injection; not routing, correct code or productivity
Does the installed plugin change graded answers end to end?	claude plugin eval against its no-plugin baseline, only when explicitly authorized and available	Routing and content together on the selected cases
Does this package or version improve engineering outcomes at acceptable cost?	Repository-selected controlled coding comparison, such as evals/skills-rpi	Endpoint outcomes and cost on selected tasks; independent completion only when required exact-subject evidence exists
Does a qualified memory update help later work?	Separate frozen-versus-updated memory transfer test	Narrow later-task reuse evidence with skill and runtime held fixed
What happened in ordinary runs?	Existing native accounting and acceptance evidence	Observational failures, repairs and cost; not causal skill benefit

Start from the caller's intended decision, not a mandatory quiz. For a behavioral question, name one observable action (a file written, tool used, criterion rejected); a belief such as “understands validation” needs translation into an action. For coding or memory questions, name unchanged task acceptance and the maintenance choice.

Prefer cases drawn from real use when authorized session history exists: a caller correcting, redoing or overriding the result right after the skill ran is the clearest sign of a miss. Label each case as drawn from use or written from the skill's own text; cases written from the text test what the author expected, not what users hit. With no usable history, say so rather than presenting text-derived cases as representative.

A skill whose result depends on the caller's answers, such as an interview, is not measured by its first turn; never run it where its questions reach a real person. Measure the dialogue as in dialogue skills.

Runners

Select the intersection of the caller's runtime authority, an available adapter and the named question. A runner's presence grants no runtime permission. If no route meets all three, return insufficient evidence with the missing route; do not automatically substitute another runtime or build a replacement runner. Forced SKILL.md injection, natural-request routing and installed-package coding comparisons answer different questions.

When Claude Code is explicitly authorized and its evaluator is available, claude plugin eval <plugin-path> --model <id> is an optional route. It runs the cases in the plugin's eval directory (evals/ by default) with the plugin and, by default (--ablation with-without), without it, scores each response with the case graders (LLM graders use --judge-model, default haiku) and reports the score delta. --runs sets repetitions per case, --max-cost-usd caps spend and --json writes per-run results. The model decides whether to load each skill, so the delta mixes routing with content. By default it also publishes its HTML report (prompts, responses and verdicts) to claude.ai and writes results under the plugin's eval directory: pass --no-publish, and point --output-dir, --json and --report at caller-selected storage. Confirm flags with claude plugin eval --help.

scripts/probe-skill.sh is the repository runner for small behavioral probes. Its live route uses scripts/lib/codex-exec.sh; select it only when Codex and the runner's required isolation support are available and authorized. It injects the exact SKILL.md bytes (or a declared prelude) into the treatment arm, grades with a deterministic discriminator and replays immutable fixtures. Loading is forced, so it measures the text's effect on one act, not routing or a complete installed package. Neither runner's result substitutes for the other. Probe forms, headroom classifications and legacy ledger rules are in behavioral probes.

scripts/probe-skill.sh, evals/ and the probe gates exist only in an AgentOps source checkout. Elsewhere, use the caller's authorized existing runner if it supports the question and record its identity and limits. With no such runner, report the measurement unsupported; do not install a platform or start another runtime to replace the missing route.

Procedure

1. Fix the decision and bounds. Name the subject package/version or qualified memory update, relevant cases, allowed runtime and existing aggregate time, trial and cost limits. Do not infer billing enforcement from token counters. Smoke runs, infrastructure retries, interrupted attempts and inner review consume the same declared envelope; a new configuration or context does not renew it. Do not launch live work without caller authorization and bounds.
2. Choose the smallest relevant measurement. Use behavioral probes for acts, coding tasks for engineering outcomes, and separate later sessions for memory. There is no universal two-effort requirement. Keep the deployed model and effort unless the caller's decision concerns effort. Retain easy regression and cost controls; do not weaken the producer to manufacture separation.
3. Freeze and calibrate. Fix task, acceptance, package, model/runtime, environment and grader identities before trials. Executable oracles must accept the intended solution and reject plausible incorrect/no-op solutions. Include genuinely correct and incomplete cases when evaluating judgment. Exposed incidents are development cases, never unseen holdouts by renaming. Broken or leaked cases invalidate affected comparisons; preserve their historical disposition when versioning a correction. If the existing capture/readout pipeline is new or changed, check parsing and scoring with a cheap known-input dry run. Consider an A/A control only when noise could change the decision, within the same authorized envelope; neither requires a fixed trial count or a new artifact. An A/A gap beyond chance is a harness, labeling or scoring defect to fix before any comparison, not noise.
4. Run within the selected consumer's bounds. Coding trials expose the actual selected package and required resources. A worktree or a prompt prohibition is not runtime isolation. Exclude operator home, production tracker, session history, sibling output and solutions; capture launched configuration and final artifacts outside the worker. Report an incompatible adapter as such; do not build a replacement platform to rescue a result.
5. Read all attempts. Use native runner results and existing accounting; collection must not require another model call or handwritten evaluation. Wrong identity, changed acceptance, contamination or ambiguous pairing cannot establish comparison proof even when a deterministic check passed.
6. Compare only supported facts. Pair by task and repetition; preserve repetitions within task clusters. Endpoint reward, worker done claim, in-workflow validator PASS and independent acceptance are different facts. Missing review, usage, billing, phase or feasibility evidence stays unknown. A worker following an instruction establishes adherence, not reduced rework or causal benefit. A passing case far from a failed boundary does not prove the boundary is repaired. Coding and memory comparisons follow coding and memory readout.
7. Recommend once and stop. First read a few treatment transcripts, not only graded outputs: turns spent on steps that do not move the task, and a helper several runs wrote for themselves, are revise targets a passing score hides. A repeated helper is a candidate bundled script under Skill Builder's occurrence rule. A concrete reproduced defect with clean controls can support a provisional narrow repair; general improvement needs held-out comparison. Do not automatically publish a lesson.

Raw trials and new proof go to caller-selected protected external non-Git storage; only public, sanitized fixtures cleared for that destination belong in Git (ADR-0016).

Output

Decision: retain | revise | remove | insufficient evidence; scope <skill, version, cases>
Question: <maintenance decision and the measurement chosen>
Setup: <runner, model, effort, grader; what differs between arms>
Attempts: <per arm: assigned, completed, crashed or infra, interrupted, reruns>
Outcomes: <paired by case and repetition; whether the skill loaded in each treatment run>
Uncertainty: <interval and method, or "directional, n=<count>"; runs per arm needed if inconclusive>
Cost: <measured time and cost per arm, or unknown>
Not proven: <confounds, missing coverage, what this measurement cannot show>

For behavioral authoring, also supply the existing probe package (probe.json, question.md, discriminator.sh, fixtures/, and a prelude only in injected-prelude mode) and its replay result. No new per-run worksheet is required.

Done when the requested measurement has reached its accepted stop, the relevant replay/oracle checks discriminate, missing coverage is explicit, and one recommendation answers the named maintenance decision. Insufficient evidence, an adverse result or an incompatible runtime can complete this evaluation; none counts as demonstrated skill benefit.

References

• Behavioral runner and conventions: scripts/probe-skill.sh, evals/skill-probes/README.md, seeding.
• Behavioral verdicts and non-verdict incidents: LEDGER.md, RUNBOOK.md.
• Existing coverage and headroom gates: check-skill-probe-coverage.sh, check-skill-probe-headroom.sh.
• Evidence and overclaim limits: ADR-0011, ADR-0016 and RPI traversal.