Compare repeated agent evaluations before and after a change to distinguish performance drops from random variation. Scripts check tool calls, identify inconsistent cases, and provide a CI gate that blocks merging when specified regression conditions are met.
中文简介正在准备中。当前显示英语简介。
Use when an agent, prompt, model or tool change needs proof it did not make the agent worse — setting up a regression eval suite, deciding whether a pass-rate drop is real or noise, grading tool-call trajectories, finding flaky eval cases, or wiring an eval gate into CI that blocks the merge. Runs repeated trials, reports pass@k and pass^k, compares candidate to baseline with paired statistics, and refuses to pass a dropped case, a critical-case regression, or a forbidden tool call.
实测评分 未实测
查看安装方法安装前可以查看交给 AI 代理的指令内容。
Portability: 3 stdlib scripts, no APIs, no LLM calls. You run the agent; these grade, count and gate.
"Pass rate went from 82% to 79%" is not a finding until you know whether three points is outside the noise. Agents are non-deterministic, so one trial per case cannot tell a regression from a coin flip, and a final answer can be right while the path to it called a tool it never should have.
# 0 - Run every case k times (k >= 3) on baseline and candidate, from a clean
# environment per trial. Write one JSONL row per trial:
# {"case_id": "refund-lookup", "trial": 1, "passed": true, "critical": true,
# "cost_usd": 0.012, "trajectory": [{"tool": "get_order", "args": {...}}]}
# 1 - Grade the path where the path matters (safety, required lookups).
python scripts/trajectory_checker.py --spec cases.json --results candidate.jsonl
# 2 - Separate flaky cases from stable ones before trusting any comparison.
python scripts/flake_detector.py --results candidate.jsonl --k 3 --quarantine-out q.json
# 3 - Gate: paired per-case comparison against baseline. Exit 4 blocks the merge.
python scripts/regression_gate.py --baseline baseline.jsonl --candidate candidate.jsonl --quarantine q.json
Step 1 uses match modes strict | unordered | subset | superset. Default to
superset (required steps present, harmless exploration allowed); grade the
outcome, not the path, unless order is the requirement. forbidden tools block
in every mode.
Step 2 reports pass@k (succeeds at least once in k) and pass^k (succeeds all k times). A user meets every trial, so pass^k is the reliability number. Cases marked critical are never quarantine candidates.
Step 3 averages trials within each case first (trials of one case are correlated), then tests paired per-case differences with a t-based 95% CI and a one-sided exact McNemar test. When the drop is real but the sample is too small to show it, the gate says how many cases it would take.
It warns, and a human decides, on an underpowered drop above --tolerance or a
cost rise above --max-cost-increase.
| Script | Role | Exit codes |
|---|---|---|
scripts/trajectory_checker.py | Grades tool-call paths: match modes, args modes, forbidden tools, call ceilings | 0 · 2 deviations · 3 bad input · 4 forbidden tool |
scripts/flake_detector.py | STABLE_PASS / STABLE_FAIL / FLAKY per case, Wilson CI, pass@k, pass^k, quarantine stubs | 0 · 2 flaky · 3 · 4 too few trials |
scripts/regression_gate.py | Paired CI, exact McNemar, critical cases, dropped cases, quarantine validity, cost | 0 PASS · 2 WARN · 3 · 4 BLOCK |
All support --sample and --output json.
references/eval_statistics.md — pass@k vs pass^k, per-case averaging, paired CI, McNemar, power (7 sources)references/trajectory_grading.md — outcome vs path, match modes, grader types (6 sources)references/flaky_cases_and_quarantine.md — flake causes, quarantine with expiry, clean trials (6 sources)assets/eval_case_spec.example.json — trajectory spec for step 1assets/sample_baseline.jsonl, assets/sample_candidate.jsonl — results formatassets/ci_eval_gate.example.yml — CI job that runs the gate and fails on exit 4还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Audit web applications for accessibility issues against WCAG 2.2 Level A and AA, the web accessibility standards. Get framework-specific code fixes, contrast checks, verification steps, and reports, with options for automated checks in development pipelines.
中文简介正在准备中。当前显示英语简介。
Plan A/B tests that compare page variations, from headlines to pricing layouts. Define a hypothesis, choose success metrics, calculate sample size and duration, and prepare launch checks and results reports to support decisions.
中文简介正在准备中。当前显示英语简介。
Create paid ad headlines and body copy tailored to each platform's character limits. Use existing performance data to develop variations, explore new messaging angles, and check copy for length, quality, and potential rejection risks.
中文简介正在准备中。当前显示英语简介。
Review code changes from three critical perspectives: production failures, maintainability, and security. Get a consolidated report that ranks findings by severity and provides merge guidance, helping examine assumptions that an ordinary self-review may overlook.
中文简介正在准备中。当前显示英语简介。
Audit and improve content intended for citation in AI answers. Review evidence, author information, and structure, generate revised content and structured metadata, and keep a local record of observed citations across AI services.
中文简介正在准备中。当前显示英语简介。
Create signed records for consequential agent actions such as deployments, deletions, and payments. Decide which actions need evidence, record the agent, action, target, and policy, then verify the resulting certificate offline to detect later changes.
中文简介正在准备中。当前显示英语简介。