Add or extend Paperclip Runner evaluations that test real sessions against a mock control service. Define observable pass conditions, validate cases and run selections, and retain source revisions, evidence, and costs for authorized live runs.
中文简介正在准备中。当前显示英语简介。
Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
安装前可以查看交给 AI 代理的指令内容。
Use this skill for the Runner Evals family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in add-product-e2e-eval.
Locate the Paperclip checkout using PAPERCLIP_ROOT when supplied, or
git rev-parse --show-toplevel from a checkout. From outside Git, inspect the
workspace roots (for example ~/paperclipai/paperclip) and verify that the
selected root contains packages/paperclip-runner and tests/runner-e2e.
Locate paperclip-evals using PAPERCLIP_EVALS_ROOT or a discovered sibling;
a worktree's parent directory need not contain that repository. Read
doc/evals.md and packages/paperclip-runner/docs/runner-protocol-live-evals.md,
then inspect the nearest existing case, roster, schema, and report test before
editing. Definitions and authored cases belong in the sibling
paperclip-evals/evals/paperclip-runner; Runner integration, aggregation,
viewer, and publication behavior belongs in packages/paperclip-runner.
Keep the control-plane boundary explicit in names and documentation.
The sibling eval README is the concrete map: cases live under cases/,
company fixtures under fixtures/, runtime/model settings under configs/,
selections under rosters/, and maintained campaign membership under
campaigns/live-direct-full.json. Update inventory/coverage mappings when the
program requires them; a new file alone does not join the maintained campaign.
From the Evals repository root, adapt these provider-free checks to the case
and roster you changed. Run the reliability-plan validator only when that
separate plan changes:
python3 evals/paperclip-runner/tools/eval_program.py validate \
--case evals/paperclip-runner/cases/get-task-context.json \
--config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
--roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
--campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
--plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
Use nearby positive and negative grader cases/fixtures to calibrate the new assertion, including malformed or missing evidence where the grader must fail closed. Preserve the existing machine disposition and grade; product, model/provider, grading, and infrastructure labels are analytical annotations, not instructions to rewrite classifiers.
Author one bounded case with a deterministic semantic assertion and an inspectable result. Declare its expected operation, state effect, provider lane/profile, timeout and retry policy, and any required evidence. Do not grade hidden reasoning, infer success from a provider terminal message, or invent conversation/tool evidence. Public output follows the reviewed projection: sanitized fixture conversation and allowlisted tool outcomes may be published; raw trusted artifacts, credentials, secrets, private references, and hidden reasoning may not.
Validate without provider calls first using the commands above and the relevant
report/render validation documented in the Runner docs. When a live run is
authorized, pin the Paperclip commit and exact 40-character
paperclip-evals commit, select the smallest useful roster, and retain the
complete provenance and cost record.
Update authoritative detailed docs when the contract or command changes, then
link from doc/evals.md rather than duplicating the Runner runbook. Keep public
reports immutable and use the reviewed projection; sanitized fixture
conversation and allowlisted tool outcomes may be public, while credentials,
secrets, private references, raw trusted payloads, and hidden reasoning must not
be exposed.
还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Add or extend Paperclip end-to-end evaluations that exercise real browser workflows across the server, database, and runner. Define expected outcomes, check independent evidence, and connect results to the existing reporting pipeline for local or Daytona execution.
中文简介正在准备中。当前显示英语简介。
Use a real browser to verify web pages and apps through short, supervised interactions. Capture screenshots, inspect errors and network requests, and document reproducible steps for UI bugs or expected behavior checks.
中文简介正在准备中。当前显示英语简介。
Read email linked to assigned Paperclip tasks, send new messages or replies through an assigned AgentMail inbox, and check delivery outcomes. Follow task permissions and preserve retry identifiers to avoid duplicate sends.
中文简介正在准备中。当前显示英语简介。
Inspect proposed changes in GitHub, GitLab, or Perforce for review feedback, failed automated checks, and incomplete descriptions. It separates actionable findings from informational or resolved comments, then helps fix issues and close addressed review threads when requested.
中文简介正在准备中。当前显示英语简介。
Create an AI agent company or team package from an idea or an existing repository. Define roles, reporting lines, skills, and work handoffs, then generate structured files that follow the Agent Companies specification.
中文简介正在准备中。当前显示英语简介。
Records an agent’s brief, unfiltered complaints about tools, permissions, or working conditions in Paperclip. It preserves the reaction without requiring a diagnosis or solution, then lets the agent continue its main task.
中文简介正在准备中。当前显示英语简介。