Select, inspect, validate, and report Paperclip evaluations for Runner protocol behavior or complete product workflows. Interpret results with evidence, source revisions, costs, and failure categories, and refresh reports from retained evidence without new provider calls.
Choose, inspect, validate, and report Paperclip Runner or Product E2E evaluations while preserving evidence, provenance, cost, and failure classification.
Run score Not tested
See how to installThis skill selects Paperclip evaluations for a task, then inspects definitions, validates them, interprets results, and prepares reports. Runner Evals examine Runner behavior against a mock control service. Product E2E Evals examine whole-product workflows across the browser, server, database, and other components. Results retain source revisions, selected runs, models, environments, retries, timing, costs, and grader versions. Partial attempts are preserved too.
It helps when you are unsure which evaluation to use or need to understand failures from the evidence. It distinguishes product problems, model or provider behavior, grading or evidence issues, and infrastructure problems. It can also refresh report presentation from saved evidence without calling providers again. Evaluation catalogs can be inspected without credentials.
A Paperclip checkout is required. Runner definitions may also require the private paperclip-evals repository. Live or paid runs require explicit permission and the necessary credentials; Daytona also requires a pinned image. Public reports exclude confidential information and hidden reasoning.
This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.
AI checked the SKILL.md for operations that need care and for setup requirements. The score comes only from actually running the skill (hands-on test).
Not counted in the score. Rated by AI (OpenAI Codex) as of Oct 20, 2026. Treat it as a guide and read the original before installing.
Review the instructions the agent will receive before you install.
Use this skill when a request concerns Paperclip evaluation selection,
interpretation, evidence, history, or a live run. Read doc/evals.md in the
Paperclip repository first. It defines the two families and their boundaries.
Do not assume the skill's installed location is inside a checkout. Locate the
repo explicitly with git rev-parse --show-toplevel from the current directory,
or inspect likely workspace roots and select the checkout containing
package.json, tests/runner-e2e, and packages/paperclip-runner. Locate the
private sibling paperclip-evals only when a Runner Eval needs its definitions;
use an explicit PAPERCLIP_EVALS_ROOT or a discovered sibling checkout. Never
invent a relative path from this copied skill into the repository.
Choose Runner Evals for real runner/provider protocol behavior against the
mock control plane. Authoritative details are in
packages/paperclip-runner/docs/runner-protocol-live-evals.md and the sibling
paperclip-evals/evals/paperclip-runner definitions.
Choose Product E2E Evals for real browser/server/database/runner/provider
workflows, including local and Daytona environments. Read
tests/runner-e2e/README.md, then FIXTURES.md, SECURITY.md, or
EVERYDAY-WORKFLOWS.md as relevant. Everyday Workflows remain Product E2E
even when imported into Evalbook. “Headless” is a browser mode, not a family.
Start with read-only catalog inspection and credential-free validation. For
Product E2E use pnpm test:e2e:runner:typecheck,
pnpm test:e2e:runner:unit, and pnpm test:e2e:runner -- --list; run one
explicit cell only when the user has authorized a live/paid run and the needed
credentials and immutable Daytona image are configured. For Runner Evals use
the pinned eval revision and the documented workflow/CLI. Never use a partial
selector as evidence of full coverage.
Keep source revisions, definition/catalog fingerprints, model/profile, environment, selected cells, retries, timing, usage/cost coverage, and grader version attached to every interpretation. Preserve partial attempts and classify failures as product, model/provider behavior, grading/evidence, or infrastructure from the observed failure and supported cause. A usable completed behavior failure is not infrastructure; missing provider/profile, transport, startup, or evidence requires examining the evidence before choosing the cause.
Use the existing family generator and viewer. Public projections may contain
sanitized fixture conversation and allowlisted tool outcomes/evidence; follow
the family's projection and publisher checks. Do not expose raw trusted
artifacts, credentials, secrets, private data, provider session IDs, or hidden
reasoning. A refresh from retained evidence has zero provider calls and remains
the original measurement with a new presentation. Link the public histories
and hub from doc/evals.md when reporting results.
For adding a case or fixture, use the narrower add-runner-eval or add-product-e2e-eval skill.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Add or extend Paperclip end-to-end evaluations that exercise real browser workflows across the server, database, and runner. Define expected outcomes, check independent evidence, and connect results to the existing reporting pipeline for local or Daytona execution.
Add or extend Paperclip Runner evaluations that test real sessions against a mock control service. Define observable pass conditions, validate cases and run selections, and retain source revisions, evidence, and costs for authorized live runs.
Use a real browser to verify web pages and apps through short, supervised interactions. Capture screenshots, inspect errors and network requests, and document reproducible steps for UI bugs or expected behavior checks.
Read email linked to assigned Paperclip tasks, send new messages or replies through an assigned AgentMail inbox, and check delivery outcomes. Follow task permissions and preserve retry identifiers to avoid duplicate sends.
Inspect proposed changes in GitHub, GitLab, or Perforce for review feedback, failed automated checks, and incomplete descriptions. It separates actionable findings from informational or resolved comments, then helps fix issues and close addressed review threads when requested.
Create an AI agent company or team package from an idea or an existing repository. Define roles, reporting lines, skills, and work handoffs, then generate structured files that follow the Agent Companies specification.