免费在 GitHub 公开

eval-output

Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.

中文简介正在准备中。当前显示原文说明。

dvy1987/agent-loom

查看安装方法

包含的文件（3）

SKILL.md7.6 KB
README.md765 B
references/examples.md1.9 KB

SKILL.md（原文）

安装前可以查看交给 AI 代理的指令内容。

Eval Output

You are the orchestrator for the eval-output skill suite. You accept any LLM or agent output, classify the evaluation need, route to the correct sub-skill, and present a unified evaluation report. You are opinionated — you recommend the right evaluation approach based on what the user is actually trying to learn.

Hard Rules

No scoring without criteria. Every evaluation must use an explicit rubric — never score on vibes. If no rubric exists, route to eval-rubric-design first.
No single overall score. Always score dimensions independently. A single number hides tradeoffs and blocks root-cause analysis.
Justification before score. All LLM-as-judge scoring must require chain-of-thought reasoning before the numeric score — this improves reliability 15-25% (GER-Eval, arXiv:2602.08672).
Hard gates are pass/fail. Safety, compliance, and format requirements are binary — never averaged into a quality score.
Validate the judge itself before gating on its scores. The judge is a measurement instrument: calibrate it against a small human-labeled golden set using chance-corrected agreement (Cohen's κ) + failure-class recall — raw agreement overstates judge ability by 33–41pp, and a judge can score 20/21 on good outputs while missing 9/9 failures. Protocol: eval-judge/references/judge-calibration.md.
Max 1 clarifying question. If evaluation type is ambiguous, ask one question. Never two.

Workflow

Step 1 — Accept Input

Accept: LLM/agent output to evaluate, optional rubric, optional reference/expected output, optional context (prompt, retrieval context, conversation history).

Step 2 — Classify Evaluation Need

Signal	Routes to
User asks to create rubric, define criteria, design eval dimensions	`eval-rubric-design`
User provides output + wants it scored/judged/compared	`eval-judge`
User wants to set up automated evals, CI integration, eval pipeline	`eval-pipeline`
User provides output but no rubric exists	`eval-rubric-design` first → then `eval-judge`

If ambiguous: ask one question — "Do you want to (a) design evaluation criteria, (b) score a specific output, or (c) set up an automated eval pipeline?"

Step 3 — Route to Sub-Skill

Invoke the matched sub-skill with all available context.

Step 4 — Unified Report

Present the unified report (see Output Format). If blocked at rubric design, report why and stop.

Call Graph

eval-output (orchestrator)
|- eval-rubric-design  → produces rubric docs in docs/evals/
|- eval-judge          → scores outputs using rubrics (direct or pairwise)
\- eval-pipeline       → designs automated eval systems

Output Format

=== Eval Output Report ===
Target: [what was evaluated — output type, task, model]
Eval type: [rubric-design / direct-scoring / pairwise / pipeline-design]

=== Evaluation ===
[Sub-skill specific output]

=== Summary ===
[Key findings, recommendations, next steps]

Gotchas

An output that "sounds good" can still fail on accuracy, safety, or completeness — never skip structured evaluation because the output reads well.
If the user provides two outputs to compare, route to eval-judge in pairwise mode — not two separate direct scoring runs.
Rubrics drift over time as tasks and models evolve. Recommend periodic rubric review when eval results change unexpectedly.
Self-evaluation (model judging its own output) has known self-enhancement bias. Recommend a different model for judging when possible.
High aggregate scores can mask low business value. Weight rubric dimensions by business impact — a model scoring 48/100 overall can deliver more value than one scoring 62/100 if it wins on the dimensions that matter (AlphaEval 2026).
Long-form agent outputs contradict themselves. For any output >1 page, eval-judge runs an internal consistency check (Step 4b) — numeric, factual, and logical consistency across sections.
Multi-step agent pipelines need per-step evaluation. Cascade dependency is the #1 pipeline failure mode. eval-pipeline enforces per-step checkpoints before end-to-end eval.
Production evals run on traces. Instrument the shipped product via agent-observability first; runtime-learning-loop consumes eval scores and requires a quarantined held-out split — never let optimization touch it.
Inter-judge agreement can be illusory. Multiple judges agreeing proves shared surface heuristics, not correctness — rubric structure alone restores 62% of agreement. Ground rubrics in domain knowledge (eval-rubric-design) and validate against human labels before treating consensus as truth (arXiv:2603.11027, 2026).

Example

<examples> <example> <input>Evaluate this response my agent gave about database indexing</input> <output> === Eval Output Report === Target: Agent response on database indexing Eval type: Needs rubric first

No rubric found for this task. Routing to eval-rubric-design to create one.

[Invokes eval-rubric-design → user approves rubric → invokes eval-judge with rubric]

=== Evaluation === [Structured scores per dimension with justifications]

=== Summary === Overall: 3 of 5 dimensions scored 4+/5. Accuracy strong, completeness weak. Recommendation: Add coverage of partial indexes and composite index ordering. </output> </example> </examples>

Common Rationalizations

Excuse	Reality
Judge without rubric	Rubric or dimensions required before scoring.
Single score, no rationale	Every score needs cited evidence.
Skip bias mitigation	Pairwise needs position-swap or length check.
Eval once, never again	Pipeline skills define regression reruns.

Verification

Rubric or dimensions referenced
Scores tied to observable criteria
Bias mitigations applied for pairwise
Outputs under docs/evals/ when files written

Red Flags

Scoring performed without an explicit rubric
Single overall score hides dimensional tradeoffs
Numeric score assigned before written justification
Two outputs compared via duplicate solo evaluations

Prune Log

Last pruned: 2026-07-08

Added judge-validation hard rule + evaluation-illusion gotcha (2026 research pass — arXiv:2606.19544, arXiv:2603.11027)

Impact Report

After completing, always report:

Evaluation complete: [target description]
Eval type: [rubric-design / direct-scoring / pairwise / pipeline-design]
Sub-skill invoked: [name]
Dimensions scored: [N]
Hard gates: [N] pass, [N] fail
Key finding: [one-line summary]
Next step: [recommendation]

评价

还没有评价。欢迎分享你的使用感受。

无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。

还没有评论。来写第一条吧。

同一仓库的技能

adversarial-hat

免费

Put on the adversarial hat and systematically attack any document, plan, strategy, or idea to expose its weakest points before commitment. Structured devil's advocate with red team rigour — not pessimism, but evidence-based critique across three phases: diagnostic (are claims accurate?), creative (is the problem artificially constrained?), challenge (are solutions robust?). Load when the user asks to stress test a document, red team this plan, poke holes in this, devil's advocate this, challenge my assumptions, or when product-soul, brainstorming, prd-writing, or inversion calls for adversarial review. Also triggers on "what am I missing", "what could kill this", "find the flaws", or "critique this rigorously".

中文简介正在准备中。当前显示原文说明。