Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.
中文简介正在准备中。当前显示原文说明。
查看安装方法安装前可以查看交给 AI 代理的指令内容。
This plugin splits eval work into targeted skills. Your job here is small: find the row below that matches the user's situation, tell the user which skill you are loading and why, then load that skill and follow its workflow from start to finish instead of improvising your own version of it.
| Situation | Skill to load |
|---|---|
| Has traces, wants to find failure modes, no established taxonomy yet | error-discovery |
| Has an existing eval pipeline and wants to know if it can be trusted | eval-audit |
| Has a known failure mode that code can check (e.g., format, schema, regex, execution) | write-code-eval |
| Has a known failure mode and wants an LLM judge for it | write-judge-prompt |
| Has an LLM judge or evaluator and wants to check its quality | validate-evaluator |
| Has no traces to review yet | generate-synthetic-data, then error-discovery |
| Wants a custom annotation interface for some other labeling task | build-review-interface |
| Wants to evaluate a RAG pipeline | evaluate-rag |
Most requests that mention error analysis with traces in hand mean error-discovery. New users with an existing pipeline usually need eval-audit first. This file holds only routing. When in doubt about which row fits, ask the user instead of guessing. The workflow lives in the targeted skill.
还没有评价。欢迎分享你的使用感受。
无需登录即可发表。欢迎分享使用感受、提问，或说明在哪些环境下无法运行。
还没有评论。来写第一条吧。
Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.
中文简介正在准备中。当前显示原文说明。
Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.
中文简介正在准备中。当前显示原文说明。
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
中文简介正在准备中。当前显示原文说明。
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
中文简介正在准备中。当前显示原文说明。
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.
中文简介正在准备中。当前显示原文说明。
Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).
中文简介正在准备中。当前显示原文说明。