Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
설치 방법 보기설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.
Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.
write-judge-prompt.| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse the output and check required fields |
| Missing or forbidden text | Match a string or pattern |
| Citation not in retrieved documents | Compare cited IDs with retrieved IDs |
| Bad tool call | Check arguments against the tool schema or run the call in a safe test environment |
| Wrong value | Compare the output with a reference value |
Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.
아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.
로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.
아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.
Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.