118 个 · 按相关度排序
Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.
中文简介正在准备中。当前显示英语简介。
Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.
中文简介正在准备中。当前显示英语简介。
Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.
中文简介正在准备中。当前显示英语简介。
Evaluate LLM applications using automated scores, human reviews, and model-based judging. Build repeatable checks to compare models or prompts, assess answer quality, and detect declines in performance before deployment.
中文简介正在准备中。当前显示英语简介。
Evaluate LLM application quality using automated scores, human reviews, and another language model as a judge. Compare models or prompts, establish baseline results, and detect quality drops through repeatable tests and statistical comparisons.
中文简介正在准备中。当前显示英语简介。
Design systems that use language models to judge AI responses. Covers scoring criteria, comparisons between two answers, bias reduction, confidence calibration, and checks against human judgments to make automated quality assessments more consistent and traceable.
中文简介正在准备中。当前显示英语简介。
Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model — change the metric and the optimizer changes behavior. Decompose by default; bool during compile, float during eval; calibrate against human; mitigate judge bias. Search keywords: LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric, RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.
中文简介正在准备中。当前显示原文说明。
Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.
中文简介正在准备中。当前显示原文说明。
Improve code, prompts, skills, or agent definitions through repeated changes and evaluation. Measure numerical targets such as latency or size, or use an LLM to judge pass-or-fail criteria, keeping changes that improve the result.
中文简介正在准备中。当前显示英语简介。
Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".
中文简介正在准备中。当前显示原文说明。
Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.
中文简介正在准备中。当前显示英语简介。
Design workflows that evaluate and refine AI outputs through self-review, separate generation and evaluation roles, tests, or weighted scoring criteria. Includes patterns for structured feedback, iteration limits, and stopping when quality stops improving.
中文简介正在准备中。当前显示英语简介。
Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.
中文简介正在准备中。当前显示英语简介。
Evaluate staged Git changes before committing using an LLM, a language model acting as a reviewer. Receive correctness, completeness, and safety scores, plus issues by severity, to decide whether to fix, commit, or stop.
中文简介正在准备中。当前显示英语简介。
Evaluate AI agents by preparing test cases, scoring responses and tool use, investigating failures, and comparing results after changes. Supports built-in checks and custom scoring rules, including evaluations performed by a language model.
中文简介正在准备中。当前显示英语简介。
Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass/fail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the evaluation lifecycle (create, update, enable/disable, delete).
中文简介正在准备中。当前显示原文说明。
Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.
中文简介正在准备中。当前显示原文说明。
Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.
中文简介正在准备中。当前显示原文说明。
依据按编程语言分类的资料，协助开发和维护Claude应用。涵盖官方SDK调用、工具集成、模型迁移、费用检查、提示词审查和评估流程。
Evaluate language models with over 100 benchmark tests covering knowledge, math, coding, safety, and image understanding. Configure runs on Docker, Slurm computing clusters, or cloud infrastructure, compare models, and export results.
中文简介正在准备中。当前显示英语简介。
Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.
中文简介正在准备中。当前显示英语简介。
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".
中文简介正在准备中。当前显示原文说明。
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
中文简介正在准备中。当前显示原文说明。
Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.
中文简介正在准备中。当前显示原文说明。