117건 · 관련도순
Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Evaluate LLM applications using automated scores, human reviews, and model-based judging. Build repeatable checks to compare models or prompts, assess answer quality, and detect declines in performance before deployment.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Evaluate LLM application quality using automated scores, human reviews, and another language model as a judge. Compare models or prompts, establish baseline results, and detect quality drops through repeatable tests and statistical comparisons.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Design systems that use language models to judge AI responses. Covers scoring criteria, comparisons between two answers, bias reduction, confidence calibration, and checks against human judgments to make automated quality assessments more consistent and traceable.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model — change the metric and the optimizer changes behavior. Decompose by default; bool during compile, float during eval; calibrate against human; mitigate judge bias. Search keywords: LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric, RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Improve code, prompts, skills, or agent definitions through repeated changes and evaluation. Measure numerical targets such as latency or size, or use an LLM to judge pass-or-fail criteria, keeping changes that improve the result.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Design workflows that evaluate and refine AI outputs through self-review, separate generation and evaluation roles, tests, or weighted scoring criteria. Includes patterns for structured feedback, iteration limits, and stopping when quality stops improving.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Evaluate staged Git changes before committing using an LLM, a language model acting as a reviewer. Receive correctness, completeness, and safety scores, plus issues by severity, to decide whether to fix, commit, or stop.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass/fail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the evaluation lifecycle (create, update, enable/disable, delete).
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Guides implementation and maintenance of Claude applications using language-specific references and official SDKs, the libraries used to call the API. Covers tool integration, streaming responses, model migration, cost reviews, prompt audits, and evaluation workflows.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Evaluate language models with over 100 benchmark tests covering knowledge, math, coding, safety, and image understanding. Configure runs on Docker, Slurm computing clusters, or cloud infrastructure, compare models, and export results.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.
한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.
Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.
한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.