'llm-as-judge' 검색 결과

Design workflows that evaluate and refine AI outputs through self-review, separate generation and evaluation roles, tests, or weighted scoring criteria. Includes patterns for structured feedback, iteration limits, and stopping when quality stops improving.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Adding self-review and revision loops to AI outputs
• Separating generation from evaluation
• Refining generated code using test results

Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Check answers for unsupported claims
• Score experiment results for correctness
• Continuously evaluate new production records

Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Build evaluation datasets from conversation logs
• Generate simulated user scenarios
• Choose scoring criteria for responses and tool use

Design systems that use language models to judge AI responses. Covers scoring criteria, comparisons between two answers, bias reduction, confidence calibration, and checks against human judgments to make automated quality assessments more consistent and traceable.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare responses from different models
• Run A/B tests for prompt changes
• Create shared scoring rubrics

Evaluate AI agents by preparing test cases, scoring responses and tool use, investigating failures, and comparing results after changes. Supports built-in checks and custom scoring rules, including evaluations performed by a language model.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Create test cases for an AI agent
• Check response quality and tool use
• Investigate failed evaluation cases

This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design criteria for using an LLM to judge AI responses through scoring or comparisons between two answers. Includes checks for order and length bias, plus validation against human judgments to assess evaluation reliability.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Build an automated evaluation pipeline for AI responses
• Compare responses after changing a prompt or model
• Define shared scoring criteria for reviewers

Evaluate LLM applications using automated scores, human reviews, and model-based judging. Build repeatable checks to compare models or prompts, assess answer quality, and detect declines in performance before deployment.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing models and prompts
• Checking quality after prompt changes
• Detecting performance declines before deployment

Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Turning observed failures into evaluation criteria
• Building code-based or language-model evaluators
• Evaluating retrieval and answer faithfulness in RAG

Improve code, prompts, skills, or agent definitions through repeated changes and evaluation. Measure numerical targets such as latency or size, or use an LLM to judge pass-or-fail criteria, keeping changes that improve the result.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Improving page load performance
• Reducing build output size
• Refining prompts against evaluation criteria

Validates Terminal User Interface (TUI) output using freeze for screenshot capture and LLM-as-judge for semantic validation. Supports both visual (PNG/SVG) and text-based validation modes.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Master context engineering for AI agent systems. Use when designing agent architectures, debugging context failures, optimizing token usage, implementing memory systems, building multi-agent coordination, evaluating agent performance, or developing LLM-powered pipelines. Covers context fundamentals, degradation patterns, optimization techniques (compaction, masking, caching), compression strategies, memory architectures, multi-agent patterns, LLM-as-Judge evaluation, tool design, and project development.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Implement a task step by step with automated LLM-as-Judge verification at the end of each phase

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate LLM systems using automated metrics, LLM-as-judge, and benchmarks. Use when testing prompt quality, validating RAG pipelines, measuring safety (hallucinations, bias), or comparing models for production deployment.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Builds an eval plan for an AI feature - error analysis on real traces, failure-mode ranking, code checks and binary LLM judges validated against human labels, CI tests, production monitoring, and a launch bar. Use when you ask how do we eval this, how do I test an AI feature, our LLM outputs are inconsistent, are the evals good enough, LLM-as-judge, hallucinations, or what accuracy is good enough to launch. Draws on 41 Lenny's Podcast guests incl. Hamel Husain, Shreya Shankar and Sander Schulhoff.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Decomposed, multi-criteria metric design for LLM pipelines. The metric IS the model — change the metric and the optimizer changes behavior. Decompose by default; bool during compile, float during eval; calibrate against human; mitigate judge bias. Search keywords: LLM-as-judge, llm as judge, eval metric, evaluation score, scoring function, rubric, RAGAS, G-Eval, judge bias, verbosity bias, how to evaluate LLM output.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate WooAIAssistant against a structured scenario suite with hard invariants + LLM-as-judge rubric scoring. Runs live against the demo store + gpt-5.1 via the woo-mobile-ai backend wrapper, writes a JSONL run record, compares against stored baselines, and surfaces regressions. Always delegated to a subagent so the main context only sees the markdown report.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when designing a study that benchmarks AI systems against a human-expert panel, before data collection. Plans the arms, decoupled rubrics with anchors, planted calibration probes, reviewer panel, inter-rater reliability targets and LLM-as-judge versus human adjudication.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.