'eval' 검색 결과

Explains how PluginEval combines structural checks and experimental model assessments into quality scores. Helps interpret low scores, identify improvement areas, set marketplace thresholds, and explain what evaluation badges actually represent.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Investigating why a skill received a low score
• Improving descriptions that define when a skill should run
• Setting score thresholds for marketplace listings

Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Build evaluation datasets from conversation logs
• Generate simulated user scenarios
• Choose scoring criteria for responses and tool use

Build evaluation suites for AI features with the Axiom AI SDK. Generate test files, scoring functions, test cases, and configurable parameters from descriptions and existing code to compare changes and monitor output quality in production.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Add regression evaluations for an AI feature
• Score classification, retrieval, and structured outputs
• Compare model and parameter configurations

Instrument Python or TypeScript agents to record runs for Failproof AI, verify local event files, and troubleshoot missing sessions. The skill also guides building evaluator workers that score completed runs on your own infrastructure.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Planning where an agent should record events
• Adding run logging to an existing agent
• Troubleshooting missing events or sessions

Diagnose slow MSBuild startup by measuring project evaluation, the work done before compilation. Inspect logs for costly file searches, imported configuration, and repeated evaluations, then identify targeted changes supported by timing evidence.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Investigating delays before compilation starts
• Checking file searches across large directories
• Analyzing deep import chains

Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Guide for diagnosing and improving MSBuild project evaluation performance. Only activate in MSBuild/.NET build context. USE FOR: builds slow before any compilation starts, high evaluation time in binlog analysis, expensive glob patterns walking large directories (node_modules, .git, bin/obj), deep import chains (>20 levels), preprocessed output >10K lines indicating heavy evaluation, property functions with file I/O ($([System.IO.File]::ReadAllText(...))), multiple evaluations per project. Covers the 5 MSBuild evaluation phases, glob optimization via DefaultItemExcludes, import chain analysis with /pp preprocessing. DO NOT USE FOR: compilation-time slowness (use build-perf-diagnostics), incremental build issues (use incremental-build), non-MSBuild build systems. INVOKES: binlog MCP server tools (evaluations, evaluation_global_properties, evaluation_properties, imports, properties); falls back to dotnet msbuild -pp:full.xml for preprocessing, /clp:PerformanceSummary.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design structured evaluation rubrics for scoring LLM and agent outputs — defining quality dimensions, scoring scales, hard gates, score descriptions, and edge cases. Load when the user asks to create an eval rubric, define evaluation criteria, design scoring dimensions, write an eval spec, or says "what should I evaluate", "design a rubric", "create eval criteria", "define quality dimensions", "evaluation rubric for", "how do I measure quality of". Sub-skill of eval-output orchestrator.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Build an end-to-end evaluation pipeline for Python AI applications using pixie-qa and real language-model calls. Define criteria, prepare realistic test data, score application runs, and analyze the evidence to produce a prioritized improvement plan.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Setting up quality checks for Python AI applications
• Creating realistic scenarios for known failure modes
• Scoring full application runs with real language-model calls

Design RAG systems that retrieve relevant documents to support LLM answers. The skill guides document splitting, meaning-based and keyword search, result reranking, and separate retrieval evaluation to improve the context passed to the model.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Designing document retrieval for LLM answers
• Reviewing document splitting strategies
• Combining keyword and meaning-based search

Design a RAG pipeline that retrieves documents to support AI answers. Analyze document splitting, select embedding models and vector databases, and evaluate retrieval quality using test queries and known relevant documents.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Designing a RAG system for internal documents
• Choosing how to split documents for search
• Comparing embedding models and vector databases

Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing two prompts on the same test cases
• Checking quality after agent configuration changes
• Turning real failures into repeatable evaluation cases

Evaluate a driving agent on CARLA Leaderboard routes after checking the required setup. Select routes and repetitions, save driving scores and infractions, record runs, and resume interrupted evaluations from a saved checkpoint.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Smoke-test a Leaderboard setup on one route
• Evaluate an agent across validation routes
• Resume an evaluation after a crash

Evaluate NVIDIA Cosmos Policy for robot manipulation in LIBERO and RoboCasa simulations. Follow GPU environment setup, configure rendering without a display, run small checks before larger benchmarks, inspect success rates, and profile inference latency.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Running a small LIBERO evaluation
• Evaluating kitchen tasks in RoboCasa
• Setting up simulation on a GPU cluster without a display

Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when evaluating a newly adapted model's accuracy/eval score. Runs evalscope eval (or the test/srt/run_eval.py fallback) against an OpenAI-compatible endpoint on datasets like gsm8k / aime, producing per-dataset reports and a score. Accuracy only — throughput/latency benchmarks belong to model-speed-benchmark. Triggers on accuracy, eval, evalscope, gsm8k, aime, run_eval — not throughput.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Suede AI eval design and coverage audit: AI-SPEC, failure-mode rubric with severity scoring, concrete pass/fail eval cases, coverage and infrastructure scores, and mechanical acceptance gates. Use when a change ships LLM, RAG, agent, classifier, prompt, or generated-media behavior, or when asked to write evals for an AI feature, design test cases for a model surface, audit existing eval coverage, or judge whether AI behavior is safe to ship. No AI-SPEC means no eval plan, and no eval plan holds the recommended ship verdict. NOT FOR: reviewing or grading the implementation code behind the AI surface (use suede-code); wiring a passing suite into CI as a required check (use suede-ci-gate); UAT of the built feature beyond the eval suite (a private Suede Labs companion, not in this pack).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

유료 평가 전에 합격 기준, 비교 조건, 교란 항목과 중단 규칙을 정합니다. 무료 예행연습과 세 항목의 유료 시험 평가를 거쳐 지출표를 후속 승인에 활용합니다.

• 지출 전에 평가 기준 정하기
• 비용 없이 예행연습 진행하기
• 세 항목의 유료 시험으로 지출 확인하기

Paperclip Runner의 동작 평가에 판정 조건, 실행 대상, 보고서 검증 데이터를 추가하고 증거와 실행 이력을 남겨 검증합니다.

• Runner 동작 평가를 추가합니다
• 정상 및 잘못된 증거로 판정을 검증합니다
• 실행 대상 목록과 지속 평가를 갱신합니다

Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Turning observed failures into evaluation criteria
• Building code-based or language-model evaluators
• Evaluating retrieval and answer faithfulness in RAG

Create and analyze AI model evaluation experiments in Arize using the ax CLI, a command-line tool. Collect real model outputs, export results, compare evaluation scores, and identify examples where a new model performs worse.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing models on the same evaluation dataset
• Finding examples that perform worse after a model change
• Exporting experiment results for further analysis

Design workflows that evaluate and refine AI outputs through self-review, separate generation and evaluation roles, tests, or weighted scoring criteria. Includes patterns for structured feedback, iteration limits, and stopping when quality stops improving.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Adding self-review and revision loops to AI outputs
• Separating generation from evaluation
• Refining generated code using test results