'eval' 검색 결과

**WORKFLOW SKILL** - Evaluate AI agent skills using structured benchmarks with YAML specs, fixture isolation, and pluggable validators. USE FOR: run waza, waza help, run eval, run benchmark, evaluate skill, test agent, generate eval suite, init eval, compare results, score agent, agent evaluation, skill testing, cross-model comparison. DO NOT USE FOR: improving skill frontmatter (use waza dev), creating new skills from scratch (use skill-creator), token counting or budget checks (use waza tokens). INVOKES: Copilot SDK executor, mock engine, code/regex validators. FOR SINGLE OPERATIONS: use waza run directly for a single benchmark.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Use when the user has run multiple evaluation skills and wants a comprehensive analysis — maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Guide for diagnosing and improving MSBuild project evaluation performance. USE FOR: builds slow before any compilation starts, high evaluation time in binlog analysis, expensive glob patterns walking large directories (node_modules, .git, bin/obj), deep import chains (>20 levels), preprocessed output >10K lines indicating heavy evaluation, property functions with file I/O ($([System.IO.File]::ReadAllText(...))), multiple evaluations per project. Covers the 5 MSBuild evaluation phases, glob optimization via DefaultItemExcludes, import chain analysis with /pp preprocessing. DO NOT USE FOR: compilation-time slowness (use build-perf-diagnostics), incremental build issues (use incremental-build), non-MSBuild build systems.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable gh-aw workflow packages in the dotnet/skills repository. Use when creating skill or workflow-package tests, writing evaluation stimuli, defining graders and rubrics, sizing an eval for statistical power, or setting up test fixture files. Handles the Vally eval.yaml schema, fixture organization, and overfitting avoidance. Do not use for running or debugging existing evals (use improve-skill-quality) nor for skills authoring (use create-skill).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Braintrust is a hosted platform for evaluating and tracing LLM applications: it runs a task over a dataset, scores the outputs, stores each run as an experiment and logs production traces. Use when a user asks to write or run a Braintrust eval, score LLM outputs with autoevals, compare prompt or model versions as experiments, run evals in CI and fail on regressions, trace OpenAI or Anthropic calls with Braintrust, manage evaluation datasets, or use the bt CLI.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Run AI agent and LLM evaluations in CI/CD pipelines — automated quality gates that fail the build when AI output quality drops. Use when someone asks to "test my AI agent", "add evals to CI", "catch prompt regressions", "compare models", "evaluate LLM output quality", "set up AI quality gates", or "benchmark my agent before deploying". Covers eval frameworks (Cobalt, Promptfoo, Braintrust), LLM-as-judge scoring, threshold-based assertions, and GitHub Actions integration.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Audit whether a paper's EVALUATION DESIGN actually measures what it claims and whether its reporting is complete — the validity layer family D (experiment-forensics) cannot reach. Three patterns: train/test leakage means the reported score may not measure generalization (HP-EVAL-LEAKAGE — adopts the Kapoor & Narayanan 8-type / 3-category leakage taxonomy; the illegitimate-proxy / sampling-bias / pretraining-contamination subtypes hand off as needs_external_check, naming but NEVER running Oren-2023 exchangeability / Shi-2023 Min-K% / Golchin-2023 Time-Travel / BIG-bench canary); a load-bearing LLM judge is conflicted (same model/family as a compared system) or unvalidated (no human-agreement, no bias control) (HP-JUDGE-VALIDITY); a declared condition/metric is dropped or switched to favor the method, or 'best' is chosen with no held-out set (HP-SELECTIVE-REPORTING). Verdict-bearing at L0/L1 from the DESCRIBED protocol — NOT repo-gated like experiment-forensics; L2 only CONFIRMS against split/preprocessing/result files. A fresh cross-model reviewer (gpt-5.6-sol xhigh, read-only, fresh thread per pass) PROPOSES findings, each span-anchored to a ledger claim_id; tools/adjudicate_findings.py DECIDES the verdict. Leakage and under-reporting are usually HONEST methodological errors — every finding describes a discrepancy to CHECK, never an accusation. An LLM generating GROUND-TRUTH labels is HP-FAKE-GT (experiment-forensics) — routed there, not here. Emits eval-design-forensics.findings.json; computes NO verdict. Detect-only. Triggers: "eval design audit", "evaluation validity", "train/test leakage", "data leakage", "is the score measuring generalization", "LLM judge bias", "is the judge validated", "selective reporting", "cherry-picked results", "评估设计审计", "评测有效性", "数据泄漏", "训练测试集泄漏", "裁判模型有没有验证", "选择性报告".

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate Jira ticket complexity + inconsistencies via the jira-evaluator agent. Read-only. Triggers: 'evaluate KEY', 'analyze ticket complexity for KEY', 'check ticket quality for KEY'. NOT for estimation (mk:jira-estimator); NOT for full RCA (mk:jira-analyst).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass/fail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the evaluation lifecycle (create, update, enable/disable, delete).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Paperclip에서 실제 브라우저부터 서버, 데이터베이스, runner까지 검증하는 E2E 평가를 추가·확장합니다. 테스트 데이터와 판정 조건, 증거를 기존 보고서에 연결합니다.

• 전체 사용자 작업 흐름의 평가 추가
• 테스트 데이터와 판정 조건 갱신
• 평가 보고서에 스크린샷과 산출물 추가

WiFi 신호로 사람의 자세를 추정하는 모델을 학습하고 평가합니다. 평균 자세를 반환하는 기준 모델과 비교하고 데이터 누수를 확인해 측정 결과를 과장 없이 보고합니다.

• 카메라 영상의 정답 라벨로 학습
• 추론 시 카메라 없는 자세 추정 평가
• 평균 자세 기준 모델과 정확도 비교

Create and run automated tests that score LLM answers against expected outputs or judging criteria. Monitor progress, inspect individual results, compare models on the same tests, and use score thresholds in CI, an automated checking workflow.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Creating tests with expected answers
• Monitoring evaluation progress
• Inspecting failed samples and scores

Create a fixed JSON evaluation suite from a repository's existing tests and check its structure for errors. Use the suite with harness-evolve to compare variants against consistent tasks, independently of the repository's normal test command.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Create a fixed evaluation suite from existing tests
• Check evaluation suite structure in CI
• Compare variants across commits using consistent tasks

Benchmark how CodeGraph affects an agent investigating a real codebase. Choose a version, language, repository, and execution mode, then compare cost, tool calls, file reads, and answer correctness with and without CodeGraph.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Evaluating a local development build
• Auditing a published CodeGraph version
• Comparing investigation cost and file reads

Build and evaluate time-to-event models with scikit-survival, including records whose events remain unobserved when follow-up ends. Keep preprocessing separate from evaluation data, compare models, predict survival probabilities, and summarize censoring-aware metrics.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Validate event indicators and follow-up times in local CSV data
• Train survival models without leaking evaluation data
• Evaluate survival probabilities with censoring-aware metrics