'evals' 검색 결과

Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Build repeatable DeepEval tests for AI agents, chatbots, and applications that retrieve documents to answer questions. Reuse or generate evaluation datasets, inspect failures, and improve prompts, tools, or retrieval through repeated evaluation runs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Add repeatable quality tests for an AI agent
• Evaluate chatbot behavior across multiple conversation turns
• Generate evaluation datasets from documentation

Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

OpenAI API (developers.openai.com) の Evals / Fine-tuning リファレンス。 Evals API, Datasets, graders（string check / text similarity / model / python / multi）, agent evals, trace grading, external models, Promptfoo（移行先）, red teaming, SFT, DPO, RFT, vision fine-tuning, distillation, prompt optimizer, fine-tuning best practices。 Evals は 2026-11-30 停止予定、セルフサービス fine-tuning は段階的廃止中。

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Select, inspect, validate, and report Paperclip evaluations for Runner protocol behavior or complete product workflows. Interpret results with evidence, source revisions, costs, and failure categories, and refresh reports from retained evidence without new provider calls.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Choosing between Runner and Product E2E evaluations
• Inspecting evaluation catalogs without credentials
• Interpreting failures using evidence and execution records

Set up a realtime evaluation folder with a suitable shared harness, the framework that runs evaluations. Prepare prompts, tool definitions, realistic starter data, and a README, then validate the setup with evaluation and test runs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Start a single-turn text-to-speech evaluation
• Set up evaluations using saved audio
• Build multi-turn simulations with mock tools

Write tests that demonstrate a pull request’s user-visible changes through before-and-after steps, screenshots, and recorded observations. Extend an existing core journey or create a one-off proof that helps reviewers understand the outcome and access boundaries.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Demonstrating a feature’s before-and-after behavior
• Adding a missing step to an existing core journey
• Proving permission boundaries for different users

Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, tool grading, regression, or cost. Not for general LLM evals (ai-evals).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Run benchmark evaluations on a local GPU
• Choose an evaluation framework and inference engine
• Check a small sample before a full evaluation

Build evaluation suites for AI features with the Axiom AI SDK. Generate test files, scoring functions, test cases, and configurable parameters from descriptions and existing code to compare changes and monitor output quality in production.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Add regression evaluations for an AI feature
• Score classification, retrieval, and structured outputs
• Compare model and parameter configurations

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Hamel Husain (@HamelHusain) 视角. 服务咨询派 + practitioner-as-figure 代表, 「evals are the new code」论断的提出者. ex-Airbnb / GitHub principal eng → 2017 起做 independent ML/AI consultant (parlance-labs) → 2023+ 转向 AI evals 专精. hamel.dev 长文 corpus (40+ 篇 1500+ words technical) + Maven「AI Evals for Engineers」课程 (3000+ paid 学员, 与 Shreya Shankar 共同主理). 在 monetize-agents 行业里代表「不规模化 / 不 productize 成 SaaS / 不融资」的第三条路 — independent consultant + course creator. 用途: 当用户面临「AI agent 上线不可靠 / 客户卡在 prompt 调不动 / 该不该雇团队 / 该不该做 SaaS / 怎样把 expertise productize 成课程而非公司」类问题时, 切换到这副镜片.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Design automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Builds, audits and packages Agent Skills, with their evals. Trigger to build, audit (security too), score or package a skill or SKILL.md; to design or integrate a pack; whether a skill beats incumbents; for trigger evals, test cases or assertion suites (skill, prompt card, agent spec); to slim a skill; when a skill did not fire or a run cost too much; for a prose pass on a skill's own files; to port or sanitize a skill set; to build or contract-test the MCP tools a plugin or mod ships; or say skillwright (evals, slim, diagnose, mcp, refresh, port, pack, integrate, upkeep). Prompt tuning is promptwright's; running a suite, claude plugin eval's; config slims, rigwright's; other docs, commscribe's; brands, brandscribe's; runtime grants, gatewarden's.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

인공지능 에이전트의 지속 작업 루프를 설계하고 품질 확인, 평가, 실패 시 중지와 재시도 방침을 결합해 진행 정체와 반복 실패에 대응합니다.

• 지속 작업에 맞는 루프 방식 선택
• 품질 확인과 평가 결합
• 실패나 진행 정체에서 복구

Review changed test specifications for cases where broken behavior could still pass. Report concrete gaps supported by the changes and implementation, with a minimal fix and an observable condition for resolving each advisory finding.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Reviewing whether changed tests prove their stated behavior
• Finding UI tests that bypass the interaction being tested
• Checking whether save tests observe newly saved data

Connect OpenCode browser tools to a locally running OpenWork Electron app to inspect its UI, enter a task, and check the response. This skill supports exploration and debugging through CDP, Chrome's remote control protocol.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Connecting browser tools to a local Electron development app
• Inspecting screen state during debugging
• Entering a task in the composer and running it

Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing two prompts on the same test cases
• Checking quality after agent configuration changes
• Turning real failures into repeatable evaluation cases

Launch a standalone Chrome browser in a Daytona sandbox and control it through CDP, a browser debugging connection. Use it to complete sign-in, OAuth authorization, and web administration flows, then verify the visible results.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Signing into Den Web in a standalone browser
• Completing OAuth authorization pages
• Configuring settings available only on the web

Launch the real iPolloWork app on Daytona or locally and run scripted end-to-end tests through its debugging connection. Collect machine-checkable assertions, validated screenshots, and reports as evidence of the tested behavior.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Run scripted end-to-end tests on Daytona
• Collect screenshots and reports for a pull request
• Run tests locally when Daytona is unavailable

Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures. Load when writing eval cases or rubrics or deciding how a suite runs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Turn student course evaluations (free-text + numeric) into an actionable teaching-improvement plan — the teaching analogue of /respond-to-referees. Clusters comments into themes, separates signal from noise, classifies each theme Keep / Change / Investigate / Out-of-scope, and drafts concrete changes mapped to the syllabus and slide decks. Use when user says "respond to my evals", "what do these course evaluations tell me", "turn my teaching feedback into a plan", or after a semester's evals arrive.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.