Results for “evals”

Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).

The English summary is being prepared. Showing the original description.

Build repeatable DeepEval tests for AI agents, chatbots, and applications that retrieve documents to answer questions. Reuse or generate evaluation datasets, inspect failures, and improve prompts, tools, or retrieval through repeated evaluation runs.

• Add repeatable quality tests for an AI agent
• Evaluate chatbot behavior across multiple conversation turns
• Generate evaluation datasets from documentation

Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.

The English summary is being prepared. Showing the original description.

OpenAI API (developers.openai.com) の Evals / Fine-tuning リファレンス。 Evals API, Datasets, graders（string check / text similarity / model / python / multi）, agent evals, trace grading, external models, Promptfoo（移行先）, red teaming, SFT, DPO, RFT, vision fine-tuning, distillation, prompt optimizer, fine-tuning best practices。 Evals は 2026-11-30 停止予定、セルフサービス fine-tuning は段階的廃止中。

The English summary is being prepared. Showing the original description.

Select, inspect, validate, and report Paperclip evaluations for Runner protocol behavior or complete product workflows. Interpret results with evidence, source revisions, costs, and failure categories, and refresh reports from retained evidence without new provider calls.

• Choosing between Runner and Product E2E evaluations
• Inspecting evaluation catalogs without credentials
• Interpreting failures using evidence and execution records

Set up a realtime evaluation folder with a suitable shared harness, the framework that runs evaluations. Prepare prompts, tool definitions, realistic starter data, and a README, then validate the setup with evaluation and test runs.

• Start a single-turn text-to-speech evaluation
• Set up evaluations using saved audio
• Build multi-turn simulations with mock tools

Write tests that demonstrate a pull request’s user-visible changes through before-and-after steps, screenshots, and recorded observations. Extend an existing core journey or create a one-off proof that helps reviewers understand the outcome and access boundaries.

• Demonstrating a feature’s before-and-after behavior
• Adding a missing step to an existing core journey
• Proving permission boundaries for different users

Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, tool grading, regression, or cost. Not for general LLM evals (ai-evals).

The English summary is being prepared. Showing the original description.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

The English summary is being prepared. Showing the original description.

Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.

• Run benchmark evaluations on a local GPU
• Choose an evaluation framework and inference engine
• Check a small sample before a full evaluation

Build evaluation suites for AI features with the Axiom AI SDK. Generate test files, scoring functions, test cases, and configurable parameters from descriptions and existing code to compare changes and monitor output quality in production.

• Add regression evaluations for an AI feature
• Score classification, retrieval, and structured outputs
• Compare model and parameter configurations

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

The English summary is being prepared. Showing the original description.

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.

The English summary is being prepared. Showing the original description.

Hamel Husain (@HamelHusain) 视角. 服务咨询派 + practitioner-as-figure 代表, 「evals are the new code」论断的提出者. ex-Airbnb / GitHub principal eng → 2017 起做 independent ML/AI consultant (parlance-labs) → 2023+ 转向 AI evals 专精. hamel.dev 长文 corpus (40+ 篇 1500+ words technical) + Maven「AI Evals for Engineers」课程 (3000+ paid 学员, 与 Shreya Shankar 共同主理). 在 monetize-agents 行业里代表「不规模化 / 不 productize 成 SaaS / 不融资」的第三条路 — independent consultant + course creator. 用途: 当用户面临「AI agent 上线不可靠 / 客户卡在 prompt 调不动 / 该不该雇团队 / 该不该做 SaaS / 怎样把 expertise productize 成课程而非公司」类问题时, 切换到这副镜片.

The English summary is being prepared. Showing the original description.

Design automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.

The English summary is being prepared. Showing the original description.

Builds, audits and packages Agent Skills, with their evals. Trigger to build, audit (security too), score or package a skill or SKILL.md; to design or integrate a pack; whether a skill beats incumbents; for trigger evals, test cases or assertion suites (skill, prompt card, agent spec); to slim a skill; when a skill did not fire or a run cost too much; for a prose pass on a skill's own files; to port or sanitize a skill set; to build or contract-test the MCP tools a plugin or mod ships; or say skillwright (evals, slim, diagnose, mcp, refresh, port, pack, integrate, upkeep). Prompt tuning is promptwright's; running a suite, claude plugin eval's; config slims, rigwright's; other docs, commscribe's; brands, brandscribe's; runtime grants, gatewarden's.

The English summary is being prepared. Showing the original description.

Design continuous agent workflows with quality checks, evaluations, and recovery controls. Helps choose a loop pattern for the task and define how to pause, audit, narrow scope, and retry when work stalls or fails.

• Choosing a loop pattern for ongoing agent work
• Combining quality checks and evaluations
• Recovering from repeated failures or stalled progress

Connect OpenCode browser tools to a locally running OpenWork Electron app to inspect its UI, enter a task, and check the response. This skill supports exploration and debugging through CDP, Chrome's remote control protocol.

• Connecting browser tools to a local Electron development app
• Inspecting screen state during debugging
• Entering a task in the composer and running it

Review changed test specifications for cases where broken behavior could still pass. Report concrete gaps supported by the changes and implementation, with a minimal fix and an observable condition for resolving each advisory finding.

• Reviewing whether changed tests prove their stated behavior
• Finding UI tests that bypass the interaction being tested
• Checking whether save tests observe newly saved data

Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.

• Comparing two prompts on the same test cases
• Checking quality after agent configuration changes
• Turning real failures into repeatable evaluation cases

Launch a standalone Chrome browser in a Daytona sandbox and control it through CDP, a browser debugging connection. Use it to complete sign-in, OAuth authorization, and web administration flows, then verify the visible results.

• Signing into Den Web in a standalone browser
• Completing OAuth authorization pages
• Configuring settings available only on the web

Launch the real iPolloWork app on Daytona or locally and run scripted end-to-end tests through its debugging connection. Collect machine-checkable assertions, validated screenshots, and reports as evidence of the tested behavior.

• Run scripted end-to-end tests on Daytona
• Collect screenshots and reports for a pull request
• Run tests locally when Daytona is unavailable

Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures. Load when writing eval cases or rubrics or deciding how a suite runs.

The English summary is being prepared. Showing the original description.

Turn student course evaluations (free-text + numeric) into an actionable teaching-improvement plan — the teaching analogue of /respond-to-referees. Clusters comments into themes, separates signal from noise, classifies each theme Keep / Change / Investigate / Out-of-scope, and drafts concrete changes mapped to the syllabus and slide decks. Use when user says "respond to my evals", "what do these course evaluations tell me", "turn my teaching feedback into a plan", or after a semester's evals arrive.

The English summary is being prepared. Showing the original description.