Results for “evaluation”

Guide for diagnosing and improving MSBuild project evaluation performance. Only activate in MSBuild/.NET build context. USE FOR: builds slow before any compilation starts, high evaluation time in binlog analysis, expensive glob patterns walking large directories (node_modules, .git, bin/obj), deep import chains (>20 levels), preprocessed output >10K lines indicating heavy evaluation, property functions with file I/O ($([System.IO.File]::ReadAllText(...))), multiple evaluations per project. Covers the 5 MSBuild evaluation phases, glob optimization via DefaultItemExcludes, import chain analysis with /pp preprocessing. DO NOT USE FOR: compilation-time slowness (use build-perf-diagnostics), incremental build issues (use incremental-build), non-MSBuild build systems. INVOKES: binlog MCP server tools (evaluations, evaluation_global_properties, evaluation_properties, imports, properties); falls back to dotnet msbuild -pp:full.xml for preprocessing, /clp:PerformanceSummary.

The English summary is being prepared. Showing the original description.

Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

The English summary is being prepared. Showing the original description.

Set up a realtime evaluation folder with a suitable shared harness, the framework that runs evaluations. Prepare prompts, tool definitions, realistic starter data, and a README, then validate the setup with evaluation and test runs.

• Start a single-turn text-to-speech evaluation
• Set up evaluations using saved audio
• Build multi-turn simulations with mock tools

Plan AI application work across six areas: prompt evaluation, context allocation, retrieval-based answer generation, agent security, evaluation pipelines, and product strategy. This overview explains the approaches; the individual workflow packages, test cases, and CI templates are not included.

• Planning prompt evaluation before deployment
• Allocating context capacity and designing retrieval-based answers
• Planning an authorized AI agent security assessment

Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

The English summary is being prepared. Showing the original description.

Select, inspect, validate, and report Paperclip evaluations for Runner protocol behavior or complete product workflows. Interpret results with evidence, source revisions, costs, and failure categories, and refresh reports from retained evidence without new provider calls.

• Choosing between Runner and Product E2E evaluations
• Inspecting evaluation catalogs without credentials
• Interpreting failures using evidence and execution records

Create a fixed JSON evaluation suite from a repository's existing tests and check its structure for errors. Use the suite with harness-evolve to compare variants against consistent tasks, independently of the repository's normal test command.

• Create a fixed evaluation suite from existing tests
• Check evaluation suite structure in CI
• Compare variants across commits using consistent tasks

Turns supplied AI agent evaluation results into reports that distinguish autonomous success, human-assisted success, failure, timeout, and invalid runs. Makes metric denominators, retries, timing, costs, and comparison conditions explicit to support evidence-based decisions.

• Separating autonomous and human-assisted success rates
• Reporting evaluations with retries and invalid runs
• Comparing agents or releases under matched conditions

Create and analyze AI model evaluation experiments in Arize using the ax CLI, a command-line tool. Collect real model outputs, export results, compare evaluation scores, and identify examples where a new model performs worse.

• Comparing models on the same evaluation dataset
• Finding examples that perform worse after a model change
• Exporting experiment results for further analysis

Build repeatable DeepEval tests for AI agents, chatbots, and applications that retrieve documents to answer questions. Reuse or generate evaluation datasets, inspect failures, and improve prompts, tools, or retrieval through repeated evaluation runs.

• Add repeatable quality tests for an AI agent
• Evaluate chatbot behavior across multiple conversation turns
• Generate evaluation datasets from documentation

Audit a completed AI development phase to check whether its planned evaluations were implemented. Produce EVAL-REVIEW.md with a score, verdict, identified gaps, and a plan to address missing coverage.

• Review evaluation coverage after an AI phase is complete
• Check implemented evaluations against AI-SPEC.md
• Document evaluation gaps and remediation steps

Add team-specific evaluation criteria to plugin-eval by designing a metric pack, a set of custom checks and measurements. Create its manifest and JSON-producing script, then run it through the evaluator using a consistent output structure.

• Add team-specific evaluation criteria
• Define custom checks and numerical metrics
• Build a metric-pack manifest and script

Guide for diagnosing and improving MSBuild project evaluation performance. USE FOR: builds slow before any compilation starts, high evaluation time in binlog analysis, expensive glob patterns walking large directories (node_modules, .git, bin/obj), deep import chains (>20 levels), preprocessed output >10K lines indicating heavy evaluation, property functions with file I/O ($([System.IO.File]::ReadAllText(...))), multiple evaluations per project. Covers the 5 MSBuild evaluation phases, glob optimization via DefaultItemExcludes, import chain analysis with /pp preprocessing. DO NOT USE FOR: compilation-time slowness (use build-perf-diagnostics), incremental build issues (use incremental-build), non-MSBuild build systems.

The English summary is being prepared. Showing the original description.

Guide for diagnosing and improving MSBuild project evaluation performance. USE FOR: builds slow before any compilation starts, high evaluation time in binlog analysis, expensive glob patterns walking large directories (node_modules, .git, bin/obj), deep import chains (>20 levels), preprocessed output >10K lines indicating heavy evaluation, property functions with file I/O ($([System.IO.File]::ReadAllText(...))), multiple evaluations per project. Covers the 5 MSBuild evaluation phases, glob optimization via DefaultItemExcludes, import chain analysis with /pp preprocessing. DO NOT USE FOR: compilation-time slowness (use build-perf-diagnostics), incremental build issues (use incremental-build), non-MSBuild build systems.

The English summary is being prepared. Showing the original description.

Investigate AI observability evaluations of both types — `hog` (deterministic code-based) and `llm_judge` (LLM-prompt-based). Find existing evaluations, inspect their configuration, run them against specific generations, query individual pass/fail results, and generate AI-powered summaries of patterns across many runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, or manage the evaluation lifecycle (create, update, enable/disable, delete).

The English summary is being prepared. Showing the original description.

Plan a paid evaluation with predefined success criteria, comparison groups, decoy items, and stopping rules. Rehearse without spending, then run a three-item paid pilot and use its spending table to inform approval.

• Define evaluation criteria before spending
• Rehearse an evaluation at zero cost
• Check spending with a three-item paid pilot

Add or extend Paperclip end-to-end evaluations that exercise real browser workflows across the server, database, and runner. Define expected outcomes, check independent evidence, and connect results to the existing reporting pipeline for local or Daytona execution.

• Adding an evaluation for a complete user workflow
• Updating test data and outcome checks
• Including screenshots and artifacts in evaluation reports

Add or extend Paperclip Runner evaluations that test real sessions against a mock control service. Define observable pass conditions, validate cases and run selections, and retain source revisions, evidence, and costs for authorized live runs.

• Adding a focused Runner protocol evaluation
• Testing grading rules with valid and invalid evidence
• Updating evaluation selections and campaign membership

Build an evaluation process for AI agents using success, reliability, cost, and latency measures. Define scoring criteria, repeatable environments, failure analysis, and regression tests that check for recurring problems to support model selection and system improvements.

• Building an initial agent evaluation process
• Comparing models on actual work tasks
• Investigating a drop in evaluation scores

Create consistent records for evaluating AI agents, including task success, invalid tool calls, token use, and actual costs. Produce CSV tables, database definitions, and JSON validation schemas to support reproducible comparisons and checks for performance declines.

• Recording agent evaluation results in a shared format
• Comparing a candidate agent with a baseline
• Tracking actual cost per solved task

Evaluate LLM applications using automated scores, human reviews, and model-based judging. Build repeatable checks to compare models or prompts, assess answer quality, and detect declines in performance before deployment.

• Comparing models and prompts
• Checking quality after prompt changes
• Detecting performance declines before deployment

Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.

• Check answers for unsupported claims
• Score experiment results for correctness
• Continuously evaluate new production records

Run repeated change-and-evaluation cycles for one predefined improvement mission within an explicit runtime limit. Save each evaluation as JSON and each decision in readable Markdown logs, with support for periodic reruns through Claude Code native cron.

• Repeating improvements under a fixed evaluation method
• Keeping evaluation results and decision logs
• Running experiments within a time limit

Set up Langfuse to record LLM application activity, costs, and response times. Manage prompt versions, organize evaluation datasets, and compare results to investigate failures and guide improvements using recorded evidence.

• Investigating failures in LLM applications
• Comparing prompt versions and evaluation scores
• Monitoring LLM costs and response times