4.5K skills · sorted by relevance
Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.
Build evaluation suites for AI features with the Axiom AI SDK. Generate test files, scoring functions, test cases, and configurable parameters from descriptions and existing code to compare changes and monitor output quality in production.
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
The English summary is being prepared. Showing the original description.
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
The English summary is being prepared. Showing the original description.
Guide for diagnosing and improving MSBuild project evaluation performance. Only activate in MSBuild/.NET build context. USE FOR: builds slow before any compilation starts, high evaluation time in binlog analysis, expensive glob patterns walking large directories (node_modules, .git, bin/obj), deep import chains (>20 levels), preprocessed output >10K lines indicating heavy evaluation, property functions with file I/O ($([System.IO.File]::ReadAllText(...))), multiple evaluations per project. Covers the 5 MSBuild evaluation phases, glob optimization via DefaultItemExcludes, import chain analysis with /pp preprocessing. DO NOT USE FOR: compilation-time slowness (use build-perf-diagnostics), incremental build issues (use incremental-build), non-MSBuild build systems. INVOKES: binlog MCP server tools (evaluations, evaluation_global_properties, evaluation_properties, imports, properties); falls back to dotnet msbuild -pp:full.xml for preprocessing, /clp:PerformanceSummary.
The English summary is being prepared. Showing the original description.
Design structured evaluation rubrics for scoring LLM and agent outputs — defining quality dimensions, scoring scales, hard gates, score descriptions, and edge cases. Load when the user asks to create an eval rubric, define evaluation criteria, design scoring dimensions, write an eval spec, or says "what should I evaluate", "design a rubric", "create eval criteria", "define quality dimensions", "evaluation rubric for", "how do I measure quality of". Sub-skill of eval-output orchestrator.
The English summary is being prepared. Showing the original description.
Build an end-to-end evaluation pipeline for Python AI applications using pixie-qa and real language-model calls. Define criteria, prepare realistic test data, score application runs, and analyze the evidence to produce a prioritized improvement plan.
Design RAG systems that retrieve relevant documents to support LLM answers. The skill guides document splitting, meaning-based and keyword search, result reranking, and separate retrieval evaluation to improve the context passed to the model.
Design a RAG pipeline that retrieves documents to support AI answers. Analyze document splitting, select embedding models and vector databases, and evaluate retrieval quality using test queries and known relevant documents.
Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.
Evaluate a driving agent on CARLA Leaderboard routes after checking the required setup. Select routes and repetitions, save driving scores and infractions, record runs, and resume interrupted evaluations from a saved checkpoint.
Evaluate NVIDIA Cosmos Policy for robot manipulation in LIBERO and RoboCasa simulations. Follow GPU environment setup, configure rendering without a display, run small checks before larger benchmarks, inspect success rates, and profile inference latency.
Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".
The English summary is being prepared. Showing the original description.
Use when evaluating a newly adapted model's accuracy/eval score. Runs evalscope eval (or the test/srt/run_eval.py fallback) against an OpenAI-compatible endpoint on datasets like gsm8k / aime, producing per-dataset reports and a score. Accuracy only — throughput/latency benchmarks belong to model-speed-benchmark. Triggers on accuracy, eval, evalscope, gsm8k, aime, run_eval — not throughput.
The English summary is being prepared. Showing the original description.
Suede AI eval design and coverage audit: AI-SPEC, failure-mode rubric with severity scoring, concrete pass/fail eval cases, coverage and infrastructure scores, and mechanical acceptance gates. Use when a change ships LLM, RAG, agent, classifier, prompt, or generated-media behavior, or when asked to write evals for an AI feature, design test cases for a model surface, audit existing eval coverage, or judge whether AI behavior is safe to ship. No AI-SPEC means no eval plan, and no eval plan holds the recommended ship verdict. NOT FOR: reviewing or grading the implementation code behind the AI surface (use suede-code); wiring a passing suite into CI as a required check (use suede-ci-gate); UAT of the built feature beyond the eval suite (a private Suede Labs companion, not in this pack).
The English summary is being prepared. Showing the original description.
Orchestrator for the eval-output skill suite — evaluate LLM and agent outputs for quality, accuracy, helpfulness, and safety using structured rubrics and LLM-as-judge techniques. Load when the user says "evaluate this output", "score this response", "run an eval", "LLM as judge", "evaluate agent output", "how good is this response", "rate this answer", "eval this", or provides an LLM output that should be assessed for quality. Single entry point for all output evaluation workflows.
The English summary is being prepared. Showing the original description.
Plan a paid evaluation with predefined success criteria, comparison groups, decoy items, and stopping rules. Rehearse without spending, then run a three-item paid pilot and use its spending table to inform approval.
Add or extend Paperclip Runner evaluations that test real sessions against a mock control service. Define observable pass conditions, validate cases and run selections, and retain source revisions, evidence, and costs for authorized live runs.
Build and run evaluations for AI applications using Phoenix. Start with observed failures, create checks in code or use a language model as a judge, run experiments, and validate automated judgments against human labels.
Design workflows that evaluate and refine AI outputs through self-review, separate generation and evaluation roles, tests, or weighted scoring criteria. Includes patterns for structured feedback, iteration limits, and stopping when quality stops improving.
Create and analyze AI model evaluation experiments in Arize using the ax CLI, a command-line tool. Collect real model outputs, export results, compare evaluation scores, and identify examples where a new model performs worse.
Design and implement RAG systems, which retrieve documents to inform AI answers. Get document preparation strategies, vector database comparisons, combined keyword and meaning-based search, result reranking, and an evaluation plan for retrieval quality.
Evaluate a local Codex skill for metadata, structure, reference links, size, and helper code quality. Get prioritized fixes and explanations, with benchmark setup for deeper analysis and measurement recommendations when observed usage is available.
Evaluate local Codex skills and plugins, understand their scores, and identify fixes to prioritize. Get explanations of token estimates, set up benchmark scenarios, and prepare workflows for measuring actual token usage.