4.5K skills · sorted by relevance
Design and implement RAG systems, which retrieve documents to inform AI answers. Get document preparation strategies, vector database comparisons, combined keyword and meaning-based search, result reranking, and an evaluation plan for retrieval quality.
Evaluate local Codex skills and plugins, understand their scores, and identify fixes to prioritize. Get explanations of token estimates, set up benchmark scenarios, and prepare workflows for measuring actual token usage.
Evaluate a local Codex skill for metadata, structure, reference links, size, and helper code quality. Get prioritized fixes and explanations, with benchmark setup for deeper analysis and measurement recommendations when observed usage is available.
Capture and review Agent Skill observations, and create, run, diagnose, or iteratively improve Skill evaluations (evals) with the skill-up CLI. Use when the user asks to record explicitly attributed Skill usage or feedback; review observations or user-supplied traces/transcripts; turn an approved observation into a regression case; evaluate, test, regress, verify, fix, improve, iterate, or evolve a Skill; add or strengthen eval cases; write eval.yaml/case.yaml; run skill-up run/validate/list-cases/report/import/init; or migrate from Anthropic evals.json. Plugin observation capture/review require either the Codex skill-up plugin or an observer-enabled DSH skill-up plugin. User-supplied external evidence can be reviewed without a plugin; evaluation remains multi-engine.
The English summary is being prepared. Showing the original description.
Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.
The English summary is being prepared. Showing the original description.
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.
The English summary is being prepared. Showing the original description.
Advanced Evaluation workflow skill. Use this skill when the user needs This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.
The English summary is being prepared. Showing the original description.
Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets".
The English summary is being prepared. Showing the original description.
Author and run offline evals for Agentforce agents: fixture format, scoring rubrics, hallucination checks, regression baselines, CI integration, prompt-change safety. Trigger keywords: agent eval, hallucination, doesn't hallucinate, agentforce offline evals, rubric for agent response. NOT for deciding which layers of testing the agent needs — use agentforce/agentforce-testing-strategy. NOT for Testing Center, AiEvaluationDefinition or sf agent test runs — use agentforce/agent-testing-and-evaluation.
The English summary is being prepared. Showing the original description.
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
The English summary is being prepared. Showing the original description.
Design continuous agent workflows with quality checks, evaluations, and recovery controls. Helps choose a loop pattern for the task and define how to pause, audit, narrow scope, and retry when work stalls or fails.
Evaluate BMad skills using your agent's configured runner. Compare results with the model alone or another version, grade outputs against criteria, and test whether descriptions trigger correctly, preserving transcripts and evidence for each run.
Evaluate how Chatbox answers questions about large attachments using synthetic and real documents. Check whether the model retrieves file content when needed, skips retrieval for unrelated requests, and investigate failures through the actual application flow.
Create evaluation drafts for existing skills using actual conversations and user corrections. Produce criteria and repeatable test cases, distinguish evidence from specification assumptions, and prepare an anonymized draft for human review.
Review completed AI work by scoring task difficulty and execution quality separately. A fixed scoring matrix, arguments for higher and lower ratings, and saved score history help examine weaknesses and detect repetitive scoring patterns across sessions.
Evaluate a machine learning GPU cluster during a cloud trial or acceptance test. Measure GPU computing performance, communication within and between machines, and local and shared storage, then compile the results into a dated Markdown report.
Evaluate a local Codex plugin to understand its scores and identify what to fix first. Review stronger and weaker skills, prepare benchmark runs, compare saved results across versions, and produce a readable report.
Diagnose why an existing skill fails evaluations or shows no clear improvement. Separate content problems from test data, execution failures, activation issues, and insufficient evaluation tasks, then apply targeted fixes and validate them.
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
The English summary is being prepared. Showing the original description.
Use when measuring or improving agent quality and performance — set up evaluators, online monitoring, CI/CD quality gates, observability, or cost optimization. Triggers on: "evaluate my agent", "add evaluator", "measure quality", "quality gate", "run evals", "agent too slow", "why is it slow", "reduce latency", "set up observability", "CloudWatch dashboard", "how much does my agent cost", "cost optimization", "logs not showing up", "logs missing", "spans not found", "eval failing", "eval error", "dev traces", "local traces", "agentcore dev traces", "traces to CloudWatch". Not for debugging errors or crashes — use agents-debug. Slow but correct routes here; broken routes to debug.
The English summary is being prepared. Showing the original description.
Design an evaluation plan for an LLM or AI feature before shipping it. Use when asked how to evaluate a prompt/model/agent, set up an eval harness, define quality metrics for an AI feature, or build a regression gate. Produces an eval plan — task definition, datasets, metrics & rubrics, baselines, automated + human evals, a pass bar, and a regression gate.
The English summary is being prepared. Showing the original description.
**WORKFLOW SKILL** - Evaluate AI agent skills using structured benchmarks with YAML specs, fixture isolation, and pluggable validators. USE FOR: run waza, waza help, run eval, run benchmark, evaluate skill, test agent, generate eval suite, init eval, compare results, score agent, agent evaluation, skill testing, cross-model comparison. DO NOT USE FOR: improving skill frontmatter (use waza dev), creating new skills from scratch (use skill-creator), token counting or budget checks (use waza tokens). INVOKES: Copilot SDK executor, mock engine, code/regex validators. FOR SINGLE OPERATIONS: use waza run directly for a single benchmark.
The English summary is being prepared. Showing the original description.
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
The English summary is being prepared. Showing the original description.
Use when the user has run multiple evaluation skills and wants a comprehensive analysis — maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.
The English summary is being prepared. Showing the original description.