4.5K skills · sorted by relevance
Build repeatable DeepEval tests for AI agents, chatbots, and applications that retrieve documents to answer questions. Reuse or generate evaluation datasets, inspect failures, and improve prompts, tools, or retrieval through repeated evaluation runs.
Add DeepEval tracing to Python AI applications so execution steps appear in Confident AI. Choose a supported integration or manual @observe instrumentation, then capture inputs, outputs, step types, tags, and metadata.
Set up a realtime evaluation folder with a suitable shared harness, the framework that runs evaluations. Prepare prompts, tool definitions, realistic starter data, and a README, then validate the setup with evaluation and test runs.
Evaluate code generation models with HumanEval, MBPP, and other benchmarks. Test generated programs, compare models across programming languages, and record pass@k scores, which estimate the chance that at least one of k solutions passes the tests.
Evaluate code generation models with HumanEval, MBPP, and multilingual benchmarks. The skill guides test execution and model comparisons using pass@k, an estimate of whether at least one of k generated solutions passes the tests.
Evaluate Hugging Face Hub models using inspect-ai or lighteval, with local inference or a provider-backed option. Choose an inference engine, start with a small test, and switch engines when model compatibility requires it.
Add team-specific evaluation criteria to plugin-eval by designing a metric pack, a set of custom checks and measurements. Create its manifest and JSON-producing script, then run it through the evaluator using a consistent output structure.
Design automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.
The English summary is being prepared. Showing the original description.
Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.
Evaluate language models with over 100 benchmark tests covering knowledge, math, coding, safety, and image understanding. Configure runs on Docker, Slurm computing clusters, or cloud infrastructure, compare models, and export results.
Evaluate language models on knowledge, math, coding, safety, and image understanding benchmarks. The skill guides reproducible runs in containers using local Docker, Slurm clusters, or cloud execution, with shared configurations for model comparison and result export.
Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.
Audit a completed AI development phase to check whether its planned evaluations were implemented. Produce EVAL-REVIEW.md with a score, verdict, identified gaps, and a plan to address missing coverage.
Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
The English summary is being prepared. Showing the original description.
Find JavaScript that executes strings as code, including eval(), new Function(), and string-based timers. Review how it runs in the browser and suggest alternatives for parsing data, choosing operations, and scheduling functions to reduce code injection risks.
Create and run evaluations in Arize using an LLM as the judge. Define scoring criteria, connect them to actual data fields, evaluate recorded operations or experiment results, and set up continuous scoring for new data.
Configure OpenTelemetry to send AI application traces, records of execution steps, to Confident AI. Reuse existing exporters, choose the regional endpoint, and add AI-specific attributes without requiring the deepeval Python package.
Improve an evaluated Codex skill by turning plugin-eval findings into prioritized changes. Refine activation descriptions, shorten SKILL.md, move lengthy details into supporting files, fix structural issues, and compare evaluation results before and after the rewrite.
Evaluate large language models on shared tests of knowledge, reasoning, and code generation with lm-evaluation-harness. Save scores and individual predictions, compare models in tables, and track performance across saved stages of model training.
Select, inspect, validate, and report Paperclip evaluations for Runner protocol behavior or complete product workflows. Interpret results with evidence, source revisions, costs, and failure categories, and refresh reports from retained evidence without new provider calls.
INVOKE THIS SKILL for LLM-as-judge evaluation workflows on Arize: creating/updating evaluators, running evaluations on spans or experiments, tasks, trigger-run, column mapping, and continuous monitoring. Use when the user says: create an evaluator, LLM judge, hallucination/faithfulness/correctness/relevance, run eval, score my spans or experiment, ax tasks, trigger-run, trigger eval, column mapping, continuous monitoring, query filter for evals, evaluator version, or improve an evaluator prompt.
The English summary is being prepared. Showing the original description.
Plan AI application work across six areas: prompt evaluation, context allocation, retrieval-based answer generation, agent security, evaluation pipelines, and product strategy. This overview explains the approaches; the individual workflow packages, test cases, and CI templates are not included.
Explains how PluginEval combines structural checks and experimental model assessments into quality scores. Helps interpret low scores, identify improvement areas, set marketplace thresholds, and explain what evaluation badges actually represent.
Evaluate AI models and agents on Google Cloud using test datasets and scoring criteria. Generate simulated user scenarios, analyze failures, and compare results across changes to guide improvements to prompts and code.