11 skills · sorted by relevance
Evaluate code generation models with HumanEval, MBPP, and other benchmarks. Test generated programs, compare models across programming languages, and record pass@k scores, which estimate the chance that at least one of k solutions passes the tests.
Evaluate code generation models with HumanEval, MBPP, and multilingual benchmarks. The skill guides test execution and model comparisons using pass@k, an estimate of whether at least one of k generated solutions passes the tests.
Evaluate AI agent outputs with code checks and model-based grading. Run repeated trials to measure reliability, compare prompts or models, and check whether changes reduce quality, with saved transcripts for reviewing the results.
Define success criteria before AI coding, then assess new capabilities and check that existing behavior still works using code, rules, models, or human review. Track repeated-attempt success metrics and reports to compare prompt changes, agent changes, or model versions.
Plan further training for AI agent models by diagnosing gaps in knowledge, output consistency, or decision making. Compare continued pretraining, supervised fine-tuning, reinforcement learning, and distillation, then design training data, feedback environments, and evaluation checks.
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
The English summary is being prepared. Showing the original description.
Eval-Driven Development (EDD) for AI workflows. pass@k metrics, capability evals, regression evals. Triggers: eval, edd, pass@k, capability, regression, benchmark.
The English summary is being prepared. Showing the original description.
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
The English summary is being prepared. Showing the original description.
Avalia modelos de geração de código em HumanEval, MBPP, MultiPL-E e 15+ benchmarks com métricas pass@k. Use ao comparar modelos de código, avaliar capacidades de codificação, testar suporte multilíngue ou medir qualidade de geração de código. Padrão da indústria do BigCode Project usado pelos leaderboards do HuggingFace.
The English summary is being prepared. Showing the original description.
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
The English summary is being prepared. Showing the original description.
Build an evaluation process for AI agents using success, reliability, cost, and latency measures. Define scoring criteria, repeatable environments, failure analysis, and regression tests that check for recurring problems to support model selection and system improvements.