“lm evaluation harness”的搜索结果

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

中文简介正在准备中。当前显示英语简介。

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

中文简介正在准备中。当前显示英语简介。

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

使用统一任务评估大语言模型的知识、推理和代码生成能力，生成多模型对比表及训练阶段的成绩趋势，并保存评估结果和每道题的回答。

• 使用MMLU和GSM8K评估模型
• 生成多个模型的成绩对比表
• 跟踪训练期间的模型成绩

Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.

中文简介正在准备中。当前显示英语简介。

• Build an evaluation set from agent traces
• Create separate checks for each failure type
• Calibrate an LLM judge against human labels

Evaluate language models with over 100 benchmark tests covering knowledge, math, coding, safety, and image understanding. Configure runs on Docker, Slurm computing clusters, or cloud infrastructure, compare models, and export results.

中文简介正在准备中。当前显示英语简介。

• Compare models using the same benchmark tests
• Run large evaluations on a Slurm cluster
• Evaluate safety and image understanding

Plan AI application work across six areas: prompt evaluation, context allocation, retrieval-based answer generation, agent security, evaluation pipelines, and product strategy. This overview explains the approaches; the individual workflow packages, test cases, and CI templates are not included.

中文简介正在准备中。当前显示英语简介。

• Planning prompt evaluation before deployment
• Allocating context capacity and designing retrieval-based answers
• Planning an authorized AI agent security assessment

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

中文简介正在准备中。当前显示原文说明。

Evaluate language models on knowledge, math, coding, safety, and image understanding benchmarks. The skill guides reproducible runs in containers using local Docker, Slurm clusters, or cloud execution, with shared configurations for model comparison and result export.

中文简介正在准备中。当前显示英语简介。

• Comparing models on the same benchmark tasks
• Running a small evaluation sample before a full benchmark
• Running large evaluations on a Slurm cluster

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

中文简介正在准备中。当前显示原文说明。

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

中文简介正在准备中。当前显示原文说明。

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

中文简介正在准备中。当前显示原文说明。

Avalia LLMs em mais de 60 benchmarks acadêmicos (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use quando benchmarking de qualidade de modelo, comparação de modelos, relatório de resultados acadêmicos ou rastreamento de progresso de treinamento. Padrão da indústria usado por EleutherAI, HuggingFace e laboratórios principais. Suporta HuggingFace, vLLM, APIs.

中文简介正在准备中。当前显示原文说明。

Evaluate code generation models with HumanEval, MBPP, and other benchmarks. Test generated programs, compare models across programming languages, and record pass@k scores, which estimate the chance that at least one of k solutions passes the tests.

中文简介正在准备中。当前显示英语简介。

• Comparing code generation models on shared benchmarks
• Measuring correctness across programming languages
• Evaluating models trained to follow instructions

Evaluate code generation models with HumanEval, MBPP, and multilingual benchmarks. The skill guides test execution and model comparisons using pass@k, an estimate of whether at least one of k generated solutions passes the tests.

中文简介正在准备中。当前显示英语简介。

• Compare code generation models on shared benchmarks
• Measure code generation performance across languages
• Evaluate instruction-tuned coding models

Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets".

中文简介正在准备中。当前显示原文说明。

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).

中文简介正在准备中。当前显示原文说明。

Rigorous benchmarking of multi-agent coordination systems: experiment design, statistical analysis, human evaluation protocols, and reproducible reporting. NOT FOR ML model evaluation (use llm-evaluation-harness), A/B testing for web products, survey design, or general data science.

中文简介正在准备中。当前显示原文说明。

Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.

中文简介正在准备中。当前显示原文说明。

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

中文简介正在准备中。当前显示原文说明。

Builds repeatable evaluation systems with golden datasets, scoring rubrics, pass/fail thresholds, and regression reports. Use for "LLM evaluation", "testing AI systems", "quality assurance", or "model benchmarking".

中文简介正在准备中。当前显示原文说明。

Create evaluation drafts for existing skills using actual conversations and user corrections. Produce criteria and repeatable test cases, distinguish evidence from specification assumptions, and prepare an anonymized draft for human review.

中文简介正在准备中。当前显示英语简介。

• Turn user corrections into repeatable test cases
• Compare promised behavior with actual usage
• Draft an evaluation from conversation history

Plan and improve AI agent systems by managing the information models receive. This collection covers memory, agent coordination, tool design, context compression, evaluation, and the operating controls needed for reliable autonomous workflows.

中文简介正在准备中。当前显示英语简介。

• Planning information management for a new AI agent
• Investigating failures in long agent conversations
• Designing agent memory and multi-agent coordination

6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.

中文简介正在准备中。当前显示原文说明。

AI Engineering Toolkit workflow skill. Use this skill when the user needs 6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

中文简介正在准备中。当前显示原文说明。