'lm evaluation harness' 검색 결과

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Assess language models with over 60 academic benchmark tests using consistent prompts and scoring. Compare models, track performance during training, and organize research results with support for HuggingFace models, vLLM, and APIs.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models on standard knowledge and reasoning tests
• Track model performance across training checkpoints
• Organize evaluation results for research reports

Evaluate large language models on shared tests of knowledge, reasoning, and code generation with lm-evaluation-harness. Save scores and individual predictions, compare models in tables, and track performance across saved stages of model training.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Evaluating models on MMLU and GSM8K
• Building comparison tables for multiple models
• Tracking scores across training checkpoints

Prepare an evaluation framework before fine-tuning a language model. Build labeled examples, checks for each failure type, and a base-model benchmark, while calibrating model-based judges against human labels to support comparisons after training.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Build an evaluation set from agent traces
• Create separate checks for each failure type
• Calibrate an LLM judge against human labels

Evaluate language models with over 100 benchmark tests covering knowledge, math, coding, safety, and image understanding. Configure runs on Docker, Slurm computing clusters, or cloud infrastructure, compare models, and export results.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare models using the same benchmark tests
• Run large evaluations on a Slurm cluster
• Evaluate safety and image understanding

Plan AI application work across six areas: prompt evaluation, context allocation, retrieval-based answer generation, agent security, evaluation pipelines, and product strategy. This overview explains the approaches; the individual workflow packages, test cases, and CI templates are not included.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Planning prompt evaluation before deployment
• Allocating context capacity and designing retrieval-based answers
• Planning an authorized AI agent security assessment

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate language models on knowledge, math, coding, safety, and image understanding benchmarks. The skill guides reproducible runs in containers using local Docker, Slurm clusters, or cloud execution, with shared configurations for model comparison and result export.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing models on the same benchmark tasks
• Running a small evaluation sample before a full benchmark
• Running large evaluations on a Slurm cluster

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Avalia LLMs em mais de 60 benchmarks acadêmicos (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use quando benchmarking de qualidade de modelo, comparação de modelos, relatório de resultados acadêmicos ou rastreamento de progresso de treinamento. Padrão da indústria usado por EleutherAI, HuggingFace e laboratórios principais. Suporta HuggingFace, vLLM, APIs.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluate code generation models with HumanEval, MBPP, and other benchmarks. Test generated programs, compare models across programming languages, and record pass@k scores, which estimate the chance that at least one of k solutions passes the tests.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Comparing code generation models on shared benchmarks
• Measuring correctness across programming languages
• Evaluating models trained to follow instructions

Evaluate code generation models with HumanEval, MBPP, and multilingual benchmarks. The skill guides test execution and model comparisons using pass@k, an estimate of whether at least one of k generated solutions passes the tests.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Compare code generation models on shared benchmarks
• Measure code generation performance across languages
• Evaluate instruction-tuned coding models

Design eval harnesses — task schemas, metrics, dataset versioning, eval-as-code patterns. Use when asked to "build an eval harness", "set up eval-as-code", or "version our eval datasets".

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Rigorous benchmarking of multi-agent coordination systems: experiment design, statistical analysis, human evaluation protocols, and reproducible reporting. NOT FOR ML model evaluation (use llm-evaluation-harness), A/B testing for web products, survey design, or general data science.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Builds repeatable evaluation systems with golden datasets, scoring rubrics, pass/fail thresholds, and regression reports. Use for "LLM evaluation", "testing AI systems", "quality assurance", or "model benchmarking".

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

Create evaluation drafts for existing skills using actual conversations and user corrections. Produce criteria and repeatable test cases, distinguish evidence from specification assumptions, and prepare an anonymized draft for human review.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Turn user corrections into repeatable test cases
• Compare promised behavior with actual usage
• Draft an evaluation from conversation history

Plan and improve AI agent systems by managing the information models receive. This collection covers memory, agent coordination, tool design, context compression, evaluation, and the operating controls needed for reliable autonomous workflows.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

• Planning information management for a new AI agent
• Investigating failures in long agent conversations
• Designing agent memory and multi-agent coordination

6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.

AI Engineering Toolkit workflow skill. Use this skill when the user needs 6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

한국어 요약은 준비 중입니다. 원문 설명을 표시하고 있습니다.