Results for “benchmark”

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarksUse when "agent testing, agent evaluation, benchmark agents, agent reliability, test agent, testing, evaluation, benchmark, agents, reliability, quality" mentioned.

The English summary is being prepared. Showing the original description.

Use when benchmarking, profiling, or A/B-testing CUDA kernels or end-to-end perf in the imp inference engine on RTX 5090 (sm_120), including refreshing tests/perf_baseline.json or publishing numbers to docs/BENCHMARKS.md and the README. Triggers on "benchmark kernel", "profile cuda", "ncu", "nsys", "kernel timing", "kernel sum", "occupancy", "bandwidth bound", "compute bound", "roofline", "perf baseline", "is this regression real", "decode dropped", "aggregate throughput", "two-image A/B", "prefill kernel A/B". Do NOT use for writing/optimizing kernel code (sm120-cuda-expert) or output-quality checks (check-degeneration).

The English summary is being prepared. Showing the original description.

Skippable Phase 0 of the Gaia evidence verification pipeline. Use only on declared need for higher-quality Stage-2 evidence: a promotion candidate, or a skill that gaia-meta-sweep flagged as under-evidenced. Given a skill id, source repo/context, and needed evidence types (benchmark-result, arxiv, peer-review, social-signal), it runs Firecrawl searches, scrapes top hits to markdown, and appends discovered sources into Phase 1 inputs. Missing benchmark evidence is recorded as benchmark-source candidates first, not direct scoring rows. Trigger phrases: "discover evidence", "find new evidence", "Phase 0", "run ev-discovery", "search for benchmarks/papers", "under-evidenced", "promotion candidate needs stronger proof". This is the ONLY ev-* skill that searches the web for NEW evidence; other ev-* skills aggregate or verify known sources. Requires Firecrawl to be usable (`firecrawl --status` authenticated OR FIRECRAWL_API_KEY set); skips gracefully when neither holds.

The English summary is being prepared. Showing the original description.

Align, run, audit, and publish GEODE benchmark comparisons against Agent-World arXiv 2604.18292v1 across MCP-Mark, BFCL V4, and tau2-bench. Use for Agent-World or AgentWorld comparison requests, cross-suite agent benchmark design, Tau2 comparator/profile diagnosis, paired thin-runtime-versus-GEODE experiments, mean-accuracy@8 replication, or benchmark comparability and artifact reviews.

The English summary is being prepared. Showing the original description.

Avalia LLMs em mais de 60 benchmarks acadêmicos (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use quando benchmarking de qualidade de modelo, comparação de modelos, relatório de resultados acadêmicos ou rastreamento de progresso de treinamento. Padrão da indústria usado por EleutherAI, HuggingFace e laboratórios principais. Suporta HuggingFace, vLLM, APIs.

The English summary is being prepared. Showing the original description.

Use when designing evaluation suites for agentic systems, choosing agent benchmarks, measuring multi-agent performance, or critiquing single-score evaluation of agents. Based on arXiv:2605.26112v1. Trigger phrases: "evaluate my agent", "benchmark agent", "measure agent reliability", "pass^k evaluation", "agent metrics beyond success rate", "trajectory quality", "memory hygiene metric", "longitudinal agent eval", "why my agent scores well on benchmarks but fails in production".

The English summary is being prepared. Showing the original description.

This skill details how to conduct cloud security audits using Center for Internet Security benchmarks for AWS, Azure, and GCP. It covers interpreting CIS Foundations Benchmark controls, running automated assessments with tools like Prowler and ScoutSuite, remediating failed controls, and maintaining continuous compliance monitoring against CIS v5 for AWS, v4 for Azure, and v4 for GCP.

The English summary is being prepared. Showing the original description.

bu skill details how to conduct cloud security audits using Center for Internet Security benchmarks for AWS, Azure, and GCP. It covers interpreting CIS Foundations Benchmark controls, running automated assessments with tools like Prowler and ScoutSuite, remediating failed controls, and maintaining continuous compliance monitoring against CIS v5 for AWS, v4 for Azure, and v4 for GCP.

The English summary is being prepared. Showing the original description.

Autonomous model validation and benchmarking. Investigates any ML model (LLM, image gen, TTS, time series, etc.), runs it on GPU cloud, evaluates quality and performance, and generates HTML reports. Use when user asks to verify, benchmark, evaluate, or test a model. Triggers on "verify model", "benchmark", "evaluate model", "test model", "run benchmark", "model evaluation", "モデルを検証", "ベンチマーク", "モデルを試して".

The English summary is being prepared. Showing the original description.

This skill details how to conduct cloud security audits using Center for Internet Security benchmarks for AWS, Azure, and GCP. It covers interpreting CIS Foundations Benchmark controls, running automated assessments with tools like Prowler and ScoutSuite, remediating failed controls, and maintaining continuous compliance monitoring against CIS v5 for AWS, v4 for Azure, and v4 for GCP.

The English summary is being prepared. Showing the original description.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

The English summary is being prepared. Showing the original description.

Cross-model benchmark for gstack skills. Runs the same prompt through Claude, GPT (via Codex CLI), and Gemini side-by-side — compares latency, tokens, cost, and optionally quality via LLM judge. Answers "which model is actually best for this skill?" with data instead of vibes. Separate from /benchmark, which measures web page performance. Use when: "benchmark models", "compare models", "which model is best for X", "cross-model comparison", "model shootout". (gstack) Voice triggers (speech-to-text aliases): "compare models", "model shootout", "which model is best".

The English summary is being prepared. Showing the original description.

When the user requests information about academic benchmarks, datasets, or research papers, particularly in machine learning, deep learning, or logical reasoning domains. This skill enables systematic research of academic benchmarks by searching web sources, downloading and analyzing arXiv papers, extracting key metadata (number of tasks, training availability, difficulty levels), and compiling comparative summaries. It triggers on requests involving dataset comparisons, benchmark analysis, or academic paper research for table creation.

The English summary is being prepared. Showing the original description.

This skill details how to conduct cloud security audits using Center for Internet Security benchmarks for AWS, Azure, and GCP. It covers interpreting CIS Foundations Benchmark controls, running automated assessments with tools like Prowler and ScoutSuite, remediating failed controls, and maintaining continuous compliance monitoring against CIS v5 for AWS, v4 for Azure, and v4 for GCP.

The English summary is being prepared. Showing the original description.

Helps build Go tests using a test-first workflow. Covers grouped test cases, HTTP response checks, performance benchmarks, fuzzing to explore generated inputs, and coverage reports showing which code was exercised by tests.

• Developing Go functions with tests written first
• Adding normal and error cases to existing tests
• Measuring execution time and memory allocations

Turn speedup requests into a measured optimization process with limits on time, cost, and variants. Compare each change against a baseline and correctness checks, then retain the fastest safe option with reproducible commands and measurements.

• Speeding up a slow operation using measured comparisons
• Comparing implementation variants under the same input conditions
• Evaluating latency, throughput, or cost per run

Turn scored competitor profiles into a report for strategic decisions. Organize market maps, comparison tables, unoccupied positions, threats, and prioritized recommendations so leaders can decide whom to compete with and how to differentiate.

• Preparing a competitive report for leadership or a board
• Combining completed competitor profiles
• Assessing room for differentiation and competitive threats

Measure and optimize claude-flow v3 startup, search, attention processing, and memory use. Compare benchmarks against performance targets, track changes over time, and identify regressions to guide further optimization.

• Compare performance before and after v3 optimization
• Evaluate vector search speed and memory use
• Track performance regressions over time

Measure throughput, response times, and resource use for systems and groups of agents. Use baseline comparisons, load tests, and performance criteria to identify slowdowns, assess behavior under pressure, and summarize results with recommendations.

• Comparing performance before and after a code change
• Finding system limits through increasing load
• Checking coordination and load balancing across agents

Investigate failed GAIA benchmark questions by inspecting execution records and classifying causes such as missing tools, reasoning errors, or answer extraction bugs. Follow targeted fixes, verify them through another run, and save findings for future debugging.

• Investigate an incorrect answer for a specific task
• Diagnose a question that timed out
• Investigate a drop in benchmark pass rate

Measure and optimize claude-flow v3 startup, search, attention processing, and memory use. Compare benchmarks against performance targets, track changes over time, and identify regressions to guide further optimization.

• Compare performance before and after v3 optimization
• Evaluate vector search speed and memory use
• Track performance regressions over time

Benchmark how CodeGraph affects an agent investigating a real codebase. Choose a version, language, repository, and execution mode, then compare cost, tool calls, file reads, and answer correctness with and without CodeGraph.

• Evaluating a local development build
• Auditing a published CodeGraph version
• Comparing investigation cost and file reads

Add a CUDA/C++ GPU operation to sgl-kernel using AOT, meaning compilation before use. Follow the implementation through PyTorch registration, Python API exposure, SGLang integration, correctness tests, and benchmarks that account for cache reuse.

• Adding a CUDA/C++ operation to sgl-kernel
• Registering an operation with PyTorch and exposing a Python API
• Testing numerical results and invalid inputs

Review SGLang Cookbook pull requests for configuration errors, documentation mismatches, benchmark issues, and broken links. Get findings for each file and an overall recommendation on whether to approve, request changes, or block the contribution.

• Reviewing a new model's Cookbook contribution
• Checking documentation against generated launch commands
• Checking benchmark definitions and measurement consistency