Skip to content
cccskills
FreePublished on GitHub

benchmark-checklist

Design, review, or report a measured performance comparison. Use for benchmark methodology, speedup or regression claims, and performance-based choices; not generic performance discussion, ordinary dependency bumps, local renames, or prose edits.

The English summary is being prepared. Showing the original description.

See how to install

Files included (3)

  • SKILL.md2.8 KB
  • agents/openai.yaml229 B
  • LICENSE1.0 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

Benchmark Checklist

Vet the claim against measured work. Use the existing benchmark harness and stay within the task's authority for runs, instrumentation, edits, and publication.

  1. State the claim, metric, units, workload, and comparison baseline. Read what the harness times, counts, and excludes. A requested quick estimate may use one run, labeled as such, but must still check errors and completed work. For that estimate, use the quick-estimate report below; steps 2–5 apply to comparisons. Choosing between options requires a comparison, not a quick estimate.
  2. Make the comparison fair: production builds and settings, matched versions, data, concurrency, cache state, warmup, and resources. Check machine load and load-generator saturation. Name untuned sides; do not select a winner from an unfair comparison.
  3. Verify correct output and count errors, timeouts, retries, and rejected work. Confirm work completed inside the timed region: await promises, consume lazy results, and verify requests, rows, or bytes actually reached their target.
  4. Identify the limiter with evidence; keep profiling overhead out of reported timings. Check physical throughput bounds and how much time the changed component could save. Investigate implausible results before reporting them.
  5. For a formal comparison, repeat each side at least five times, interleaving sides to reduce drift. Report the median and range using the harness's statistics where available. Treat differences within noise as no measurable difference. Relate microbenchmarks to the end-to-end path users wait on.

For a requested quick estimate, report the observed value, units, environment, reproduction command, errors and completed work, and label it as a single run without a comparison verdict. Missing correctness or work evidence makes even a quick estimate inconclusive.

For a formal comparison, finish with faster, slower, no measurable difference, or inconclusive; include run count, units, spread, environment, limiter evidence, and reproduction command. Mark missing correctness, work, fairness, or limiter evidence as inconclusive. Separate a measured result from an untested explanation. Stop when the claim is supported at its stated scope or the remaining evidence gap is explicit.

Adapted from Lauren Tan's pstack benchmark-checklist, under the MIT license.

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

adhx

Free

Read public X/Twitter post URLs as structured evidence through the ADHX API. Use for relevant user-provided or research-discovered x.com, twitter.com, or adhx.com links. ADHX does not search X, and long-form Article content may be incomplete.

The English summary is being prepared. Showing the original description.

bioinformatist/codex-base9Updated Oct 16, 2026

Trace what an actual code change could break beyond its diff when interfaces, serialized data, database schemas, configuration, caches, lifecycle ordering, or cross-language consumers are involved. Use for explicit impact analysis; not local renames, prose edits, ordinary dependency bumps, or generic architecture discussion.

The English summary is being prepared. Showing the original description.

bioinformatist/codex-base9Updated Oct 16, 2026

Shared vocabulary for designing deep modules, interfaces, seams, adapters, leverage, and locality. Use when designing or reshaping module boundaries, making code more testable, or evaluating interface depth.

The English summary is being prepared. Showing the original description.

bioinformatist/codex-base9Updated Oct 16, 2026

Plan or implement retirement of a system, API, feature, persisted configuration, or database shape. Use for compatibility cutovers, explicit deprecation review, or an explicit unreleased replacement decision, including determining whether consumers or data exist; not ordinary dependency bumps, local renames, prose edits, or generic architecture discussion.

The English summary is being prepared. Showing the original description.

bioinformatist/codex-base9Updated Oct 16, 2026

Disciplined diagnosis loop for hard bugs, regressions, flaky failures, and performance problems with unclear cause. Use for root-cause debugging after a concrete symptom exists; do not use for routine implementation or speculative cleanup.

The English summary is being prepared. Showing the original description.

bioinformatist/codex-base9Updated Oct 16, 2026

Route requests for current library, framework, SDK, API, CLI, or cloud-service documentation through available documentation tools and primary sources.

The English summary is being prepared. Showing the original description.

bioinformatist/codex-base9Updated Oct 16, 2026

See all skills by bioinformatist

Report a problem with this skill