본문으로 건너뛰기
cccskills
무료GitHub에 공개

cost-benchmark

Run a prepared set of test cases to check agent-booster success rates and routing decisions. Optionally compare results with Gemini or Claude models, then save detailed measurements and summaries as JSON records for later verification.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

원문원문 설명 보기

Run the corpus benchmark — booster locally, optional Gemini/Sonnet/Opus baselines — and persist a verifiable measured-vs-claimed table

ruvnet/ruflo
연동 도구OpenAIGemini
설치 방법 보기

이럴 때 유용합니다

  • Checking success rates before a release
  • Verifying routing after adding test cases
  • Checking performance claims against measurements
  • Comparing costs with external model baselines

포함된 파일(1)

  • SKILL.md2.9 KB

SKILL.md(원문)

설치하기 전에 에이전트에게 전달되는 지시 내용을 확인할 수 있습니다.

Cost Benchmark

Runs scripts/bench.mjs against the structural+adversarial corpus and writes per-case + summary results to docs/benchmarks/runs/. This is the verification gate that backs every measurable claim in cost-booster-edit / cost-booster-route.

When to use

  • Before publishing a release — verify booster win rate didn't regress.
  • After expanding bench/booster-corpus.json — confirm new cases route correctly.
  • When auditing a "claimed upstream" tag — flip it to "verified" once the bench supports it.
  • On a cost question ("is Sonnet 4.6 cheaper than Opus 4.7 for these tasks?") — re-run with BENCH_ANTHROPIC=1.

Steps

  1. Run the bench from v3/ (where agent-booster resolves):

    ( cd v3 && node ../plugins/ruflo-cost-tracker/scripts/bench.mjs )                  # booster only — free, ~85 ms
( cd v3 && BENCH_LLM_BASELINE=1 node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Gemini 2.0 Flash (cheap)
( cd v3 && BENCH_LLM_BASELINE=1 BENCH_ANTHROPIC=1 \
     node ../plugins/ruflo-cost-tracker/scripts/bench.mjs )                          # + Sonnet 4.6 + Opus 4.7

  2. Inspect the markdown summary printed to stdout. The gate metric is winRate (Tier 1 cases). Adversarial cases are tracked separately as escalationRate.

  3. Persisted output lands at:

    • docs/benchmarks/runs/latest.json — pointer to the most recent run
    • docs/benchmarks/runs/<ISO-timestamp>.json — historical record

  4. Read it back in subsequent skills (e.g. cost-report step 2 reads latest.json for live tier-spend numbers).

Smoke gates

  • winRate ≥ 0.80 on Tier 1 cases (smoke step 23). Lower the threshold by editing scripts/smoke.sh.
  • escalationRate is reported but ungated — adversarial cases are diagnostic.

Env overrides

Env varDefaultPurpose
BENCH_LLM_BASELINEunset=1 runs the OpenAI-compat baseline
BENCH_LLM_MODELmodels/gemini-2.0-flashOverride the OpenAI-compat model
BENCH_LLM_BASE_URLGemini OpenAI shimOverride endpoint
BENCH_ANTHROPICunset=1 runs Anthropic baseline (Sonnet 4.6 + Opus 4.7)
BENCH_ANTHROPIC_MODELSclaude-sonnet-4-6,claude-opus-4-7Comma-separated Claude IDs
BENCH_OUTtimestamped fileOverride output path
BENCH_QUIET=1unsetSuppress markdown summary

API keys auto-pulled from gcloud secrets (GOOGLE_AI_API_KEY, ANTHROPIC_API_KEY); override with BENCH_LLM_API_KEY / BENCH_ANTHROPIC_API_KEY.

Cross-references

ADR-0002 §"Decision 1" / §"Riskiest assumption" · cost-booster-edit/SKILL.md (verification table consumes this skill's output) · cost-report/SKILL.md step 2 (reads runs/latest.json).

리뷰

아직 리뷰가 없습니다. 사용해 본 소감을 남겨 주세요.

댓글

로그인 없이 쓸 수 있습니다. 사용 후기, 질문, 작동하지 않은 환경 등을 편하게 남겨 주세요.

아직 댓글이 없습니다. 첫 댓글을 남겨 보세요.

보내면 바로 공개됩니다. 이메일 주소나 전화번호 같은 개인 정보는 쓰지 마세요. URL은 2개까지입니다.

0 / 1,000자

이 양식은 reCAPTCHA로 보호되며 Google 개인정보처리방침 및 서비스 약관이 적용됩니다.

같은 저장소의 스킬

add-model-descriptions

무료

Compare models available through the HuggingFace router with chat-ui configuration, research missing models, and write short English descriptions of their architecture and capabilities. Add entries to both production and development configuration files and commit the changes.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

  • Adding newly available router models to chat-ui
  • Writing concise descriptions of model capabilities
  • Updating production and development model entries together
ruvnet/ruflo7.7만2026년 10월 18일 업데이트

adr-create

무료

Create an Architecture Decision Record, a document for recording important design choices, with the next sequential number. It generates a Markdown template, registers the record in AgentDB, and links related decisions found through search.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

  • Record the reasoning behind adopting a technology
  • Start an ADR for an API or data model change
  • Document an infrastructure decision
ruvnet/ruflo7.7만2026년 10월 18일 업데이트

adr-index

무료

Build a searchable index and relationship graph from ADRs, records of architectural decisions. Import two supported document formats, refresh existing metadata without duplicate relationships, and review summaries of statuses, missing references, and inconsistencies.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

  • Index ADRs imported from another project
  • Refresh a graph that differs from files on disk
  • Set up ADR tracking in an existing codebase
ruvnet/ruflo7.7만2026년 10월 18일 업데이트

adr-reindex

무료

Rebuild the index of ADRs, records of architectural decisions, after files or relationship lines are deleted. It clears stored records and relationships, indexes the files still present, and checks that the final record count matches.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

  • Rebuild the index after deleting an ADR file
  • Remove stale relationships after editing an ADR
  • Reconcile the ADR index on a schedule
ruvnet/ruflo7.7만2026년 10월 18일 업데이트

adr-review

무료

Review code changes against accepted Architecture Decision Records, documents that capture agreed design choices. Report violations, outdated references, compliant changes, and areas without ADR coverage, then suggest code fixes or a new decision record where appropriate.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

  • Check a pull request before merging
  • Review major code changes against agreed design decisions
  • Find references to superseded ADRs
ruvnet/ruflo7.7만2026년 10월 18일 업데이트

adr-verify

무료

Check saved architecture decision records (ADRs) for broken references, circular replacement relationships, and inconsistent statuses. Use the results to verify imported records or stop a CI check when selected integrity rules fail.

한국어 요약은 준비 중입니다. 영어 요약을 표시하고 있습니다.

  • Checking ADR integrity after import
  • Blocking CI when ADR issues are found
  • Reviewing ADR data before a release
ruvnet/ruflo7.7만2026년 10월 18일 업데이트

ruvnet의 스킬 모두 보기

이 스킬의 문제 신고