Run a prepared set of test cases to check agent-booster success rates and routing decisions. Optionally compare results with Gemini or Claude models, then save detailed measurements and summaries as JSON records for later verification.
Run the corpus benchmark — booster locally, optional Gemini/Sonnet/Opus baselines — and persist a verifiable measured-vs-claimed table
Review the instructions the agent will receive before you install.
Runs scripts/bench.mjs against the structural+adversarial corpus and writes per-case + summary results to docs/benchmarks/runs/. This is the verification gate that backs every measurable claim in cost-booster-edit / cost-booster-route.
bench/booster-corpus.json — confirm new cases route correctly.BENCH_ANTHROPIC=1.Run the bench from v3/ (where agent-booster resolves):
( cd v3 && node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # booster only — free, ~85 ms
( cd v3 && BENCH_LLM_BASELINE=1 node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Gemini 2.0 Flash (cheap)
( cd v3 && BENCH_LLM_BASELINE=1 BENCH_ANTHROPIC=1 \
node ../plugins/ruflo-cost-tracker/scripts/bench.mjs ) # + Sonnet 4.6 + Opus 4.7
Inspect the markdown summary printed to stdout. The gate metric is winRate (Tier 1 cases). Adversarial cases are tracked separately as escalationRate.
Persisted output lands at:
docs/benchmarks/runs/latest.json — pointer to the most recent rundocs/benchmarks/runs/<ISO-timestamp>.json — historical recordRead it back in subsequent skills (e.g. cost-report step 2 reads latest.json for live tier-spend numbers).
winRate ≥ 0.80 on Tier 1 cases (smoke step 23). Lower the threshold by editing scripts/smoke.sh.escalationRate is reported but ungated — adversarial cases are diagnostic.| Env var | Default | Purpose |
|---|---|---|
BENCH_LLM_BASELINE | unset | =1 runs the OpenAI-compat baseline |
BENCH_LLM_MODEL | models/gemini-2.0-flash | Override the OpenAI-compat model |
BENCH_LLM_BASE_URL | Gemini OpenAI shim | Override endpoint |
BENCH_ANTHROPIC | unset | =1 runs Anthropic baseline (Sonnet 4.6 + Opus 4.7) |
BENCH_ANTHROPIC_MODELS | claude-sonnet-4-6,claude-opus-4-7 | Comma-separated Claude IDs |
BENCH_OUT | timestamped file | Override output path |
BENCH_QUIET=1 | unset | Suppress markdown summary |
API keys auto-pulled from gcloud secrets (GOOGLE_AI_API_KEY, ANTHROPIC_API_KEY); override with BENCH_LLM_API_KEY / BENCH_ANTHROPIC_API_KEY.
ADR-0002 §"Decision 1" / §"Riskiest assumption" · cost-booster-edit/SKILL.md (verification table consumes this skill's output) · cost-report/SKILL.md step 2 (reads runs/latest.json).
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Compare models available through the HuggingFace router with chat-ui configuration, research missing models, and write short English descriptions of their architecture and capabilities. Add entries to both production and development configuration files and commit the changes.
Create an Architecture Decision Record, a document for recording important design choices, with the next sequential number. It generates a Markdown template, registers the record in AgentDB, and links related decisions found through search.
Build a searchable index and relationship graph from ADRs, records of architectural decisions. Import two supported document formats, refresh existing metadata without duplicate relationships, and review summaries of statuses, missing references, and inconsistencies.
Rebuild the index of ADRs, records of architectural decisions, after files or relationship lines are deleted. It clears stored records and relationships, indexes the files still present, and checks that the final record count matches.
Review code changes against accepted Architecture Decision Records, documents that capture agreed design choices. Report violations, outdated references, compliant changes, and areas without ADR coverage, then suggest code fixes or a new decision record where appropriate.
Check saved architecture decision records (ADRs) for broken references, circular replacement relationships, and inconsistent statuses. Use the results to verify imported records or stop a CI check when selected integrity rules fail.