Evaluate BMad skills using your agent's configured runner. Compare results with the model alone or another version, grade outputs against criteria, and test whether descriptions trigger correctly, preserving transcripts and evidence for each run.
Runs the evals of a BMad skill through the agent harness the user works in and reports what they show: baseline against the bare model, variant against a stripped or prior version, quality against a rubric, and trigger accuracy of the description with an optional optimization loop. Use when the user asks to evaluate, benchmark, test the triggers of, grade, or compare versions of a skill, or when the bmad-toolsmith skill asks for an eval of a skill it built.
Review the instructions the agent will receive before you install.
You run a skill's evals and report what they say. Cite specific findings, flag evals that pass for trivial reasons, and never widen a tolerance to make a run look like it succeeded. No model or harness is named in this skill: the harness a project's evals run through is recorded in this skill's customization at the first run.
| Mode | Question it answers | Script / reference |
|---|---|---|
| baseline | Does the skill beat the bare model on the same input? | references/eval-format.md, scripts/run_evals.py |
| variant | Does a section earn its place, or does a stripped version do as well? | references/eval-format.md, scripts/run_evals.py |
| quality | Does the output meet the named rubric? | references/grader.md, references/eval-format.md |
| trigger | Does the description fire on the right queries and stay quiet on the rest? | references/harness.md, scripts/run_triggers.py |
Baseline runs every case with the skill staged and with nothing staged. Variant runs the skill against the one at --variant-path. Quality grades one config against its rubric. Trigger measures real firing; references/description-optimization.md improves the description across rounds.
A case is input + rubric + optional state_prefix + optional fixture files; the state_prefix is a bracketed prime prepended to the input that places the skill mid-workflow in one shot. Cases live in <skill>/evals/cases.json, trigger queries as {query, should_trigger} in <skill>/evals/triggers.json. The full shape and the strong-versus-weak expectation taxonomy are in references/eval-format.md. A skill-creator evals.json converts with uv run {skill-root}/scripts/convert_cases.py <evals.json> --output <cases.json> (--to skill-creator reverses it).
uv run {project-root}/_bmad/scripts/resolve_config.py --project-root {project-root} --key core.output_folder and customization with uv run {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --project-root {project-root} --key workflow. Absent keys are fine. {reports_folder} is {workflow.reports_folder} with {output_folder} filled in.<skill-path>/SKILL.md exists; halt with a clear error if not.workflow.harness.command came back empty, work out the harness facts for the CLI you are running in per references/harness.md, prove them on one case, and record them by invoking the bmad-customize skill (install: npx skills add bmad-code-org/BMAD-METHOD --skill bmad-customize); the runner reads them from there. When nobody is at the keyboard, stop and show the table to record instead.<skill-path>/evals/, then <project-root>/evals/<skill-name>/, then anywhere under <project-root>/evals/. If nothing is found, halt and say so; the runner does not invent cases.The user asks for a run in <mode> mode on <skill> (the directory holding SKILL.md), with a variant skill for variant mode. The project root is the first ancestor of the skill holding _bmad/ or .git/; the output dir is {reports_folder} when BMad is set up, else ~/bmad-evals/. Every run is its own timestamped folder there, so runs never collide. Each case runs in a clean room, a temporary folder outside the project with a copy of the skill staged into its workspace and an environment built from scratch, so host config, prior runs, the project's instruction files and its installed skills cannot bias the result. The runner reads the harness from customization itself; in a project without BMad, pass --harness <json> with the same keys.
Baseline, variant and quality:
uv run {skill-root}/scripts/run_evals.py \
--cases <cases-file> --skill-path <skill> --output-dir <dir> \
--mode quality|baseline|variant [--variant-path <skill>] \
--label <skill-name>-<mode> [--runs N]
--runs defaults to 1 for a quick look; a result counts as settled only at 3 or more. Case subsets, timeouts and workers are in the script docstrings. Exit 0 is a complete run; 1 means a case errored or timed out or the command was not found, 3 that no harness is recorded; a trigger query failing its threshold is a result, not an error.
Trigger:
uv run {skill-root}/scripts/run_triggers.py \
--skill-path <skill> --queries <queries-file> --output-dir <dir> \
[--runs-per-query N]
For quality, spawn the grader in references/grader.md per case with its rubric, transcript path, artifacts dir (the case's cwd/) and grading_path of <case-folder>/grading.json. Relay its rubric feedback. If a grader errors, mark the case grading_error, never a default verdict.
When --runs is greater than one, uv run {skill-root}/scripts/aggregate_benchmark.py --baseline <run-dir>/<config-a> --variant <run-dir>/<config-b> gives mean, sample standard deviation, min, max and delta (--runs <run-dir>/<config> for one config).
When a run comes back weak and the user wants the skill improved from it, follow references/self-improvement.md.
Each run is a dated folder under the output dir, with run.json naming the harness command; each case folder, and each trigger attempt, holds its prompt, the transcript (what the harness printed), stderr.txt, cwd/ (the workspace after the run), timing.json (written the moment the invocation ends) and grading.json when quality ran. Never delete, overwrite or rotate a run folder. Tell the user where it is when you finish.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Get guidance on using installed BMad modules and choosing the next skill for a task. The skill also handles installation setup, updates, repairs, module additions, artifact migration, and selecting the active initiative being worked on.
Revisit a recent answer, plan, draft, or decision using selected critique methods. The skill offers relevant approaches, explains what they reveal, and proposes improvements that are applied only after the user accepts them.
Work with Mary, a business analyst persona, to research markets, examine competitors, and clarify requirements. Use evidence-based discussion to turn vague needs into concrete specifications that can guide the next steps of a project.
Turn product requirements and user experience plans into technical architecture with Winston, a system architect persona. Discuss technology choices and their trade-offs, with an emphasis on established tools, developer productivity, and designs that can be implemented.
Implement approved development stories and code changes with Amelia, a senior software engineer persona. Follow a test-first process: write a failing test, make it pass, then improve the code while checking every acceptance criterion.
Work with John, a product manager persona, to clarify requirements through user interviews and stakeholder alignment. Turn a product vision into a PRD, a product requirements document, and small, validated increments for development.