gz-adr-evaluate

Purpose

Structured quality evaluation for ADRs and their OBPI decompositions. This skill provides rubrics, challenge protocols, and a red-team prompt that form a blocking QC step between ADR authoring and human proposal/defense review.

It can be invoked for newly authored ADRs or for retroactive evaluation of an existing ADR package.

Common Rationalizations

Thought	Reality
"The ADR looks comprehensive, it'll score well"	Comprehensiveness does not equal quality. The rubric measures decision justification, not document length.
"The CLI gave a high score, no need for manual review"	The CLI uses heuristics that produce false negatives. Manual review supersedes CLI pre-screen.
"This is a retroactive evaluation, I'll be lenient"	The rubric is the rubric. Retroactive evaluation with relaxed standards produces misleading scorecards.
"The red-team challenges don't apply to this simple ADR"	Every challenge must be engaged. N/A is not acceptable. Simple ADRs often have unexamined assumptions.
"One dimension scored 1, but the overall weighted score is above 3.0"	Any dimension scoring 1 must be revised regardless of the weighted total.
"I'll skip the OBPI scoring since the ADR scored well"	ADR quality and OBPI quality are independent assessments. Both are required.

Red Flags

• EVALUATION_SUBSTANCE.md written without reading the actual ADR document
• A judged scorecard hand-written into EVALUATION_SCORECARD.md, which gz adr evaluate regenerates and will destroy (GHI #769)
• CLI pre-screen score accepted without manual verification (Steps 3–4 skipped)
• Manual score differs from CLI score but no explanation of the heuristic mismatch
• Red-team challenges marked N/A instead of engaged
• ADR proceeds to proposal/defense with a NO GO or CONDITIONAL GO verdict

---

Persona

Active driver: pipeline-orchestrator — read .gzkit/personas/pipeline-orchestrator.md and adopt its behavioral identity before executing this skill. Evaluation is a sequenced ceremony (score dimensions → engage red-team → render verdict); step-discipline keeps the rubric honest. Evaluation is independent judgment, not confirmation. Score what you read, not what you hope.

Persona Dispatch

Evaluation is the highest-risk read-only judgment ceremony — the rubric's 8 dimensions split cleanly along the persona functional boundaries, and a single driver scoring its own scoring is the precise optimistic-bias defect spec-reviewer's anti-traits name. Dispatch the following personas to produce independent dimension scores the driver synthesizes:

Persona	Function in this ceremony	Invoked at
spec-reviewer	Scores spec/requirement dimensions of the rubric: requirement clarity, target-scope falsifiability, OBPI decomposition coverage, REQ-trace integrity — anything that asks "is the spec honest, falsifiable, traceable?"	Steps 3–4 (manual rubric scoring)
quality-reviewer	Scores architectural dimensions: decision justification, alternatives considered, SOLID-analogues for ADR design, single-responsibility per ADR, decision boundaries, maintainability of the proposed surface	Steps 3–4 (manual rubric scoring)
narrator	Composes the operator-facing scorecard and frames red-team challenge findings — each challenge's finding rendered as evidence-to-decision, not as raw analysis	Step 5 (red-team challenges, if invoked) and final scorecard render

Personas not dispatched: implementer (evaluation is pre-implementation review — no code exists to write or evaluate).

The receipt machinery that would attest these dispatches is not yet authored (GHI #846; ADR-pool.obpi-pipeline-dispatch-attestation is Superseded into ADR-0.0.73, which delivered no receipts). This ceremony does have a dispatch channel: gzkit.adr_eval_dispatch reports NOT DISPATCHED absent a receipt and never infers dispatch from the presence of scores (GHI #770). See § Degraded mode.

Degraded mode — when dispatch cannot run (binding, GHI #770)

Dispatch is sometimes genuinely unavailable — a session that forbids subagents, a headless or cron run, a harness without the Agent tool. Until GHI #770 there was no compliant path for those runs and no way to mark the output, so the mandate was simply skipped: on 2026-08-07 an evaluation of ADR-0.35.0 ran single-driver, produced 8 dimension scores and a GO verdict, and nothing recorded that the mandated dispatch had not happened. An un-compliable mandate gets worked around.

The degraded mode is therefore declared and legitimate, never silent:

1. Running single-driver is permitted. Do not fabricate a dispatch, and do not abandon the evaluation.
2. The scorecard states it mechanically. render_scorecard_markdown always emits a --- Persona Dispatch --- channel: one row per mandated persona, NOT DISPATCHED / no dispatch receipt recorded absent a receipt, plus a DISPATCH MODE: SINGLE-DRIVER verdict. This is not something you remember to write — the renderer emits it unconditionally, and tests/test_adr_eval_dispatch.py fails closed if a dispatched and an undispatched scorecard could ever render identically.
3. A single-driver scorecard is not an independent review. Say so when relaying the verdict; do not present it as one.

A dispatch is credited only from a recorded receipt (gzkit.adr_eval_dispatch), never inferred from the presence of scores — the same discipline the substance channel applies to judge verdicts (GHI #624). Nothing emits that receipt yet, so every scorecard truthfully reads SINGLE-DRIVER until that machinery is authored (GHI #846); the channel populates then with no change here.

Persona doctrine reference: ADR-0.0.11-persona-driven-agent-identity-frames (Validated).

Trust Model

Layer 1 - Evidence Gathering. This tool reads ADR documents and produces evaluation scorecards. It does not modify ADR or brief content.

• Reads: ADR document, OBPI briefs, evaluation framework template
• Writes: EVALUATION_SUBSTANCE.md in the ADR directory (the judge's file; EVALUATION_SCORECARD.md is machine-owned and regenerated by the CLI)
• Books: one adr-evaluation ledger event per successful gz adr evaluate run (ADR-0.0.26, OBPI-0.0.26-01)
• Does NOT modify: ADR content, brief content, or registries

---

Invocation

/gz-adr-evaluate ADR-X.Y.Z             # evaluate a specific ADR
/gz-adr-evaluate ADR-X.Y.Z --red-team  # include the 10-challenge red-team protocol

Arguments:

Argument	Required	Description
adr_id	Yes	ADR identifier (for example ADR-0.13.0)
--red-team	No	Include the 10-challenge adversarial review protocol

---

When to Use

• After drafting a new ADR and its OBPIs
• Before a Draft ADR goes to human proposal/defense review
• When benchmarking the quality of an existing ADR package
• When you want a structured red-team pass against scope, evidence, and decomposition

---

Procedure

Step 1: Run CLI deterministic scoring (pre-screen)

uv run gz adr evaluate ADR-X.Y.Z

This produces structural quality scores (8 ADR dimensions, OBPI scores) and writes an initial EVALUATION_SCORECARD.md.

The CLI score is a pre-screen, never a gate. The CLI uses pattern-matching heuristics (keyword detection, section structure) that can under-score or over-score dimensions when an ADR distributes content across non-standard sections. Common false negatives:

• Problem Clarity: CLI looks for before/after language in Intent — misses ADRs that carry depth in Rationale or Agent Context Frame
• Decision Justification: CLI looks for numbered items in Decision — misses bullet-point decisions or justification distributed across Decision + Alternatives Considered

Mandatory next step: Always proceed to Step 2 regardless of CLI verdict. The agent reads the ADR, analyzes every CLI dimension score for false-positive and false-negative artifacts, and produces the authoritative manual scorecard that supersedes the CLI pre-screen.

Step 2: Locate the ADR and its OBPIs

1. Resolve the ADR document under docs/design/adr/**/ADR-X.Y.Z-*/ADR-X.Y.Z-*.md
2. List all OBPI briefs in obpis/ (preferred) or briefs/ (legacy)
3. Read the evaluation framework from assets/ADR_EVALUATION_FRAMEWORK.md
4. Read the CLI-generated EVALUATION_SCORECARD.md — note each dimension score and the CLI's stated findings for analysis in Step 3

Step 3: Score ADR Quality (Part 1 - 8 Dimensions)

Read Part 1 of the framework and score the ADR on each dimension (1-4 scale):

#	Dimension	Weight
1	Problem Clarity	15%
2	Decision Justification	15%
3	Feature Checklist Completeness	15%
4	OBPI Decomposition Quality	15%
5	Lane Assignment Correctness	10%
6	Scope Discipline	10%
7	Evidence Requirements	10%
8	Architectural Alignment	10%

For each dimension, work through the checklist items in the framework and score based on how many checklist items pass with path-level evidence.

CLI reconciliation (mandatory per dimension): Compare the agent's manual score against the CLI pre-screen score. When they differ, the scorecard MUST document why — naming the specific CLI heuristic that misfired (false positive or false negative) and the evidence that justifies the override. The manual score is authoritative; the CLI score is recorded for traceability.

Step 4: Score OBPI Quality (Part 2 - 5 Dimensions)

For each OBPI, score on 5 dimensions (1-4 scale):

Dimension	Question
Independence	Can this OBPI be completed without waiting for others?
Testability	Can completion be verified with commands?
Value	What concrete capability would be lost if this OBPI were removed?
Size	Is this a 1-3 day work unit?
Clarity	Could a different agent implement this without ambiguity?

Step 5: Run Red-Team Challenges (Optional - Part 3)

If --red-team is specified, or if the evaluator wants stronger adversarial review, work through all 10 structured challenges from the framework.

Every challenge must be engaged. N/A is not acceptable.

Step 6: Determine Verdict

Apply the framework thresholds:

ADR Weighted Total	Verdict
>= 3.0	GO - Ready for proposal/defense review
2.5 to < 3.0	CONDITIONAL GO - Address weaknesses, then re-evaluate
< 2.5	NO GO - Structural revision required

OBPI threshold: Average >= 3.0 per OBPI. Any OBPI scoring 1 on any dimension must be revised.

Red-team threshold: <= 2 failures = GO, 3-4 = CONDITIONAL GO, >= 5 = NO GO.

Step 7: Record Substance Scorecard

Write EVALUATION_SUBSTANCE.md in the ADR directory using the summary template from the framework.

Never write EVALUATION_SCORECARD.md by hand (GHI #769). That path is machine-owned: gz adr evaluate rewrites it wholesale on every run, so a judged scorecard authored there is destroyed by the next invocation. The two files measure different things and GHI #624 forbids compositing them — the split makes that separation structural instead of a warning banner the writer ignores. The closeout NO-GO gate reads this file first, so a recorded NO GO now survives regeneration.

Cite the CLI pre-screen for traceability rather than restating it as your own finding. Include:

• CLI pre-screen verdict and weighted total (for traceability)
• All ADR dimension scores with weighted totals and rationale — when the manual score differs from the CLI score, state the CLI score, the manual score, and the specific heuristic mismatch that caused the divergence
• All OBPI dimension scores with averages
• Red-team challenge results when run
• Overall verdict (GO / CONDITIONAL GO / NO GO)
• Action items for any deficiencies

Low-Score Footer Guidance

Append the footer under the condition gz validate --evaluation-justify-binding fails closed on (ADR-0.0.26 Decision 2): any dimension scores below low_score_threshold (it ships as < 3.0; the file below, never this line, is authoritative), or at least red_team_count_threshold red-team challenges fired — both read from data/eval_feedback_thresholds.json. The weighted total is not the trigger: a single dimension below threshold fails the gate while the total stays healthy. The footer requires an OBPI brief under the evaluated ADR to exist, and names it:

> Consider: uv run -m gzkit justify OBPI-<X.Y.Z>-<NN> --save

When the ADR has no OBPI brief yet there is no footer to append; the evaluation is answered instead on the ADR itself as a draft slug — its id with dots as dashes:

> Consider: uv run -m gzkit justify --draft "<what the evaluation found>" --draft-slug adr-<x>-<y>-<z> --save

Only these two forms discharge the gate: the walkthrough's own frontmatter must name the evaluated ADR, an OBPI under it, or its draft slug (operator ruling 2026-09-14, GHI #996). A walkthrough on a tracking GHI does NOT — the GHI's link to the ADR lives on GitHub, which the gate does not consult — so never route the footer to one.

Amendment record (REQ-0.0.19-04-05, 2026-09-14, GHI #996). That attested REQ asserts this block cites the < 3.0 threshold, requires a tracking GHI or OBPI to exist, and suggests uv run -m gzkit justify. All three still hold. What the operator's ruling retired is the weighted total as the trigger and the tracking-GHI route: the footer now fires on the gate's own condition and names only an OBPI, because a GHI walkthrough never discharged the gate it answers.

A trigger means the ADR's structural weakness will leave implementing agents guessing on at least one OBPI. The pre-execution walkthrough (gz-justify skill) surfaces the hidden ambiguity before promotion, and the lifecycle refuses to advance the ADR past Pending until a qualifying walkthrough exists. Skipping this footer on a triggered evaluation is the adjacent rationalization pattern the walkthrough exists to close.

Step 8: Gate Decision

• GO: proceed to human proposal/defense review
• CONDITIONAL GO: revise the ADR or OBPIs, then re-run evaluation
• NO GO: return to authoring; do not proceed to proposal/defense

---

External Red-Team Review

For adversarial review by a separate model:

1. Read the red-team model prompt from Part 4 of the framework
2. Copy the prompt and append the target ADR document
3. Send it to a separate model
4. Review the challenge report and address findings before proposal/defense

---

Assets

• Evaluation Framework: assets/ADR_EVALUATION_FRAMEWORK.md
  • Part 1: ADR quality rubric (8 weighted dimensions)
  • Part 2: OBPI quality rubric (5 dimensions)
  • Part 3: Red-team challenge protocol (10 challenges)
  • Part 4: Red-team model prompt
  • Part 5: Scoring summary template

---

Failure Modes

Failure	Cause	Resolution
Framework not found	assets/ADR_EVALUATION_FRAMEWORK.md missing	Repair skill directory structure
ADR not found	No ADR package matches the requested ID	Verify ADR exists and uses canonical naming
No OBPIs found	obpis/ and briefs/ are empty	ADR must have co-created OBPIs before evaluation
Scorecard not written	Path or permission error	Verify ADR directory is writable
NO GO verdict	ADR has structural deficiencies	Revise ADR and re-run evaluation before proposal/defense

---

Acceptance Rules

• All 8 ADR dimensions are scored with evidence-based rationale
• All OBPIs are scored on all 5 dimensions
• EVALUATION_SUBSTANCE.md is written to the ADR directory (never EVALUATION_SCORECARD.md, which the CLI regenerates — GHI #769)
• Verdict follows the threshold rules with no manual override
• NO GO blocks progression to human proposal/defense

---

Related Skills

Skill	Relationship
gz-adr-create	Authoring workflow that should invoke evaluation before proposal
gz-adr-audit	Post-completion audit; downstream phase
gz-adr-closeout-ceremony	Closeout occurs after implementation, not authoring

---

References

• Evaluation framework: assets/ADR_EVALUATION_FRAMEWORK.md
• ADR lifecycle: docs/governance/GovZero/adr-lifecycle.md
• GovZero charter: docs/governance/GovZero/charter.md
• Parity origin: the AirlineOps gz-adr-evaluate skill (sibling repository; not resolvable from this one)

---

Related ADRs

• ADR-0.0.19 — Pre-execution reasoning walkthrough. The Low-Score Footer Guidance section routes operators from a triggered evaluation into the gz-justify walkthrough so hidden ambiguity is surfaced before the ADR is promoted into active work.