ai-spec-coverage-mapper

Overview

Coverage tools (per lcov-analysis in the qa-test-reporting plugin) report which lines are tested. They don't report which spec sections are tested.

A spec section like "AC-1.4: Already-applied promo shows 'Already applied'" might be:

• Covered by a test (good).
• Partially covered (the AC says "Already applied"; the test asserts "Already used" - semantic drift).
• Not covered.

This skill uses an LLM to map tests to spec sections semantically - the mapping human coverage tools cannot produce.

When to use

• The team has a spec doc + a test suite and wants spec-coverage visibility.
• Before a release sign-off - to answer "which requirements are actually covered?"
• An audit / compliance review needs spec-test traceability.
• After AC drift - tests written months ago for ACs that have since changed.

Scope is mapping tests that already exist and naming the gaps - not authoring tests for the gaps (that is ai-test-generator).

How to use

1. Point the mapper at the spec doc + the test globs, and set the AC-ID extraction pattern.
2. Run the LLM mapper - it reads the spec and the tests and emits, per AC ID, a coverage tier (full / partial / none) plus the covering test names.
3. Read the coverage matrix - every partial and none row is a gap.
4. File one action item per gap (fix the drift, or add the missing test).
5. Schedule it weekly and verify the LLM's claims - see references/continuous-coverage-and-verification.md.

Worked example - checkout promo spec

Inputs. Point the mapper at the spec and the tests, and declare how AC IDs are written in the spec:

spec_path: "docs/specs/checkout.md"
test_globs:
  - "tests/checkout/**/*.spec.ts"
  - "features/checkout/*.feature"
ac_extraction:
  pattern: "AC-(\\d+\\.\\d+):"   # AC IDs in the spec

Run the mapper. The LLM reads both the spec and the tests and classifies each AC:

# scripts/ai-coverage.py
import openai

spec_text = read(spec_path)
ac_list = extract_acs(spec_text)
test_files = read_all(test_globs)

system_prompt = """
You map AC IDs to tests. For each AC ID, identify:
- which test files cover it
- which test names within those files cover it
- coverage tier: full | partial | none

If partial, explain what aspect is missing.
"""

response = openai.chat.completions.create(
    model='gpt-4',
    messages=[
        {'role': 'system', 'content': system_prompt},
        {'role': 'user', 'content': f"Spec:\n{spec_text}\n\nTests:\n{test_files}"},
    ],
)
print(response.choices[0].message.content)

Output. A coverage matrix per AC, the action items, and a trend vs. the prior run:

## Spec -> test coverage map

**Spec:** `docs/specs/checkout.md`
**ACs:** 12
**Tests inventoried:** 47

### Coverage matrix

| AC ID    | Description                              | Coverage | Tests                                               |
|----------|------------------------------------------|----------|-----------------------------------------------------|
| AC-1.1    | Valid promo applies discount               |   ✅ full | `promo.spec.ts > "applies WELCOME10"`              |
| AC-1.2    | Expired promo shows error                   |   ✅ full | `promo.spec.ts > "shows error for EXPIRED50"`      |
| AC-1.3    | Invalid format shows "Code not found"        |   ✅ full | `promo.spec.ts > "rejects NOTREAL"`                  |
| AC-1.4    | Already-applied promo shows "Already applied" |   ⚠ partial | `promo.spec.ts > "rejects duplicate"`. ⚠ Test asserts "Already used" - message drift from AC. |
| AC-1.5    | Promo applies before tax                    |   ❌ none |                                                       |
| AC-2.1    | Stripe webhook delivery retried              |   ✅ full | `webhook.spec.ts > "retries on 500"`               |
| AC-2.2    | Stripe webhook delivery DLQ after 3 fails    |   ❌ none |                                                       |
| ...

### Action items

| AC ID   | Action                                          |
|---------|-------------------------------------------------|
| AC-1.4   | Update test message assertion to match AC ("Already applied"). |
| AC-1.5   | Add test asserting promo applies before tax.    |
| AC-2.2   | Add test for DLQ-after-3-fails behavior.        |

### Coverage trend

(Compare with prior run)
- AC count: 12 (was 10)
- Full coverage: 9/12 = 75% (was 8/10 = 80%)
- 2 new ACs added in this PR; both uncovered.

Each partial / none row becomes one action item; the two new uncovered ACs are the highest-priority gaps to close before sign-off.

Anti-patterns

Anti-pattern	Why it fails	Fix
Trusting LLM's "full coverage" claim without verification	Hallucination; tests don't actually cover the AC.	Spot-check + cross-reference every high-priority AC (see references).
Running on the entire codebase repeatedly	Cost + slow.	Filter to changed ACs / tests since last run.
One-shot mapping; never updated	Drift; mapping stale.	Schedule weekly (see references).
No action items per gap	Coverage gaps surface but nothing happens.	One action item per gap (see the worked example).

Limitations

• LLM quality varies. Same caveats as ai-test-generator.
• Spec format matters. Well-structured ACs (numbered, clearly scoped) map cleanly; vague specs produce vague mappings.
• No code-execution verification. The LLM reads code; doesn't run it. A test that imports correctly but throws at runtime may be classified "covered."
• Test naming influences mapping quality. Per-test names like "test_1" don't help the LLM identify what's covered.

References

• references/continuous-coverage-and-verification.md - weekly CI scheduling and how to verify the LLM's coverage claims.
• ai-test-generator - sister skill: generates tests for the gaps this skill identifies.
• gherkin-from-stories (in the qa-bdd plugin, ATDD mode) - for tag-based AC traceability without LLM.
• test-coverage-targeter (in the qa-test-reporting plugin) - its coverage debt ledger tracks line-coverage debt; complementary to spec coverage.