Benchmark-Source Verification (ev-benchmark-verification)
Phase 2B runs after /ev-star-verification and before /ev-adversarial-audit. It is benchmark-only and read-only: it classifies existing benchmark-result registry rows and optional benchmark-source candidate manifests without mutating the registry, the catalog, or the evidence lake.
Policy
registry/benchmark-sources.json is the benchmark allow/blacklist surface. Current lanes collapse to three values:
verified — CI-reproduced or verifier-attested benchmark evidence; Trust Magnitude lane multiplier 2.0x.reported — public claim or mirrored leaderboard evidence accepted by the human gate; Trust Magnitude lane multiplier 1.0x.rejected — blacklisted, disputed, pending, retired, candidate, unknown, or otherwise not approved; Trust Magnitude multiplier 0x.
Legacy aliases are accepted during migration: ci-reproduced and verifier-attested normalize to verified; mirrored normalizes to reported; pending, unknown, retired, and candidate normalize to rejected/no scoring. A catalog entry with status: rejected is the blacklist and always scores zero.
Reported rows do not need runAt, datasetHash, or benchmarkInputHash. Verified rows still need reproducibility fields. appliesToGenericSkillRefs records which generic skill IDs a benchmark applies to; it does not create a named-skill score by itself.
Command
PYTHONPATH=src python evidence/scripts/verify_benchmark_sources.py \
--catalog registry/benchmark-sources.json \
--registry-root . \
--check \
--report /tmp/ev-phase2b-benchmark-report.md
With candidate sources discovered during Phase 0 or issue review:
PYTHONPATH=src python evidence/scripts/verify_benchmark_sources.py \
--catalog registry/benchmark-sources.json \
--registry-root . \
--candidate-manifest /tmp/benchmark-candidates.jsonl \
--check \
--report /tmp/ev-phase2b-benchmark-report.md
Candidate manifest entries may be JSON or JSONL objects with fields such as:
{
"target": "firecrawl/firecrawl-research-index",
"source": "https://www.firecrawl.dev/blog/research-index-launch",
"benchmarkId": "alphaxiv-arxivqa@v1.0",
"status": "candidate",
"missing": [],
"notes": "Candidate becomes reported only after the human gate approves it."
}
Firecrawl Research Index / alphaXiv ArXivQA is the model case: a public reported benchmark can count at 1.0x after the human gate, but a candidate report alone is not permission to ingest.
Human gate
Machines classify. Humans approve or reject.
After Phase 2B benchmark-source verification, Phase 3 adversarial audit, and Phase 4 link validation, stop before benchmark catalog promotion or /gaia-ingest-batch. A human reviewer must approve reported benchmark evidence or mark it rejected/blacklisted. Do not treat a green Phase 2B report as permission to ingest.
Output
The report summarizes:
scoring-eligible rows that may count toward Trust Magnitude.reported, rejected-row, candidate-only, blocked, and unknown-benchmark rows that need human review or score zero.- rejected/retired usage and scoring-lane misuse as hard blockers.
- missing verified-lane reproducibility fields and malformed percentiles as findings.
Candidate-only manifest entries are never hard blockers by themselves; they are backlog items for human review.