Evidence Verification Pipeline (ev-pipeline)
Orchestrates the evidence verification phases that take raw evidence from collected to audited and link-checked before ingestion. It operates on the evidence/ data lake only and does not mutate registry files.
Type-First Evidence Lake Contract (#1148)
The evidence lake is type-first. The primary working set for every phase is evidence/by-type/<canonical-evidence-type>.md using the canonical evidence vocabulary: repo-own, github-stars-own, social-signal, benchmark-result, arxiv, peer-review, proxy-containment, verifier-attestation, fusion-recipe, and self-attestation.
Legacy evidence/tier_*.md files may still exist as coexistence artifacts for older tooling, but they are not the semantic routing key. New collection, audit, compile, and handoff work must route by evidence type.
Multi-Target Peer-Review Source Packets (#1418)
When one legitimate peer-review URL truly reviews multiple named skills, materialize a scratch peer-review partition instead of duplicating registry rows or hand-writing generated lake files:
python evidence/scripts/peer_review_source_packets.py \
--manifest /path/to/manifest.json \
--by-type-dir /tmp/ev1418/by-type
The packet contract is peer-review-only, expands to one temporary row per reviewed skill in peer-review.md, and may repeat the same URL once per target skill when the source genuinely covers each one. Reject strength/scoring fields anywhere in the packet (trustNumber, grade, class, tier, level, stars, rank). Do not commit scratch manifests or generated partitions without human approval.
graph TD
Phase0[Phase 0: ev-discovery (skippable)] -->|Append discovered rows| A
A[Phase 1: ev-collection] -->|Materialize + compile by evidence type| B[Phase 2: ev-star-verification]
B -->|Verify live stars, no rank repartition| B2[Phase 2B: ev-benchmark-verification]
B2 -->|Classify benchmark sources + candidate manifests| C[Phase 3: ev-adversarial-audit]
C -->|Audit by-type files| D[Phase 4: ev-link-validation]
D -->|Validate URL health| E[Source Report & Ingestion Handoff]
Phase Responsibilities
- Phase 0 —
ev-discovery (skippable): searches for new Stage-2 evidence on declared need and appends discovered rows into source inputs for Phase 1. It does not score or rank evidence.
- Phase 1 —
ev-collection: materializes primary partitions under evidence/by-type/ and compiles evidence/unified_evidence_lake.md from those by-type files. It may emit tier_*.md only as coexistence output.
- Phase 2 —
ev-star-verification: verifies live GitHub stargazer counts and records stale/inflated metrics. It must not repartition the lake by rank; by-type partitions remain primary.
- Phase 2B —
ev-benchmark-verification: verifies benchmark-result rows and optional candidate manifests against registry/benchmark-sources.json. It classifies benchmark lanes (verified = CI/verifier reproduced, 2.0x; reported = public claim or mirrored evidence accepted by the human gate, 1.0x; rejected = blacklist/no scoring, 0x), catalog status, unknown benchmarkIds, missing verified-lane reproducibility fields, and malformed percentiles without mutating the registry. Legacy aliases normalize safely: ci-reproduced/verifier-attested → verified, mirrored → reported, and pending/candidate/retired/unknown → rejected. appliesToGenericSkillRefs says which generic capability buckets a benchmark evaluates; it does not create a named-skill score.
- Phase 3 —
ev-adversarial-audit: audits by-type files for bad URLs, subjective wording, proxy/source mismatches, stale notes, benchmark catalog misuse/vendor-claim leakage, and star/evidence inconsistencies.
- Phase 4 —
ev-link-validation: validates URL health. validate_sources.py remains a temporary coexistence URL-health shim while the lake is type-first.
Additive loop — no green gate
Evidence is additive, not pass/fail. New rows can arrive from Stage 1 minimum rows (github-stars-own, repo-own, self-attestation) or from skippable Phase 0 discovery (benchmark-result, arxiv, peer-review, richer social-signal). Benchmark rows add a dedicated human gate: after Phase 2B, Phase 3, and Phase 4, humans must approve a candidate as reported benchmark evidence or reject/blacklist it; verified rows require CI/verifier reproducibility. Machines classify, humans approve or reject. A benchmark catalog entry may name generic refs via appliesToGenericSkillRefs, but it creates no named-skill score and no Trust Magnitude contribution until a named benchmark-result row lands in a scoring lane. Trust Magnitude and rank are recomputed at appraisal time.
Outputs and Handoff
Generated evidence outputs are review artifacts, not registry mutation. Do not commit generated evidence/by-type/**, evidence/tier_*.md, source reports, validation reports, unified lake files, seeds, or collector rows without a human gate. For L4-approved intake, pass a reviewed manifest of live, correctly scoped rows to /gaia-ingest-batch.