Evidence Discovery (ev-discovery)
Different from pre-curation /ev-scout: scouting happens before
topology mapping and casts a wide contextual net, including rivals and
negative evidence. This Phase 0 is a later, optional evidence-development
step for selected Stage-2 sources. Scout leads are unverified and non-scoring
until the canonical verification and human approval sequence completes.
Phase 0 is optional. It searches for new, higher-quality Stage-2 evidence only when a maintainer declares a need.
Type-First Evidence Lake Contract (#1148)
The evidence lake is type-first. Discovery feeds Phase 1, which materializes evidence/by-type/<canonical-evidence-type>.md. Legacy evidence/tier_*.md files may still exist as coexistence artifacts, but they are not the semantic routing key.
Discovery Scope
Prioritize new evidence rows for:
benchmark-resultarxivpeer-review- richer
social-signal
Discovery may also identify better repo-own or github-stars-own sources, but it must not assign authoritative rank, tier, grade, or Trust Magnitude.
Benchmark candidate handoff
When discovery finds missing benchmark evidence, create a benchmark-source candidate record for Phase 2B review instead of appending a scoring benchmark-result row from a vendor page. The candidate should name the target skill, source URL, proposed benchmarkId, proposed generic applicability (appliesToGenericSkillRefs) when known, status: candidate, missing reproducibility fields, and notes. No discovered vendor page may become a scoring row until a human promotes the source in registry/benchmark-sources.json and approves /gaia-ingest-batch. A promoted registered benchmark source is still generic-applicable, non-scoring catalog metadata only until a named benchmark-result row satisfies reproducibility fields and approved provenance. Firecrawl-style vendor benchmark claims are candidate-only or registered-only until reproducibility fields, allowed provenance, and human approval exist.
Handoff to Phase 1
Append discovered raw rows to the appropriate source inputs with a canonical or normalizable evidence type. Phase 1 is responsible for normalizing aliases and materializing evidence/by-type/<type>.md. Do not hand-write generated by-type partitions as if they were source rows.
If one discovered peer-review source legitimately covers multiple named skills, hand Phase 1 a scratch multi-target packet manifest for evidence/scripts/peer_review_source_packets.py rather than cloning the same review into multiple intake mutations. The helper is peer-review-only, allows the same URL once per reviewed skill, rejects strength/scoring fields (trustNumber, grade, class, tier, level, stars, rank), and should write only temporary by-type output.
Preflight (#1788)
Firecrawl usability is not gated on the FIRECRAWL_API_KEY env var alone — the
CLI can be authenticated via stored browser/CLI credentials independent of it.
Check both before deciding to skip Phase 0:
firecrawl --status # Authenticated? credits remaining?
echo "${FIRECRAWL_API_KEY:+set}" # non-empty output if the env var is set
Proceed with discovery if either check succeeds. Skip Phase 0 gracefully
only when firecrawl --status reports unauthenticated and the env var is
unset/empty.
Commands
Quote every URL and query — unquoted ?/&/glob characters expand under zsh,
this repo's shell, and silently turn a single request into a file-glob error.
mkdir -p .firecrawl
firecrawl search "<query>" -o .firecrawl/result.json --json
firecrawl scrape '<url>' --json -o .firecrawl/scrape.json
gh api -X GET 'repos/<owner>/<repo>/git/trees/main' -f recursive=1
The gh api form above (path + -f recursive=1 instead of a literal
?recursive=1 in the URL) avoids the same zsh glob-expansion problem for
GitHub tree listings.
Retry rule
Retry is a status-code rule, not a named-host list — a source is not
"unresolved" just because it wasn't on someone's retry shortlist:
- On any
401, 402, 403, 429, or SSL/TLS failure from a direct fetch,
retry once via firecrawl scrape before marking the source dead.
- Do not maintain or consult a list of specific domains to retry — apply the
rule to every failing URL, every time.
Guardrails
- Requires Firecrawl usability per the Preflight above (
firecrawl --status
OR FIRECRAWL_API_KEY); skip gracefully only when both fail.
- Do not run as a broad web crawl without declared need.
- Do not mutate registry files.
- Do not append direct scoring
benchmark-result rows from discovered vendor pages; route them through benchmark-source candidates and Phase 2B first.
- Do not treat generic-applicable registered benchmark sources as Trust Magnitude evidence; they are non-scoring until a named row satisfies reproducibility and approved provenance.
- Do not commit generated collector/by-type/report artifacts without a human gate.