Guide GAIA benchmark runs from environment checks and cost estimates to result comparison. Create a signed submission package compatible with HAL, including agent traces, metadata, and an audit report that checks for benchmark exploitation.
Walk through a complete GAIA benchmark→submit flow — from key resolution through HAL-compatible package generation
This skill guides you through running GAIA, a benchmark for evaluating agent performance, and creating a submission package compatible with HAL. It covers environment checks, cost estimates, evaluation runs, submission file generation, leaderboard comparisons, and saving findings, in that order. Packages contain results, agent execution traces, model and related information, an audit report, and signed verification information.
Use it to check your preparation before a first evaluation or turn an existing results file into a submission package. You can specify the evaluation level, number of questions, and models to use. You can also keep a history of results and costs, compare scores, and record what you learned.
You need ANTHROPIC_API_KEY, HF_TOKEN, Node.js 20 or later, and a built CLI. If the estimated cost exceeds $5, the skill asks for confirmation before running. Critical findings in the submission audit normally stop package generation. Audit checks without enough instrumentation data are treated as verification gaps, rather than passes.
This overview was written by AI (OpenAI Codex) from the public SKILL.md. Check the original below for the exact behavior.
AI rated this skill on four points using only the contents of its SKILL.md. Stars and popularity are not taken into account.
The total counts each point as A = 25, B = 15, C = 5. Rated by AI (OpenAI Codex) as of Oct 21, 2026. Treat it as a guide and read the original before installing.
Review the instructions the agent will receive before you install.
Walk Claude Code through every step needed to go from a clean environment to a signed, HAL-compatible submission package ready to upload to the Princeton GAIA leaderboard.
When the user wants to:
Before starting, confirm these are available:
| Requirement | Check |
|---|---|
ANTHROPIC_API_KEY | echo ${ANTHROPIC_API_KEY:0:8}… (should show sk-ant-…) |
HF_TOKEN | echo ${HF_TOKEN:0:5}… (should show hf_…) |
| Node.js 20+ | node --version |
| CLI built | node v3/@claude-flow/cli/bin/cli.js --version |
# Run all pre-flight checks
/gaia validate
If any check fails, resolve it before continuing.
Ask the user for their configuration:
claude-sonnet-4-6)/gaia cost --level=$LEVEL --limit=$LIMIT --models=$MODELS --voting=$VOTING
If projected cost > $5, show the estimate and ask: "This run will cost approximately $X. Proceed? (y/N)"
/gaia run --level=$LEVEL --limit=$LIMIT --models=$MODELS --voting=$VOTING
While running, progress is reported every 5 questions:
[12/53] 22.7% (5 passed of 22 scored) — est. remaining: $0.18
Store the run summary in memory for history tracking:
npx @claude-flow/cli@latest memory store \
--namespace gaia-runs \
--key "run-$(date +%Y%m%d-%H%M)" \
--value '{"level":$LEVEL,"model":"$MODEL","total":$TOTAL,"passed":$PASSED,"pass_rate":$RATE,"est_cost_usd":$COST}'
/gaia submit --results=~/.cache/ruflo/gaia/results-latest.json
This produces:
submission-<date>-<sha>/
├── results.jsonl ← HAL-compatible, one JSON per line
├── trajectories.jsonl ← full agent traces
├── metadata.json ← harness info, model, tool catalogue
├── audit-report.json ← ADR-167 pre-submission exploit-audit report
├── manifest.md.json ← Ed25519-signed witness (signs audit-report.json's hash)
└── README.md ← human summary + leaderboard comparison
Post-RDI (UC Berkeley broke 8 agent benchmarks — GAIA to ~98% — without solving
a task), a signature alone is not enough: it proves the bytes are untampered,
not that the score was earned. /gaia submit therefore runs a deterministic,
$0 exploit audit before signing and refuses to build the leaderboard package
on a CRITICAL failure unless --allow-dirty is passed. The audit report is
signed into the witness manifest as an ADR-103 fix marker, so a ruflo GAIA
submission attests both transport-integrity and earning-integrity.
If the gate blocks, treat it as a real finding — inspect audit-report.json
(answer-leakage, no-work pass, oracle leakage, grader monkey-patching, an
answer-key read outside the dataset dir, or dynamic eval/exec of task content in
the runner) rather than reaching for --allow-dirty. The static source-scan
family (answer-key-reads, dynamic-eval, judge-injection) enforces today with no
trajectory instrumentation; the trajectory-fed checks the current schema cannot
feed are reported as harness_gaps (ADR-167 §7), not passes.
/gaia leaderboard --level=$LEVEL
/gaia history
Interpret the gap between ruflo's score and the leaderboard top-10.
Identify the primary failure mode (tool gap, reasoning miss, extraction bug)
using the /gaia-debugging skill if needed.
npx @claude-flow/cli@latest hooks post-task \
--task-id "gaia-submission-$(date +%Y%m%d)" \
--success true \
--train-neural true
Store any discovered patterns:
npx @claude-flow/cli@latest memory store \
--namespace gaia-patterns \
--key "submission-notes-$(date +%Y%m%d)" \
--value "Level $LEVEL, $MODEL: $NOTES"
This skill is intentionally structured to be benchmark-agnostic. The phase headers (validate → estimate → run → package → compare → learn) apply to SWE-bench, WebArena, and HumanEval with only phase 3-4 details changing.
No reviews yet. Tell others what you thought.
No login needed. Share how it worked for you, ask a question, or note where it did not run.
No comments yet. Be the first to write one.
Compare models available through the HuggingFace router with chat-ui configuration, research missing models, and write short English descriptions of their architecture and capabilities. Add entries to both production and development configuration files and commit the changes.
Create an Architecture Decision Record, a document for recording important design choices, with the next sequential number. It generates a Markdown template, registers the record in AgentDB, and links related decisions found through search.
Build a searchable index and relationship graph from ADRs, records of architectural decisions. Import two supported document formats, refresh existing metadata without duplicate relationships, and review summaries of statuses, missing references, and inconsistencies.
Rebuild the index of ADRs, records of architectural decisions, after files or relationship lines are deleted. It clears stored records and relationships, indexes the files still present, and checks that the final record count matches.
Review code changes against accepted Architecture Decision Records, documents that capture agreed design choices. Report violations, outdated references, compliant changes, and areas without ADR coverage, then suggest code fixes or a new decision record where appropriate.
Check saved architecture decision records (ADRs) for broken references, circular replacement relationships, and inconsistent statuses. Use the results to verify imported records or stop a CI check when selected integrity rules fail.