Harbor Benchmarks

Build a bounded mathematical experiment with a hidden, replayable verifier. Jacobian availability is an experimental condition, not a prerequisite for task admission. Task correctness depends on the submitted mathematics, not the agent's tool choices or transcript.

Choose the artifact

• Put meaningful agent-facing mathematical tasks with replayable outcomes in Harbor, including capability gaps and tool-free baselines.
• Put provider/environment feasibility spikes under tests/fixtures/providers/.
• Keep public known-answer regressions and human-reviewed proof corpora separate from held-out comparative scoring.

For submission design, mathematical predicates, scoring, or verifier repairs, use verifier-evaluations. Ordinary packaging and execution do not require that skill. A task's instruction and schema must expose its complete protocol while keeping solutions, Oracle fixtures, verifier code, and host caches hidden.

Package and validate

Each task is a direct dataset child with one member record:

benchmarks/datasets/<dataset>/<task-id>/
benchmarks/datasets/<dataset>/members/<task-id>.toml

Use frozen offline inputs, pinned images and dependencies, and an explicit execution profile. Verifier Dockerfiles build from tests/; avoid parent COPY, host paths, floating tags, and symlinks. For the affected contract, use benchmark contracts. For local commands and planner selection, use the Harbor section of CONTRIBUTING.md.

make harbor-plan BASE=origin/main
make harbor-prepare-task DATASET=<dataset> TASKS="<task-id>"
make harbor-validate-task DATASET=<dataset> TASKS="<task-id>"

Preparation formats selected task Python and refreshes contracts/checksums; validation runs the selected static, host, and Oracle checks without a model agent. Run the selected verifier's behavioral attacks when its contract changes. Report any deferred affected Oracle coverage.

For task-local support migrations, older contracts, checksum updates, or an intentional snapshot, read packaging details. Do not create a snapshot merely because a task contract changed.

Complete the requested task preparation and validation, repairing failures caused by the change within the authorized scope. Report the command, task digest, selected scope, Oracle result, and remaining evidence gaps. Distinguish local validation from a causal model comparison; model calls and external publication require the applicable user authorization.