Remote builds and worktrees

When comparing providers, RAM/GPU capacity or hourly versus monthly budgets, consult the dated provider capability and cost snapshot. Use it to shortlist offers, then verify the current region, stock and billing terms before provisioning.

Use service selection and security admission to distinguish full VMs from managed build/container services. AWS workers additionally follow the AWS operating runbook; an available AWS credential does not establish a usable worker or cleanup coverage.

Before planning or repeating remote readiness checks, apply validation overhead: bound blocking time, reuse evidence for unchanged prerequisites, and prefer a representative real job whose valid result contributes to the requested work.

Use the coordinator checkout as the integration point. A worker owns a bounded job and isolated workspace; successful remote edits return to the coordinator as reviewable commits before the work is considered integrated.

Before batch or long-running jobs, apply the batch failure and retry policy. Close affected worker queues on confirmed setup blockers or near-total failures; bounded retries require a transient cause or a verified change to the prerequisite.

Read synchronization before staging source, collecting remote edits, or cleaning a worker. Read host tuning when selecting capacity, bootstrapping services, or repairing a failed job. Experience notes distinguish observed incidents from recommendations.

Before selecting a worker or exposing credentials, delegated identities, tunnels or sensitive source/data, read remote host secrets and assurance. Apply its standing maintainer approvals, direct-copy decisions and locally authorized default SSH connections without asking again. Additional exposure uses that contract's authorization gates. Before transfers or cleanup, read custody mechanics; ledger each destination. SSH private identities and agent sockets stay on the coordinator.

For repeated provisioning and failed-stage recovery, follow remote host recovery. Invoke pinned pnpm through Corepack in non-interactive environments. Prefer supervisor operations over pattern-based process termination; missing command output can be self-kill and must be reconciled before replay. Use the shared resource watchdog for campaign-owned rented workers; it never authorizes deletion of a borrowed host.

Choose the smallest useful offload

• Read the destination project's instructions and package scripts. Identify the requested revision, exact build/test command, required OS/architecture, artifact destination, and completion signal.
• Reuse the user's designated hosts and existing authorization. Available SSH credentials alone do not authorize host changes, spending, or service interruption. Discover only the relevant configured routes; do not scan networks for additional hosts.
• Classify each host by proven capability: full build/test executor, artifact verifier, or unavailable. A reachable host without container access or application credentials may still run static checks or verify result hashes. Report that contribution accurately.
• Before renting capacity, use manage-remote-resources for budget, ownership, independent expiry, and verified deletion. This skill does not provision hosts. A CPU build needs no GPU, model call, judge, browser recording, or production database unless its actual command requires them.

Establish a reproducible workspace

1. Record coordinator and worker Git status, exact base commit, and relevant existing services. Choose a task-owned checkout and output directory on a verified persistent filesystem with sufficient free space and inodes. A path named /tmp may be a small tmpfs.
2. Use a separate clone or worktree when the existing checkout contains concurrent edits. A linked worktree's .git file points back to its parent repository: create worktrees on the destination host; do not copy one wholesale and assume it is portable.
3. Stage the exact revision and an explicit list of any required uncommitted/untracked source. Identify such a snapshot with its base and patch/manifest hashes; never label it a clean commit build. Preserve remote edits before refreshing source.
4. Install the project's pinned runtime and lockfile dependencies on the destination. Keep mutable build output, runtime data, and service namespaces isolated per job. Reuse caches only where their platform/runtime compatibility and concurrency behavior are known.
5. Capture the effective service environment by non-secret settings: runtime paths, working directory, container socket, cache paths, ports and configuration identity. Inject only workload-needed credentials covered by the canonical contract into protected configuration, ledger their authority and cleanup state, and keep values out of command lines, logs and images. A whole .env or credential directory is not a source-transfer asset.
6. Resolve the pinned runtime from the same non-interactive SSH or service environment that will run the job. Record the absolute executable and prepend its directory to the supervised job's explicit PATH; an interactive shell finding node does not prove that systemd, launchd, cron, or a non-login SSH command will. In this repository, source scripts/codex-env.sh when appropriate, then persist the resolved runtime path in the generated service environment rather than relying on shell startup files.

Run and recover

• Start a supervised job with a stable run ID, durable log, PID/service identity, timeout, and explicit exit status. Confirm the actual child process and output directory; a launcher returning successfully is not a completed build.
• Begin with one representative job. Raise concurrency from measured CPU, available memory, disk/I/O, and service capacity while reserving capacity for existing workloads. Build workers inside each job also count toward total concurrency.
• On SSH disconnect, inspect the same job's durable state before resubmitting. A quiet process within its deadline is not automatically dead. Preserve errors and partial output before bounded recovery.
• Repair ordinary dependency/configuration gaps in place within authorization and budget. Encode a verified fix in the coordinator's bootstrap, service configuration, or source rather than relying on remote shell history.
• After source changes, rebuild the exact revision. If this task includes deployment, restart only its owned service after build success, verify its build ID and real endpoint, and inspect a fresh log window. A build-only request does not imply deployment authorization.
• Compare speed only with matched commands, revisions, and cache conditions, including staging/setup time. A successful Linux build does not prove macOS packaging or browser behavior.

Finish on the coordinator

Collect logs, exit status, runtime/platform details, source/build identity, output hashes, and any remote source changes. Follow the synchronization reference to make dedicated commits, run relevant coordinator checks, and record remote-only validation separately. Commit bootstrap fixes independently from product changes when each is coherent.

Remove only owned temporary services, tunnels and workspaces after outputs and source changes are preserved; apply canonical custody rules to credentials and record removal proof or the exact accepted-risk disposition. Recheck ownership and worktree state immediately before removal. Retain unresolved edits; use the paid-resource contract for deadline-driven teardown even when collection fails.

Report which hosts performed which checks, passed/failed/deferred results, coordinator commit IDs, remaining remote drift, artifact locations, and cleanup state. A host-only fix with no coordinator commit remains unfinished integration work.