Debug a CI failure on an OS you are not on

You are on Linux and a test fails only on macos-latest (or you are on a Mac and it fails only on ubuntu-latest). The slow way is a pull request per attempt: each push reruns the whole matrix, builds everything, and takes about fifteen minutes to tell you one thing. Do not do that.

The fast way is a dispatchable workflow on your own fork, scoped to the smallest run that answers your question. You iterate by pushing to a fork branch and re-dispatching. No pull request is involved until you have the answer.

The two things that make it fast

1. Scope the run. Do not run the matrix, the lint job, and the whole suite to learn one test. Run one OS, no lint, and one test. full-tests.yml takes three inputs for exactly this:

   gh workflow run full-tests.yml -f ref=<branch|sha> -f os=macos-latest -f lint=false -f filter=<substring>
   

   • os picks a single runner (macos-latest, ubuntu-latest, or both).
   • lint=false skips fmt and clippy.
   • filter is a substring of the cargo test name (not the test-binary name). It runs only the tests whose name matches.
   • Build time does not shrink with filter (the build is --all-targets for the trampoline). The saving is one OS instead of two, no lint, and far fewer tests that execute. To also cut build time, scope the build in a throwaway workflow with --test <binary> or -p <crate>.

2. For an environment question, build nothing. Most "green here, red there" failures are the environment, not the code: a different active toolchain, a different system path, a pointer file, a library location. A workflow that only runs shell commands returns in about fifteen seconds, so you can ask five questions in the time one build takes. Print what you suspect differs:

   • macOS: sw_vers, xcode-select -p, xcrun --find <tool>, which -a <tool>, otool -L <bin>, readlink of any pointer files, the relevant env vars, and for an interpreter python3 -c 'import sys,os; print(sys.executable, os.path.realpath(sys.executable))'.
   • Linux: uname -a, the distribution release, which -a <tool>, readlink -f <bin>, ldd <bin>, and the mount or namespace state that matters.

Two fork rules, or the dispatch fails

• A pull request from a fork runs the base repository's workflow, not your fork's copy. So a workflow edit on a fork branch does nothing on a PR. It only takes effect through a manual dispatch on the fork (or after it merges to the base).
• workflow_dispatch needs the workflow file on the fork's default branch to be dispatchable at all. Push the workflow to the fork default branch once; after that you can dispatch it against any branch or sha with -f ref=....

The loop

1. Confirm the divergence: check out the exact CI commit, run the test locally, see it pass. Now you know the code is fine and the environment differs. Stop reading the diff.
2. Push a no-build probe workflow to your fork default branch. Dispatch it. Read what the runner's environment holds that yours does not.
3. Form one hypothesis. Test it with the smallest scoped run: one OS, no lint, one filtered test, or the probe with one more command. Change one thing per run.
4. Read the result. gh run watch <id>, or gh api repos/<owner>/<fork>/actions/jobs/<jobid>/logs for the raw log. The runner's own log names the cause; do not infer it from a test timeout.
5. When a run confirms the fix, land it and do one final scoped dispatch to prove it green.

When the failure is containment (deny-default box)

If the failing test starts a contained workload and it dies at startup (a leaf launch, a Seatbelt or namespace refusal, a startup-marker timeout), the environment difference is usually a path the box's floor does not grant on the runner. Two extra moves apply, and the full method plus a worked example (the macOS /usr/bin/python3 xcode-select leaf) live in docs/dev/debug-os-specific-ci-failures.md:

• Reproduce the box's floor by hand. On macOS write a sandbox-exec profile from the agent and leaf cells in crates/containment/src/containment-data/os-paths.macos.json; on Linux reconstruct the launcher's binds or run the trampoline directly. Validate the reconstruction passes locally before you trust its runner failure.
• Read the kernel's own denial, not the test's timeout: on macOS sudo log show --style syslog --last 3m --predicate 'eventMessage CONTAINS "deny"'; on Linux dmesg, journalctl, or an strace of the refused syscall.