/testing — real meshes, real conditions, one run
Canonical references:
WritingTests.md ·
TestStateIsolation.md ·
CqrsAndContentAccess.md ·
SatelliteEntityPatterns.md.
The standards
No mocking. Use the real mesh test bases — never mock IMessageHub, IMeshService, or core
interfaces (#1810).
Always run_in_background: true for test runs (they take minutes).
Never --verbosity minimal when tests may fail — it hides stack traces.
Never Task.Delay to wait for propagation. A fixed sleep races CI load: too short → flakes, too
long → wastes minutes across the suite. Wait on the actual condition via
stream.Where(...).FirstAsync().Timeout(...). When the source is request/response (not an
observable), wrap the re-query in
Observable.Interval(50.Milliseconds()).StartWith(0L).SelectMany(...).Where(predicate).FirstAsync().Timeout(...).
Hand-rolled while + Task.Delay(50) poll loops are forbidden. Sanctioned Task.Delay uses: forcing
distinct timestamps for sort assertions, and "wait to confirm nothing happened" negative tests where
there's no positive signal to filter for. See WritingTests.md → "Polling loops around QueryAsync".
A negative that HAS a later positive fact to end on (a sentinel through the same pipeline, the owner's
answer to a follow-up request, a response that follows the work) uses
.Should().Within(t).NotEmitBefore(settled) instead of NotEmit(window) — a passing NotEmit spends
its whole window, 36 s on CI at TestTimeouts.Quick (#6817). WritingTests.md → "A negative assertion
that spends a fixed window when a settled signal exists".
Never assert "exactly N change events" on a stream backed by pg_notify or any change feed that
can race the initial-snapshot path. Filter on the emission shape (e.g.
.Where(c => c.ChangeType == QueryChangeType.Initial)), not the count.
🚨 A reachability / GC assertion joins DisposalCompleted BEFORE it collects. Dispose() only
STARTS a hub's teardown — it posts ShutdownRequest(Quiescing) and returns — so on return the hub is
still rooted by its own in-flight shutdown. WeakReference + Dispose() + GC.Collect() therefore
measures the teardown's speed, not the reference graph, and it fails GREEN: #3321's version passed
inside a 485-test project and failed alone on the same binary. Take the weak reference and drop the
strong one inside a [MethodImpl(NoInlining)] helper, return the completion observable rather than
the object, await it, then collect. Suite-green / alone-red with nothing rebuilt is its own signal
— and the PASSING run is the control: it names the ambient wait the test was silently relying on.
See WritingTests.md → "A reachability assertion that collects before teardown has finished".
🚨 NEVER re-run a test (single or suite) unless code under test has changed. Re-running to "see
if it was a flake" hides the bug — flakes are real races. Either fix the race or pin the failure
with a smaller repro; do not retry. The only exceptions: (a) the test harness itself crashed
(MSBuild MSB4166, infrastructure error — a re-run is the same input), (b) the previous run was
killed by the user before completion.
🚨 .ToTask() is FORBIDDEN in tests too (maintainer, 2026-08-30 — the earlier test exemption is retracted: a Task completed inside an Rx pipeline resumes inline on the signalling thread, still inside the trampoline, so the bridge changes what the test measures). 🚨 Awaiting the observable DIRECTLY is not the alternative — Rx's awaiter is an AsyncSubject<T> that completes its continuation from inside OnCompleted and resumes inline just the same (InlineResumptionMechanismTest); this line used to say it was, and DirectObservableAwaitSites.allow is the 228-site inventory that advice produced. Assert on the stream through MeshWeaver.Reactive.Assertions, or wait through .Await(ct) / .ObserveCompletion(reportLateFault, ct) — see
/async.
No static collections, in test/ as much as src/ — a Clear() added "for test isolation" is
the proof of the bug, not the fix.
Configuration and its blind spot
xUnit v3 config (test/xunit.runner.json): parallelizeAssembly: false,
parallelizeTestCollections: false, maxParallelThreads: 1, methodTimeout: 30000 (30 s).
🚨 methodTimeout only bounds time spent INSIDE a test method. A wedge in fixture construction,
class init, or teardown is outside it and runs unbounded — on 2026-08-04 an orphaned local run sat
at a pegged core for 25+ minutes and, together with a leaking e2e container, drove the colima VM to
128 MB free / load average 195, OOM-killing unrelated containers.
So always give a local test run its own wall-clock cap — but not with timeout: neither
timeout nor gtimeout exists on this macOS host, so timeout 20m dotnet test … runs nothing
and reports command not found. Background the run and hold the deadline yourself against date -u
— the five-step shape is in /worktree → "A verification step that cannot
fail".
CI has the equivalent at two levels — its runners are Linux, where timeout does exist: timeout 8m per project inside the shard loop, and timeout-minutes: 20 on the shard job itself for a wedge
BETWEEN projects (which the per-project cap cannot see). If a run hits either, it is stuck — do not
raise the bound, find what is not completing (AGENTS.md → "No band-aids").
Running tests
dotnet build test/MeshWeaver.Hosting.Monolith.Test/MeshWeaver.Hosting.Monolith.Test.csproj
dotnet test test/MeshWeaver.Hosting.Monolith.Test --no-build
dotnet test test/MeshWeaver.Graph.Test --filter "FullyQualifiedName~AccessAssignment" --no-build
Build first, every time: dotnet test --no-restore/--no-build against a project this worktree
has never built exits 0 with no output and no .trx — an unmissable-looking pass that ran
nothing. One project per dotnet build invocation (several args is an MSB1008 no-op).
Workflow: run once in background → read failures → fix → run once more.
In-mesh suites: a node's Tests area — run it ONCE, as an activity
🚨 Rendering a Tests area RUNS the suite. Every get @<node>/area/Tests (or render_area)
opens a fresh subscription, so polling the area re-runs every live case, files their side effects
again, and can still show a verdict an earlier run cached. The area streams now (every case pending
→ the running case with its elapsed time and output → the verdict), and every case is bounded, so a
hung case fails as timed out: no verdict within Ns instead of leaving the page on Rendering ….
From a terminal or an agent: memex tests @<node> (exit 0 passed, 1 failed, 4 no verdict in time),
or the MCP run_tests tool — both drive MeshOperations.RunTests, ONE subscription written to a
TestRun activity you poll with get, which starts nothing. Only a verdict frame counts: a
progress frame (tests-running) shows ✔/✖, never ✅/❌ or N/M passed. Full reference:
WritingTests.md, "A Tests
area streams its progress".
Test triage — when CI fails
DO NOT run entire test projects. Iterate one test at a time:
- Read failed test names from CI logs (
gh run view <id> --log).
dotnet build <project>.csproj — in a fresh worktree, skipping this makes step 3 a silent no-op
(exit 0, no output, no .trx).dotnet test <project> --filter "FullyQualifiedName~<TestName>" --no-build.- No skipping — CI-only failures catch real timing/state bugs.
🚨 Read the run's evidence before reasoning about it. The shard artifact carries three things and
they answer different questions:
- the per-project
.trx — output lives in <Output><TextMessages>, not <StdOut>; the native
xUnit v3 writer does not use StdOut, so finding it empty says nothing;
collected-logs/_meshweaver-test-trace.log — the ONLY log that survives a host killed at the
wall-clock cap, carrying TEST_START/TEST_END window markers and [FAULT] records with stacks,
joinable by pid= and timestamp;- the classified
[CI] <name> exit=<n> markers.
A host that CRASHED is written into the trx as a <project>.HOST_CRASHED failure, so no summary can
report a pass over a dead process. Full map: WritingTests.md → "Reading a CI Failure".
A hub-handler test that hangs, or a message that disappears, is a message-flow problem, not a
timeout problem — see /debug and
DebuggingMessageFlow.md
first. Never rerun a hung test "to see".
Test bases and fixtures
MonolithMeshTestBase (recommended) — full integration with persistence, messaging, DI; use
AwaitResponseAsync(request, ...) for request/response in tests.HubTestBase — message routing / layout tests; assert on the stream, e.g. await hub.Observe<TResponse>(request).Should().Within(TestTimeouts.Convergence).Emit("the response arrives"), or wait through the one bridge: await hub.Observe<TResponse>(request).FirstAsync().Timeout(TestTimeouts.Convergence).Await(ct). Never .ToTask(), and never a bare await hub.Observe<TResponse>(request).FirstAsync() — Rx's own awaiter resumes inline on the signalling thread exactly as .ToTask() does.MeshWeaver.Hosting.Orleans.TestBase — the core Orleans cluster machinery (test cluster,
disposal drain, shutdown-race suppression). OrleansMeshTestBase is the ONE base a suite
derives from; which cluster it gets is Bootstrap => MeshBootstrap.Orleans(…) plus
SiloConfiguratorType. The AI-flavoured Orleans rig (the derived fixture and the swappable
chat-client factory) ships with the AI engine in
MeshWeaver.Plugins, not here (#2276).
Use hub.Observe(...), not RegisterCallback/AwaitResponse — those overloads are
[Obsolete] and deadlock. Tests use MonolithMeshTestBase.AwaitResponseAsync(...).
DevLogin and access control
MonolithMeshTestBase auto-logs in rbuergi@systemorph.com as Admin. Available helpers:
TestUsers.Admin, TestUsers.SampleUsers(), builder.AddSampleUsers().
For per-user access-control tests, use
accessService.SetCircuitContext(new AccessContext { ObjectId = "...", Name = "..." }) before
creating test data; set null after.
Node types available in a test mesh
From AddGraph(): Markdown, Code, Group, User, VUser, Role, Notification, Approval,
AccessAssignment, GroupMembership, PartitionAccessPolicy, ActivityLog, UserActivity,
Comment, Redirect.
Agent, Skill, Thread and ThreadMessage are declared by the AI engine module, which lives
in MeshWeaver.Plugins (#2276) — a core-only test mesh does not have them.
Custom types: builder.AddMeshNodes(new MeshNode("MyType") { Name = "My Type" }) in
ConfigureMesh.
Checklist