Agent Ops Debugging — Autonomous Agent Operational Debugging Patterns
Source: Distilled from GEODE operational debugging sessions (2026-03-14)
Philosophy: "Not crashing" and "working correctly" are different problems.
Historical incident: ADR-008.
The referenced Blog 25 (25-operational-debugging-four-layer-fix.md) is absent
from this checkout; it is retained here only as historical provenance.
Current contract: construction and lifecycle.
5 Pattern Overview
| # | Pattern | One-line Principle | Applicable Layer |
|---|
| D1 | Safe Default Anti-pattern | Safe defaults guarantee stability but not correctness | All layers |
| D2 | Multi-gap Root Cause | Individual gaps are harmless but N gaps combined manifest — individual tests cannot catch them | Pipeline |
| D3 | Ownership and Context Lifetime | Compose services explicitly; bind and reset request-local context | Wiring + Runtime |
| D4 | Execution Boundary | Choose explicit snapshots or context propagation according to ownership | Runtime + Concurrency |
| D5 | Degradation ≠ Correctness | Graceful skip/fallback and functional correctness must be verified separately | Verification |
D1. Safe Default Anti-pattern
Principle
When defaults are set to safe values like True, None, or "", the system does not crash but returns results different from the intended behavior. This type of bug does not appear in error logs, so it is discovered late.
Diagnostic Criteria
Q: If this code path operates with the default value, is the result "normal" or "degraded"?
| Default Type | Normal Case | Degraded Case |
|---|
dry_run=True | Returns fixture when no API key | Returns only fixture even with API key present |
return None | Optional feature not in use | Required feature silently disabled |
log.debug(skip) | Optional external integration skipped | Required integration silent fail |
Application Pattern
# BAD — always safe but always degraded
dry_run = args.get("dry_run", True)
# GOOD — default determined by system state
dry_run = args.get("dry_run", force_dry_run) # force_dry_run is readiness-based
Use the owner's declared default and distinguish normal, degraded, and failed
outcomes. The example above describes the 2026-03-14 incident; credential
availability alone does not authorize live calls or changing a dry-run request.
D2. Multi-gap Root Cause Analysis
Principle
The most difficult operational bugs to find are those that manifest only when N independent gaps exist simultaneously. Each gap is individually harmless or has separate safeguards, so they are not discovered by unit tests.
Analysis Framework
1. Symptom definition: "On which path, under which conditions, does the result differ from expectations"
2. Path comparison: "Identify divergence points compared to the normally working path"
3. Gap enumeration: "Verify whether each divergence point has an independent gap"
4. Overlap determination: "Must all gaps exist simultaneously for the bug to manifest?"
Case: Sub-agent dry-run (3-gap)
Gap 1: Handler default = True (hardcoded)
→ Alone: Solvable if LLM passes dry_run=False
Gap 2: dry_run not defined in tool schema
→ Alone: LLM cannot know the parameter
Gap 3: ContextVar thread isolation
→ Alone: Handler cannot query readiness
All 3 present → sub-agent path always runs in dry-run
Fix the smallest root cause, then verify whether the remaining gaps are
actually harmless. Do not close them from the existence of one fix alone.
D3. Service Ownership and Request Context
Principle
Current GEODE composes services in core.wiring through constructors or existing
typed contexts. Do not restore the historical ContextVar service locator.
ContextVars may hold request identity, diagnostics, or request-local state;
their binding and reset must match the logical request lifetime.
Entry Point Checklist
Typical entry points for an agent system:
[ ] CLI single command
[ ] REPL interactive loop (e.g., `geode` → interactive)
[ ] Hosted runtime turn
[ ] Sub-agent task, worker thread, or subprocess
[ ] Server or scheduled-job entry point
[ ] Test fixture (e.g., pytest conftest.py)
For the affected entry points, trace the service constructor to its consumer
and the request-context binder to its reset, including errors and cancellation.
Reuse the existing composition owner; entry-point count alone does not justify
another bootstrap layer or DI container.
D4. Crossing Task, Thread, and Process Boundaries
Principle
Check how the actual execution primitive propagates context. A copied context
does not synchronize mutable services, and a captured value can become stale.
Pass services explicitly and choose the lifetime of value snapshots deliberately.
Historical pattern — ADR-008, 2026-03-14
def make_handler(*, force_dry_run: bool = True):
"""force_dry_run is captured in the closure — same regardless of which thread executes."""
def handler(task_type: str, args: dict) -> dict:
dry_run = args.get("dry_run", force_dry_run)
...
return handler
# Historical call site: fix readiness state at handler creation time
readiness = _get_readiness()
handler = make_handler(force_dry_run=readiness.force_dry_run)
Current decision criteria
| Value | Boundary treatment | Verify |
|---|
| Service dependency | Constructor or existing typed context | Correct owner and teardown |
| Immutable per-turn policy | Explicit snapshot at the owning turn boundary | A later config change affects the intended next turn |
| Request identity or diagnostics | Scoped binding or explicit context propagation | Correlation preserved; reset on success/error/cancel |
| Cross-process input | Existing serialized request contract | No assumption that in-process ContextVars cross the boundary |
The historical closure fixed one readiness lookup. It is not a recommendation
to capture mutable credentials or long-lived policy at handler construction.
D5. Degradation ≠ Correctness
Principle
Graceful degradation is a system stability pattern. It must be verified separately from functional correctness.
Verification Matrix
For all external dependencies:
| Dependency | Expected behavior when present | Expected behavior when absent | Actual behavior |
|------------|-------------------------------|-------------------------------|-----------------|
| Provider | authorized call uses selected source | explicit error or explicitly requested fixture mode | ??? |
| MCP | tool list loaded | skip + warning | ??? |
| Redis | L1 cache used | direct L2 query | ??? |
Test both columns with offline fakes first. Live verification requires the
task's authorization; merely finding credentials does not authorize it.
Distinction Criteria
Stability test: "Does the system not crash when dependency X is absent?"
Correctness test: "Does the system actually use X when dependency X is present?"
Verify the declared outcome in both cases. A required dependency may correctly
fail closed instead of returning a successful-looking fallback.
Debugging Workflow
When encountering a symptom of "it works but not correctly" during operations:
1. Symptom → Identify the affected layer (infrastructure/UI/DI/pipeline)
2. Compare with the normal path to find divergence points
3. Check defaults at divergence points — D1 Safe Default applicable?
4. Determine single cause vs Multi-gap — D2 applicable?
5. ContextVar access failure? — D3/D4 applicable?
6. After fix, verify both "when present" + "when absent" — D5