QA Resilience

Quick Reference

Symptom	Start With
slow or hanging dependency	deadline and timeout budget
transient dependency failure	bounded retry with jitter and retry budget
sustained dependency failure	circuit breaker and fallback
rate-limited dependency	honor Retry-After, expose degraded behavior, and test quota paths intentionally
one bad host in a healthy pool	outlier detection or endpoint ejection
queue or pool saturation	bulkheads, concurrency limits, load shedding
static concurrency limit keeps going stale under changing capacity	adaptive concurrency limit with a static backstop; verify the goodput plateau — load-shedding-backpressure.md
non-critical feature outage	graceful degradation or feature flag fallback
resilience validation	deterministic fault injection before chaos

When to Use This Skill

• retries, deadlines, hedging, breakers, bulkheads, and overload protection
• degraded-mode UX or API behavior
• service-mesh or gateway resilience policy
• chaos engineering, game days, DR drills, and fault injection
• release gates based on failure behavior, not only happy-path load tests

Route Elsewhere

• simple CRUD or single-process utilities -> ordinary error handling may be enough
• frontend-only failure behavior -> software-frontend
• service implementation details -> software-backend
• telemetry instrumentation -> qa-observability
• broader platform and incident operating model -> ops-devops-platform

---

Workflow

1. Identify the critical user journeys and the dependencies that can break them.
2. Define the contract per dependency:
   • timeout or deadline budget
   • retry ownership
   • breaker or outlier policy
   • concurrency and queue limits
   • degraded behavior if the dependency is unavailable
3. Decide where the policy lives:
   • app code
   • client library
   • mesh or gateway
4. Test in stages:
   • design review or checker output establishes policy coherence only
   • deterministic fault injection proves the response to a named controlled fault
   • staged chaos in non-production proves bounded behavior and recovery in that environment
   • narrow prod canary or game day proves only the observed target path and window, with guardrails
5. Define pass or fail signals:
   • burn rate
   • p95 or p99
   • fallback rate
   • breaker transitions
   • shed volume
   • recovery time
6. Record the injected fault, steady-state hypothesis, blast-radius limit, abort condition, observation window, recovery result, and dependencies simulated rather than observed.

---

Selective verification and hazard analysis

• Use formal methods when the question is whether a recovery/failover protocol preserves an invariant under concurrent faults, or when a counterexample is needed. Return the property, initial states, transitions/environment assumptions, checked bound, result/trace, and model-to-code limits. A finite invariant check does not establish liveness or recovery time. Skip for routine timeout/retry configuration and named fault tests whose behavior is already directly observable.
• Use safety engineering when retries, rollback, failover, operators, or other functioning controls can interact to cause a specified unacceptable loss, such as duplicate financial effects or destructive recovery. Return loss → hazard → unsafe control action and context → constraint → owner → verification evidence, including residual unknowns. Skip for ordinary availability or latency tuning without a harmful interaction question. Uptime and a hazard diagram do not prove safety.

Keep resilience policy, fault-injection execution, recovery evidence, and release recommendations with this skill; use the foundation only for the named gap.

Pattern Rules

• retries happen at one layer only
• deadlines come before retries
• hedging is only for idempotent or cancellation-safe reads
• overload handling must shed early instead of collapsing late
• overload tests pass on goodput plateauing past capacity, not on "503s appeared"; shed responses are retried only at the retry-owning layer, within budget, after Retry-After
• in regulated firms (EU DORA, UK FCA/PRA operational resilience), test each important business service against its impact tolerance, third parties included; counsel owns scoping — see disaster-recovery-testing.md
• readiness and liveness must stay bounded and shallow
• resilience behaviors belong in targeted checks; do not let rate-limit or degraded-mode coverage leak into unrelated happy-path suites
• prod experiments require blast-radius limits, abort criteria, dashboards, and owners

Failure Modes to Validate

• timeouts and deadline propagation
• retry storms and duplicate side effects
• partial dependency outages
• slow downstreams and long-tail latency
• queue buildup and connection-pool exhaustion
• one-bad-host behavior inside a pool
• degraded-mode responses and stale-data fallbacks
• rate-limit handling, Retry-After, and client backoff expectations
• visible state convergence after recovery or backend resets
• failover and failback behavior
• metastable failure: a self-sustaining feedback loop (retry amplification, cache-miss stampede, queue backlog, connection-pool churn) that does not self-resolve after the original trigger clears — see references/cascading-failure-prevention.md

---

Testing Ladder

Deterministic first

• inject latency, errors, timeouts, malformed payloads, and unavailable endpoints in controlled tests
• verify the intended control activates and the wrong controls do not

Chaos second

• start in non-production
• use a small blast radius and a fixed time window
• stop immediately on error-budget or customer-impact breach

DR and game days last

• validate RTO and RPO claims explicitly
• rehearse recovery ownership, not only technical failover

---

Operational Guardrails

• every experiment needs a stated hypothesis and steady-state metric
• every run should capture timestamps, targets, blast radius, and dashboard links
• telemetry fields for retries, breaker transitions, hedging, shedding, and fallback are part of the resilience contract
• releases should be gated on reliability behavior, not only resource usage

Anti-Patterns

• no timeouts
• retries at every hop
• fixed-interval or unbounded retries
• fixed per-call retry count with no system-wide retry budget (does not tighten as error rate rises)
• retries without idempotency
• hedging unsafe writes
• no bulkheads or queue bounds
• deep readiness or liveness checks
• silent degraded mode
• untested failover plans
• happy-path-only load testing

---

Scripts

Script	Purpose
scripts/resilience_checker.py	Scores declared pattern coverage and reports gaps; rejects malformed claims

Typical usage:

python scripts/resilience_checker.py assess --input data/sample-service-profile.json
python scripts/resilience_checker.py gaps --input data/sample-service-profile.json
python scripts/resilience_checker.py report --input data/sample-service-profile.json --output resilience-report.md

Supply actual JSON booleans and explicit hedging N/A rationale. The heuristic score describes declared coverage; it does not establish fault behavior or recovery. See scripts/README.md for the input format and scoring logic.

ASCII Flow

Resilience request
  -> Identify failure mode: slow, transient, sustained, overloaded, or degraded
  -> Set deadlines, retry budgets, isolation, and fallback ownership
  -> Add telemetry for saturation, errors, latency, and degraded behavior
  -> Validate with deterministic fault injection before chaos experiments
  -> Gate release on recovery evidence, SLO impact, and rollback path
  -> Document runbook actions and residual risk

Navigation

Foundation applied recipes

• references/reliability-theory-applied.md
• references/distributed-systems-applied.md

Core references

• references/circuit-breaker-patterns.md
• references/retry-patterns.md
• references/timeout-policies.md
• references/deadlines-hedging.md
• references/bulkhead-isolation.md
• references/load-shedding-backpressure.md
• references/gateway-mesh-resilience.md
• references/graceful-degradation.md
• references/health-check-patterns.md
• references/cascading-failure-prevention.md
• references/disaster-recovery-testing.md

Operational resources

• references/resilience-checklists.md
• references/chaos-engineering-guide.md
• references/chaos-tooling-recipes.md
• references/idempotency-key-design.md
• references/resilience-telemetry.md
• references/slo-as-code.md
• references/ai-llm-resilience-failure-modes.md
• assets/runbooks/resilience-runbook-template.md
• assets/testing/fault-injection-playbook.md
• assets/testing/template-resilience-test-plan.md
• data/sample-service-profile.json

Related Skills

• ops-devops-platform
• software-backend
• software-architecture-design
• qa-observability
• qa-debugging
• software-security-appsec
• data-sql-optimization
• dev-api-design

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.