QA Observability

Use telemetry as a QA signal and a debugging substrate. Treat logs, metrics, traces, and profiles as evidence for test outcomes, release readiness, and production regressions.

Core references live in data/sources.json; load the task-specific guide from Navigation.

Quick Start (Default)

If key context is missing, ask for: critical user journeys, service/dependency inventory, environments (local/staging/prod), current telemetry stack, and current SLO/SLA commitments.

1. Establish the minimum bar: correlation IDs, structured logs, traces, and golden metrics (latency, traffic, errors, saturation).
2. Verify the telemetry transport with one known request or fault: application emission, context propagation, collector acceptance, and backend query. Stop the claim at the last observed stage; static SDK or collector configuration proves configuration only.
3. Make failures diagnosable: every integration or E2E failure should capture a trace link or trace ID plus correlated logs, and critical degraded paths should expose structured error metadata such as rate-limit codes, retry hints, and state-transition markers.
4. When dashboards, SLOs, or alerting are in scope, define the relevant SLI/SLO and error-budget policy, verify that the dashboard selects the known telemetry, and test alert evaluation and routing with a synthetic condition and a controlled non-production receiver. Do not notify real responders without explicit authorization. Record backend-query, dashboard-selection, rule-evaluation, route, and delivery results independently; basic instrumentation can stop after backend-query proof.
5. Produce the artifacts that match the task: a readiness checklist, SLO definition, or alert rules using assets/checklists/template-observability-readiness-checklist.md, assets/monitoring/slo/slo-definition.yaml, and assets/monitoring/slo/prometheus-alert-rules.yaml. Preserve environment, service revision, trace/request ID, query window, sampling decision, and the result of each stage exercised.

Default QA stance

• Treat telemetry as acceptance criteria, especially for integration and E2E flows.
• Require correlation: request ID plus trace ID across service boundaries.
• For critical journeys, make auth redirects, rate limits, and state-sync lag diagnosable with structured codes or attributes instead of opaque text-only errors.
• Prefer SLO-based release gates and burn-rate alerts over raw infrastructure thresholds.
• Treat sampling, cardinality, retention, and cost as quality constraints.
• Redact PII and secrets by default in logs, spans, and attributes.
• Use OpenTelemetry semantic conventions instead of custom attribute names, and add manual spans at business workflow boundaries: auto-instrumentation alone does not show them.
• Treat logs and profiles as ecosystem-dependent in OpenTelemetry: confirm language and backend support before promising a vendor-neutral implementation.
• The OTel Span Events API (Span.AddEvent, Span.RecordException) is being deprecated in favour of log-based events (OTel blog). Write new event instrumentation via the Logs API; existing span event data remains functional during the gradual transition.

Expert Judgment (what a checklist misses)

• Missing telemetry: Start with one critical journey: a structured request log, golden metrics at its entry point, and a propagated trace. Query a known request before using dashboards or alerts as evidence.
• Logs or traces: Use traces for request causality and latency, logs for business state transitions, and metrics for rates and percentiles. Verify retention, sampling, and audit requirements before treating logs or traces as durable records.
• Sampling: Tail sampling can only retain spans that reach the sampler. Head-dropped traces cannot be recovered downstream; route all spans of a trace to the same tail-sampling collector and account for buffering. Use unsampled request metrics for SLIs. For configuration constraints, read OpenTelemetry sampling and references/sampling-strategies.md.
• Paging: Consolidate duplicate symptoms into actionable SLO alerts. Measure page load and response quality; use the service's staffing and error-budget policy rather than a universal alert-count cap. See references/alerting-strategies.md.
• Cardinality: Bound metric labels such as user IDs and raw paths; backend cost depends on the signal and storage design. Retain useful log/trace dimensions deliberately, with a stated retention and ingestion budget. Look up the chosen backend's billing dimensions and limits before sizing it.
• Debugging boundary: Use telemetry to locate the affected component and request context, then use a debugger when code-level execution evidence is needed.

Workflow

1. Establish the baseline: logs, metrics, traces, correlation, and at least one diagnosable critical journey.
2. Instrument with OpenTelemetry: auto-instrument first, then add manual spans for business workflow boundaries.
3. Verify context propagation across HTTP, queues, and RPC boundaries.
4. Define SLIs/SLOs, error budgets, and burn-rate alerts from the canonical burn-rate table, which also covers low-traffic services.
5. Make failures diagnosable: attach trace links, key logs, and relevant metrics to failed tests.
6. Add performance evidence only after telemetry is trustworthy: profiling, load tests, exemplars, and baselines.

Quick reference

Task	Recommended default	Notes
Tracing	OpenTelemetry + Collector + OTLP-compatible backend	Jaeger and Tempo are both fine backends; keep Collector as the default routing layer
Metrics	Prometheus + Grafana	Use latency histograms. Check native histogram documentation for the deployed version's scrape and storage settings; treat rollout as an infrastructure change and test dashboards and alerts first
Logging	Structured JSON to stdout/stderr + Collector/filelog pipeline	Never log secrets or high-cardinality IDs as labels; OTel Logs SDK maturity varies by language — check status page before adopting direct SDK path
Reliability gates	SLIs/SLOs + error budgets + burn-rate alerts	Gate releases on sustained burn and material regressions
Performance	Continuous profiling + load tests + budgets	For stable production profiling use Pyroscope or Parca; check OTel Profiles status before critical production use, and see references/performance-profiling-guide.md
Zero-code visibility	eBPF-based instrumentation where feasible	Before choosing Beyla/OBI, check Beyla setup requirements and the selected release for kernel, permissions, runtime, and backend compatibility
LLM / AI agent visibility	OTel GenAI semconv + cost metrics + eval events	GenAI conventions live in github.com/open-telemetry/semantic-conventions-genai; check the repo for their stability status before shipping, pin a semconv version, and expect renames. See references/tools-ebpf-apm.md and ai-coding-agents-observability-evals for agent traces and evals

Navigation

Open these guides when needed:

If the user needs...	Read	Also use
A minimal production baseline	references/core-observability-patterns.md	assets/checklists/template-observability-readiness-checklist.md
Current Node or Python instrumentation	references/opentelemetry-best-practices.md	assets/opentelemetry/nodejs/opentelemetry-nodejs-setup.md, assets/opentelemetry/python/opentelemetry-python-setup.md
Working trace propagation across services	references/distributed-tracing-patterns.md	assets/checklists/template-observability-readiness-checklist.md
SLOs, burn-rate alerts, and release gates	references/slo-design-guide.md (canonical burn-rate table)	assets/monitoring/slo/slo-definition.yaml, assets/monitoring/slo/prometheus-alert-rules.yaml, rule tests (promtool test rules), assets/monitoring/grafana/grafana-dashboard-slo.json
Profiles as telemetry (continuous profiling, exemplars, OTel Profiles)	references/performance-profiling-guide.md	Diagnosis: software-performance; load tests: qa-testing-performance; starter assets: assets/performance/backend/template-nodejs-profiling-config.js, assets/performance/frontend/template-lighthouse-ci.json, assets/load-testing/load-testing-k6.js, assets/load-testing/template-load-test-artillery.yaml
A maturity model and roadmap	references/observability-maturity-model.md	assets/checklists/template-observability-readiness-checklist.md
What to avoid and how to fix it	references/anti-patterns-best-practices.md	assets/checklists/template-observability-readiness-checklist.md
Alert design and fatigue reduction	references/alerting-strategies.md	assets/monitoring/slo/prometheus-alert-rules.yaml
Dashboard hierarchy and layout	references/dashboard-design-patterns.md	assets/monitoring/grafana/template-grafana-dashboard-observability.json
Structured logging and cost control	references/log-aggregation-patterns.md	assets/observability/template-logging-setup.md
RED vs USE vs Golden Signals — choosing a metrics framework	references/methods-red-use-golden.md	references/slo-design-guide.md
Sampling strategies, tail sampling, exemplars	references/sampling-strategies.md	references/opentelemetry-best-practices.md
eBPF and APM tool stubs (Beyla, Pixie, Honeycomb, SigNoz, Coroot), LLM/AI agent observability	references/tools-ebpf-apm.md	data/sources.json
Debugging a production anomaly with no hypothesis and no reproduction	references/core-analysis-loop.md	../qa-debugging/references/production-debugging-patterns.md
Reliability, queueing, control, or information theory applied to telemetry	references/reliability-theory-applied.md, references/queueing-theory-applied.md, references/control-theory-applied.md, references/information-theory-applied.md	the matching foundations-* skill

Curated sources: data/sources.json.

Avoid

• Logging secrets or high-cardinality IDs as metric labels
• Inventing custom attribute names when semantic conventions exist
• Adding instrumentation without a sampling and cardinality strategy
• Trusting auto-instrumentation alone for business workflow visibility; add manual spans at business workflow boundaries, not duplicate route spans
• Picking vendors from memory: start from data/sources.json and check current docs or releases

Scripts

Stdlib-only Python CLI tools; run with Python 3.9+. Regression tests require pytest.

Script	Purpose
scripts/observability_scorer.py	Maturity scoring, SLO error budget analysis, and readiness report generation

The scorer accepts complete-window SLO measurements only; its status thresholds are local heuristics, not multi-window paging decisions. Supply target_pct strictly between 0 and 100, positive integer window_days, and either both integer event counts (total_events positive, 0 <= good_events <= total_events) or an explicit finite current_availability_pct in 0..100. Counts take precedence over a supplied percentage; zero events cannot establish health. Percentage-only inputs leave remaining event budget unknown. Maturity scores are supplied self-assessments, not runtime proof; each provided dimension needs an integer score in its documented range. Invalid or missing required inputs fail with an error.

Regression tests: scripts/test_observability_scorer.py. Run python3 -m pytest scripts/test_observability_scorer.py; set OBSERVABILITY_SCORER_SCRIPT to an alternate script path when comparing a saved implementation.

Quick start:

# Score observability maturity (0–100) across 6 signal dimensions
python scripts/observability_scorer.py maturity \
  --input data/sample-observability-profile.json

# Calculate SLO error budget burn rates and status flags
python scripts/observability_scorer.py slo \
  --input data/sample-slo-data.json

# Full readiness report (maturity + SLO) written to a Markdown file
python scripts/observability_scorer.py report \
  --input data/sample-observability-profile.json \
  --slos  data/sample-slo-data.json \
  --output report.md

See scripts/README.md for full CLI reference and input schema.

Data

Sample input files for the scripts.

File	Description
data/sample-observability-profile.json	Realistic B2B SaaS service observability assessment (checkout-service, Node.js)
data/sample-slo-data.json	Five SLO definitions with current availability and event counts
data/sources.json	Curated reference sources for the skill

Related Skills

Skill	Purpose
ops-devops-platform	Infrastructure monitoring, Kubernetes, CI/CD
data-sql-optimization	Database query optimization and indexing
qa-debugging	Application-level debugging and stack traces
qa-testing-strategy	Test strategy design and coverage
qa-resilience	Resilience patterns, retries, and circuit breakers
software-architecture-design	Architecture decisions
software-performance	Performance diagnosis and profiling
qa-testing-performance	Load, stress, and soak tests; performance CI gates
ai-coding-agents-observability-evals	Agent traces and eval pipelines

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.