Observability Agent - Intent-based Router

Scheduling

Goal

Route, design, tune, and review observability work across MELT+P signals, layers, boundaries, vendor categories, transport choices, meta-observability, and incident forensics.

Intent signature

• User asks for observability, telemetry, OTel, metrics, logs, traces, profiles, SLOs, RUM, APM, incident forensics, trace propagation, transport tuning, or observability-as-code.
• User needs vendor/category routing or observability architecture instead of a single vendor's already-covered setup.

When to use

• Setting up an observability pipeline (OTel SDK + Collector + vendor backend)
• Designing traceability across service and domain boundaries (W3C propagators, baggage, multi-tenant, multi-cloud)
• Tuning transport layer (UDP/MTU, OTLP gRPC vs HTTP, Collector DaemonSet vs sidecar topology)
• Running incident forensics (6-dimension localization: code / service / layer / host / region / infra)
• Selecting a vendor category (OSS full-stack vs commercial SaaS vs high-cardinality specialist vs profiling specialist)
• Implementing observability-as-code (Grafana Jsonnet dashboards, PrometheusRule CRD, OpenSLO YAML, SLO burn-rate alerts)
• Meta-observability (pipeline self-health, clock skew detection, cardinality guardrails, retention matrix)
• Covering the MELT+P signal set: metrics, logs, traces, profiles (OTEP 0239), cost (OpenCost), audit (SOC2/ISO), privacy (GDPR/PIPA)
• Migrating off deprecated tools (Fluentd → Fluent Bit or OTel Collector, per CNCF 2025-10 guide)

When NOT to use

• LLM ops (prompt versioning, evals, gen_ai span deep dive); use Langfuse, Arize Phoenix, LangSmith, or Braintrust directly
• Data pipeline lineage: use OpenLineage + Marquez, dbt test, or Airflow lineage backends
• IoT / hardware / datacenter physical-layer telemetry (IPMI, BMC, SNMP); use vendor DCIM tooling (Nlyte, Sunbird, Device42)
• Chaos engineering orchestration: use Chaos Mesh, Litmus, Gremlin, or ChaosToolkit (this skill consumes their telemetry; it does not orchestrate chaos)
• GPU / TPU infrastructure observability: use NVIDIA DCGM Exporter + Prometheus
• Software supply chain (SBOM, attestation): use sigstore (cosign / rekor), in-toto framework, SLSA level attestations
• Incident response workflow (on-call rotation, paging, escalation); use PagerDuty, OpsGenie, or Grafana OnCall
• Single-vendor setup already fully covered by that vendor's own published skill; invoke the vendor skill directly

Expected inputs

• Observability intent, target system, architecture boundary, signals, vendor context, and incident symptoms if any
• Existing OTel/collector/vendor configs, dashboards, SLOs, trace/log/metric examples, or deployment topology

Expected outputs

• Routed observability guidance, setup/migration/tuning plan, incident-forensics path, alerting/SLO guidance, or observability-as-code recommendations
• Transport, meta-observability, privacy, audit, and retention checks
• Vendor delegation target when appropriate

Dependencies

• OTel/W3C/CNCF references and resources under resources/
• Vendor categories, matrix, standards, incident forensics, meta-observability, transport, layers, boundaries, and signal guides

Control-flow features

• Branches by intent, vendor category, layer/boundary/signal matrix, transport topology, privacy/audit risk, and incident localization dimension
• May read/write observability config and docs; generally delegates vendor-specific implementation
• Requires live status verification for load-bearing CNCF/vendor currency

Structural Flow

Entry

1. Classify the intent: setup, migrate, investigate, alert, trace, tune, or route.
2. Identify layers, boundaries, signals, and vendor category.
3. Load only the relevant resource guide(s).

Scenes

1. PREPARE: Classify intent and matrix coverage.
2. ACQUIRE: Read configs, topology, telemetry examples, or incident signals.
3. REASON: Route vendor/category, tune transport, assess meta-observability, or localize incident.
4. ACT: Produce setup/migration/tuning/alert/trace/forensics guidance or config changes.
5. VERIFY: Check pipeline health, clock skew, cardinality, retention, privacy, and audit concerns.
6. FINALIZE: Report route, evidence, risks, and handoff references.

Transitions

• If a vendor-owned skill fully covers setup, delegate instead of duplicating docs.
• If Fluentd appears, recommend Fluent Bit or OTel Collector migration.
• If incident investigation is requested, use 6-dimensional localization.
• If transport tuning appears, load transport-specific resources.

Failure and recovery

• If live CNCF/vendor status is load-bearing, verify current status.
• If telemetry samples are missing, provide instrumentation/collection steps before analysis.
• If scope belongs to out-of-scope domains, route to external authoritative tools.

Exit

• Success: observability path is routed, evidence-backed, and checks are explicit.
• Partial success: missing telemetry, stale vendor status, or external-domain handoff is explicit.

Logical Operations

Actions

Action	SSL primitive	Evidence
Classify observability intent	SELECT	Intent rules
Read telemetry/config evidence	READ	OTel/vendor configs, dashboards, samples
Route vendor/category	SELECT	Vendor categories
Infer coverage gaps	INFER	Matrix and signal/boundary mapping
Validate meta-observability	VALIDATE	Clock, cardinality, retention, health
Write guidance/config	WRITE	OaC/config/docs when requested
Notify result	NOTIFY	Routed recommendation

Tools and instruments

• OTel/CNCF/W3C standards references
• Vendor categories, matrix, incident forensics, meta-observability, transport and signal guides
• Optional CLI/config tooling from the target stack

Canonical workflow path

1. Classify intent: setup, migrate, investigate, alert, trace, tune, or route.
2. Select layer/boundary/signal coverage from `resources/matrix.md`.
3. Load the specific vendor, transport, incident, or signal guide before producing guidance.

When CNCF/vendor status is load-bearing, verify live state at https://landscape.cncf.io.

Resource scope

Scope	Resource target
CODEBASE	Observability config, dashboards, alert rules, instrumentation
LOCAL_FS	Resource guides and generated docs
NETWORK	Vendor/CNCF status and telemetry backends when checked
USER_DATA	Incident symptoms, logs, metrics, traces, profiles

Preconditions

• Observability intent and system boundary are identifiable.
• Relevant telemetry/config evidence is available or missing evidence is stated.

Effects and side effects

• May recommend or modify observability config, dashboards, alerts, and instrumentation docs.
• May route to vendor-owned skills or external tools.

Guardrails

1. Classify intent before routing: every query goes through intent classification; setup | migrate | investigate | alert | trace | tune | route
2. Category-first, not vendor-registry: delegate to vendor-owned skills via resources/vendor-categories.md; do not duplicate their documentation
3. Transport tuning is the moat: UDP/MTU thresholds, OTLP protocol selection, Collector topology, and sampling recipes are in-skill depth that other skills do not cover
4. Meta-observability is non-negotiable: always validate pipeline self-health, clock sync (< 100 ms drift), cardinality, and retention before declaring setup complete
5. CNCF-first preference: Prometheus, Jaeger, Thanos, Fluent Bit, OpenFeature (Graduated 2024-11), Flagger, Falco (Graduated); OpenTelemetry, Cortex, OpenCost (Incubating)
6. Fluentd is deprecated: per CNCF 2025-10 migration guide, recommend Fluent Bit or OTel Collector for all new and migration work
7. W3C Trace Context as default propagator: translate per cloud (AWS X-Ray X-Amzn-Trace-Id, GCP Cloud Trace, Datadog, Cloudflare, Linkerd) via boundaries/cross-application.md
8. Privacy before features: PII redaction, sampling-aware baggage rules, and compliance (SOC2/ISO immutable audit + GDPR/PIPA erasure) are applied at collection, not only at storage
9. Domain-level trust: all vendor and tool references are timestamped as of 2026-Q2; verify live status at https://landscape.cncf.io
10. No stub in final deliverable: scaffolds are editing anchors only during build phase; remove before output

Out of Scope (use external tools)

The combinations below are outside this skill's boundary. The external tools listed are authoritative for each domain.

Domain	External tools
LLM ops / gen_ai observability	Langfuse, Arize Phoenix, LangSmith, Braintrust
Data pipeline lineage	OpenLineage + Marquez, dbt test, Apache Airflow lineage
L1/L2 physical / datacenter hardware	Nlyte, Sunbird, Device42; SNMP exporters where Prometheus bridge is needed
L5 Session / L6 Presentation full TLS inspection	Wireshark (packet-level), Cloudflare Radar (TLS ecosystem data), vendor TLS inspection tooling
Chaos engineering orchestration	Chaos Mesh, Litmus, Gremlin, ChaosToolkit
GPU / AI infra (DCGM, NVIDIA)	NVIDIA DCGM Exporter + Prometheus; OTel GPU semconv (Development, not production-ready)
Software supply chain (SBOM, attestation)	sigstore (cosign / rekor), in-toto framework, SLSA level attestations
Incident response workflow (paging, rotation)	PagerDuty, OpsGenie, Grafana OnCall
Fluentd (primary tool)	Deprecated CNCF 2025-10; use Fluent Bit or OTel Collector

Architecture (4 x 4 x 7 matrix)

                  User / Other Skill Query
                            |
                            v
              +-----------------------------+
              |      Intent Classifier      |
              |  setup | migrate | investigate
              |  alert | trace | tune | route|
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |      Vendor Router          |
              |  category-first delegation  |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |   vendor-categories.md      |
              |   (a) OSS Full-Stack        |
              |   (b) Commercial SaaS APM   |
              |   (c) High-Cardinality      |
              |   (d) Profiling Specialist  |
              |   (e) SIEM / Enterprise Logs|
              |   (f) FinOps / Cost         |
              |   (g) Feature Flags/Rollout |
              |   (h) Log Pipeline          |
              |   (i) Time Series Storage   |
              |   (j) Crash Analytics       |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |  Matrix Coverage Selector   |
              |  4 Layers x 4 Boundaries    |
              |  x 7 Signals = 112 cells    |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |  Transport Depth /          |
              |  Meta-observability         |
              |  UDP, OTLP, Collector,      |
              |  cardinality, clock skew    |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |  Incident Forensics         |
              |  6-dim localization:        |
              |  code/service/layer/host/   |
              |  region/infra               |
              +-----------------------------+

Layers (4): L3-network, L4-transport, mesh, L7-application Boundaries (4): multi-tenant, cross-application, slo, release Signals (7): metrics, logs, traces, profiles, cost, audit, privacy

See resources/matrix.md for the full 112-cell coverage map with N/A markers for invalid combinations.

Routes (Intent)

Intent	Primary target	Fallback
setup	resources/vendor-categories.md → vendor-owned skill	Generic OTel semconv in resources/standards.md
migrate	CNCF 2025-10 guide + resources/vendor-categories.md §(h)	OTel Collector bridge config
investigate	resources/incident-forensics.md (MRA + 6-dim localization)	signals/traces.md + signals/logs.md
alert	boundaries/slo.md (burn-rate alert rules)	resources/observability-as-code.md
trace	boundaries/cross-application.md (propagator matrix)	layers/mesh.md (zero-code auto-instrumentation)
tune	transport/ (4 files: UDP/MTU, OTLP, topology, sampling)	resources/meta-observability.md (cardinality guardrails)
route	boundaries/multi-tenant.md + transport/collector-topology.md	boundaries/cross-application.md (data residency)

Invocation

Standalone:

/oma-observability "set up OTel stack on Kubernetes"
/oma-observability --migrate "move from Fluentd to Fluent Bit"
/oma-observability --investigate "5xx spike in ap-northeast-2"
/oma-observability --alert "configure SLO burn-rate alert for checkout API"
/oma-observability --trace "W3C propagator across AWS + GCP boundary"
/oma-observability --tune "UDP statsd MTU throughput limit"
/oma-observability --route "multi-tenant log isolation with data residency"

Shared invocation (from other skills):

1. State intent: setup | migrate | investigate | alert | trace | tune | route
2. Pass the user query string
3. Receive routed guidance or a vendor-skill delegation target

How to Execute

Follow resources/execution-protocol.md step by step. See resources/examples.md for end-to-end walkthroughs. Use resources/intent-rules.md for intent classification reference. Use resources/matrix.md for coverage navigation across layers, boundaries, and signals. Use resources/vendor-categories.md for vendor delegation and category selection. Before submitting, run resources/checklist.md.

Integrations with OMA Ecosystem

Skill	Integration point
oma-debug	On failure: pull traces + logs by request_id → trigger resources/incident-forensics.md 6-dim localization playbook
oma-qa	Canary post-deploy loop via chrome-devtools MCP: console errors + Core Web Vitals trend; INP/LCP/CLS from layers/L7-application/web-rum.md
oma-tf-infra	Terraform modules for OTel Collector, Grafana, and Loki stack provisioning
oma-scm	Deployment SHA → service.version OTel attribute + release marker events; see boundaries/release.md
oma-backend	Propagator and baggage rules cross-referenced in backend.md ruleset; DB N+1 + Kafka patterns in signals/traces.md
oma-frontend	layers/L7-application/web-rum.md INP/LCP/CLS checklist cross-referenced in frontend.md ruleset
oma-mobile	layers/L7-application/mobile-rum.md offline-queuing pattern cross-referenced in mobile.md ruleset
oma-db	signals/traces.md DB patterns (N+1, connection pool) cross-referenced in database.md ruleset

Versioning & Deprecation

• Spec version pinning: otel_spec / otel_semconv keys in each file's frontmatter document the assumed version. If content depends on a specific attribute stability tier, the tier is stated inline.
• Update triggers (not scheduled):
  • OTel semconv promotion (Development → RC → Stable) affecting attributes cited in this skill → update resources/standards.md and the affected file, bump minor version.
  • Attribute deprecation → replace across all citing files; migration note in resources/standards.md.
  • CNCF status change for a vendor/project named in vendor-categories.md (Graduated / Archived / acquired) → update the vendor table.
• Authoritative live state: https://landscape.cncf.io for CNCF project status. This skill does not promise to track it on any schedule; verify at use time if the information is load-bearing.
• No per-file review stamps: earlier drafts carried last_reviewed / next_review frontmatter. Those were removed because no automated enforcement exists; relying on voluntary manual review produces stale stamps that misrepresent currency. Git history (git log path/to/file) is the source of truth for when a file was last changed.

Contribution Protocol

• Do NOT pre-declare future OMA skill names in user-facing documentation. If OMA-native coverage becomes warranted for an out-of-scope domain, evaluate and name it at that point.
• File edits follow the ownership matrix in docs/plans/designs/005-oma-observability.md §Ownership. CTO co-signs changes to standards.md, matrix.md, anti-patterns.md.
• Run resources/checklist.md §1 Setup validation before merging.

References

• Execution steps: resources/execution-protocol.md
• Intent classification: resources/intent-rules.md
• Coverage matrix: resources/matrix.md
• Standards (OTel spec, W3C, ISO): resources/standards.md
• Vendor categories: resources/vendor-categories.md
• Incident forensics: resources/incident-forensics.md
• Meta-observability: resources/meta-observability.md
• Observability-as-code: resources/observability-as-code.md
• Anti-patterns catalog (72 entries): resources/anti-patterns.md
• Checklist: resources/checklist.md
• Examples: resources/examples.md
• Transport:
  • resources/transport/udp-statsd-mtu.md
  • resources/transport/otlp-grpc-vs-http.md
  • resources/transport/collector-topology.md
  • resources/transport/sampling-recipes.md
• Layers:
  • resources/layers/L3-network.md
  • resources/layers/L4-transport.md
  • resources/layers/mesh.md
  • resources/layers/L7-application/web-rum.md
  • resources/layers/L7-application/mobile-rum.md
  • resources/layers/L7-application/crash-analytics.md
  • resources/layers/L7-application/waf.md
• Boundaries:
  • resources/boundaries/multi-tenant.md
  • resources/boundaries/cross-application.md
  • resources/boundaries/slo.md
  • resources/boundaries/release.md
• Signals:
  • resources/signals/metrics.md
  • resources/signals/logs.md
  • resources/signals/traces.md
  • resources/signals/profiles.md
  • resources/signals/cost.md
  • resources/signals/audit.md
  • resources/signals/privacy.md