Overview

Datadog is the all-in-one observability platform: infrastructure monitoring, APM tracing, log management, RUM, synthetics, and security — unified by tagging. Its strength is correlation with minimal setup: install the agent, get metrics/traces/logs linked automatically. Its weakness is the bill, which scales with hosts, ingested logs, and custom metrics.

Using Datadog well means: tag everything consistently, be deliberate about what's ingested (logs and custom metrics are the cost drivers), and build monitors/SLOs on symptoms. This skill covers the platform's core, cost control, and operational patterns.

When to use

• Setting up Datadog (agents, integrations, APM).
• Designing tagging strategy and dashboards.
• Writing monitors (thresholds, anomaly, composite).
• Defining SLOs and error budgets.
• Controlling Datadog costs (logs, custom metrics, retention).
• Correlating traces, logs, and metrics in investigations.
• Choosing between Datadog and Prometheus/Grafana/ELK.

Core concepts

• Unified tagging. Tags (env:prod, service:api, version:abc) unify metrics, traces, and logs. A consistent tagging scheme is the highest-leverage Datadog decision — inconsistent tags make correlation impossible.
• The Agent. Host/container agent collecting metrics, traces (APM), and logs. DaemonSet in Kubernetes; trace client libraries in apps. One agent version policy — keep them updated.
• APM. Distributed tracing with automatic instrumentation for common frameworks; custom spans for business operations. Trace search and analytics on span tags; link traces to logs via trace IDs.
• Log management. Ingest, parse (pipelines/grok), index selectively. Ingestion is the cost driver — index what you search, archive the rest to object storage (rehydration when needed).
• Custom metrics. The other cost driver — billed per unique metric+tag combination per month. Cardinality discipline (same rule as Prometheus) plus metrics-without-limits (distributions) where appropriate.
• Monitors. Threshold, anomaly, forecast, and composite monitors; multi-alert vs simple-alert (per-group or aggregate). Alert on symptoms; include runbook links and tags for routing.
• SLOs and error budgets. Time-slice or monitor-based SLOs with error-budget alerts — the framework for "how reliable are we" that both engineering and product understand.
• Dashboards and notebooks. Timeboards (shared, versioned-ish) vs screenboards; notebooks for investigations and postmortems. Template variables for reuse; keep golden dashboards curated.
• Synthetics. API tests and browser tests from global locations — proactive monitoring of user journeys, not just backend health. Test the checkout flow, not just /health.
• RUM. Real-user monitoring for frontend performance and errors — Web Vitals, session replay. Privacy implications need review (masking, consent).
• Watchdog. ML-driven anomaly detection surfacing issues without predefined monitors — triage its findings, don't auto-page on them initially.
• Service catalog and Scorecards. Service ownership, docs, and health scorecards — the organizational layer that keeps observability maintained as teams grow.
• Sensitive data. Log scrubbing (obfuscation rules) for PII/secrets — configure before sensitive data flows, not after an incident.
• Cost controls. Usage attribution by team (tags!), log exclusion filters, metrics cardinality management, retention settings, and regular bill reviews with owners per cost center.
• Incident management. Declared incidents with timelines, responders, and retrospectives — the coordination layer that turns alerts into resolved outages.
• Cloud cost management. Cost attribution alongside observability — the same tags showing spend per service, so cost and performance live in one view.

Practical workflow

1. Define the tagging scheme. env, service, version, team on everything — agent config, APM, logs. Enforce via admission checks or CI; audit tag coverage.

2. Deploy agents consistently. DaemonSet with log collection in K8s; APM enabled; dogstatsd for custom app metrics; keep agent versions current.

3. Control ingestion. Log pipelines parsing only needed fields; exclusion filters dropping debug/health-check noise before indexing; archive everything to storage; custom-metrics cardinality review.

   • Rule of thumb: if nobody searches it, don't index it — archive and rehydrate on demand.

4. Build golden dashboards. Per service: RED metrics, saturation, dependencies, deploy markers (via events API from CI). Template variables for env/service.

5. Write monitors on symptoms. Error-rate, latency, and saturation monitors with multi-alert grouping; anomaly detection for seasonal patterns; composite monitors for "really broken" paging.

   { "name": "[prod] High error rate — shop-api",
     "type": "query alert",
     "query": "avg(last_5m):sum:trace.http.request.errors{env:prod,service:shop-api}.as_rate() / sum:trace.http.request.hits{env:prod,service:shop-api}.as_rate() > 0.05",
     "message": "@pagerduty Runbook: https://wiki.example.com/runbooks/high-error-rate" }
   

6. Define SLOs. A few meaningful SLOs per critical service (availability, latency); error-budget burn alerts; review in planning — budgets guide prioritization.

7. Add synthetics for user journeys. API and browser tests for critical flows from relevant locations; alert on failure with the same severity as backend pages.

8. Attribute and control cost. Usage dashboards by team/service; monthly reviews; log rehydration instead of permanent indexing for cold data; right-size retention.

   exclusion filter: drop health-check noise before indexing

   Query: status:ok service:api @http.url:/health

   Action: exclude from index (still archived to object storage)

Common pitfalls

• Inconsistent tagging — correlation impossible; define and enforce the scheme early.
• Indexing everything — log bills exploding; exclusion filters + archive + rehydrate.
• Custom metric cardinality — per-user/per-request tags; the bill scales with unique combinations.
• Monitors on causes — paging for CPU; alert on user-facing symptoms.
• Monitors without runbooks — pages with no guidance; link runbooks.
• Simple-alert when multi-alert needed — one alert for 50 hosts; group appropriately.
• No SLOs — reliability debates without data; define a few meaningful SLOs.
• Watchdog auto-paging — ML findings paging before trust is built; triage first.
• PII in logs — sensitive data ingested unscrubbed; obfuscation rules from day one.
• Stale monitors — alerts for decommissioned services; audit monitors with service lifecycle.
• RUM without privacy review — session replay capturing sensitive input; masking and consent.
• No cost attribution — one bill, no owners; tag-based usage dashboards per team.
• Synthetics only on /health — green while checkout is broken; test real user journeys.
• No exclusion filters — debug and health-check logs indexed at full price; filter noise before indexing.
• Tag sprawl — hundreds of low-value tags multiplying custom-metric costs; audit tag cardinality regularly.