Skip to content
cccskills
FreePublished on GitHub

incident-triage-runbook

The SRE team's runbook for triaging production latency and error-rate incidents. Use this whenever investigating an incident, a latency spike, elevated error rates, or when asked "what caused X" about a production service.

The English summary is being prepared. Showing the original description.

See how to install

Files included (1)

  • SKILL.md2.4 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

Incident triage

If you change the order below, say why in #sre.

Order of operations

  1. Pull deploys for the last 6h. Don't open the log first.
  2. Line the deploy timestamps up against p99_latency_ms / error_rate for the paged service. State the gap ("deploy 14:31, p99 moves 14:33").
  3. If a deploy lines up: pull the diff, read it. Check for the stuff in the next section.
  4. Then grep the log to confirm. Don't grep to fish.
  5. No deploy lines up → the service may be a victim. Check db_pool_utilization across checkout/cart/auth/inventory, then the upstream deps (db-primary, db-inventory, idp.internal). Find what moved first.

Ask get_metrics for a name that doesn't exist and it lists the ones that do.

Things that have burned us

In rough order of how often:

  • per-row query where there used to be a batch
  • cache decorator removed "temporarily"
  • new query, no index
  • blocking call in an async handler
  • retry loop with no backoff

Write-up

One line at the bottom of your reply:

Root cause: <sha> — one sentence on the mechanism.

If it wasn't a deploy, put the component or upstream dep where the sha goes (db-primary, stripe-api, whatever). Still one sentence.

Everything above that line is evidence. Keep it short; the long version goes in the postmortem doc.

If you were handed an incident id, also save the structured version to /workspace/findings/<INCIDENT_ID>.json so the postmortem can be built from it:

{
  "incident_id": "INC-0000",
  "service": "checkout",
  "title": "one line, what users saw",
  "root_cause": {"ref": "<sha or component>", "summary": "one sentence on the mechanism"},
  "timeline": [{"time": "14:31:18", "event": "what happened"}],
  "evidence": ["one fact per line, with the number or timestamp that proves it"],
  "ruled_out": ["what you checked that turned out not to be the cause, and why"],
  "impact": {"metric": "p99_latency_ms", "baseline": 65, "peak": 3640, "unit": "ms"},
  "fix": "the immediate remediation",
  "caused_by_incident": null
}

Set caused_by_incident to another page's id when your service is a victim of it.

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

How to pull SEC EDGAR data with the edgartools Python package — company lookup, filings, XBRL financial statements, and filing sections like Item 1A risk factors. Use whenever a task involves SEC filings, 10-K/10-Q data, or company financials.

The English summary is being prepared. Showing the original description.

anthropics/cwc-workshops2.2KUpdated Oct 17, 2026

How to produce a demand forecast for a SKU, and when to delegate that to a subagent vs. compute it yourself. Load this for any task involving "forecast", "how much will we sell", "next month", promos, or seasonal SKUs.

The English summary is being prepared. Showing the original description.

anthropics/cwc-workshops2.2KUpdated Oct 17, 2026

mining

Free

Where diamonds spawn in Minecraft 1.20.

The English summary is being prepared. Showing the original description.

anthropics/cwc-workshops2.2KUpdated Oct 17, 2026

Fixed-format templates for Slack alerts, supplier emails, and escalations. Load this whenever the task is "notify", "alert", "email", or "tell ops".

The English summary is being prepared. Showing the original description.

anthropics/cwc-workshops2.2KUpdated Oct 17, 2026

How the SRE team writes a postmortem PDF. Use this whenever asked for a postmortem, an incident report, a write-up to share, or a PDF of what happened.

The English summary is being prepared. Showing the original description.

anthropics/cwc-workshops2.2KUpdated Oct 17, 2026

How to decide whether and how much to reorder a SKU. Load this whenever a task involves reorder recommendations, purchase orders, or "should we restock" questions.

The English summary is being prepared. Showing the original description.

anthropics/cwc-workshops2.2KUpdated Oct 17, 2026

See all skills by anthropics

Report a problem with this skill