Runbook Generator
Operation runbooks that on-call engineers can follow under pressure. Based on Google SRE practices, PagerDuty incident response, and post-mortem culture.
Workflow
Follow these steps in order when generating a runbook:
- Discover — Ask the 5 Required Discovery questions (below) to scope the runbook
- Classify — Map the incident to the Severity Matrix; if unclear start at Sev2
- Template — Instantiate the Runbook Template with service-specific commands, time-boxes, and contacts
- War Room — Configure Slack channel, Zoom bridge, shared doc, status page, and customer comms
- Verify — Run through the Production Checklist before marking complete
- Archive — Store the runbook in the team's on-call repo, test in a drill within the quarter
When the ask is a post-mortem (not a runbook), skip steps 3-4 and use the Post-Mortem Template instead.
2. Required Discovery
- Service/System: What is the runbook about? (API, database, cache, queue)
- Incident types: What can go wrong? (down, degraded, slow, data loss)
- Environment: Production, staging, or both?
- Team structure: Primary, secondary, escalation contacts
- Existing monitoring: Alerts, dashboards, runbooks
3. Severity Matrix
| Severity | Definition | Response SLA |
|---|
| Sev1 | Service down, users impacted | Respond within 15 min |
| Sev2 | Degraded performance, partial impact | Respond within 1 hour |
| Sev3 | Minor issue, no user impact | Next business day |
| Sev4 | Internal tooling, non-critical | Per team schedule |
4. Runbook Template
## Runbook: [Incident Type]
### Detection
How this incident is typically discovered:
- Alert: [Prometheus/Grafana/Datadog alert]
- User symptom: [what users see or report]
- Automated detection: [auto-remediation]
### Initial Response (first 5 min)
1. Acknowledge alert (PagerDuty/Opsgenie). Time: < 2 min
2. Determine severity. Time: < 1 min
3. Assign owner. Create incident channel (#inc-sev1). Time: < 2 min
4. Post initial status:
"Investigating [issue] affecting [scope]. Will update in 15 min."
5. Start diagnosis timer. Time: < 1 min
### Diagnosis (decision tree)
1. Check [primary dashboard]: expected [X], current [Y]
2. Check logs: `kubectl logs -n [ns] -l app=[service] --tail 200`
3. Check database health: `SELECT count(*) FROM pg_stat_activity`
4. Check cache/queue latency: `redis-cli --latency`
5. IF [error in logs] → Runbook A
6. IF [latency > threshold] → Runbook B
7. IF unknown → Escalate
### Resolution Procedures
Each time-boxed with exact commands and verification:
#### Runbook A: [Name] (10 min)
```bash
# Step 1 (2 min)
kubectl rollout restart deployment/[service]
# Verify (1 min)
kubectl rollout status deployment/[service]
Runbook B: [Alternative]
Verification Checklist
Escalation
| Timebox | Action | Contact |
|---|
| 0-15 min | Primary on-call | @name / phone |
| 15-30 min | Secondary on-call | @name / phone |
| 30-60 min | Engineering manager | @name |
| 60+ min | VP Engineering | @name |
Post-Incident Recovery
## 5. War Room Setup
### Communications
- **Slack channel**: `#inc-sev1-[incident-name]`
- **Zoom bridge**: [link] (permanent war room)
- **Shared doc**: Google Doc or Notion page for live notes
- **Status page**: Update immediately on detection
- **Customer comms**: Template below
### Status Page Updates
Investigating: We're aware of [issue] affecting [scope]. Investigating root cause.
Monitoring: Deployed fix for [root cause]. Monitoring closely.
Resolved: [Issue] has been resolved. All systems operational.
### Customer Communication
Subject: [Service] incident — [date]
We experienced [description] from [start] to [end] ([duration]).
Root cause: [one sentence]
Impact: [specific metrics]
What we're doing:
- [Fix deployed]
- [Monitoring improvements]
- [Process changes]
We apologize for the disruption.
## 6. Post-Mortem Template
Post-Mortem: [Date] — [Title]
Severity: Sev[1-4]
Duration: [detection] → [resolution] ([total])
Impact: [users/customers] for time
Timeline (UTC)
- Time: [What happened]
- Time: [Response started]
Root Cause
One paragraph. System-focused, not person-focused.
Contributing Factors
- Monitoring gaps
- Process gaps
- Knowledge gaps
Action Items
| Action | Owner | Due | Type |
|---|
| [action] | @name | Date | prevent/detect/respond |
What Went Well
What Went Wrong
What We'll Do Differently
## Error Handling
| Scenario | Behaviour | Guidance |
|----------|-----------|----------|
| User doesn't know the service/platform | Ask clarifying questions one at a time (not a list) | Start with "what service is this for?" |
| User says "just give me a template" | Output a blank Runbook Template with placeholder brackets | Skip Discovery, fill later |
| User asks for both runbook + post-mortem at once | Generate runbook first, then offer post-mortem | "I'll write the runbook now. Do you have an incident in mind for the post-mortem?" |
| Service has no existing monitoring | Flag the gap, offer to add basic health-check instructions | Note alerts must be configured separately |
| Runbook for a third-party/SaaS dependency | Include vendor status page check; mark resolution as "vendor fixes" | Focus on detection + escalation, not resolution |
| User says "I don't know the escalation contacts" | Use generic roles (Primary on-call, Secondary, EM, VP Eng) | Leave bracketed placeholders |
| Multiple teams involved | Add a RACI section to the runbook | Clarify who decides vs who executes |
| Runbook already exists, user wants update | Treat as revision: read current, diff, propose changes | Flag deprecations, test outdated commands |
## 8. Production Checklist
Before marking a runbook complete, verify every item:
- [ ] **Commands are copy-paste ready** — no unsubstituted variables (`[ns]` only in section headers/notes, never in commands)
- [ ] **Every step has a time-box** — engineer knows when to escalate
- [ ] **Decision tree is shallow** — max 3-4 levels; deeper means split into sub-runbooks
- [ ] **Multiple resolution paths** — at least 2 common failure modes covered
- [ ] **One runbook per incident type** — don't combine "DB failover" and "deployment rollback"
- [ ] **Verification step after every resolution** — how to confirm the fix worked
- [ ] **Escalation contacts listed** — primary + backup, both Slack and phone
- [ ] **Status page templates included** — Investigating / Monitoring / Resolved
- [ ] **Runbook tested in a drill** — within the current quarter, not when incident strikes
- [ ] **README section added** — link to runbook from team's on-call repo
- [ ] **Secrets checked** — no passwords, API keys, or tokens in commands; use env vars
- [ ] **Reviewed by secondary on-call** — fresh eyes catch assumptions
## Anti-Patterns
| Anti-pattern | Why It Fails | Fix |
|-------------|-------------|-----|
| Steps that say "fix the issue" | Too vague under pressure | Tell HOW, not what — exact commands |
| Commands with unsubstituted variables | Copy-paste fails, engineer wastes time retyping | Every command must run as-is |
| No verification step | Fix might not work, but nobody knows | Add check after every resolution |
| Runbooks >6 months stale | Commands rot, trust erodes, people ignore them | Schedule quarterly review in team calendar |
| Runbooks nobody tested | First test happens during the actual incident | Require one drill per quarter per runbook |
| Too many steps (>15) | Cognitive overload during Sev1 | Split into sub-runbooks, keep shallow |
| No time-boxes | Engineer doesn't know when to escalate | Every diagnosis/resolution step has a max time |
| Single resolution path | Assumes first idea works | Always have Plan B in the same runbook |
| Post-mortem blames people | Blame culture kills incident reporting | "What broke the system?" not "Who broke it?" |
| Skipping customer comms | Stakeholders hear "we're down" from social media | Draft customer email as part of template |
## Incident Timeline Template
[HH:MM] Alert triggered: <alert name> from <monitoring system>
[HH:MM] On-call acknowledged: <name>
[HH:MM] War room opened: <Slack channel/Zoom link>
[HH:MM] Initial diagnosis: <symptoms observed>
[HH:MM] Impact confirmed: <users affected, services degraded>
[HH:MM] Mitigation applied: <action taken>
[HH:MM] Monitoring confirms recovery
[HH:MM] All-clear declared
[HH:MM+1] Post-mortem scheduled: <date>
## Post-Mortem Template
Incident Post-Mortem: <title>
Date: YYYY-MM-DD | Duration: Xh Ym | Severity: Sev1/2/3
Timeline
[HH:MM] ...
Root Cause
What system/process failure caused this.
Impact
Users affected, data loss, revenue impact.
Detection
How was this found? (alert, user report, manual)
Resolution
What fixed it.
Action Items
Lessons Learned
What would prevent this next time.
## Sources
- Google SRE Book: "Monitoring Distributed Systems"
- Google SRE Workbook: Incident Response
- PagerDuty incident response documentation
- Atlassian post-mortem best practices
- Microsoft SRE practices
- AWS Well-Architected Framework: Operational Excellence
## Checklist
- [ ] Skill loads without errors in the AI agent
- [ ] YAML frontmatter is valid (description, compatibility, audience)
- [ ] Workflow section provides clear step-by-step instructions
- [ ] Error handling section covers common failure modes
- [ ] All referenced files (references/, scripts/, assets/) exist
- [ ] Skill triggers correctly for intended use cases
- [ ] No broken links or missing resources