Overview
Based on the Google SRE Book. A runbook is the operational contract for a service: how to start it, stop it, diagnose it, and recover it. Google SRE's standard: a runbook should be executable by an on-call engineer who has never seen the service before. If it requires tribal knowledge, it's not a runbook - it's an assumption.
The test: can an on-call engineer follow this runbook at 3am with a degraded service, no time to research, and adrenaline affecting their cognition? If not, it's not operational-grade.
Workflow
Step 1: Write the service overview
Service: [your service]
Owner team: [your team]
On-call rotation: [who gets paged]
SLO: [availability target, e.g. 99.9% uptime]
Dependencies: [upstream services this depends on]
Dependents: [downstream services that depend on this]
Dashboards: [links]
Logs: [how to access]
Alerts: [link to alerting config]
Step 2: Document steady-state operations
What does normal look like?
Health check:
# Command to verify the service is healthy
curl https://[your service]/health
# Expected response: {"status": "ok", "version": "x.y.z"}
Key metrics (normal ranges):
| Metric | Normal | Warning | Critical |
|---|
| Error rate | < 0.1% | 0.1-1% | > 1% |
| p99 latency | < 200ms | 200-500ms | > 500ms |
| CPU usage | < 60% | 60-80% | > 80% |
Step 3: Document alert runbooks
For each alert, write a response procedure:
Alert: [Alert name]
Severity: [SEV1/SEV2/SEV3]
Fires when: [condition]
Impact: [what users experience]
Diagnosis:
1. Check [dashboard link] for [specific signal]
2. Run: [command] - Expected: [output]
3. Check: [logs location] for [error pattern]
Common causes:
A. [Cause] - Fix: [command or action]
B. [Cause] - Fix: [command or action]
If none of the above: escalate to [team/person]
Step 4: Document common operations
List every routine operational task with exact commands:
Restart the service:
kubectl rollout restart deployment/[service-name] -n [namespace]
kubectl rollout status deployment/[service-name] -n [namespace]
Scale up:
kubectl scale deployment/[service-name] --replicas=[n] -n [namespace]
Rollback a deploy:
kubectl rollout undo deployment/[service-name] -n [namespace]
# Verify rollback succeeded:
kubectl rollout status deployment/[service-name] -n [namespace]
Step 5: Document recovery procedures
For each failure mode, step-by-step recovery:
Failure: [service is unresponsive / returning 5xx / high latency]
Verify: [how to confirm this is the problem]
Recover:
1. [first action]
2. [second action]
3. [verify recovery: command - expected output]
Escalate if: [condition that means this runbook is insufficient]
Step 6: Document escalation paths
Who to call when the runbook doesn't work:
| Situation | Contact | How |
|---|
| Database issue | [DB team] | [PagerDuty / Slack] |
| Network issue | [Infra team] | [contact] |
| Vendor outage | [Vendor support] | [ticket URL] |
Step 7: Add a testing section
How does an on-call engineer practice with this runbook in a non-emergency?
- Chaos engineering scenarios
- Runbook review frequency (recommend: quarterly)
- Last tested: [date]
Anti-Patterns
1. Runbook that requires tribal knowledge
Bad: "Check if the DB is having issues." (how? where? what to look for?)
Good: "Run psql -h [host] -U [user] -c 'SELECT count(*) FROM pg_stat_activity' - if > 100 connections, connection pool exhaustion is likely."
2. Commands without expected output
Bad: "Run the health check command."
Good: "Run: curl https://[your service]/health - Expected: {"status":"ok"}. If you see {"status":"degraded"}, proceed to Step 3."
3. No escalation path
Bad: Runbook ends with "investigate further."
Good: Every runbook has an explicit "escalate if" condition and names who to escalate to.
4. Runbook never tested
Bad: Runbook written once, never validated in a real incident.
Good: Run quarterly fire drills. Note the last tested date in the runbook.
Quality Checklist