post-mortem-author
Overview
Per google-sre-postmortem:
"A postmortem is 'a written record of an incident, its impact,
the actions taken to mitigate or resolve it, the root cause(s),
and the follow-up actions to prevent the incident from
recurring.'"
"Blameless postmortems are a tenet of SRE culture."
(google-sre-postmortem)
"Writing a postmortem is not punishment - it is a learning
opportunity for the entire company." (google-sre-postmortem)
The blameless framing is load-bearing. Per
google-sre-postmortem, the document must "focus on
identifying the contributing causes of the incident without
indicting any individual or team for bad or inappropriate
behavior" - assuming "everyone involved in an incident had good
intentions and did the right thing with the information they
had."
When to use
Per google-sre-postmortem, common triggers include:
"user-visible downtime, data loss, on-call interventions,
extended resolution times, and monitoring failures."
Author a post-mortem after every such incident - not just sev-1.
Lower-severity incidents accumulate context that prevents the
sev-1.
Step 1 - Author the document
Copy the full section skeleton from
references/post-mortem-document-template.md
and fill every section. The required sections, in order:
- Header - status, severity, authors, dates, reviewers.
- Summary - 2-3 sentences: what, who, how long, what was done.
- Impact - users affected, revenue (deferred vs lost), SLO debt, reputational.
- Timeline - chronological events with UTC timestamps and a source per row.
- Root cause - what happened in detail, with the system as the subject, never a person.
- Contributing factors - every condition that allowed the incident; list all, since incidents rarely have one cause.
- What went well - the positives; per google-sre-postmortem, post-mortems should call these out too.
- Action items - the load-bearing section (Step 3).
- Lessons learned and Postmortem trigger - what is known now, and which trigger criteria the incident met.
The worked example below fills this skeleton for a real SEV-2.
Step 2 - Blameless review
Per google-sre-postmortem: "Postmortems are not punishment."
Reviewers should:
- Reject blame language. "Bob deployed without testing" → "The
pre-deploy testing didn't cover the null-metadata case."
- Focus on systems, not individuals. "Why did Bob fix this
fast?" → "What enabled fast diagnosis? (Sentry stack trace +
the team's incident-response training.)"
- Surface contributing factors openly. Multiple factors usually
contribute; document all.
Step 3 - Action item discipline
Action items must have:
- Owner (one named person; not "the team").
- Due date (concrete; not "next sprint when we have time").
- Priority (P0/P1/P2/P3; ties to the action's importance).
- Success criterion (measurable; "X done" not "we'll think about X").
The action items are the post-mortem's value. Without them, the
document is paperwork.
Step 4 - Approval + closure
The post-mortem isn't "done" until:
- Reviewers (eng manager, SRE lead, Product) sign off.
- Action items are added to the team's tracker (Jira / Linear /
GitHub issues).
- P1/P2 action items have a target completion before the next
release.
The post-mortem is "closed" when all action items ship - typically
2-4 weeks. A 6-month-old open post-mortem is a process failure.
Step 5 - Storage
docs/postmortems/
├── INC-1234-stripe-webhook-2026-05-04.md
├── INC-1235-cache-invalidation-2026-05-12.md
├── INC-summary-2026-Q2.md ← rollup
└── README.md
Markdown + git. Quarterly rollup identifies patterns:
## Q2 2026 incident summary
**Total incidents:** 12
**SEV-1:** 1
**SEV-2:** 6
**SEV-3:** 5
**Patterns:**
- 4 of 12 (33%) were "test gap" - the failing condition wasn't
in the test suite. Action: invest in
test-coverage-targeter
+ property-based testing.
- 3 of 12 (25%) involved canary metrics; 2 of those proceeded
through canary gate. Action: review thresholds (per AI-2 from
INC-1234).
- ...
Worked example - INC-1234 Stripe webhook failure (SEV-2)
The skeleton from Step 1, filled for a real incident.
Summary. A v1.4.5 deploy introduced a null-metadata crash in the Stripe
webhook handler; ~12,400 customers (4.3% of MAU) hit failed checkout
completions for 23 minutes until rollback.
Impact. ~$140,000 in delayed (not lost) orders; 32% of the monthly
availability budget burned; 47 support tickets.
Timeline (excerpt).
| Time (UTC) | Event | Source |
|---|
| 14:00 | Deploy of v1.4.5 to canary (5% traffic) | CD pipeline log |
| 14:23 | First Sentry alert: NullPointerException at WebhookHandler:42 | Sentry |
| 14:30 | Canary window ends within thresholds; promoted to 100% | CD pipeline |
| 14:42 | PagerDuty SLO burn-rate alert; incident declared SEV-2 | PagerDuty |
| 14:58 | Rollback complete; error rate returning to baseline | Datadog |
Root cause (what, not who). The v1.4.5 handler added a path for Stripe's
payment_intent.partially_funded event that called
payment.metadata.get("internal_id"); for ~3% of events metadata was null,
the exception was uncaught, the handler returned 500, and Stripe stopped
retrying, so fulfillment never triggered. The canary stage saw the error rate
rise (0.4% vs 0.3% baseline) but stayed under the 1.5x rollback threshold, so
prod-canary-validator returned PROCEED with WARNING and the gate was acked.
Contributing factors. (1) test gap - no unit test for the null-metadata
case; (2) canary threshold too lenient for a low baseline; (3) staging carries
almost no Stripe webhook traffic, so the new event type was never exercised
pre-deploy.
Action items.
| ID | Action | Owner | Priority | Due | Success criterion |
|---|
| AI-1 | Unit test for partially_funded with null metadata | Bob | P1 | 2 days | Test in WebhookHandlerTest.kt fails against the bug, passes after |
| AI-2 | Tighten canary error-rate threshold 1.5x -> 1.3x | SRE | P2 | 1 sprint | canary-thresholds.yml updated; one normal canary passes |
| AI-3 | Staging fixture covering all Stripe event types | Bob | P2 | 1 sprint | Staging "events by type" metric shows all types > 0 |
What went well. Sentry caught the regression at 14:23, well before the
PagerDuty page; rollback finished in 7 minutes, inside RTO. Diagnosis came from
the Sentry stack trace alone, with no production debugging.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
| Blame language | Defeats the blameless principle; team stops authoring post-mortems honestly. | Per Google SRE: focus on contributing causes, not individuals (Step 2). |
| Action items without owner / due date | Nobody acts; same incident recurs. | All four fields required (Step 3). |
| Skipping post-mortems for "small" incidents | Lower-severity context that prevents big incidents is missed. | Author per google-sre-postmortem trigger criteria (Step 1). |
| Post-mortem stored in private docs | Org learning capped at the team. | Public to org (per Google SRE pattern). |
| One-shot post-mortem with no follow-up | "Closed" but action items stale; recurrence likely. | Track action items in tracker (Step 4); post-mortem closed only when all done. |
| Post-mortem authored 2+ weeks after incident | Memory faded; details lost. | Author within 5 business days. |
Limitations
- Honesty depends on culture. A team that fears blame will
produce sanitized post-mortems. Leadership must reinforce
blamelessness in word and action.
- Post-mortems don't prevent the next incident. They prevent
the same class of incident. Pair with proactive practices
(chaos testing, threat modeling).
- Time investment. A SEV-2 post-mortem typically takes 4-8
hours to author + 2-4 hours of review. Budget accordingly.
References
- gsp - Google SRE blameless post-mortem definition,
triggers, blamelessness principle, learning-opportunity framing.
prod-canary-validator - sibling: canary verdict; post-mortems often surface threshold
tuning needs.