Incident Response

Quick Reference

Need	Go to
Classify the incident severity	## Severity Classification
Run the end-to-end response loop	## Workflow
Coordinate as incident commander	## Incident Commander Checklist
Send status or stakeholder updates	## Communication Templates
Use templates and runbooks	## Navigation

Scope

Use For	Do NOT Use For
Severity classification and escalation	Infrastructure provisioning or CI/CD
On-call runbook design and review	Application resilience patterns (retries, circuit breakers)
Incident commander workflows	Monitoring/alerting tool setup
Status page and stakeholder communication	Root cause analysis of code bugs
Blameless postmortem facilitation	Performance benchmarking
MTTD/MTTR measurement and improvement	Security incident forensics (use software-security-appsec)

Severity Classification

Read the service's paging policy and runbook before assigning severity, acknowledgement deadlines, or update cadence. If unavailable, mark the following as an illustrative starting policy and assign an owner to confirm it.

Level	Criteria	Response time	Who is paged
SEV1	Revenue-impacting, data loss, full outage	Immediate	On-call + incident commander + leadership
SEV2	Degraded service, partial outage, SLO breach	15 min	On-call + incident commander
SEV3	Minor degradation, workaround available	1 hour	On-call
SEV4	Cosmetic, no user impact, internal tooling	Next business day	Ticket owner

Workflow

1) DETECT — Alert fires or user report received
   - Acknowledge the alert within SLA
   - Open an incident channel (Slack/Teams)
   - Assign incident commander if SEV1/SEV2

   DECLARE-AND-MOBILIZE (T+0, SEV1/SEV2):
   - Auto-create or manually open a dedicated #inc-YYYYMMDD-[short-name] channel
   - Assign roles at declaration: IC, comms lead, scribe — do not wait for volunteers
   - Assign the first status update and its deadline using the service's communication policy
   - Trigger postmortem document creation at incident close (template auto-populated
     with title, severity, IC, timeline start — not at the writing stage)
   - Use the team's configured incident tooling; confirm its working escalation path

2) TRIAGE — Assess impact and classify severity
   - Confirm affected systems, user segments, blast radius
   - Check recent deployments, config changes, dependency status
   - Update status page with initial assessment
   - If SLO/error-budget tracking is active: post current budget consumed in channel

3) MITIGATE — Stop the bleeding
   - Prioritize mitigation over root cause
   - Rollback, feature-flag off, scale up, redirect traffic
   - Communicate ETA or "investigating" to stakeholders

4) RESOLVE — Confirm recovery
   - Verify the primary user outcome, service SLI, dependency health, and queued or delayed work remain within bounds for at least one meaningful traffic or processing cycle
   - Check data integrity separately when writes, ordering, or retries were affected
   - Record the observation window and residual degradation; a deploy success, pod health, falling error rate, or quiet alert stream alone is not recovery
   - Update status page to "resolved"
   - Close the incident channel
   - Note total error-budget consumed in the postmortem summary line

5) LEARN — Blameless postmortem
   - Write timeline (what happened, when, who did what)
   - Identify contributing factors (not "root cause")
   - Define action items with owners and deadlines
   - Share postmortem with the team
   - For SEV1/SEV2: use the Howie structured-interview approach before group debrief
     (see references/postmortem-facilitation.md)
   - When every component worked as designed but the interaction failed: CAST
     (see ../foundations-safety-engineering/references/cast-incident-analysis.md)

Incident Commander Checklist

• Open dedicated incident channel
• Assign roles: IC, comms lead, subject matter experts
• Set a timer for the service's agreed status-update cadence
• Keep a running timeline in the channel
• Delegate investigation — IC coordinates, does not debug
• Post status page updates at each phase transition
• Apply the recovery checks in Workflow before calling "resolved"
• Schedule postmortem using the team's review deadline

Incident Commander Decisions

• Ambiguous severity: declare when plausible user harm from delayed coordination exceeds the disruption of mobilizing responders; de-escalate after bounding impact. The cost comparison in decision-theory-applied.md P2 is illustrative, not proof that escalation always wins.
• Communication: send material changes immediately and retain the promised next-update time even when nothing changed. Keep raw investigation detail in the responder channel; send impact, progress, and next update to stakeholders.
• Conflicting mitigations: name one action owner and surface incompatible proposals before execution. If investigation stops producing updates, request the hypothesis, evidence, and next checkpoint.
• Blameless review: preserve factual actions and timestamps, then explain the information and constraints available to responders. Rewriting every action as “the system allowed” can erase evidence. Track action closure separately from postmortem publication and assign owners to each contributing factor.

Communication Templates

Status Page — Investigating

[Service Name] — Investigating
We are aware of [brief description of impact].
Our team is investigating and will provide updates every [interval].
Started: [timestamp UTC]

Status Page — Resolved

[Service Name] — Resolved
The issue affecting [brief description] has been resolved.
Duration: [start] to [end] ([total minutes])
[If approved for this audience: postmortem publication date or next follow-up.]

Stakeholder Update (SEV1/2)

Subject: [SEV-N] [Service] incident update — [status]

Impact: [who is affected, what they see]
Current status: [mitigating / investigating / resolved]
Next update: [time UTC]
Actions taken: [bulleted list]
Incident lead: [name]

Metrics

Targets below are example starting points; set real targets from your own incident baseline.

Metric	What it measures	Example target
MTTD (Mean Time to Detect)	Impact starts → confirmed detection (alert or user report)	< 5 min for SEV1
MTTA (Mean Time to Acknowledge)	Alert fires → responder acknowledges	< 15 min for SEV1
MTTM (Mean Time to Mitigate)	Acknowledge → user impact stops	< 30 min for SEV1
MTTR (Mean Time to Resolve)	Impact starts → full recovery (local definition)	< 1 hour for SEV1
Postmortem completion rate	Incidents with published postmortems	100% for SEV1/2
Action item close rate	Postmortem actions completed on time	> 80% within deadline

Distributional caveat: Davidovič's report finds that sparse, variable incident samples can make means, medians, and tail percentiles unreliable for detecting improvements. Define timestamps consistently, show cohort size, and measure the response step a change targets. See incident-metrics-guide.md when designing the measurement.

Anti-Patterns

Avoid	Do Instead
Blaming individuals in postmortems	Focus on systems, processes, and contributing factors
Reviewing only full outages	Apply the team's review criteria; include near misses or repeated incidents when they expose a failure pattern
IC also debugging	IC coordinates and communicates; SMEs debug
"Root cause" singular thinking	Most incidents have multiple contributing factors
Silent status pages during incidents	Update even if the update is "still investigating"
Action items without owners or deadlines	Every action item gets a name and a date
Blameless language over a blame-shaped narrative	Keep factual actions and explain responder context — see Incident Commander Decisions
Action items logged but never revisited ("action-item graveyard")	Track action-item close rate as its own metric, separate from postmortem completion rate

Regulatory Reporting Overlays

During triage, send plausible regulatory exposure to legal/compliance and record awareness, classification, materiality determination, and submission timestamps separately. This skill supplies the operational timeline; counsel confirms scope and filing deadlines.

Regime	Trigger to record	Lookup before calculating a deadline
EU DORA	Awareness of the ICT incident and classification as major	RTS 2025/301 Article 5: initial deadline uses both awareness and classification, with a separate rule for classification after the awareness window. Intermediate timing runs from initial submission; final timing from the intermediate or latest updated intermediate report. Check entity-specific holiday rules and competent-authority instructions. ITS 2025/302 supplies incident forms/templates; ITS 2024/2956 concerns the third-party register.
EU NIS2	Awareness of a significant incident	Directive 2022/2555 Article 23: early warning without undue delay and within 24 hours of awareness; incident notification without undue delay and within 72 hours of awareness; final report within 1 month after submitting the incident notification. Check national implementing law and the competent CSIRT's instructions, trust-service-provider notification exceptions, and the ongoing-incident progress/final-report sequence.
SEC Form 8-K Item 1.05	Domestic registrant's materiality determination	SEC disclosure guidance and the applicable form: file within 4 business days after determining the incident is material; make that determination without unreasonable delay. Discovery is a separate timestamp. Foreign private issuers use the Form 6-K framework. Counsel confirms permitted delays and issuer scope.

Navigation

References

• runbook-design-guide.md — How to write effective on-call runbooks
• on-call-practices.md — On-call rotation, escalation, and fatigue management
• postmortem-facilitation.md — Running blameless postmortems (includes Howie standard for SEV1/SEV2)
• incident-metrics-guide.md — Load when defining duration metrics or evaluating response improvements; covers timestamp definitions and sample uncertainty
• slo-incident-triggering.md — Error-budget burn rate as incident trigger; burn rate → severity mapping; postmortem prioritization by budget remaining
• references/control-theory-applied.md — Control-theory applied recipes for incident response: stable autoscaler retune, cascading-failure containment, recovery throttling.
• references/decision-theory-applied.md — Decision-theory applied recipes for incident response: paging thresholds via EU, rollback as real option, MAB runbook ordering.

Assets

Use these as copy-paste starters; fill placeholders before sharing with stakeholders.

• incident-channel-template.md — Use at T+0 when opening a SEV1/SEV2 incident channel; post as the first pinned message to assign roles and set next-update time.
• postmortem-template.md — Use at incident close to create the postmortem document; populate title, severity, IC, and timeline start immediately (do not wait until the writing session).
• runbook-template.md — Use when authoring or reviewing a service runbook; also reference during triage to confirm runbook coverage gaps as action items.

Related Skills

Skill	Relationship
ops-devops-platform	Infrastructure, CI/CD, deployment automation
qa-resilience	Application-level fault tolerance patterns
qa-observability	Monitoring, alerting, SLI/SLO setup
qa-debugging	Systematic debugging during incidents
software-security-appsec	Security incident handling

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.